From ae71d21824743c48f1344b0a0246c53a3e57e29b Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Fri, 9 Oct 2026 16:31:06 +0800 Subject: [PATCH 1/4] Rewrite task instructions as direct user requests --- .agents/AGENTS.md | 7 +++- .agents/skills/create-searchswe-task/SKILL.md | 16 ++++---- .../references/instruction-rewrite.md | 39 ++++++++++++------- .../references/model-evaluation.md | 2 +- .../references/task-authoring.md | 4 +- .../references/validation.md | 2 +- .../scripts/scaffold_task.py | 6 ++- .../scripts/test_scaffold_task.py | 6 +-- tasks/task-1-1/instruction.md | 14 +++---- tasks/task-1-2/instruction.md | 12 +++--- tasks/task-1-3/instruction.md | 6 +-- tasks/task-1-4/instruction.md | 10 ++--- tasks/task-2-1/README.md | 2 +- tasks/task-2-1/instruction.md | 10 ++--- tasks/task-2-2/instruction.md | 10 ++--- tasks/task-2-3/instruction.md | 8 ++-- tasks/task-2-4/instruction.md | 8 ++-- tasks/task-2-5/instruction.md | 12 +++--- 18 files changed, 96 insertions(+), 78 deletions(-) diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index aec554d..cc1517a 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -28,8 +28,11 @@ root unless expressed as Markdown links. contributors' changes. Ask about uncertainties that affect task design or authorization, not information already supplied by the user. 2. For task contributions, let the skill drive design → structured source → - scenario rewrite → grader checks → configured model evaluation. Preserve `raw-instruction.md` and audit all its conditions - in the agent-facing `instruction.md`; the skill defines both gates. + user-request rewrite → grader checks → configured model evaluation. Preserve + `raw-instruction.md` and audit all its conditions in the agent-facing + `instruction.md`. Describe the task, available inputs, resource constraints, + acceptance criteria and optimization objective directly, without invented + requester identities or narrative scenes; the skill defines both gates. Before scaffolding, inspect one or two closest reviewed packages under `tasks/` as read-only structural precedents, selected by mode, grading shape, resources and hardware. Current repository contracts and validators override diff --git a/.agents/skills/create-searchswe-task/SKILL.md b/.agents/skills/create-searchswe-task/SKILL.md index f608c41..07b916d 100644 --- a/.agents/skills/create-searchswe-task/SKILL.md +++ b/.agents/skills/create-searchswe-task/SKILL.md @@ -5,7 +5,7 @@ description: Create or substantially revise Search-SWE task packages with CPU/GP # Create a Search-SWE Task -Deliver a reproducible task whose structured source, scenario instruction, +Deliver a reproducible task whose structured source, natural-language request, environment, submission interface, and grading agree. Validate the setting with an actual configured coding-model run as well as grader checks. This skill contains its authoring guidance and helpers; no outer AGENTS.md, other skill, or historical Harbor workspace is @@ -112,10 +112,12 @@ Follow the detailed rules in `references/task-authoring.md`: 1. Define the submission interface and grader behavior in `raw-instruction.md`. Preserve an existing structured instruction there before rewriting. Follow [references/instruction-rewrite.md](references/instruction-rewrite.md) to write - the agent-facing `instruction.md`: a named person in a concrete physical - setting asking their coding agent for help in long, natural prose. Preserve - every condition and exact interface; do not reveal hidden answers. Audit the - two documents for semantic equivalence before running a model. + the agent-facing `instruction.md`: a direct user request in connected prose + describing the task, current data and starting system, available resources, + constraints, acceptance criteria and optimization objective. Omit + self-introductions, invented identities and fictional scenes. Preserve every + condition and exact interface; do not reveal hidden answers. Audit the two + documents for semantic equivalence before running a model. 2. When adding fixed data/models or external services, read [references/assets-and-resources.md](references/assets-and-resources.md). Build exact manifests and read-only phase-specific mounts. Keep hidden inputs, @@ -137,7 +139,7 @@ diagnosis; retry after a relevant fix, not in an unbounded loop. After local package, asset and grader checks, follow [references/model-evaluation.md](references/model-evaluation.md). Run the final -scenario instruction with the configured coding model and any required judges; +rewritten instruction with the configured coding model and any required judges; inspect its trajectory, artifacts, grader result and judge health to validate the task setting. An infrastructure failure does not measure difficulty, and one low-scoring model run does not establish that a task is impossible. @@ -152,7 +154,7 @@ asking again. Ordinary local checks may proceed. Report the task path, mode/hardware, commands actually executed and their results, formal reference tasks consulted (or that none was close), intentional -reuse/deviations, the raw-to-scenario equivalence review, known-good/negative-case +reuse/deviations, the source-to-request equivalence review, known-good/negative-case evidence, the exact evaluated revision/model/effort, observed scores and judge health, and every unrun validation layer. Never equate a scaffold, a dry-run, or a static release check with a solvable diff --git a/.agents/skills/create-searchswe-task/references/instruction-rewrite.md b/.agents/skills/create-searchswe-task/references/instruction-rewrite.md index 78b8703..c8a1227 100644 --- a/.agents/skills/create-searchswe-task/references/instruction-rewrite.md +++ b/.agents/skills/create-searchswe-task/references/instruction-rewrite.md @@ -11,7 +11,7 @@ For an existing task, first copy the original instruction byte for byte to structured specification there first. Keep the source organized enough to review paths, interfaces, outputs, limits, scoring and restrictions precisely. For later task changes, update the source deliberately and regenerate the -scenario; never silently change a requirement during a stylistic rewrite. +request; never silently change a requirement during a stylistic rewrite. Read the actual environment, resource documents and verifier before rewriting. If the source conflicts with them, record the discrepancy and resolve the @@ -21,19 +21,28 @@ snapshot when migrating existing tasks. ## Write the request -Give the requester a name, a plausible role, a concrete physical workplace and -a reason to need this system. Write as that person explaining the problem to -their coding agent: a long, coherent natural-language request with connected -paragraphs, practical context and requirements woven into the story. The -setting should suit the supplied data; do not invent its provenance or claim -that a benchmark corpus contains a different kind of material. +Write the request a user would give their coding agent to get the work done. +Start with what needs to be built or improved, then connect the current data +and supplied starting system, available resources, restrictions, required +deliverables, acceptance criteria and optimization objective in natural prose. +Explain what counts as better using the actual metric, baseline, quality gates +and resource tradeoffs in the source. Include enough detail to preserve the +whole contract; length should come from those requirements, not filler. + +Do not introduce a named requester, a self-introduction such as "I'm Alex", an +invented role, physical workplace, personal history or fictional motivation. +A direct request such as "I need a system that..." is fine. Use context grounded +in the supplied task, such as vectors that are already computed or a starter +whose retrieval quality needs improvement. Do not invent dataset provenance, +usage claims or a story around the work. Use no headings, enumerated requirements, bullet lists, tables, directory trees or fenced code blocks in the final instruction. Exact commands, paths, field names and JSON examples can appear inline within sentences. Turning -each old bullet into a paragraph or adding a story before the old checklist -does not complete the rewrite. The prose should require the agent to extract -and connect requirements across the request, while remaining unambiguous. +each old bullet into an isolated sentence does not complete the rewrite. +Connect requirements where they belong in the workflow so the request reads +naturally and remains unambiguous. Avoid deliberate obscurity, repetitive +reminders and decorative opening or closing paragraphs. Preserve every source condition, including optional permissions and exceptions: @@ -49,8 +58,8 @@ Preserve every source condition, including optional permissions and exceptions: - Every visible validation file and referenced resource document, the hidden split boundary, metric/formula, failure behavior and scope of zero scores. -Narrative details are motivation, never extra acceptance criteria. Do not add -new deadlines, hardware, output fields, evaluation claims or solution hints. +Do not add new deadlines, hardware, output fields, evaluation claims or solution +hints, or manufacture task requirements to make the request sound realistic. Do not drop repeated conditions unless their full meaning survives elsewhere. Keep the original language unless the requested rewrite includes translation. Do not refer the agent to `raw-instruction.md` or copy it into the agent image @@ -59,10 +68,12 @@ or mounts; doing so exposes an alternate structured prompt. ## Review before evaluation Make a small author-only correspondence record: source condition or line range, -the scenario paragraph containing it, and any discrepancy. Review both ways: +the request paragraph containing it, and any discrepancy. Review both ways: every raw condition must appear in the prose, and every prose obligation must come from the source or an explicitly resolved contract correction. Read the -whole request for natural flow after this check. +whole request for natural flow after this check. Confirm it focuses on the +work, inputs, resources, constraints and measurable outcome, without an +invented persona or setting. Literal comparisons of paths, flags, IDs and numbers catch omissions but cannot prove semantic equivalence. Inspect negations, exceptions, time accounting, diff --git a/.agents/skills/create-searchswe-task/references/model-evaluation.md b/.agents/skills/create-searchswe-task/references/model-evaluation.md index 1555837..a02b6f9 100644 --- a/.agents/skills/create-searchswe-task/references/model-evaluation.md +++ b/.agents/skills/create-searchswe-task/references/model-evaluation.md @@ -78,7 +78,7 @@ Check all of the following, not just a directory name or reward: trials completed, and zero errored/running/pending trials. - Trial logs, agent exit, artifact collection and the separate verifier agree with that terminal state. The transcript actually uses the selected model, - effort and scenario prompt. + effort and reviewed instruction. - Required judges ultimately returned valid, parseable decisions, with no unresolved authentication, transport, rate-limit, parse or answer-judging errors. Record recovered internal retries separately from full trial attempts. diff --git a/.agents/skills/create-searchswe-task/references/task-authoring.md b/.agents/skills/create-searchswe-task/references/task-authoring.md index d998d85..4607d09 100644 --- a/.agents/skills/create-searchswe-task/references/task-authoring.md +++ b/.agents/skills/create-searchswe-task/references/task-authoring.md @@ -113,7 +113,7 @@ tasks// Also provide the repository conventions that apply to the task: ```text -├── raw-instruction.md # Structured source for the scenario instruction +├── raw-instruction.md # Structured source for the natural-language request ├── .gitignore # Ignore /data/, /models/, Python caches ├── README.md # Author-facing overview and provenance ├── environment/ @@ -251,7 +251,7 @@ both versions: Do not hide a correctness requirement only in the grader. Conversely, do not mention `solution/`, reveal hidden cases/labels, or tell the agent how reward is implemented. Keep the structured source outside agent mounts and Docker COPY -paths; never point the scenario instruction to it as an easier alternate task. +paths; never point the rewritten instruction to it as an easier alternate task. Keep `environment/docs/environment.md` synchronized with the actual base image, task additions, paths, CPU/GPU capability, and available commands. An diff --git a/.agents/skills/create-searchswe-task/references/validation.md b/.agents/skills/create-searchswe-task/references/validation.md index 27d7092..6fd7882 100644 --- a/.agents/skills/create-searchswe-task/references/validation.md +++ b/.agents/skills/create-searchswe-task/references/validation.md @@ -28,7 +28,7 @@ for an authorized remote inspection or PR handoff, not local task authoring. assumptions were inherited without independent evidence. - Review `raw-instruction.md` against `instruction.md` using [instruction-rewrite.md](instruction-rewrite.md). Account for every source - condition in the scenario and trace graded conditions to the verifier. + condition in the request and trace graded conditions to the verifier. Confirm the running agent only receives the rewritten instruction. Public limits must not exist only in hidden tests. - Parse `task.toml` and `assets.json`; verify unique identity, correct mode, diff --git a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py index 7776053..2034807 100755 --- a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py @@ -131,8 +131,10 @@ def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: "instruction.md": clean( f""" Replace this scaffold for {task_id} after completing raw-instruction.md. - Write a long natural-language request from a named person in a real - physical setting to their coding agent. Use prose paragraphs without + Write a direct natural-language request describing the task, current + data and starting system, resources, constraints, acceptance criteria, + and optimization objective. Omit self-introductions, invented people, + roles, and fictional scenes. Use connected prose paragraphs without headings, lists, tables, or fenced blocks. Preserve every source condition and exact interface; follow references/instruction-rewrite.md in the authoring skill and remove this author-only prompt before use. diff --git a/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py index 2074d36..a6dca36 100644 --- a/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py @@ -41,10 +41,10 @@ def test_all_mode_hardware_pairs_and_fail_closed_entrypoint(self): self.assertEqual(result.returncode, 0, result.stderr) task = self.repo / "tasks" / name raw = (task / "raw-instruction.md").read_text() - scenario = (task / "instruction.md").read_text() + instruction = (task / "instruction.md").read_text() self.assertTrue(raw.strip()) - self.assertTrue(scenario.strip()) - self.assertNotEqual(raw, scenario) + self.assertTrue(instruction.strip()) + self.assertNotEqual(raw, instruction) config = tomllib.loads((task / "task.toml").read_text()) self.assertEqual(config["metadata"]["task_type"], kind) self.assertEqual(config["environment"]["gpus"], gpus) diff --git a/tasks/task-1-1/instruction.md b/tasks/task-1-1/instruction.md index b2be730..6f12369 100644 --- a/tasks/task-1-1/instruction.md +++ b/tasks/task-1-1/instruction.md @@ -1,15 +1,15 @@ -I'm Maya Chen, and I'm working at a desk in our university library's digital services room, trying to turn the document collection on this machine into something people can actually search. When someone types a natural-language question, I need an executable retrieval system that finds and ranks the most relevant documents in the supplied corpus. My priority is retrieval quality on questions we have held back, within the execution interface and resource limits of this environment. Please take this on as my coding agent and leave me a system I can hand over for evaluation. +Build an executable retrieval system over the supplied document corpus. Given a natural-language query, it should retrieve and rank the most relevant corpus documents. The goal is to maximize retrieval quality on held-out queries while satisfying the executable interface and resource constraints described here. -Before choosing your approach, read /task/docs/environment.md and /task/docs/available_resources.md. They describe the installed Python environment, system runtime, available packages and tools, and the retrieval and generative API resources that are actually permitted, including their model allowlist, providers, endpoints, credential handling, usage restrictions and jailbreak penalties. Those documents are read-only task data. You may use an external model or service only if that resource document explicitly permits it, and only in its documented retrieval or query-transformation role. An unlisted provider, model or endpoint is not an option. Harbor injects the documented credentials at runtime; please never put them into the task package or Docker image. +Before choosing your approach, read /task/docs/environment.md and /task/docs/available_resources.md. They describe the installed Python environment, system runtime, available packages and tools, and the retrieval and generative API resources that are actually permitted, including their model allowlist, providers, endpoints, credential handling, usage restrictions and jailbreak penalties. Those documents are read-only task data. You may use an external model or service only if that resource document explicitly permits it, and only in its documented retrieval or query-transformation role. An unlisted provider, model or endpoint is not an option. Harbor injects the documented credentials at runtime; do not put them into the task package or Docker image. The complete collection is /task/data/corpus.jsonl. For development, /task/data/validation/queries.jsonl contains public queries in the same record schema as the hidden queries, and /task/data/validation/ground_truth.jsonl contains their normalized labels. Use the supplied corpus and task data to build real retrieval. Do not obtain relevance judgments or results from hidden labels, hard-coded query-to-document mappings, precomputed query-to-result mappings or answer files, external datasets containing evaluation results or judgments, an external search engine, or an external retrieval service. The resource permissions above do not authorize those shortcuts. The questions used for the final evaluation and their private relevance judgments are held out, disjoint from the public development examples, and never copied into your environment. -All submission files you create or change must stay under /app, and /task is read-only. Please budget your implementation, validation and debugging together: the agent phase allows at most 120 minutes. I need executable files named /app/build.sh and /app/run.sh at the end. You may keep implementation modules in /app/src and implementation notes or self-test details in /app/README.md, but those two additions are optional. During verification the submission runs as a non-root user and /app is read-only; the supplied --index-dir is the only writable location for persistent and runtime-generated artifacts. Design around that boundary rather than expecting extra writable fixed locations. +All submission files you create or change must stay under /app, and /task is read-only. Implementation, validation and debugging must fit within the 120-minute agent phase. The final submission must contain executable /app/build.sh and /app/run.sh files. You may keep implementation modules in /app/src and implementation notes or self-test details in /app/README.md, but those two additions are optional. During verification the submission runs as a non-root user and /app is read-only; the supplied --index-dir is the only writable location for persistent and runtime-generated artifacts. -The handover starts with the verifier invoking /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. This must construct every required local index and runtime artifact and return exit status 0 when they are ready. A background service is fine if useful, but the verifier will not keep a foreground process alive for you: build.sh must finish successfully before any query invocation begins. The --index-dir argument identifies the place for persistent indexes and other build artifacts, and the verifier must not need to know any additional fixed paths. +The verifier starts by invoking /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. This must construct every required local index and runtime artifact and return exit status 0 when they are ready. A background service is fine if useful, but the verifier will not keep a foreground process alive for you: build.sh must finish successfully before any query invocation begins. The --index-dir argument identifies the place for persistent indexes and other build artifacts, and the verifier must not need to know any additional fixed paths. -Once that build succeeds, the query entry point is invoked as /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 3. Process every supplied query and write exactly one result object for each input query to the requested output. Choose whatever internal protocol or service endpoint works with your build process and this command-line interface. I need JSONL output, where a line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}; the single result shown illustrates an item's fields, while an invocation with --top-k 3 must actually return exactly three items for every valid query. +Once that build succeeds, the query entry point is invoked as /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 3. Process every supplied query and write exactly one result object for each input query to the requested output. Choose whatever internal protocol or service endpoint works with your build process and this command-line interface. Write JSONL output, where a line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}; the single result shown illustrates an item's fields, while an invocation with --top-k 3 must actually return exactly three items for every valid query. -Please preserve the corresponding input identifier in query_id. Each returned doc_id must identify a document in the supplied corpus, each score must be finite and numeric, and a query must never contain duplicate document IDs. Rank results from highest to lowest score, using corpus order to resolve equal scores. These details matter to the downstream consumer just as much as having build.sh and run.sh present and executable. +Please preserve the corresponding input identifier in query_id. Each returned doc_id must identify a document in the supplied corpus, each score must be finite and numeric, and a query must never contain duplicate document IDs. Rank results from highest to lowest score, using corpus order to resolve equal scores. -After your work, Harbor runs the submission with the task runtime and private test split. It checks genuine corpus retrieval, all the resource and model/provider/endpoint/credential/usage restrictions, successful execution through the specified interfaces, and the JSONL structure, coverage of queries, result counts, duplicate handling, document IDs, scores and ranking. Invalid output or failure at an executable gate receives score 0. For a valid submission, the only final retrieval metric is Accuracy@3: a query earns 1 when at least one relevant corpus document appears in its first three returned results and 0 otherwise, and the displayed task score is 100 times the average of those query values. That is the useful system I want you to leave on this machine. +After your work, Harbor runs the submission with the task runtime and private test split. It checks genuine corpus retrieval, all the resource and model/provider/endpoint/credential/usage restrictions, successful execution through the specified interfaces, and the JSONL structure, coverage of queries, result counts, duplicate handling, document IDs, scores and ranking. Invalid output or failure at an executable gate receives score 0. For a valid submission, the only final retrieval metric is Accuracy@3: a query earns 1 when at least one relevant corpus document appears in its first three returned results and 0 otherwise, and the displayed task score is 100 times the average of those query values. diff --git a/tasks/task-1-2/instruction.md b/tasks/task-1-2/instruction.md index 3f8affc..7a5dca4 100644 --- a/tasks/task-1-2/instruction.md +++ b/tasks/task-1-2/instruction.md @@ -1,15 +1,15 @@ -I'm Daniel Wu, sitting beside the test workstation in our campus search lab. The collection has already been converted into vectors, but we still need a working way to find the right items when someone supplies a query vector. Please build that search system for me. I care about retrieving and ranking the most relevant corpus items on held-out queries under the evaluator's resource constraints, and you can choose any indexing, storage and retrieval architecture that satisfies the interface and those constraints. +The corpus and development queries are already represented by precomputed vectors. Build an executable search system that takes a query vector and retrieves and ranks the most relevant corpus items. Maximize retrieval quality on held-out queries within the evaluator's resource limits; you may choose any indexing, storage and retrieval architecture that satisfies those limits and the required interfaces. -The files on this machine are the material you have to work with. /task/data/vectors.f32 contains 1,500,000 normalized float32 corpus vectors, each with dimension 1024. /task/data/metadata.jsonl has 1,500,000 rows in one-to-one correspondence with them, and /task/data/vector_config.json describes that vector collection. The public development vectors are /task/data/validation/query_vectors.f32, with matching metadata at /task/data/validation/query_metadata.jsonl and configuration at /task/data/validation/vector_config.json. You also have /task/data/validation/queries.jsonl for the public query records and /task/data/validation/ground_truth.jsonl for their normalized labels. The final questions use held-out query vectors and private relevance judgments, disjoint from these public examples and not copied into your environment. +/task/data/vectors.f32 contains 1,500,000 normalized float32 corpus vectors, each with dimension 1024. /task/data/metadata.jsonl has 1,500,000 rows in one-to-one correspondence with them, and /task/data/vector_config.json describes that vector collection. The public development vectors are /task/data/validation/query_vectors.f32, with matching metadata at /task/data/validation/query_metadata.jsonl and configuration at /task/data/validation/vector_config.json. You also have /task/data/validation/queries.jsonl for the public query records and /task/data/validation/ground_truth.jsonl for their normalized labels. The final evaluation uses held-out query vectors and private relevance judgments, disjoint from these public examples and not copied into your environment. -Please read /task/docs/environment.md before implementing anything so you know the installed Python environment, system runtime, packages and tools. That document is read-only task data, as is /task generally. Create or modify submission files only under /app. Retrieval must use only the supplied precomputed vectors and task data: no external models, retrieval services, external datasets containing evaluation results, or precomputed query-to-result mappings. I need genuine vector retrieval, not answers obtained from hidden labels, hard-coded query-to-document mappings, precomputed answer files, or external datasets containing the evaluation judgments. Plan your implementation, validation and debugging within the agent's total allowance of 120 minutes. +Please read /task/docs/environment.md before implementing anything so you know the installed Python environment, system runtime, packages and tools. That document is read-only task data, as is /task generally. Create or modify submission files only under /app. Retrieval must use only the supplied precomputed vectors and task data: no external models, retrieval services, external datasets containing evaluation results, or precomputed query-to-result mappings. Implement genuine vector retrieval over the supplied corpus. Do not obtain answers from hidden labels, hard-coded query-to-document mappings, precomputed answer files, or external datasets containing the evaluation judgments. Plan your implementation, validation and debugging within the agent's total allowance of 120 minutes. -At handover, /app/build.sh and /app/run.sh must both be executable. Optional implementation modules can go in /app/src, and optional notes and self-test details in /app/README.md. Keep in mind that verification runs your submission as a non-root user with /app read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts. The evaluator must not have to know another fixed path for the indexes, runtime files or other build artifacts. +The final submission must include executable /app/build.sh and /app/run.sh files. Optional implementation modules can go in /app/src, and optional notes and self-test details in /app/README.md. Keep in mind that verification runs your submission as a non-root user with /app read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts. The evaluator must not have to know another fixed path for the indexes, runtime files or other build artifacts. -To prepare the system, the verifier will call /app/build.sh --vectors /task/data/vectors.f32 --metadata /task/data/metadata.jsonl --config /task/data/vector_config.json --index-dir /app/index. Have that command construct every required index and runtime artifact, start the search service, and return exit status 0 only when the service is ready to accept queries. The whole build process, including index construction and service startup, must finish within 120 seconds. The service must then remain available for repeated run.sh calls; returning from build.sh cannot mean shutting down the thing those calls need. +To prepare the system, the verifier will call /app/build.sh --vectors /task/data/vectors.f32 --metadata /task/data/metadata.jsonl --config /task/data/vector_config.json --index-dir /app/index. Have that command construct every required index and runtime artifact, start the search service, and return exit status 0 only when the service is ready to accept queries. The whole build process, including index construction and service startup, must finish within 120 seconds. The service must then remain available for repeated run.sh calls. The query invocation looks like /app/run.sh --index-dir /app/index --query-vectors /task/data/validation/query_vectors.f32 --query-metadata /task/data/validation/query_metadata.jsonl --query-config /task/data/validation/vector_config.json --output /tmp/results.jsonl --top-k 3. Process every query supplied to that invocation and write exactly one result object for each input query. Your internal protocol or service endpoint is your choice provided it works with build.sh and this command-line interface. The verifier may start run.sh separately for every query. Each such invocation has just 0.5 seconds of end-to-end wall-clock time, measured from process start through process exit, so the entire invocation must fit rather than merely the search calculation inside it. The output is JSONL. A line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}, with exactly three result items when --top-k 3 is supplied. Preserve the corresponding input query's identifier as query_id. Every item needs a doc_id from the supplied corpus and a finite numeric score; no document ID may repeat within a query. Order the items by descending score and break equal-score ties using corpus order. -After the agent phase, Harbor uses the task runtime and private test split to check retrieval integrity, executable entry points, build completion and service readiness, query execution, and the validity of the JSONL structure, query coverage, result counts, duplicate handling, document IDs, scores and ranking. A bad output or failed run.sh invocation gives that query zero credit, while the verifier continues with the other queries. A hidden query earns 1 only if its invocation exits successfully within 0.5 seconds, its output is valid, and at least one relevant document is in the first three results; otherwise it earns 0. The primary metric is the average of these latency-gated Accuracy@3 values. The normalized reward is that average, and the displayed score is 100 times the average. A failed build or failed retrieval-integrity check makes the overall score 0. Please leave me an implementation that meets this whole operating arrangement, so we can judge its retrieval quality fairly. +After the agent phase, Harbor uses the task runtime and private test split to check retrieval integrity, executable entry points, build completion and service readiness, query execution, and the validity of the JSONL structure, query coverage, result counts, duplicate handling, document IDs, scores and ranking. A bad output or failed run.sh invocation gives that query zero credit, while the verifier continues with the other queries. A hidden query earns 1 only if its invocation exits successfully within 0.5 seconds, its output is valid, and at least one relevant document is in the first three results; otherwise it earns 0. The primary metric is the average of these latency-gated Accuracy@3 values. The normalized reward is that average, and the displayed score is 100 times the average. A failed build or failed retrieval-integrity check makes the overall score 0. diff --git a/tasks/task-1-3/instruction.md b/tasks/task-1-3/instruction.md index fa5fa4f..58c25e5 100644 --- a/tasks/task-1-3/instruction.md +++ b/tasks/task-1-3/instruction.md @@ -1,10 +1,10 @@ -I'm Elena Park, working from a desk in our research institute's reading room with a collection of scientific papers that I need to consult repeatedly. I want you, my coding agent, to build an executable retrieval-augmented question answering system over the PDFs supplied here. When I ask a scientific question, it should give a concise answer and identify the paper supporting it. Please aim for the best answer correctness and source attribution on held-out questions; you may choose any method that respects the execution interface and resources. +Build an executable retrieval-augmented question answering system over the supplied scientific-paper PDFs. Given a scientific question, it should return a concise answer and identify the paper supporting that answer. The goal is to maximize answer correctness and source attribution quality on held-out questions. You may use any method that satisfies the executable interface and resource constraints. There are 300 scientific-paper PDFs in /task/data/corpus. Public development questions are in /task/data/validation/queries.jsonl and use the same record schema as hidden questions; /task/data/validation/golden_answers.jsonl supplies their reference answers and evidence document IDs. Use these papers to answer, without precomputed query-to-answer mappings, hidden labels, hard-coded evaluation answers, or external datasets or services containing evaluation answers. The held-out queries and private relevance judgments are disjoint from the development examples and unavailable in your environment. Before implementing the system, read /task/docs/environment.md for the installed Python environment, runtime, packages and system tools, and /task/docs/available_resources.md for the optional retrieval and generative APIs, exact model allowlists, provider and endpoint restrictions, usage rules, runtime credential handling and jailbreak penalties. Follow those restrictions rather than assuming an unlisted service, model or endpoint is allowed; unauthorized datasets and services are prohibited. Harbor supplies the documented credentials at runtime, and they must never be put in the task package or Docker image. These documents and all of /task are read-only. Create or change submission files only under /app, and finish implementation, validation and debugging within the 120-minute agent allowance. -I need executable /app/build.sh and /app/run.sh files in the final handover. You can organize optional modules under /app/src and leave optional implementation and self-test notes in /app/README.md. The preparation call is /app/build.sh --corpus /task/data/corpus --index-dir /app/index. Build all required indexes and runtime artifacts, start the search service and return exit status 0 only after it is ready for queries. It must stay available through repeated run.sh invocations. Use the supplied --index-dir for persistent indexes and other build artifacts, and honor supplied paths generally: the verifier's index and output directories may differ from the example locations. The build must finish within 3600 seconds. +The final submission must include executable /app/build.sh and /app/run.sh files. You can organize optional modules under /app/src and leave optional implementation and self-test notes in /app/README.md. The preparation call is /app/build.sh --corpus /task/data/corpus --index-dir /app/index. Build all required indexes and runtime artifacts, start the search service and return exit status 0 only after it is ready for queries. It must stay available through repeated run.sh invocations. Use the supplied --index-dir for persistent indexes and other build artifacts, and honor supplied paths generally: the verifier's index and output directories may differ from the example locations. The build must finish within 3600 seconds. After a successful build, the query interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Whatever internal protocol or service endpoint you use must work with the build process and this interface. Process every supplied query and write exactly one result object for each. The verifier builds once and then invokes run.sh once for each hidden query, with up to five invocations in parallel. They share the task's CPU, memory, index and service resources. Each query has 900 seconds, counted from the start of its run.sh process; waiting for an available slot does not use up that query's individual allowance. @@ -12,4 +12,4 @@ For the output, write one JSON object per line in the form {"query_id":"query-id Harbor will transfer /app into a separate verifier with the same runtime and the private split after you finish; your build.sh must restart the service there. The verifier phase has 14,400 seconds overall, including the arrangement of a 3,600-second build and 900 seconds per query with up to five concurrent queries. It checks retrieval and answer integrity, resource compliance, executable interfaces and output validity. A failed execution or invalid output gives score 0 to the entire submission, rather than just dropping the affected question. -The final metric is LLMJudgeAccuracy, equally averaged over every hidden query. For a query to earn 1, its evidence document must match the reference and an LLM judge must accept its answer as semantically equivalent to the reference answer; otherwise it earns 0. The answer judge only evaluates an answer after its evidence document matches. The normalized reward is the mean of those query values, and the displayed score is 100 times that reward. An independent trajectory audit also checks compliance with the task and resource restrictions; a failed audit makes the entire submission's reward and score 0. That combination of a useful answer and the right source is what I need from this system. +The final metric is LLMJudgeAccuracy, equally averaged over every hidden query. For a query to earn 1, its evidence document must match the reference and an LLM judge must accept its answer as semantically equivalent to the reference answer; otherwise it earns 0. The answer judge only evaluates an answer after its evidence document matches. The normalized reward is the mean of those query values, and the displayed score is 100 times that reward. An independent trajectory audit also checks compliance with the task and resource restrictions; a failed audit makes the entire submission's reward and score 0. diff --git a/tasks/task-1-4/instruction.md b/tasks/task-1-4/instruction.md index c1716cc..1e6a0b0 100644 --- a/tasks/task-1-4/instruction.md +++ b/tasks/task-1-4/instruction.md @@ -1,19 +1,19 @@ -I'm Sofia Alvarez, and I'm sitting in our archive's reading room with a long PDF open on the workstation beside me. Finding the passage I need by scrolling has become impractical. Please build an executable evidence-localization system over the single PDF supplied here: when I describe what I'm looking for in natural language, it should return the physical pages most likely to contain the relevant evidence. I want the best localization quality on held-out queries, and you may use any retrieval architecture that fits the executable interface and resource limits. +Build an executable evidence-localization system for the supplied single long PDF. Given a natural-language query describing the evidence to find, return the physical pages in that PDF that best contain it. Maximize localization quality on held-out queries using any retrieval architecture that satisfies the executable interface and resource constraints. The corpus directory /task/data/corpus contains exactly one PDF, the same long document used for development and hidden evaluation. Public queries are in /task/data/validation/queries.jsonl, following the same record schema as hidden queries, and /task/data/validation/golden_answers.jsonl contains their reference physical pages and supporting excerpts. A page means the 1-based physical page index in the PDF. Please build genuine localization over this document, without hidden labels, hard-coded query-to-location mappings, precomputed query-to-answer mappings or answer files, external datasets containing evaluation judgments, or external search or answer services that supply evidence locations or evaluation answers. Read /task/docs/environment.md and /task/docs/available_resources.md before implementing. They describe the installed Python environment, runtime, packages and system tools, and optional retrieval and generative resources, including exact model allowlists, provider and endpoint restrictions, runtime environment-variable and credential handling, usage rules and jailbreak penalties. Only listed resources may be used according to those restrictions; an unlisted model, provider or endpoint is not implicitly permitted. Harbor injects the documented credentials at runtime, and you must not put them in the package or Docker image. Both documents are read-only task data. Treat all of /task as read-only and create or modify submission files only under /app. Your implementation, validation and debugging must fit into the agent's 120-minute allowance. -At the end, leave executable /app/build.sh and /app/run.sh entry points. Optional modules may live in /app/src and optional implementation notes and self-test details in /app/README.md. The verifier prepares the system by invoking /app/build.sh --corpus /task/data/corpus --index-dir /app/index. That command must create all required index and runtime artifacts, start the search service, and return exit status 0 only when the service is ready for queries. The service must continue working across repeated run.sh calls. The supplied --index-dir is where persistent indexes and other build artifacts belong. Honor the passed paths: verification mounts the same PDF at a different location, and its index and output directories may differ from these examples. +Provide executable /app/build.sh and /app/run.sh entry points. Optional modules may live in /app/src and optional implementation notes and self-test details in /app/README.md. The verifier prepares the system by invoking /app/build.sh --corpus /task/data/corpus --index-dir /app/index. That command must create all required index and runtime artifacts, start the search service, and return exit status 0 only when the service is ready for queries. The service must continue working across repeated run.sh calls. The supplied --index-dir is where persistent indexes and other build artifacts belong. Honor the passed paths: verification mounts the same PDF at a different location, and its index and output directories may differ from these examples. Once build.sh has returned successfully, the query interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Each input line looks like {"query_id":"opaque-query-id","query":"Describe the passage to locate."}. Query IDs are opaque, all queries concern the sole PDF, and no target field is supplied. Process every input query and write exactly one result object per query. The verifier invokes run.sh separately for each query, with up to five concurrent processes sharing the index and any service started by the build. Choose any internal protocol or endpoint that works with this build lifecycle and command-line interface. Each output line must be a JSON object like {"query_id":"query-id","results":[{"page":12,"evidence":"text from page 12","score":0.9},{"page":13,"evidence":"text from page 13","score":0.8},{"page":25,"evidence":"text from page 25","score":0.7},{"page":26,"evidence":"text from page 26","score":0.6},{"page":30,"evidence":"text from page 30","score":0.5}]}. Preserve the input query_id and return exactly five items when invoked with --top-k 5. Each item needs a valid physical page from this PDF, a non-empty evidence string from that page, and a finite numeric score. Pages must be unique within the query, results must be in descending score order, and equal scores must be resolved by ascending page number. The evidence text must actually correspond to the reported page. -When you're done, Harbor transfers /app to a fresh verifier environment with the same runtime. That environment gets the same PDF and private queries, but not the public validation queries or labels. Agent-phase processes are not transferred, so build.sh must restart the service. Submission commands execute as an unprivileged user; anything needed at evaluation time must already be in the base image or installed under /app. +After the agent phase, Harbor transfers /app to a fresh verifier environment with the same runtime. That environment gets the same PDF and private queries, but not the public validation queries or labels. Agent-phase processes are not transferred, so build.sh must restart the service. Submission commands execute as an unprivileged user; anything needed at evaluation time must already be in the base image or installed under /app. -The verifier gives build.sh 2,400 seconds. Every query has its own run.sh process with a 900-second limit and up to five such processes may run concurrently, but they all share a 1,800-second execution budget. Questions still unfinished or not yet launched when that shared deadline arrives score zero. Time spent queuing does not consume a query's individual allowance, although the shared deadline keeps advancing. The runner gets an additional 60 seconds for cleanup and result collection, and all of this sits within a 10,800-second verifier phase. Please account for those different clocks in the service design. +The verifier gives build.sh 2,400 seconds. Every query has its own run.sh process with a 900-second limit and up to five such processes may run concurrently, but they all share a 1,800-second execution budget. Questions still unfinished or not yet launched when that shared deadline arrives score zero. Time spent queuing does not consume a query's individual allowance, although the shared deadline keeps advancing. The runner gets an additional 60 seconds for cleanup and result collection, and all of this sits within a 10,800-second verifier phase. The verifier checks the executable entry points and service startup, real evidence localization, and each query's JSONL structure, result count, page numbers, evidence, scores, duplicates and ranking. A failed or timed-out query scores zero while the others continue; missing or invalid output also scores zero for that query and stays in the average's denominator. A failure to build the shared index or initialize verification prevents every query from scoring. Page relevance is checked deterministically against gold labels, with no model judging answer correctness. A model audit checks trajectory compliance only. -For a valid submission that passes that audit, the only localization metric is Recall@5. For one query this is the number of relevant evidence pages retrieved in the top five divided by the number of relevant evidence pages for that query. Reward is average Recall@5 with failed or invalid queries contributing zero, so it lies between 0 and 1. The evaluation report displays 100 times average Recall@5 before the trajectory gate; a failed trajectory audit sets the final reward to zero. Hidden queries and their private page-relevance judgments concern this same PDF but differ from the public examples and are not copied into your environment. Their IDs are independent opaque identifiers, unique across splits, and their order says nothing about page position. I need the returned pages to be supported by the document itself. +For a valid submission that passes that audit, the only localization metric is Recall@5. For one query this is the number of relevant evidence pages retrieved in the top five divided by the number of relevant evidence pages for that query. Reward is average Recall@5 with failed or invalid queries contributing zero, so it lies between 0 and 1. The evaluation report displays 100 times average Recall@5 before the trajectory gate; a failed trajectory audit sets the final reward to zero. Hidden queries and their private page-relevance judgments concern this same PDF but differ from the public examples and are not copied into your environment. Their IDs are independent opaque identifiers, unique across splits, and their order says nothing about page position. diff --git a/tasks/task-2-1/README.md b/tasks/task-2-1/README.md index d25f950..bb810c3 100644 --- a/tasks/task-2-1/README.md +++ b/tasks/task-2-1/README.md @@ -129,7 +129,7 @@ assets, credentials, or hardware. ## Instruction source -`raw-instruction.md` preserves the original structured instruction. The scenario +`raw-instruction.md` preserves the original structured instruction. The request in `instruction.md` corrects its `/tast/model/bge-reranker-large` path typo to the actual read-only mount `/task/models/bge-reranker-large`; the fixed model and all candidate/model restrictions are unchanged. diff --git a/tasks/task-2-1/instruction.md b/tasks/task-2-1/instruction.md index 989d9e9..852b3f0 100644 --- a/tasks/task-2-1/instruction.md +++ b/tasks/task-2-1/instruction.md @@ -1,13 +1,13 @@ -I'm Arun Mehta, working at the review table in our financial research office. We have a long-document search pipeline already, but the ordering of its initial results needs work. Please improve the supplied reranking system for me. Each query comes with a fixed BM25 Top-100 set of candidate documents, and I want you to rerank those candidates with our supplied local BAAI/bge-reranker-large model to maximize retrieval quality on held-out financial queries while respecting the interfaces and resources. +Improve the supplied long-document reranking system. Each query already has a fixed BM25 Top-100 set of candidate documents, which must be reranked with the supplied local BAAI/bge-reranker-large model. The goal is to maximize retrieval quality on held-out financial queries by improving the reranking pipeline within the required interfaces and resource limits. -The candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The model is mounted at /task/models/bge-reranker-large; older task notes spell this /tast/model/bge-reranker-large, and that spelling refers to the same supplied model, not permission to fetch a replacement. Read /task/docs/environment.md and /task/docs/available_resources.md for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. +The candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The supplied model is mounted at /task/models/bge-reranker-large. Read /task/docs/environment.md and /task/docs/available_resources.md for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. -You have the full corpus at /task/data/corpus.jsonl, public development queries at /task/data/validation/queries.jsonl in the same record schema used by hidden queries, and their normalized labels at /task/data/validation/ground_truth.jsonl. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. This needs to be a genuine reranking pipeline over the candidates we provide. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. +You have the full corpus at /task/data/corpus.jsonl, public development queries at /task/data/validation/queries.jsonl in the same record schema used by hidden queries, and their normalized labels at /task/data/validation/ground_truth.jsonl. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. Create or modify submission files only under /app. Treat /task, /tests and the task-provided documents as read-only. The total agent allowance for implementation, validation and debugging is 120 minutes. The final package needs executable /app/build.sh and /app/run.sh files; optional implementation modules can go in /app/src and optional notes and self-test details in /app/README.md. During verification you run as a non-root user and /app is read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts, including preprocessing and cache state, and the verifier must not need to know any additional fixed paths. -The verifier starts with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. Please have it prepare all required corpus, model and runtime artifacts, start the reranking service, and return exit status 0 only when the system can process queries. It must keep that service available across repeated run.sh calls. After build.sh succeeds, queries arrive through /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Every query record carries its fixed BM25 Top-100 candidates; rerank only those candidates and write exactly one result object for each input query. The internal protocol or service endpoint is your choice as long as it works with the build process and the specified interface. +The verifier starts with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must prepare all required corpus, model and runtime artifacts, start the reranking service, and return exit status 0 only when the system can process queries. It must keep that service available across repeated run.sh calls. After build.sh succeeds, queries arrive through /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Every query record carries its fixed BM25 Top-100 candidates; rerank only those candidates and write exactly one result object for each input query. The internal protocol or service endpoint is your choice as long as it works with the build process and the specified interface. Write JSONL output with a line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. That example shows the fields of a result item; --top-k 5 requires exactly five items for each valid query. Keep the corresponding input identifier in query_id. Each doc_id must identify one of that query's supplied candidates and each score must be finite and numeric. Document IDs must not repeat within a query. Sort by descending score and, for equal scores, preserve original candidate order as the tie-break. -After the agent phase, Harbor uses the task runtime and private split to verify genuine reranking, the fixed candidate set, and use of the unchanged supplied local model. It checks that build.sh and run.sh are executable, waits for the build to return successfully, invokes the query entry point and validates the JSONL structure, query coverage, count of results, duplicates, candidate document IDs, scores and ranking. Invalid output or failure at an executable gate gives a zero score. A valid submission is scored only by Accuracy@5: a query earns 1 if at least one relevant corpus document is among its first five results and 0 otherwise, and the final score is 100 times the average of those values. Please improve how well we order the allowed candidates within those boundaries. +After the agent phase, Harbor uses the task runtime and private split to verify genuine reranking, the fixed candidate set, and use of the unchanged supplied local model. It checks that build.sh and run.sh are executable, waits for the build to return successfully, invokes the query entry point and validates the JSONL structure, query coverage, count of results, duplicates, candidate document IDs, scores and ranking. Invalid output or failure at an executable gate gives a zero score. A valid submission is scored only by Accuracy@5: a query earns 1 if at least one relevant corpus document is among its first five results and 0 otherwise, and the final score is 100 times the average of those values. diff --git a/tasks/task-2-2/instruction.md b/tasks/task-2-2/instruction.md index 2f77460..aba6fae 100644 --- a/tasks/task-2-2/instruction.md +++ b/tasks/task-2-2/instruction.md @@ -1,9 +1,9 @@ -I'm Nora Singh, sitting at a workstation in our software engineering lab where we are trying to make code easier to find. We already have an embedder and a corpus to work from, and I'd like you to improve that supplied code-retrieval embedder using the corpus and training data provided here. The handover I need is a checkpoint, because the evaluation will use exact Top-1 retrieval on held-out queries to see whether your changes help. +Improve the supplied code-retrieval embedder using the provided corpus and training data, and deliver the result as a model checkpoint. It will be compared with the original checkpoint using exact Top-1 retrieval on held-out queries. Larger improvement in held-out Accuracy@1 is better, subject to preserving the supplied backbone. -Please read /task/docs/environment.md and /task/docs/available_resources.md first. The supplied backbone is mounted read-only under /task/models, and your starting point must be /task/models/bge-base-en-v1.5. Preserve its architecture, parameter names, parameter shapes and parameter count. I am asking for an improved checkpoint within that same backbone, so keep those properties intact while you work on retrieval quality. All of /task is read-only. +Please read /task/docs/environment.md and /task/docs/available_resources.md first. The supplied backbone is mounted read-only under /task/models, and your starting point must be /task/models/bge-base-en-v1.5. Preserve its architecture, parameter names, parameter shapes and parameter count. All of /task is read-only. -The corpus and training data are under /task/data. There is also a fixed public validation split of 10 queries with labels under /task/data/validation for development. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable, so this work and the final handover must rely on what is supplied locally. +The corpus and training data are under /task/data. There is also a fixed public validation split of 10 queries with labels under /task/data/validation for development. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable; both development and the submitted checkpoint must rely on the supplied local resources. -Write the final model to /app/submission/model as a complete, self-contained offline checkpoint. Both AutoTokenizer.from_pretrained and AutoModel.from_pretrained must be able to load it with local_files_only=True. Anything you expect the verifier to use has to be in the submission it collects: it ignores files outside /app/submission. Please do not leave a checkpoint that requires some uncollected file elsewhere on the workstation in order to load. +Write the final model to /app/submission/model as a complete, self-contained offline checkpoint. Both AutoTokenizer.from_pretrained and AutoModel.from_pretrained must be able to load it with local_files_only=True. The verifier collects only /app/submission and ignores files outside that directory, so the checkpoint must load without depending on uncollected files. -You have a 12-hour agent phase. Task-specific submission verification has a 1-hour limit, which excludes the separate integrity judge. A timeout of the agent phase, the task-specific verification, or the integrity judge results in score 0. Once you finish, the checkpoint is copied into a separate verifier environment, where its architecture is checked against the supplied backbone. The verifier evaluates the original and submitted checkpoints with the same fixed embedding implementation and exact-search implementation. Larger improvement in held-out Accuracy@1 is better. That comparison, within the unchanged backbone contract, is the improvement I need you to deliver. +You have a 12-hour agent phase. Task-specific submission verification has a 1-hour limit, which excludes the separate integrity judge. A timeout of the agent phase, the task-specific verification, or the integrity judge results in score 0. Once you finish, the checkpoint is copied into a separate verifier environment, where its architecture is checked against the supplied backbone. The verifier evaluates the original and submitted checkpoints with the same fixed embedding implementation and exact-search implementation. Larger improvement in held-out Accuracy@1 is better. diff --git a/tasks/task-2-3/instruction.md b/tasks/task-2-3/instruction.md index 8ef4afa..ef4062d 100644 --- a/tasks/task-2-3/instruction.md +++ b/tasks/task-2-3/instruction.md @@ -1,11 +1,11 @@ -I'm Leo Tan, working at the evaluation workstation in our information retrieval lab. We have a fixed collection of document vectors, and I need you to build the query side that searches them well. Please base the query encoder on the supplied 0.6B backbone and aim to maximize held-out Accuracy@1 while meeting the fixed runtime gate. The document representations are already settled; the work I am handing to you is the query-side retrieval system around them. +The document vectors are already fixed. Build a query-side retrieval system for searching them, with a query encoder based on the supplied 0.6B backbone. Maximize held-out Accuracy@1 while meeting the fixed runtime gate and preserving the document vectors. Read /task/docs/environment.md and /task/docs/available_resources.md before you begin. The fixed vectors, corpus mapping and public validation split are under /task/data, and the supplied query backbone and reference starter model are under /task/models. Use /task/models/Qwen3-Embedding-0.6B as the basis of your query-side system. Treat all of /task as read-only and put the complete submission under /app/submission. The vectors supplied through --doc-vectors must remain unchanged: do not modify, replace, re-encode or reorder them. Preserve the row correspondence between /task/data/doc.npy and /task/data/corpus.jsonl. -Agent-stage network access is available, but the system you submit must be self-contained and run without network access. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers, or external retrieval, reranking, generation or answer services. The hidden retrieval queries and their private relevance labels are held out, disjoint from the public validation split and unavailable during the agent phase. Please use the supplied material to develop something that genuinely generalizes to those unseen queries. +Agent-stage network access is available, but the system you submit must be self-contained and run without network access. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers, or external retrieval, reranking, generation or answer services. The hidden retrieval queries and their private relevance labels are held out, disjoint from the public validation split and unavailable during the agent phase. -You have 12 hours for the agent phase. Task-specific submission verification has a 70-minute limit, excluding the separate integrity judge. If the agent phase, task-specific verification or integrity judge times out, the score is 0. The runtime gate is part of the evaluation, so quality improvements have to fit the fixed operating arrangement as well. +You have 12 hours for the agent phase. Task-specific submission verification has a 70-minute limit, excluding the separate integrity judge. If the agent phase, task-specific verification or integrity judge times out, the score is 0. The verifier's call is /app/submission/run.sh --doc-vectors /task/data/doc.npy --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 1. Please make run.sh executable, process every query exactly once in input order, write deterministic results, exit successfully and leave no background processes. The output is JSONL with each line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. With --top-k 1, return exactly one corpus document and a finite numeric score. Ranking is by descending score, with corpus order breaking ties. Include the executable entry point and every required local file under /app/submission, since that directory alone is copied into the verifier. -After you finish, the verifier runs both the fixed reference starter and your submission on the same private queries and the same document vectors. It checks vector integrity, output format, determinism and background processes. Submissions that meet the fixed runtime gate are evaluated on held-out Accuracy@1. Please leave me a complete local query-side system that can be compared under those shared conditions. +After you finish, the verifier runs both the fixed reference starter and your submission on the same private queries and the same document vectors. It checks vector integrity, output format, determinism and background processes. Submissions that meet the fixed runtime gate are evaluated on held-out Accuracy@1. diff --git a/tasks/task-2-4/instruction.md b/tasks/task-2-4/instruction.md index 02c03cb..07eb0f2 100644 --- a/tasks/task-2-4/instruction.md +++ b/tasks/task-2-4/instruction.md @@ -1,4 +1,4 @@ -I'm Grace Lin, working at a desk in our university's research support office. The questions people bring me often combine several constraints, and returning a few documents that each address only an isolated part is not enough. We have a ReAct retrieval starter over the supplied corpus, and I want you to improve it so that a multi-constraint question produces exactly five distinct corpus documents that best address the combined information need. Our objective is Gold Recall@5 on held-out questions, within the interfaces and resources described here. +Optimize the supplied ReAct retrieval starter over the corpus. For each question combining multiple constraints, the system must select exactly five distinct corpus documents that best address the combined information need. The goal is to maximize Gold Recall@5 on held-out questions while preserving the interfaces and resource constraints. The starter lives under /app/starter, with executable /app/build.sh and /app/run.sh entry points. Read /app/starter/README.md to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If OPENROUTER_API_KEY is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective. @@ -6,14 +6,14 @@ Please read /task/docs/environment.md for the installed Python environment, runt The complete corpus is /task/data/corpus.jsonl. There are 20 public development queries in /task/data/validation/queries.jsonl, using the same record schema as the hidden questions, and their gold relevance labels are /task/data/validation/qrels_gold.txt in TREC format, query_id 0 docid 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment. -The preparation interface is /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must finish within 600 seconds, having built all required indexes and runtime artifacts and started the search service. Return exit status 0 only when that service is ready for queries, and keep it alive across repeated run.sh calls. Use the supplied --index-dir for persistent indexes and other build artifacts; the verifier must not have to know any additional fixed paths. The final handover needs executable build.sh and run.sh under /app. You can also keep optional modules in /app/src and optional implementation or self-test notes in /app/README.md. +The preparation interface is /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must finish within 600 seconds, having built all required indexes and runtime artifacts and started the search service. Return exit status 0 only when that service is ready for queries, and keep it alive across repeated run.sh calls. Use the supplied --index-dir for persistent indexes and other build artifacts; the verifier must not have to know any additional fixed paths. The final submission must contain executable build.sh and run.sh under /app. You can also keep optional modules in /app/src and optional implementation or self-test notes in /app/README.md. After a successful build, the query call is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Process every supplied query and write exactly one result object per query. You may choose any internal protocol or service endpoint that works with build.sh and that command-line interface. The verifier builds once and invokes run.sh once per hidden query, with up to five invocations in parallel. The search service and entry point must support those concurrent invocations sharing the same --index-dir, with separate input and output files. They share the task's CPU, memory, index and service resources. -Each query is allowed at most 20 retrieval rounds. A search action returning a candidate set counts as one round; putting independent searches into a batch does not combine them into a single round. There is no individual query time limit, but the whole query execution shares a 15,300-second budget and the verifier phase has a 15,600-second limit. Design your planning and retrieval around those shared bounds as well as the build limit. +Each query is allowed at most 20 retrieval rounds. A search action returning a candidate set counts as one round; putting independent searches into a batch does not combine them into a single round. There is no individual query time limit, but the whole query execution shares a 15,300-second budget and the verifier phase has a 15,600-second limit. The output is JSONL, with a line like {"query_id":"example-1","doc_ids":["100","101","102","103","104"]}. query_id must be a string preserving the input query's identifier. doc_ids must be an array of exactly five non-empty strings, all distinct and matching actual docid values in /task/data/corpus.jsonl. These are the final five selected documents even if your system went through many internal retrieval rounds. Emit exactly one record per input query, with no missing, duplicate or extra query IDs. Harbor transfers /app to a separate verifier with the same runtime and private split after the agent phase. Your build.sh must restart the service there. The verifier checks genuine corpus retrieval, compliance with the resource/model/provider/usage restrictions and the build, shared-time and 20-round limits, executable entry points, and output validity. Failed execution or invalid output receives score 0 for the entire submission. -The only final metric is Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward. Please make the starter more effective at assembling those five documents while keeping the whole handover runnable under the stated conditions. +The only final metric is Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward. diff --git a/tasks/task-2-5/instruction.md b/tasks/task-2-5/instruction.md index bbadf99..8174266 100644 --- a/tasks/task-2-5/instruction.md +++ b/tasks/task-2-5/instruction.md @@ -1,16 +1,16 @@ -I'm Ben Carter, at a workstation in our search engineering lab watching a sparse retrieval job take longer than we can justify. I want you to optimize the supplied system over its large corpus. Each document and query is represented by a set of opaque term IDs, and the system needs to return relevant corpus documents while reducing the wall time for the complete query workload. You can replace the starter index or retrieval algorithm, provided the executable interface and output contract stay intact. +Optimize the supplied sparse retrieval system over its large corpus. Each document and query is represented by a set of opaque term IDs. The system must return relevant corpus documents while reducing total search wall time for the complete query workload. You may replace the starter index or retrieval algorithm as long as the executable interface and output contract are preserved. -Quality comes first in this comparison. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the corrected, unpruned starter wall time measured by the verifier in the same environment. I am interested in that complete externally measured run, not an internal timing number that omits some of the work. +Quality comes first in this comparison. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the corrected, unpruned starter wall time measured by the verifier in the same environment. Please read /task/docs/environment.md for the installed runtime and resource limits, /task/docs/available_resources.md for the local dependency policy, and /task/docs/index_format.md before changing the starter index. That format is an implementation detail and may be replaced. All of /task is read-only, and submission files must only be created or modified under /app. The agent phase allows up to 120 minutes. Do not use precomputed hidden query-to-document mappings, hidden relevance judgments, or external datasets or services containing evaluation answers. The formal build and search run offline and must not need credentials, remote models, package downloads or external services. -There is an implementation constraint I need you to preserve throughout the optimization: all submission logic must be Python source code. build.sh and run.sh may be thin Bash wrappers for argument parsing and launching Python, but indexing, retrieval, scoring, pruning and ranking must all be Python. Do not add or compile native submission source, including C, C++, Rust or Go, and do not have build.sh generate a native executable. Preinstalled Python packages such as NumPy are available through their normal APIs. +Implement all submission logic in Python source code. build.sh and run.sh may be thin Bash wrappers for argument parsing and launching Python, but indexing, retrieval, scoring, pruning and ranking must all be Python. Do not add or compile native submission source, including C, C++, Rust or Go, and do not have build.sh generate a native executable. Preinstalled Python packages such as NumPy are available through their normal APIs. -Both the submission build and search path must be single-process and single-threaded. Do not use multiprocessing, worker pools, threading, joblib, Ray or parallel query execution. The verifier environment is limited to one CPU, so extra threads cannot gain more aggregate CPU throughput, but that CPU limit does not itself make parallel execution authorized or valid. Please obtain the improvement within this execution model. +Both the submission build and search path must be single-process and single-threaded. Do not use multiprocessing, worker pools, threading, joblib, Ray or parallel query execution. The verifier environment is limited to one CPU, so extra threads cannot gain more aggregate CPU throughput, but that CPU limit does not itself make parallel execution authorized or valid. The full corpus is /task/data/corpus.jsonl. Public development material includes /task/data/validation/queries.jsonl for validation queries, /task/data/validation/qrels.tsv for relevance judgments, /task/data/validation/stats.json for term and document-frequency statistics, and /task/data/validation/reference_top100.jsonl for regression rankings. The formal hidden query split and its relevance judgments belong to the verifier. Document and query files are JSONL: a document looks like {"doc_id":"document-id","terms":["term-id", "term-id"]} and a query like {"query_id":"query-id","terms":["term-id", "term-id"]}. The repeated placeholder in those examples denotes the shape of the array; actual terms within a row are unique opaque strings. Their lexical value, apparent numeric value, input order and position do not encode relevance. -Leave executable /app/build.sh and /app/run.sh files as the handover. Optional Python modules may live in /app/src and optional notes in /app/README.md. The verifier builds with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /path/to/index. Build retrieval state from the supplied corpus and create every persistent index and runtime artifact beneath that supplied --index-dir. Return status 0 only when the index is ready. The build may use public statistics next to the corpus, but it must not require additional undeclared paths or undeclared persistent files from the verifier. +Provide executable /app/build.sh and /app/run.sh files in the final submission. Optional Python modules may live in /app/src and optional notes in /app/README.md. The verifier builds with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /path/to/index. Build retrieval state from the supplied corpus and create every persistent index and runtime artifact beneath that supplied --index-dir. Return status 0 only when the index is ready. The build may use public statistics next to the corpus, but it must not require additional undeclared paths or undeclared persistent files from the verifier. Once the build succeeds, the search interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Required runtime state must come from the supplied index directory and query file. Process each supplied query exactly once, preserve input query order, and write one result object per query. A JSONL output line looks like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. query_id must identify its corresponding input query. Return at most 100 results, with unique doc_id values that occur in the supplied corpus and scores that are finite and numeric. Order results by descending score, resolving equal scores by ascending lexicographic order of the opaque document ID. Fewer than 100 results is valid, though it can reduce Recall@100. @@ -18,4 +18,4 @@ The latency baseline used by the verifier is the corrected, unpruned starter. It For evaluation, the verifier rebuilds your submission from the full corpus and runs it on a private query split. It checks retrieval integrity, executable behavior, valid output, retrieval quality and search wall time. Hidden results must achieve NDCG@10 of at least 0.89 and qrels Recall@100 of at least 0.99. Candidate and corrected-starter wall time are measured by external process wrappers under the same workload and resource allocation. The latency ratio is candidate wall time divided by that verifier-measured starter wall time. -NDCG and Recall are hard gates: failure of either one, build/run behavior, output validation or the trajectory jailbreak judge gives reward 0. Otherwise a latency ratio at or below 0.25 earns reward 1; strictly between 0.25 and 0.35 it earns (0.35 - latency ratio) / 0.10; and at or above 0.35 it earns 0. Submission-reported lookup counters and internal timings are diagnostics only and cannot replace the verifier's wall-time measurement. Please leave me an implementation whose measured end-to-end search is faster while meeting those quality and execution conditions. +NDCG and Recall are hard gates: failure of either one, build/run behavior, output validation or the trajectory jailbreak judge gives reward 0. Otherwise a latency ratio at or below 0.25 earns reward 1; strictly between 0.25 and 0.35 it earns (0.35 - latency ratio) / 0.10; and at or above 0.35 it earns 0. Submission-reported lookup counters and internal timings are diagnostics only and cannot replace the verifier's wall-time measurement. From b570ffece8e60e3cd16d71300cc60a4e7e666bb2 Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Fri, 9 Oct 2026 20:54:15 +0800 Subject: [PATCH 2/4] Use descriptive task names across packages and tooling Rename nine task packages and pin assets to the migrated Hugging Face paths. Select tasks by name, flatten submission paths, and update evaluation, scaffolding, promotion, and validation tools with regression coverage. --- .../scripts/scaffold_task.py | 87 +- .../scripts/test_scaffold_task.py | 168 +-- .../scripts/maintainer_helpers.py | 58 +- .../scripts/test_maintainer_skill.py | 51 +- .env.example | 10 +- .github/workflows/task-checks.yml | 2 +- .gitignore | 8 +- scripts/check_release.py | 16 +- scripts/check_submission.py | 146 +- scripts/download_assets.py | 6 +- scripts/run_task.py | 4 +- scripts/task_paths.py | 14 +- scripts/tests/egress_adapter_smoke.py | 2 +- scripts/tests/egress_full_trial.py | 2 +- scripts/tests/test_egress_contract.py | 8 +- scripts/tests/test_egress_launcher.py | 8 +- scripts/tests/test_network_policies.py | 48 +- scripts/tests/test_release.py | 50 +- scripts/tests/test_rewardkit_deepseek.py | 42 +- scripts/tests/test_submission_workflow.py | 269 ++-- scripts/tests/test_task_images.py | 4 +- tasks/{task-1-1 => agentic-search}/.gitignore | 0 tasks/{task-2-4 => agentic-search}/README.md | 28 +- .../{task-2-4 => agentic-search}/assets.json | 12 +- .../environment/Dockerfile | 0 .../environment/build.sh | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../environment/run.sh | 0 .../environment/starter/README.md | 2 +- .../environment/starter/build.sh | 0 .../environment/starter/run.sh | 0 .../environment/starter/src/build_index.py | 0 .../environment/starter/src/run_search.py | 0 .../environment/starter/src/search_service.py | 0 .../instruction.md | 0 .../raw-instruction.md | 2 +- tasks/{task-2-4 => agentic-search}/task.toml | 3 +- .../tests/Dockerfile | 0 .../tests/data/qrels_gold.txt | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/evaluate.py | 2 +- .../tests/run_submission.py | 6 +- .../tests/test.sh | 0 .../tests/test_evaluate.py | 2 +- .../tests/test_parallel_submission.py | 2 +- .../tests/test_run_submission.py | 2 +- .../tests/test_starter.py | 0 .../tests/test_verify.py | 2 +- .../tests/verify.py | 4 +- .../.gitignore | 0 .../README.md | 10 +- .../assets.json | 16 +- .../environment/.dockerignore | 0 .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../environment/starter/train.sh | 0 .../instruction.md | 0 .../bge-base-en-v1.5}/configuration.json | 0 .../raw-instruction.md | 2 +- .../task.toml | 3 +- .../tests/Dockerfile | 0 .../tests/data/ground_truth.jsonl | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 2 +- .../tests/grader.py | 2 +- .../tests/jailbreak_judge/codex.toml | 0 .../tests/rewardkit_deepseek.py | 0 .../tests/run_evaluation.sh | 2 +- .../tests/test.sh | 2 +- .../.gitignore | 0 .../README.md | 13 +- .../assets.json | 12 +- .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../environment/starter/README.md | 0 .../environment/starter/build.sh | 0 .../environment/starter/run.sh | 0 .../environment/starter/src/__init__.py | 0 .../environment/starter/src/build_index.py | 0 .../environment/starter/src/server.py | 0 .../instruction.md | 0 .../bge-reranker-large/.gitattributes | 0 .../bge-reranker-large}/configuration.json | 0 .../raw-instruction.md | 2 +- .../task.toml | 3 +- .../tests/.dockerignore | 0 .../tests/Dockerfile | 0 .../tests/README.md | 2 +- .../tests/data/ground_truth.jsonl | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 2 +- .../tests/grader.py | 6 +- .../tests/jailbreak_judge/codex.toml | 0 .../tests/rewardkit_deepseek.py | 0 .../tests/test.sh | 2 +- .../.gitignore | 0 .../{task-1-4 => long-pdf-evidence}/README.md | 10 +- .../assets.json | 12 +- .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../instruction.md | 0 .../raw-instruction.md | 2 +- .../{task-1-4 => long-pdf-evidence}/task.toml | 3 +- .../tests/.dockerignore | 0 .../tests/Dockerfile | 0 .../tests/configure_trajectory_judge.py | 0 .../tests/data/golden_answers.jsonl | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 4 +- .../tests/grader.py | 10 +- .../tests/jailbreak_judge/codex.toml | 2 +- .../tests/rewardkit_deepseek.py | 0 .../tests/run_queries.py | 0 .../tests/test.sh | 4 +- .../.gitignore | 0 .../README.md | 11 +- .../assets.json | 32 +- .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../instruction.md | 0 .../raw-instruction.md | 2 +- .../task.toml | 3 +- .../tests/.dockerignore | 0 .../tests/Dockerfile | 0 .../tests/data/ground_truth.jsonl | 0 .../tests/data/query_metadata.jsonl | 0 .../tests/data/query_vectors.f32 | Bin .../tests/data/vector_config.json | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 4 +- .../tests/grader.py | 8 +- .../tests/jailbreak_judge/codex.toml | 0 .../tests/rewardkit_deepseek.py | 0 .../tests/test.sh | 2 +- .../.gitignore | 0 .../README.md | 8 +- .../assets.json | 16 +- .../environment/.dockerignore | 0 .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../environment/starter/README.md | 2 +- .../environment/starter/run.sh | 0 .../environment/starter/src/__init__.py | 1 + .../environment/starter/src/common.py | 0 .../environment/starter/src/run_search.py | 0 .../instruction.md | 0 .../Qwen3-Embedding-0.6B/README.md | 0 .../Qwen3-Embedding-0.6B/configuration.json | 0 .../Qwen3-Embedding-4B/configuration.json | 0 .../raw-instruction.md | 2 +- .../task.toml | 3 +- .../tests/Dockerfile | 0 .../tests/data/ground_truth.jsonl | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 2 +- .../tests/grader.py | 14 +- .../tests/jailbreak_judge/codex.toml | 2 +- .../tests/rewardkit_deepseek.py | 0 .../tests/run_evaluation.sh | 4 +- .../tests/starter/README.md | 2 +- .../tests/starter/run.sh | 0 .../tests/starter/src/__init__.py | 1 + .../tests/starter/src/common.py | 0 .../tests/starter/src/run_search.py | 0 .../tests/test.sh | 2 +- .../.gitignore | 0 .../README.md | 12 +- .../assets.json | 12 +- .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../instruction.md | 0 .../raw-instruction.md | 2 +- .../task.toml | 7 +- .../tests/.dockerignore | 0 .../tests/Dockerfile | 0 .../tests/data/ground_truth.jsonl | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 2 +- .../tests/grader.py | 8 +- .../tests/jailbreak_judge/codex.toml | 0 .../tests/rewardkit_deepseek.py | 0 .../tests/test.sh | 2 +- .../.gitignore | 0 .../README.md | 10 +- .../assets.json | 1208 ++++++++--------- .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 0 .../instruction.md | 0 .../raw-instruction.md | 2 +- .../task.toml | 3 +- .../tests/.dockerignore | 0 .../tests/Dockerfile | 0 .../tests/configure_trajectory_judge.py | 0 .../tests/data/golden_answers.jsonl | 0 .../tests/data/queries.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 4 +- .../tests/grader.py | 10 +- .../tests/jailbreak_judge/codex.toml | 2 +- .../tests/rewardkit_deepseek.py | 0 .../tests/run_queries.py | 0 .../tests/test.sh | 4 +- .../tests/test_run_queries.py | 0 .../README.md | 36 +- .../assets.json | 20 +- .../environment/Dockerfile | 0 .../environment/docker-compose.yaml | 0 .../environment/docs/available_resources.md | 0 .../environment/docs/environment.md | 4 +- .../environment/docs/index_format.md | 2 +- .../environment/starter/build.sh | 0 .../environment/starter/run.sh | 0 .../starter/src/sparse_retrieval.py | 0 .../instruction.md | 6 +- .../raw-instruction.md | 10 +- .../task.toml | 3 +- .../tests/Dockerfile | 0 .../tests/data/categories.json | 0 .../tests/data/corpus_ids.txt | 0 .../tests/data/hidden_queries.jsonl | 0 .../tests/data/qrels.tsv | 0 .../tests/data/reference_metadata.json | 0 .../tests/data/reference_top100.jsonl | 0 .../tests/docker-compose.yaml | 0 .../tests/finalize_reward.py | 0 .../tests/grader.py | 2 +- .../tests/jailbreak_judge/codex.toml | 2 +- .../tests/rewardkit_deepseek.py | 0 .../tests/run_with_metrics.py | 0 .../tests/starter/build.sh | 0 .../tests/starter/run.sh | 0 .../tests/starter/src/sparse_retrieval.py | 0 .../tests/test.sh | 2 +- .../environment/starter/src/__init__.py | 1 - tasks/task-2-3/tests/starter/src/__init__.py | 1 - 257 files changed, 1351 insertions(+), 1345 deletions(-) rename tasks/{task-1-1 => agentic-search}/.gitignore (100%) rename tasks/{task-2-4 => agentic-search}/README.md (89%) rename tasks/{task-2-4 => agentic-search}/assets.json (68%) rename tasks/{task-2-4 => agentic-search}/environment/Dockerfile (100%) rename tasks/{task-2-4 => agentic-search}/environment/build.sh (100%) rename tasks/{task-1-2 => agentic-search}/environment/docker-compose.yaml (100%) rename tasks/{task-2-3 => agentic-search}/environment/docs/available_resources.md (100%) rename tasks/{task-1-1 => agentic-search}/environment/docs/environment.md (100%) rename tasks/{task-2-4 => agentic-search}/environment/run.sh (100%) rename tasks/{task-2-4 => agentic-search}/environment/starter/README.md (98%) rename tasks/{task-2-4 => agentic-search}/environment/starter/build.sh (100%) rename tasks/{task-2-4 => agentic-search}/environment/starter/run.sh (100%) rename tasks/{task-2-4 => agentic-search}/environment/starter/src/build_index.py (100%) rename tasks/{task-2-4 => agentic-search}/environment/starter/src/run_search.py (100%) rename tasks/{task-2-4 => agentic-search}/environment/starter/src/search_service.py (100%) rename tasks/{task-2-4 => agentic-search}/instruction.md (100%) rename tasks/{task-2-4 => agentic-search}/raw-instruction.md (99%) rename tasks/{task-2-4 => agentic-search}/task.toml (96%) rename tasks/{task-2-4 => agentic-search}/tests/Dockerfile (100%) rename tasks/{task-2-4 => agentic-search}/tests/data/qrels_gold.txt (100%) rename tasks/{task-2-4 => agentic-search}/tests/data/queries.jsonl (100%) rename tasks/{task-2-4 => agentic-search}/tests/docker-compose.yaml (100%) rename tasks/{task-2-4 => agentic-search}/tests/evaluate.py (98%) rename tasks/{task-2-4 => agentic-search}/tests/run_submission.py (98%) rename tasks/{task-2-4 => agentic-search}/tests/test.sh (100%) rename tasks/{task-2-4 => agentic-search}/tests/test_evaluate.py (98%) rename tasks/{task-2-4 => agentic-search}/tests/test_parallel_submission.py (97%) rename tasks/{task-2-4 => agentic-search}/tests/test_run_submission.py (94%) rename tasks/{task-2-4 => agentic-search}/tests/test_starter.py (100%) rename tasks/{task-2-4 => agentic-search}/tests/test_verify.py (92%) rename tasks/{task-2-4 => agentic-search}/tests/verify.py (94%) rename tasks/{task-1-2 => code-embedding-finetuning}/.gitignore (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/README.md (94%) rename tasks/{task-2-2 => code-embedding-finetuning}/assets.json (91%) rename tasks/{task-2-2 => code-embedding-finetuning}/environment/.dockerignore (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/environment/Dockerfile (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/environment/docker-compose.yaml (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/environment/docs/available_resources.md (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/environment/docs/environment.md (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/environment/starter/train.sh (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/instruction.md (100%) rename tasks/{task-2-1/model-metadata/bge-reranker-large => code-embedding-finetuning/model-metadata/bge-base-en-v1.5}/configuration.json (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/raw-instruction.md (98%) rename tasks/{task-2-2 => code-embedding-finetuning}/task.toml (95%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/Dockerfile (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/data/ground_truth.jsonl (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/data/queries.jsonl (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/docker-compose.yaml (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/finalize_reward.py (97%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/grader.py (99%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/jailbreak_judge/codex.toml (100%) rename tasks/{task-1-1 => code-embedding-finetuning}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/run_evaluation.sh (85%) rename tasks/{task-2-2 => code-embedding-finetuning}/tests/test.sh (99%) rename tasks/{task-1-3 => long-document-reranking}/.gitignore (100%) rename tasks/{task-2-1 => long-document-reranking}/README.md (94%) rename tasks/{task-2-1 => long-document-reranking}/assets.json (92%) rename tasks/{task-2-1 => long-document-reranking}/environment/Dockerfile (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/docker-compose.yaml (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/docs/available_resources.md (100%) rename tasks/{task-1-2 => long-document-reranking}/environment/docs/environment.md (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/starter/README.md (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/starter/build.sh (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/starter/run.sh (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/starter/src/__init__.py (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/starter/src/build_index.py (100%) rename tasks/{task-2-1 => long-document-reranking}/environment/starter/src/server.py (100%) rename tasks/{task-2-1 => long-document-reranking}/instruction.md (100%) rename tasks/{task-2-1 => long-document-reranking}/model-metadata/bge-reranker-large/.gitattributes (100%) rename tasks/{task-2-2/model-metadata/bge-base-en-v1.5 => long-document-reranking/model-metadata/bge-reranker-large}/configuration.json (100%) rename tasks/{task-2-1 => long-document-reranking}/raw-instruction.md (99%) rename tasks/{task-2-1 => long-document-reranking}/task.toml (95%) rename tasks/{task-2-1 => long-document-reranking}/tests/.dockerignore (100%) rename tasks/{task-1-1 => long-document-reranking}/tests/Dockerfile (100%) rename tasks/{task-2-1 => long-document-reranking}/tests/README.md (92%) rename tasks/{task-2-1 => long-document-reranking}/tests/data/ground_truth.jsonl (100%) rename tasks/{task-2-1 => long-document-reranking}/tests/data/queries.jsonl (100%) rename tasks/{task-2-1 => long-document-reranking}/tests/docker-compose.yaml (100%) rename tasks/{task-2-1 => long-document-reranking}/tests/finalize_reward.py (97%) rename tasks/{task-2-1 => long-document-reranking}/tests/grader.py (98%) rename tasks/{task-2-1 => long-document-reranking}/tests/jailbreak_judge/codex.toml (100%) rename tasks/{task-1-2 => long-document-reranking}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-2-1 => long-document-reranking}/tests/test.sh (99%) rename tasks/{task-1-4 => long-pdf-evidence}/.gitignore (100%) rename tasks/{task-1-4 => long-pdf-evidence}/README.md (94%) rename tasks/{task-1-4 => long-pdf-evidence}/assets.json (67%) rename tasks/{task-1-1 => long-pdf-evidence}/environment/Dockerfile (100%) rename tasks/{task-1-4 => long-pdf-evidence}/environment/docker-compose.yaml (100%) rename tasks/{task-1-4 => long-pdf-evidence}/environment/docs/available_resources.md (100%) rename tasks/{task-1-3 => long-pdf-evidence}/environment/docs/environment.md (100%) rename tasks/{task-1-4 => long-pdf-evidence}/instruction.md (100%) rename tasks/{task-1-4 => long-pdf-evidence}/raw-instruction.md (99%) rename tasks/{task-1-4 => long-pdf-evidence}/task.toml (96%) rename tasks/{task-1-3 => long-pdf-evidence}/tests/.dockerignore (100%) rename tasks/{task-1-2 => long-pdf-evidence}/tests/Dockerfile (100%) rename tasks/{task-1-3 => long-pdf-evidence}/tests/configure_trajectory_judge.py (100%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/data/golden_answers.jsonl (100%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/data/queries.jsonl (100%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/docker-compose.yaml (100%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/finalize_reward.py (94%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/grader.py (98%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/jailbreak_judge/codex.toml (96%) rename tasks/{task-1-3 => long-pdf-evidence}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/run_queries.py (100%) rename tasks/{task-1-4 => long-pdf-evidence}/tests/test.sh (99%) rename tasks/{task-2-1 => memory-constrained-dense-retrieval}/.gitignore (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/README.md (94%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/assets.json (64%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/environment/Dockerfile (100%) rename tasks/{task-1-3 => memory-constrained-dense-retrieval}/environment/docker-compose.yaml (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/environment/docs/available_resources.md (100%) rename tasks/{task-1-4 => memory-constrained-dense-retrieval}/environment/docs/environment.md (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/instruction.md (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/raw-instruction.md (99%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/task.toml (95%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/.dockerignore (100%) rename tasks/{task-1-3 => memory-constrained-dense-retrieval}/tests/Dockerfile (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/data/ground_truth.jsonl (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/data/query_metadata.jsonl (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/data/query_vectors.f32 (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/data/vector_config.json (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/docker-compose.yaml (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/finalize_reward.py (93%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/grader.py (98%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/jailbreak_judge/codex.toml (100%) rename tasks/{task-1-4 => memory-constrained-dense-retrieval}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-1-2 => memory-constrained-dense-retrieval}/tests/test.sh (98%) rename tasks/{task-2-2 => query-encoder-alignment}/.gitignore (100%) rename tasks/{task-2-3 => query-encoder-alignment}/README.md (95%) rename tasks/{task-2-3 => query-encoder-alignment}/assets.json (95%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/.dockerignore (100%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/Dockerfile (100%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/docker-compose.yaml (100%) rename tasks/{task-2-4 => query-encoder-alignment}/environment/docs/available_resources.md (100%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/docs/environment.md (100%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/starter/README.md (96%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/starter/run.sh (100%) create mode 100644 tasks/query-encoder-alignment/environment/starter/src/__init__.py rename tasks/{task-2-3 => query-encoder-alignment}/environment/starter/src/common.py (100%) rename tasks/{task-2-3 => query-encoder-alignment}/environment/starter/src/run_search.py (100%) rename tasks/{task-2-3 => query-encoder-alignment}/instruction.md (100%) rename tasks/{task-2-3 => query-encoder-alignment}/model-metadata/Qwen3-Embedding-0.6B/README.md (100%) rename tasks/{task-2-3 => query-encoder-alignment}/model-metadata/Qwen3-Embedding-0.6B/configuration.json (100%) rename tasks/{task-2-3 => query-encoder-alignment}/model-metadata/Qwen3-Embedding-4B/configuration.json (100%) rename tasks/{task-2-3 => query-encoder-alignment}/raw-instruction.md (98%) rename tasks/{task-2-3 => query-encoder-alignment}/task.toml (96%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/Dockerfile (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/data/ground_truth.jsonl (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/data/queries.jsonl (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/docker-compose.yaml (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/finalize_reward.py (97%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/grader.py (95%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/jailbreak_judge/codex.toml (90%) rename tasks/{task-2-1 => query-encoder-alignment}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/run_evaluation.sh (98%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/starter/README.md (96%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/starter/run.sh (100%) create mode 100644 tasks/query-encoder-alignment/tests/starter/src/__init__.py rename tasks/{task-2-3 => query-encoder-alignment}/tests/starter/src/common.py (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/starter/src/run_search.py (100%) rename tasks/{task-2-3 => query-encoder-alignment}/tests/test.sh (99%) rename tasks/{task-2-3 => reasoning-query-rewriting}/.gitignore (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/README.md (92%) rename tasks/{task-1-1 => reasoning-query-rewriting}/assets.json (67%) rename tasks/{task-1-3 => reasoning-query-rewriting}/environment/Dockerfile (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/environment/docker-compose.yaml (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/environment/docs/available_resources.md (100%) rename tasks/{task-2-1 => reasoning-query-rewriting}/environment/docs/environment.md (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/instruction.md (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/raw-instruction.md (99%) rename tasks/{task-1-1 => reasoning-query-rewriting}/task.toml (88%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/.dockerignore (100%) rename tasks/{task-1-4 => reasoning-query-rewriting}/tests/Dockerfile (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/data/ground_truth.jsonl (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/data/queries.jsonl (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/docker-compose.yaml (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/finalize_reward.py (97%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/grader.py (98%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/jailbreak_judge/codex.toml (100%) rename tasks/{task-2-2 => reasoning-query-rewriting}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-1-1 => reasoning-query-rewriting}/tests/test.sh (99%) rename tasks/{task-2-4 => scientific-paper-qa}/.gitignore (100%) rename tasks/{task-1-3 => scientific-paper-qa}/README.md (94%) rename tasks/{task-1-3 => scientific-paper-qa}/assets.json (67%) rename tasks/{task-1-4 => scientific-paper-qa}/environment/Dockerfile (100%) rename tasks/{task-2-4 => scientific-paper-qa}/environment/docker-compose.yaml (100%) rename tasks/{task-1-3 => scientific-paper-qa}/environment/docs/available_resources.md (100%) rename tasks/{task-2-4 => scientific-paper-qa}/environment/docs/environment.md (100%) rename tasks/{task-1-3 => scientific-paper-qa}/instruction.md (100%) rename tasks/{task-1-3 => scientific-paper-qa}/raw-instruction.md (99%) rename tasks/{task-1-3 => scientific-paper-qa}/task.toml (96%) rename tasks/{task-1-4 => scientific-paper-qa}/tests/.dockerignore (100%) rename tasks/{task-2-1 => scientific-paper-qa}/tests/Dockerfile (100%) rename tasks/{task-1-4 => scientific-paper-qa}/tests/configure_trajectory_judge.py (100%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/data/golden_answers.jsonl (100%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/data/queries.jsonl (100%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/docker-compose.yaml (100%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/finalize_reward.py (94%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/grader.py (98%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/jailbreak_judge/codex.toml (95%) rename tasks/{task-2-3 => scientific-paper-qa}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/run_queries.py (100%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/test.sh (99%) rename tasks/{task-1-3 => scientific-paper-qa}/tests/test_run_queries.py (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/README.md (81%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/assets.json (66%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/Dockerfile (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/docker-compose.yaml (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/docs/available_resources.md (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/docs/environment.md (89%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/docs/index_format.md (97%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/starter/build.sh (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/starter/run.sh (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/environment/starter/src/sparse_retrieval.py (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/instruction.md (86%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/raw-instruction.md (95%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/task.toml (95%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/Dockerfile (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/data/categories.json (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/data/corpus_ids.txt (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/data/hidden_queries.jsonl (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/data/qrels.tsv (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/data/reference_metadata.json (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/data/reference_top100.jsonl (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/docker-compose.yaml (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/finalize_reward.py (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/grader.py (99%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/jailbreak_judge/codex.toml (95%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/rewardkit_deepseek.py (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/run_with_metrics.py (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/starter/build.sh (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/starter/run.sh (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/starter/src/sparse_retrieval.py (100%) rename tasks/{task-2-5 => sparse-retrieval-pruning}/tests/test.sh (99%) delete mode 100644 tasks/task-2-3/environment/starter/src/__init__.py delete mode 100644 tasks/task-2-3/tests/starter/src/__init__.py diff --git a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py index 2034807..f34f461 100755 --- a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py @@ -16,10 +16,7 @@ "gpu": "docker.io/hanhainebula/search-swe-base:gpu-cu13.0-py3.12-1.0.0", } -TASK_TYPES = { - "implementation": "create", - "optimization": "optimize", -} +TASK_NAME = r"(?!all$)(?!task-)[a-z][a-z0-9]*(?:-[a-z0-9]+){0,4}" def clean(text: str) -> str: @@ -61,11 +58,9 @@ def compose(hardware: str, *, environment: bool) -> str: return body -def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: +def files_for(task_name: str, hardware: str) -> dict[str, str]: image = IMAGES[hardware] gpus = 1 if hardware == "gpu" else 0 - task_type = TASK_TYPES[mode] - mode_label = mode.title() return { ".gitignore": clean( """ @@ -77,9 +72,9 @@ def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: ), "README.md": clean( f""" - # {task_id} + # {task_name} - **Mode:** {mode_label} + **Task:** `{task_name}` Replace this scaffold text with an author-facing overview of the objective, dataset/model provenance and license, metric, resource @@ -92,7 +87,7 @@ def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: + "\n", "raw-instruction.md": clean( f""" - # Task: {task_id} + # Task: {task_name} ## Task Description @@ -130,7 +125,7 @@ def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: ), "instruction.md": clean( f""" - Replace this scaffold for {task_id} after completing raw-instruction.md. + Replace this scaffold for {task_name} after completing raw-instruction.md. Write a direct natural-language request describing the task, current data and starting system, resources, constraints, acceptance criteria, and optimization objective. Omit self-introductions, invented people, @@ -149,14 +144,13 @@ def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: ] [task] - name = "search-swe/{task_id}" + name = "search-swe/{task_name}" version = "0.1.0" description = "Replace with the task's observable objective." authors = [{{ name = "Search-SWE" }}] keywords = ["information-retrieval", "search"] [metadata] - task_type = "{task_type}" primary_metric = "Replace with the primary metric" [agent] @@ -249,14 +243,8 @@ def files_for(task_id: str, hardware: str, mode: str) -> dict[str, str]: def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( - "task_id", - help="Lowercase task directory name, for example task-1-x-1", - ) - parser.add_argument( - "--mode", - choices=sorted(TASK_TYPES), - default="implementation", - help="Engineering mode; defaults to implementation", + "task_name", + help="Descriptive lowercase task name, at most five hyphen-separated words", ) parser.add_argument( "--hardware", @@ -270,16 +258,16 @@ def parse_args() -> argparse.Namespace: default=Path.cwd(), help="Search-SWE repository root; defaults to the current directory", ) - parser.add_argument("--submission-first-name", help="Explicit ASCII first name, not a username") - parser.add_argument("--author", action="append", help="Actual author name; repeat for coauthors (required for submissions)") + parser.add_argument("--formal", action="store_true", help="Maintainer-only scaffold under tasks/; default is task-submissions/") + parser.add_argument("--author", action="append", required=True, help="Actual author name; repeat for coauthors") return parser.parse_args() def main() -> int: args = parse_args() - if not re.fullmatch(r"task-[a-z0-9]+(?:-[a-z0-9]+)*", args.task_id): + if not re.fullmatch(TASK_NAME, args.task_name): print( - "error: task_id must match task-", + "error: use at most five lowercase hyphen-separated words, without a task- prefix; all is reserved", file=sys.stderr, ) return 2 @@ -292,39 +280,20 @@ def main() -> int: if (root / "tasks").is_symlink() or (root / "task-submissions").is_symlink(): print("error: task roots must not be symlinks", file=sys.stderr) return 2 - destination = root / "tasks" / args.task_id - if args.submission_first_name is not None: - first_name = args.submission_first_name - temporary = re.fullmatch(r"task-([12])-x-([1-9][0-9]*)", args.task_id) - if (not first_name.isascii() or any(c in first_name for c in "/\\.") - or not temporary): - print("error: use an ASCII first name and task-<1|2>-x-", file=sys.stderr) - return 2 - expected_mode = {"1": "implementation", "2": "optimization"}[temporary[1]] - if args.mode != expected_mode: - print(f"error: {args.task_id} requires --mode {expected_mode}; hardware is independent", file=sys.stderr) - return 2 - slug = re.sub(r"[^a-z0-9]+", "-", first_name.lower()).strip("-") - if not re.fullmatch(r"[a-z][a-z0-9]*(?:-[a-z0-9]+)*", slug): - print("error: provide an ASCII transliteration beginning with a letter", file=sys.stderr) - return 2 - if not args.author or any(not name.strip() or name.strip().lower() in - ("search-swe", "your name", "author") for name in args.author): - print("error: submissions require actual --author names", file=sys.stderr) - return 2 - parent = root / "task-submissions" - parent.mkdir(exist_ok=True) - namespace = parent / slug - if namespace.is_symlink() or (namespace.exists() and not namespace.is_dir()): - print(f"error: contributor namespace must be a real directory: {namespace}", file=sys.stderr) - return 2 - namespace.mkdir(exist_ok=True) - destination = namespace / args.task_id.removeprefix("task-") - if destination.exists() or destination.is_symlink(): - print(f"error: refusing to overwrite existing path: {destination}", file=sys.stderr) - return 1 - - generated = files_for(args.task_id, args.hardware, args.mode) + if any(not name.strip() or name.strip().lower() in + ("search-swe", "your name", "author") for name in args.author): + print("error: actual --author names are required", file=sys.stderr) + return 2 + for directory in ("tasks", "task-submissions"): + existing = root / directory / args.task_name + if existing.exists() or existing.is_symlink(): + print(f"error: task name already exists; refusing overwrite: {existing}", file=sys.stderr) + return 1 + parent = root / ("tasks" if args.formal else "task-submissions") + parent.mkdir(exist_ok=True) + destination = parent / args.task_name + + generated = files_for(args.task_name, args.hardware) if args.author: authors = ", ".join('{ name = ' + json.dumps(name.strip(), ensure_ascii=False) + ' }' for name in args.author) @@ -348,7 +317,7 @@ def main() -> int: print( f"Created {destination.relative_to(root)} " - f"({args.mode}, {args.hardware})" + f"({args.hardware})" ) print("Next: replace scaffold text, implement the verifier, add exact assets,") print("and follow the skill's references/validation.md through all layers.") diff --git a/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py index a6dca36..fc8d867 100644 --- a/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py @@ -1,4 +1,4 @@ -"""Offline checks for a relocated skill; no Docker build or paid API calls.""" +"""Offline relocation and generation checks; no Docker builds or paid API calls.""" from pathlib import Path import re @@ -24,104 +24,83 @@ def setUp(self): (self.repo / "docker").mkdir() (self.repo / "docker/README.md").write_text("# Project fixture\n") - def scaffold(self, name, *args): + def scaffold(self, name, *args, authors=True): return subprocess.run( - [sys.executable, str(self.skill / "scripts/scaffold_task.py"), - name, "--repo-root", str(self.repo), *args], + [sys.executable, str(self.skill / "scripts/scaffold_task.py"), name, + "--repo-root", str(self.repo), + *(["--author", "Alice Example"] if authors else []), *args], cwd=self.root, capture_output=True, text=True, ) - def test_all_mode_hardware_pairs_and_fail_closed_entrypoint(self): - for mode, kind in (("implementation", "create"), ("optimization", "optimize")): - for hardware, tag, gpus in (("cpu", "cpu-py3.12-1.0.0", 0), - ("gpu", "gpu-cu13.0-py3.12-1.0.0", 1)): - with self.subTest(mode=mode, hardware=hardware): - name = f"task-{mode}-{hardware}" - result = self.scaffold(name, "--mode", mode, "--hardware", hardware) - self.assertEqual(result.returncode, 0, result.stderr) - task = self.repo / "tasks" / name - raw = (task / "raw-instruction.md").read_text() - instruction = (task / "instruction.md").read_text() - self.assertTrue(raw.strip()) - self.assertTrue(instruction.strip()) - self.assertNotEqual(raw, instruction) - config = tomllib.loads((task / "task.toml").read_text()) - self.assertEqual(config["metadata"]["task_type"], kind) - self.assertEqual(config["environment"]["gpus"], gpus) - self.assertEqual(config["verifier"]["environment_mode"], "separate") - for phase in ("environment", "tests"): - self.assertEqual((task / phase / "Dockerfile").read_text().splitlines()[0], - f"FROM docker.io/hanhainebula/search-swe-base:{tag}") - overlay = (task / phase / "docker-compose.yaml").read_text() - self.assertEqual("driver: nvidia" in overlay, bool(gpus)) - script = task / "tests/test.sh" - self.assertTrue(script.read_bytes().startswith(b"#!/usr/bin/env bash\n")) - subprocess.run(["bash", "-n", str(script)], check=True) - # Redirect only the log directory into the temporary workspace. - log = task / "test-log" - local = task / "local-test.sh" - local.write_text(script.read_text().replace( - "reward_dir=/logs/verifier", f"reward_dir={shlex.quote(str(log))}")) - local.chmod(0o755) - failed = subprocess.run([str(local)], capture_output=True, text=True) - self.assertEqual(failed.returncode, 1) - self.assertEqual(float((log / "reward.txt").read_text()), 0) + def test_cpu_gpu_generation_and_fail_closed_entrypoint(self): + for hardware, tag, gpus in (("cpu", "cpu-py3.12-1.0.0", 0), + ("gpu", "gpu-cu13.0-py3.12-1.0.0", 1)): + with self.subTest(hardware=hardware): + name = f"example-{hardware}-search" + result = self.scaffold(name, "--hardware", hardware, "--author", "Coauthor Example") + self.assertEqual(result.returncode, 0, result.stderr) + task = self.repo / "task-submissions" / name + self.assertNotEqual((task / "raw-instruction.md").read_text(), + (task / "instruction.md").read_text()) + config = tomllib.loads((task / "task.toml").read_text()) + self.assertEqual(config["task"]["name"], f"search-swe/{name}") + self.assertEqual([a["name"] for a in config["task"]["authors"]], + ["Alice Example", "Coauthor Example"]) + self.assertNotIn("task_type", config["metadata"]) + self.assertEqual(config["environment"]["gpus"], gpus) + self.assertEqual(config["verifier"]["environment_mode"], "separate") + for phase in ("environment", "tests"): + self.assertEqual((task / phase / "Dockerfile").read_text().splitlines()[0], + f"FROM docker.io/hanhainebula/search-swe-base:{tag}") + self.assertEqual("driver: nvidia" in (task / phase / "docker-compose.yaml").read_text(), bool(gpus)) + script = task / "tests/test.sh" + subprocess.run(["bash", "-n", str(script)], check=True) + log = task / "test-log" + local = task / "local-test.sh" + local.write_text(script.read_text().replace("reward_dir=/logs/verifier", + f"reward_dir={shlex.quote(str(log))}")) + local.chmod(0o755) + failed = subprocess.run([str(local)], capture_output=True, text=True) + self.assertEqual(failed.returncode, 1) + self.assertEqual(float((log / "reward.txt").read_text()), 0) + + def test_name_rules_authors_and_removed_options(self): + for name in ("../escape", "/absolute", "Élodie", "123", "all", "task-1-1", "task-1-x-1", + "task-example", "Upper-Case", "one--two", "-prefix", "suffix-", "one-two-three-four-five-six"): + self.assertNotEqual(self.scaffold(name).returncode, 0, name) + self.assertNotEqual(self.scaffold("missing-author", authors=False).returncode, 0) + self.assertNotEqual(self.scaffold("bad-author", "--author", "Search-SWE", authors=False).returncode, 0) + self.assertNotEqual(self.scaffold("old-mode", "--mode", "implementation").returncode, 0) + self.assertNotEqual(self.scaffold("old-namespace", "--submission-first-name", "Alice").returncode, 0) + for name in ("retrieval", "one-two-three-four-five", "bm25-search"): + self.assertEqual(self.scaffold(name).returncode, 0) - def test_existing_paths_and_invalid_mode_are_rejected(self): - self.assertEqual(self.scaffold("task-new").returncode, 0) - marker = self.repo / "tasks/task-new/keep.txt" + def test_names_cannot_collide_across_submission_and_formal_roots(self): + self.assertEqual(self.scaffold("example-search").returncode, 0) + marker = self.repo / "task-submissions/example-search/keep.txt" marker.write_text("keep") - self.assertNotEqual(self.scaffold("task-new").returncode, 0) + self.assertNotEqual(self.scaffold("example-search").returncode, 0) + self.assertNotEqual(self.scaffold("example-search", "--formal").returncode, 0) self.assertEqual(marker.read_text(), "keep") - link = self.repo / "tasks/task-link" - target = self.root / "absent" - link.symlink_to(target) - self.assertNotEqual(self.scaffold("task-link").returncode, 0) - self.assertFalse(target.exists()) - self.assertNotEqual(self.scaffold("task-other", "--mode", "repair").returncode, 0) - self.assertFalse((self.repo / "tasks/task-other").exists()) - - def test_submission_mode_reuses_namespace_and_explicit_name_suffix(self): - self.assertNotEqual(self.scaffold("task-1-x-1", "--submission-first-name", "Alice").returncode, 0) - for task_id in ("task-1-x-1", "task-1-x-2", "task-2-x-1"): - mode = "optimization" if task_id.startswith("task-2") else "implementation" - result = self.scaffold(task_id, "--submission-first-name", "Alice", "--author", "Alice Example", - "--author", "Coauthor Example", "--mode", mode) - self.assertEqual(result.returncode, 0, result.stderr) - task = self.repo / "task-submissions/alice" / task_id.removeprefix("task-") - config = tomllib.loads((task / "task.toml").read_text()) - self.assertEqual(config["task"]["name"], f"search-swe/{task_id}") - self.assertEqual([a["name"] for a in config["task"]["authors"]], ["Alice Example", "Coauthor Example"]) - duplicate = self.scaffold("task-1-x-1", "--submission-first-name", "Alice", - "--author", "Alice Example") - self.assertNotEqual(duplicate.returncode, 0) - result = self.scaffold("task-1-x-1", "--submission-first-name", "Alice-2", - "--author", "Another Alice") - self.assertEqual(result.returncode, 0, result.stderr) - self.assertTrue((self.repo / "task-submissions/alice-2/1-x-1/task.toml").is_file()) + self.assertEqual(self.scaffold("formal-search", "--formal").returncode, 0) + self.assertTrue((self.repo / "tasks/formal-search/task.toml").is_file()) + self.assertNotEqual(self.scaffold("formal-search").returncode, 0) - def test_submission_names_and_roots_are_safe(self): - for name in ("../escape", "/absolute", "Élodie", "123"): - self.assertNotEqual(self.scaffold("task-1-x-1", "--submission-first-name", name, - "--author", "Real Author").returncode, 0) - for task_id in ("task-1-x", "task-1-x-0", "task-1-6"): - self.assertNotEqual(self.scaffold(task_id, "--submission-first-name", "Alice", - "--author", "Alice").returncode, 0) - (self.repo / "task-submissions").mkdir() - (self.repo / "task-submissions/alice").symlink_to(self.root) - self.assertNotEqual(self.scaffold("task-1-x-1", "--submission-first-name", "Alice", - "--author", "Alice").returncode, 0) - shutil.rmtree(self.repo / "task-submissions") - (self.repo / "task-submissions").symlink_to(self.root) - self.assertNotEqual(self.scaffold("task-1-x-1", "--submission-first-name", "Alice", - "--author", "Alice").returncode, 0) + def test_symlink_roots_and_destinations_are_rejected(self): + submissions = self.repo / "task-submissions" + submissions.mkdir() + (submissions / "linked-search").symlink_to(self.root / "absent") + self.assertNotEqual(self.scaffold("linked-search").returncode, 0) + self.assertFalse((self.root / "absent").exists()) + shutil.rmtree(submissions) + submissions.symlink_to(self.root) + self.assertNotEqual(self.scaffold("example-search").returncode, 0) + submissions.unlink() shutil.rmtree(self.repo / "tasks") (self.repo / "tasks").symlink_to(self.root) - self.assertNotEqual(self.scaffold("task-new").returncode, 0) + self.assertNotEqual(self.scaffold("example-search").returncode, 0) def test_instruction_links_stay_inside_relocated_skill(self): - # Markdown links are instructional dependencies; project paths in code - # refer to the target checkout and intentionally remain external inputs. for document in self.skill.rglob("*.md"): for link in re.findall(r"\]\(([^)]+)\)", document.read_text()): if "://" in link or link.startswith("#"): @@ -130,21 +109,6 @@ def test_instruction_links_stay_inside_relocated_skill(self): self.assertTrue(target.is_relative_to(self.skill), (document, link)) self.assertTrue(target.is_file(), (document, link)) - def test_submission_category_matches_mode_independent_of_hardware(self): - for category, mode in ((1, "implementation"), (2, "optimization")): - wrong = "optimization" if category == 1 else "implementation" - result = self.scaffold(f"task-{category}-x-1", "--submission-first-name", "Alice", - "--author", "Alice", "--mode", wrong) - self.assertNotEqual(result.returncode, 0) - self.assertFalse((self.repo / "task-submissions").exists()) - ordinal = 1 - for category, mode in ((1, "implementation"), (2, "optimization")): - for hardware in ("cpu", "gpu"): - result = self.scaffold(f"task-{category}-x-{ordinal}", "--submission-first-name", "Alice", - "--author", "Alice", "--mode", mode, "--hardware", hardware) - self.assertEqual(result.returncode, 0, result.stderr) - ordinal += 1 - if __name__ == "__main__": unittest.main() diff --git a/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py b/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py index bf2c206..bfbc15a 100644 --- a/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py +++ b/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py @@ -12,9 +12,9 @@ import subprocess import tomllib -SLUG = r"[a-z][a-z0-9]*(?:-[a-z0-9]+)*" -SUBMISSION = re.compile(rf"task-submissions/({SLUG})/([12])-x-([1-9][0-9]*)") -FORMAL = re.compile(r"task-[12]-[1-9][0-9]*") +TASK_NAME = r"(?!all(?:/|$))(?!task-)[a-z][a-z0-9]*(?:-[a-z0-9]+){0,4}" +SUBMISSION = re.compile(rf"task-submissions/({TASK_NAME})") +FORMAL = re.compile(TASK_NAME) def safe_path(root, value): """Accept a canonical repo-relative path, never symlinks (even internal ones).""" @@ -78,18 +78,17 @@ def read_manifest(task): def git(repo, *args): return subprocess.run(["git", *args], cwd=repo, check=True, capture_output=True, text=True).stdout -def promote(repo, source, target_id, phase, dry_run=False): +def promote(repo, source, target_name, phase, dry_run=False): repo = checked_path(repo) if phase not in ("rename", "finalize"): raise ValueError("Unknown promotion phase") match = SUBMISSION.fullmatch(source) - if not match or not FORMAL.fullmatch(target_id): - raise ValueError("Use task-submissions//<1|2>-x- and task-<1|2>-") - if target_id.split("-")[1] != match[2]: - raise ValueError("Task category mismatch") + if not match or not FORMAL.fullmatch(target_name): + raise ValueError("Use task-submissions/ and the same task-name (at most five lowercase words)") + if target_name != match[1]: + raise ValueError("Promotion preserves the task name; rename the submission before promotion") src = safe_path(repo, source) - target = safe_path(repo, f"tasks/{target_id}") - old_id = f"task-{match[2]}-x-{match[3]}" + target = safe_path(repo, f"tasks/{target_name}") if git(repo, "status", "--porcelain"): raise ValueError("Promotion requires a clean worktree and index; commit reviewed work first") if phase == "rename": @@ -97,12 +96,12 @@ def promote(repo, source, target_id, phase, dry_run=False): raise ValueError("Source must exist and target must not exist; refusing overwrite") no_symlinks(src) config = tomllib.loads((src / "task.toml").read_text()) - if config.get("task", {}).get("name") != f"search-swe/{old_id}": - raise ValueError("Source temporary task name does not match category") - print(f"git mv {source} tasks/{target_id}") + if config.get("task", {}).get("name") != f"search-swe/{target_name}": + raise ValueError("Source task name does not match its directory") + print(f"git mv {source} tasks/{target_name}") if not dry_run: target.parent.mkdir(exist_ok=True) - git(repo, "mv", "--", source, f"tasks/{target_id}") + git(repo, "mv", "--", source, f"tasks/{target_name}") print("STOP: review and commit ONLY this pure rename; then run finalize with the same source and target.") return if src.exists() or not target.is_dir(): @@ -117,30 +116,28 @@ def promote(repo, source, target_id, phase, dry_run=False): for index in range(0, len(fields), 3): chunk = fields[index:index + 3] if (len(chunk) != 3 or chunk[0] != "R100" or not chunk[1].startswith(source + "/") - or chunk[2] != f"tasks/{target_id}/" + chunk[1][len(source) + 1:]): + or chunk[2] != f"tasks/{target_name}/" + chunk[1][len(source) + 1:]): raise ValueError("HEAD must contain only 100% identical source-to-target renames") renamed.add(chunk[2]) - tracked = set(filter(None, git(repo, "ls-files", "-z", "--", f"tasks/{target_id}").split("\0"))) + tracked = set(filter(None, git(repo, "ls-files", "-z", "--", f"tasks/{target_name}").split("\0"))) if not tracked or renamed != tracked: raise ValueError("Pure rename commit must include every tracked package file") config = tomllib.loads((target / "task.toml").read_text()) - if config.get("task", {}).get("name") != f"search-swe/{old_id}": - raise ValueError("Expected the unchanged temporary task name after rename") + if config.get("task", {}).get("name") != f"search-swe/{target_name}": + raise ValueError("Expected the same task name after promotion") changes = [] text_suffixes = {".md", ".toml", ".json", ".yaml", ".yml", ".sh", ".py", ".txt", ".cfg", ".ini"} for name in sorted(tracked): path = repo / name - if old_id in path.relative_to(target).as_posix(): - raise ValueError(f"Manual filename review required: {name}") raw = path.read_bytes() - if old_id.encode() not in raw and source.encode() not in raw: + if source.encode() not in raw: continue if path.suffix not in text_suffixes and path.name not in ("Dockerfile", ".gitignore", ".dockerignore"): raise ValueError(f"Manual review required for nonstandard text file: {name}") text = raw.decode("utf-8") - updated = text.replace(source, f"tasks/{target_id}").replace(old_id, target_id) - if old_id in updated or source in updated: - raise ValueError(f"Temporary reference remains: {name}") + updated = re.sub(re.escape(source) + r"(?![a-z0-9-])", f"tasks/{target_name}", text) + if re.search(re.escape(source) + r"(?![a-z0-9-])", updated): + raise ValueError(f"Submission reference remains: {name}") changes.append((path, updated)) for number, (before, after) in enumerate(zip(text.splitlines(), updated.splitlines()), 1): if before != after: @@ -148,7 +145,7 @@ def promote(repo, source, target_id, phase, dry_run=False): if not dry_run: for path, updated in changes: path.write_text(updated) - print("Temporary package references remaining after planned changes: 0.") + print("Submission package references remaining after planned changes: 0.") print("Review every change, external asset paths/images and repository task inventories. " "Publish approved assets, pin the official SHA, validate, then commit finalization in this SAME PR.") @@ -167,8 +164,8 @@ def prepare(snapshot, task, data, output): no_symlinks(tree) if output.exists(): raise ValueError("Upload output must not exist; refusing overwrite") - if not FORMAL.fullmatch(task.name): - raise ValueError("Assign the final task ID before preparing official publication") + if task.parent.name != "tasks" or not FORMAL.fullmatch(task.name): + raise ValueError("Promote to tasks/ before preparing official publication") config = tomllib.loads((task / "task.toml").read_text()) if config.get("task", {}).get("name") != f"search-swe/{task.name}": raise ValueError("Finalize task.toml before preparing official publication") @@ -221,13 +218,13 @@ def promotion_main(default_repo=None): parser = argparse.ArgumentParser(description="Two-phase offline promotion; never commits or publishes") parser.add_argument("phase", choices=("rename", "finalize")) parser.add_argument("source") - parser.add_argument("target_id") + parser.add_argument("target_name") parser.add_argument("--repo-root", type=Path, default=default_repo, required=default_repo is None) parser.add_argument("--dry-run", action="store_true") args = parser.parse_args() try: repo = checked_path(args.repo_root) - promote(repo, args.source, args.target_id, args.phase, args.dry_run) + promote(repo, args.source, args.target_name, args.phase, args.dry_run) except (OSError, ValueError, subprocess.CalledProcessError) as error: parser.error(str(error)) return 0 @@ -249,6 +246,9 @@ def upload_main(legacy=False): task = checked_path(repo / task) if not task.is_relative_to(repo): raise ValueError("Task must be inside --repo-root") + relative = task.relative_to(repo) + if len(relative.parts) != 2 or relative.parts[0] != "tasks": + raise ValueError("Official publication requires tasks/ under --repo-root") count = prepare(args.official_manifest, task, args.new_data, args.output) except (OSError, ValueError, KeyError, TypeError) as error: parser.error(str(error)) diff --git a/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py b/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py index f56f84a..c16a914 100644 --- a/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py +++ b/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py @@ -104,18 +104,18 @@ def task(self): (self.repo / ".gitignore").write_text("__pycache__/\n*.py[cod]\n") (self.repo / "base.txt").write_text("base\n") self.git("add", "."); self.git("commit", "-qm", "base") - self.source = "task-submissions/alice/1-x-1" + self.source = "task-submissions/example-search" task = self.repo / self.source task.mkdir(parents=True) - (task / "task.toml").write_text('[task]\nname = "search-swe/task-1-x-1"\nauthors = ["Alice Example", "Bob Example"]\n') - (task / "instruction.md").write_text("# task-1-x-1\n\nOriginal contributor content\n") + (task / "task.toml").write_text('[task]\nname = "search-swe/example-search"\nauthors = ["Alice Example", "Bob Example"]\n') + (task / "instruction.md").write_text("# example-search\n\ntask-submissions/example-search\nOriginal contributor content\n") (task / "assets.json").write_text('{"schema_version":1,"files":[]}\n') self.git("add", ".") self.git("commit", "-qm", "Write submission", "--author", "Alice Example ") return task def promote(self, phase, *extra): - return self.cli("promote_task.py", "--repo-root", self.repo, phase, self.source, "task-1-6", *extra) + return self.cli("promote_task.py", "--repo-root", self.repo, phase, self.source, "example-search", *extra) def test_capture_relocated_paginated_untrusted_and_read_only(self): # Dirty local content is untouched; dangerous PR text remains inert. @@ -191,7 +191,7 @@ def test_partial_check_set_and_intrafile_diff_are_not_verified(self): def test_promotion_relocation_collision_history_and_authors(self): task = self.task() - target = self.repo / "tasks/task-1-6" + target = self.repo / "tasks/example-search" target.mkdir() self.assertNotEqual(self.promote("rename").returncode, 0) target.rmdir() @@ -205,30 +205,47 @@ def test_promotion_relocation_collision_history_and_authors(self): result = self.promote("finalize") self.assertEqual(result.returncode, 0, result.stderr) self.assertIn('authors = ["Alice Example", "Bob Example"]', (target / "task.toml").read_text()) - self.assertIn("search-swe/task-1-6", (target / "task.toml").read_text()) + self.assertIn("search-swe/example-search", (target / "task.toml").read_text()) self.git("add", "."); self.git("commit", "-qm", "Finalize") - self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", "tasks/task-1-6/instruction.md")) - self.assertIn("author Alice Example", self.git("blame", "--line-porcelain", "tasks/task-1-6/instruction.md")) + self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", "tasks/example-search/instruction.md")) + self.assertIn("author Alice Example", self.git("blame", "--line-porcelain", "tasks/example-search/instruction.md")) # Explicit target is mandatory even when cwd is a Search-SWE checkout. self.assertNotEqual(self.cli("promote_task.py", "rename", self.source, "task-1-9").returncode, 0) def staging(self): - target = self.repo / "tasks/task-1-6" + target = self.repo / "tasks/example-search" target.mkdir(parents=True) - (target / "task.toml").write_text('[task]\nname="search-swe/task-1-6"\nauthors=["Alice Example"]\n') + (target / "task.toml").write_text('[task]\nname="search-swe/example-search"\nauthors=["Alice Example"]\n') payload = b"licensed fixture data\n" entry = {"path": "data/corpus.txt", "size_bytes": len(payload), "sha256": hashlib.sha256(payload).hexdigest(), "source": {"repo_id": "search-swe/Search-SWE", "repo_type": "dataset", "revision": None, - "filename": "tasks/task-1-6/corpus.txt"}} + "filename": "tasks/example-search/corpus.txt"}} (target / "assets.json").write_text(json.dumps({"schema_version": 1, "files": [entry]})) data = self.root / "new-data" file = data / entry["source"]["filename"] file.parent.mkdir(parents=True); file.write_bytes(payload) - old = {"path": "tasks/task-1-1/old.txt", "size_bytes": 99, "sha256": "b" * 64, "license": "preserve"} + old = {"path": "tasks/reasoning-query-rewriting/old.txt", "size_bytes": 99, "sha256": "b" * 64, "license": "preserve"} snapshot = self.root / "official.json" snapshot.write_text(json.dumps({"schema_version": 1, "files": [old], "extra": "preserve"})) return target, data, file, snapshot, old + def test_official_staging_rejects_submissions_and_nested_formal_paths(self): + target, data, _, snapshot, _ = self.staging() + submission = self.repo / "task-submissions/example-search" + submission.parent.mkdir() + target.rename(submission) + args = ("--repo-root", self.repo, "--official-manifest", snapshot, + "--new-data", data, "--output", self.out) + result = self.cli("prepare_hf_upload.py", *args, "--task-path", submission.relative_to(self.repo)) + self.assertNotEqual(result.returncode, 0) + self.assertIn("requires tasks/", result.stderr) + nested = self.repo / "nested/tasks/example-search" + nested.parent.mkdir(parents=True) + submission.rename(nested) + result = self.cli("prepare_hf_upload.py", *args, "--task-path", nested.relative_to(self.repo)) + self.assertNotEqual(result.returncode, 0) + self.assertFalse(self.out.exists()) + def test_incremental_staging_relocated_hashes_and_manifest_preservation(self): target, data, file, snapshot, old = self.staging() args = ("--repo-root", self.repo, "--task-path", target.relative_to(self.repo), @@ -239,7 +256,7 @@ def test_incremental_staging_relocated_hashes_and_manifest_preservation(self): self.assertEqual(merged["files"][0], old) self.assertEqual(merged["extra"], "preserve") self.assertEqual({p.relative_to(self.out).as_posix() for p in self.out.rglob("*") if p.is_file()}, - {"manifest.json", "tasks/task-1-6/corpus.txt"}) + {"manifest.json", "tasks/example-search/corpus.txt"}) self.assertNotEqual(self.cli("prepare_hf_upload.py", *args, "--output", self.out).returncode, 0) snapshot.write_text(json.dumps(merged)) self.assertNotEqual(self.cli("prepare_hf_upload.py", *args, "--output", self.root / "collision").returncode, 0) @@ -262,10 +279,10 @@ def test_legacy_wrapper_cli_compatibility(self): def legacy(name, *args): return subprocess.run([sys.executable, str(scripts / name), *map(str, args)], cwd=self.root, capture_output=True, text=True) - self.assertEqual(legacy("promote_task.py", "rename", self.source, "task-1-6").returncode, 0) + self.assertEqual(legacy("promote_task.py", "rename", self.source, "example-search").returncode, 0) self.git("commit", "-qm", "Pure rename") - self.assertEqual(legacy("promote_task.py", "finalize", self.source, "task-1-6").returncode, 0) - shutil.rmtree(self.repo / "tasks/task-1-6") + self.assertEqual(legacy("promote_task.py", "finalize", self.source, "example-search").returncode, 0) + shutil.rmtree(self.repo / "tasks/example-search") target, data, file, snapshot, old = self.staging() result = legacy("prepare_hf_upload.py", "--task-path", target, "--official-manifest", snapshot, "--new-data", data, "--output", self.out) @@ -292,7 +309,7 @@ def test_staging_rejects_symlinks_traversal_extra_files_and_external_target(self snapshot.write_text(json.dumps({"schema_version": 1, "files": [old]})) self.assertNotEqual(self.cli("prepare_hf_upload.py", *args).returncode, 0) snapshot.write_text(json.dumps({"schema_version": 1, "files": []})) - outside = self.root / "task-1-6" + outside = self.root / "example-search" shutil.copytree(target, outside) result = self.cli("prepare_hf_upload.py", "--repo-root", self.repo, "--task-path", outside, "--official-manifest", snapshot, "--new-data", data, "--output", self.out) diff --git a/.env.example b/.env.example index 00fb52c..a506ee5 100644 --- a/.env.example +++ b/.env.example @@ -41,16 +41,16 @@ AGENT_CODEX_CONFIG= # --------------------------------------------------------------------------- # 2. RewardKit 0.2.0 trajectory judge # --------------------------------------------------------------------------- -# Required by every current task except task-2-4, regardless of which coding +# Required by every current task except agentic-search, regardless of which coding # agent is selected. The judge is fixed to deepseek/deepseek-v4.1-flash on # OpenRouter. Use an OpenRouter key for VERIFIER_OPENAI_API_KEY. VERIFIER_OPENAI_BASE_URL=https://openrouter.ai/api/v1 VERIFIER_OPENAI_API_KEY= # --------------------------------------------------------------------------- -# 3. Task-1-3 answer-equivalence judge +# 3. Task scientific-paper-qa answer-equivalence judge # --------------------------------------------------------------------------- -# Required only by task-1-3. Use a Chat Completions endpoint allowed by the +# Required only by scientific-paper-qa. Use a Chat Completions endpoint allowed by the # task verifier (OpenRouter or DeepSeek); this judge is configured independently. ANSWER_JUDGE_MODEL_NAME= ANSWER_JUDGE_BASE_URL=https://openrouter.ai/api/v1 @@ -60,8 +60,8 @@ ANSWER_JUDGE_API_KEY= # 4. Optional APIs available to submissions # --------------------------------------------------------------------------- # Fill only when the selected task and your implementation use that service. -# task-1-1 uses its own restricted OpenRouter key. -TASK_1_1_OPENROUTER_API_KEY= +# reasoning-query-rewriting uses its own restricted OpenRouter key. +REASONING_QUERY_REWRITING_OPENROUTER_API_KEY= # OPENROUTER_API_KEY: tasks 1-3, 1-4, and 2-4. # JINA_API_KEY: tasks 1-1, 1-3, 1-4, and 2-4. diff --git a/.github/workflows/task-checks.yml b/.github/workflows/task-checks.yml index 58ceff7..9ab1038 100644 --- a/.github/workflows/task-checks.yml +++ b/.github/workflows/task-checks.yml @@ -22,7 +22,7 @@ jobs: - run: python -m pip install -r scripts/requirements.txt PyYAML python-dotenv - run: python scripts/check_release.py - run: python scripts/check_submission.py - - name: One contributor namespace per PR (including promoted packages) + - name: Submission history and same-PR promotion if: github.event_name == 'pull_request' env: BASE_SHA: ${{ github.event.pull_request.base.sha }} diff --git a/.gitignore b/.gitignore index f3f9721..eeb3fed 100644 --- a/.gitignore +++ b/.gitignore @@ -6,9 +6,9 @@ /.venv/ tasks/*/data/ tasks/*/models/ -task-submissions/*/*/data/ -task-submissions/*/*/models/ -task-submissions/*/*/jobs/ -task-submissions/*/*/solution/ +task-submissions/*/data/ +task-submissions/*/models/ +task-submissions/*/jobs/ +task-submissions/*/solution/ __pycache__/ *.py[cod] diff --git a/scripts/check_release.py b/scripts/check_release.py index 9bcb67f..5e8fd67 100644 --- a/scripts/check_release.py +++ b/scripts/check_release.py @@ -10,7 +10,7 @@ import tomllib from download_assets import REPO, matches, read_manifest, relative_path -from task_paths import no_symlinks, safe_path +from task_paths import FORMAL, no_symlinks, safe_path def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False, *, packages=None): @@ -45,6 +45,12 @@ def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False errors.append(f"{task.name}: missing or empty {name}") try: config = tomllib.loads((task / "task.toml").read_text()) + if not FORMAL.fullmatch(task.name): + errors.append(f"Invalid task name: {task.name}; use at most five lowercase hyphen-separated words without a task- prefix") + if config.get("task", {}).get("name") != f"search-swe/{task.name}": + errors.append(f"{task.name}: expected task name search-swe/{task.name}") + if "task_type" in config.get("metadata", {}): + errors.append(f"{task.name}: remove metadata.task_type; describe the task through its objective and keywords") if not config.get("task", {}).get("version"): errors.append(f"{task.name}: missing task version") manifest = read_manifest(task) @@ -88,8 +94,8 @@ def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False if not path.is_file(): continue # A tracked deletion is absent from the next commit. parts = Path(name).parts - task_payload = ((len(parts) > 3 and parts[0] == "tasks" and parts[2] in ("data", "models", "jobs", "solution")) - or (len(parts) > 4 and parts[0] == "task-submissions" and parts[3] in ("data", "models", "jobs", "solution"))) + task_payload = (len(parts) > 3 and parts[0] in ("tasks", "task-submissions") + and parts[2] in ("data", "models", "jobs", "solution")) if name in runtime_paths or task_payload: errors.append(f"Downloaded runtime asset would enter Git: {name}") if path.stat().st_size >= 100 * 1024**2: @@ -101,8 +107,8 @@ def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False text = path.read_text() except UnicodeError: continue - if name.startswith("tasks/") and re.search(r"task-[12]-x-[1-9][0-9]*\b", text): - errors.append(f"Unfinalized temporary task ID in formal package: {name}") + if name.startswith("tasks/") and re.search(r"task-submissions/[a-z0-9-]+", text): + errors.append(f"Unfinalized submission path in formal package: {name}") if secret.search(text): errors.append(f"Possible credential in Git file: {name} (value omitted)") if runtime_paths: diff --git a/scripts/check_submission.py b/scripts/check_submission.py index 945e374..e14a5b9 100644 --- a/scripts/check_submission.py +++ b/scripts/check_submission.py @@ -11,30 +11,27 @@ from check_release import check_release from download_assets import REPO -from task_paths import SUBMISSION, no_symlinks, safe_path, select_task +from task_paths import FORMAL, SUBMISSION, no_symlinks, safe_path, select_task def discover(repo): root = safe_path(repo, "task-submissions") no_symlinks(root) # Include malformed paths and incomplete package directories, not just valid TOMLs. - return sorted(set(root.glob("*/*/")) | {p.parent for p in root.rglob("task.toml")}) + return sorted(set(root.glob("*/")) | {p.parent for p in root.rglob("task.toml")}) -def submission_namespace(path): - """Return a namespace from a canonical package path, rejecting unsafe history names.""" - if "\\" in path: +def submission_root(path): + """Validate a historical filename and return its flat submission package path.""" + if "\\" in path or any(part in ("", ".", "..") for part in path.split("/")): raise ValueError(f"Unsafe task-submissions path: {path!r}") parts = path.split("/") - if any(part in ("", ".", "..") for part in parts): - raise ValueError(f"Unsafe task-submissions path: {path!r}") - if parts[0] != "task-submissions": - raise ValueError(f"Expected task-submissions path: {path!r}") - if len(parts) == 2 and parts[1] == "README.md": + if path == "task-submissions/README.md": return None - if len(parts) < 4 or not re.fullmatch(SUBMISSION, "/".join(parts[:3])): + if (len(parts) < 3 or not SUBMISSION.fullmatch("/".join(parts[:2])) + or (parts[-1] == "task.toml" and len(parts) != 3)): raise ValueError(f"Malformed task submission path in PR history: {path!r}") - return parts[1] + return "/".join(parts[:2]) def git_names(repo, *args): @@ -52,39 +49,77 @@ def git_object_exists(repo, commit, path): def history_submission_audit(repo, base): - """Inspect every commit and merge parent; return namespaces and reused task roots.""" + """Track path reuse and unchanged, separate promotion commits across every parent.""" commits = subprocess.run( ["git", "rev-list", "--reverse", "--topo-order", f"{base}..HEAD"], cwd=repo, check=True, capture_output=True, text=True, ).stdout.splitlines() - namespaces = set() - additions = {} + additions, promoted = {}, set() for commit in commits: parents = subprocess.run( ["git", "show", "-s", "--format=%P", commit], cwd=repo, check=True, capture_output=True, text=True, ).stdout.split() changed = set() - if parents: - for parent in parents: - changed.update(git_names( - repo, "diff-tree", "--no-commit-id", "-r", "--name-only", - "-z", "--no-renames", parent, commit, "--", "task-submissions", - )) - else: - changed.update(git_names( - repo, "diff-tree", "--root", "--no-commit-id", "-r", - "--name-only", "-z", "--no-renames", commit, "--", "task-submissions", - )) + for parent in parents: + names = git_names(repo, "diff-tree", "--no-commit-id", "-r", "--name-only", + "-z", "--no-renames", parent, commit) + changed.update(name for name in names if name.startswith("task-submissions/")) + if len(parents) != 1: + continue # A promotion must be its own single-parent commit. + for name in names: + parts = name.split("/") + if len(parts) != 3 or parts[0] != "tasks" or parts[2] != "task.toml": + continue + task_name = parts[1] + if not FORMAL.fullmatch(task_name): + continue + source, target = f"task-submissions/{task_name}", f"tasks/{task_name}" + if (git_object_exists(repo, parent, target) + or git_object_exists(repo, commit, source) + or not git_object_exists(repo, parent, source) + or not git_object_exists(repo, commit, target)): + continue + if any(not (p.startswith(source + "/") or p.startswith(target + "/")) for p in names): + continue + trees = subprocess.run( + ["git", "rev-parse", f"{parent}:{source}", f"{commit}:{target}"], + cwd=repo, check=True, capture_output=True, text=True, + ).stdout.splitlines() + if len(trees) == 2 and trees[0] == trees[1]: + promoted.add(task_name) + if not parents: + changed.update(git_names(repo, "diff-tree", "--root", "--no-commit-id", "-r", + "--name-only", "-z", "--no-renames", commit, "--", "task-submissions")) for name in changed: - namespace = submission_namespace(name) - if namespace is not None: - namespaces.add(namespace) + submission_root(name) if (name.endswith("/task.toml") and git_object_exists(repo, commit, name) and all(not git_object_exists(repo, parent, name) for parent in parents)): additions[name] = additions.get(name, 0) + 1 - return namespaces, sorted(name.rsplit("/", 1)[0] - for name, count in additions.items() if count > 1) + return promoted, sorted(name.rsplit("/", 1)[0] for name, count in additions.items() if count > 1) + + +def renamed_formal_packages(repo, base): + """Recognize existing formal packages renamed in this PR, including legacy names.""" + fields = git_names(repo, "diff", "--name-status", "-z", "--find-renames=50%", + f"{base}...HEAD", "--", "tasks") + renamed, index = set(), 0 + while index < len(fields): + status = fields[index] + index += 1 + source = fields[index] + index += 1 + if not status.startswith(("R", "C")): + continue + target = fields[index] + index += 1 + old, new = Path(source), Path(target) + if (status.startswith("R") and len(old.parts) == len(new.parts) == 3 + and old.name == new.name == "task.toml" + and git_object_exists(repo, base, source) + and not git_object_exists(repo, "HEAD", str(old.parent))): + renamed.add(new.parent.name) + return renamed def check_package(repo, task, *, formal=False, require_authors=True): @@ -95,12 +130,9 @@ def check_package(repo, task, *, formal=False, require_authors=True): if not formal and not match: raise ValueError("Not a submission path") config = tomllib.loads((task / "task.toml").read_text()) - expected = task.name if formal else f"task-{match[2]}-x-{match[3]}" - category = re.fullmatch(r"task-([12])-(?:x-[1-9][0-9]*|[1-9][0-9]*)", expected) - if category: - kind = {"1": "create", "2": "optimize"}[category[1]] - if config.get("metadata", {}).get("task_type") != kind: - errors.append(f"Category {category[1]} requires metadata.task_type = {kind}") + expected = task.name + if not formal and (repo / "tasks" / expected).exists(): + errors.append(f"Task name collides with an existing formal package: {expected}") if config.get("task", {}).get("name") != f"search-swe/{expected}": errors.append(f"Expected task name search-swe/{expected}") authors = config.get("task", {}).get("authors", []) @@ -124,14 +156,11 @@ def check_package(repo, task, *, formal=False, require_authors=True): text = path.read_text() except UnicodeError: continue - ids = set(re.findall(r"\btask-[12]-(?:[1-9][0-9]*|x-[1-9][0-9]*)\b", text)) - if not formal and ids - {expected}: - errors.append(f"{name}: unexpected task IDs {sorted(ids - {expected})}") - if formal and any(re.fullmatch(r"task-[12]-x-[1-9][0-9]*", value) for value in ids): - errors.append(f"{name}: temporary task ID remains") - for ref in re.findall(r"task-submissions/[a-z0-9-]+/[12]-x-[1-9][0-9]*", text): + if re.search(r"\btask-[12]-(?:[1-9][0-9]*|x-[1-9][0-9]*)\b", text): + errors.append(f"{name}: obsolete numbered task reference") + for ref in re.findall(r"task-submissions/[a-z0-9-]+", text): if formal or ref != task.relative_to(repo).as_posix(): - errors.append(f"{name}: another submission path: {ref}") + errors.append(f"{name}: another or unfinalized submission path: {ref}") if path.suffix == ".py": ast.parse(text, filename=name) if path.suffix == ".sh": @@ -162,7 +191,7 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("task_path", nargs="?", help="Repository-relative submission; otherwise check all") parser.add_argument("--merge-ready", action="store_true", help="Fail if any submission task.toml remains") - parser.add_argument("--base", help="Full PR base commit for contributor-namespace and new-formal-package checks") + parser.add_argument("--base", help="Full PR base commit for submission history and formal-package checks") args = parser.parse_args() errors = [] try: @@ -170,18 +199,6 @@ def main(): for task in tasks: errors.extend(f"{task.relative_to(REPO)}: {error}" for error in check_submission(REPO, task)) remaining = list((REPO / "task-submissions").rglob("task.toml")) - current_namespaces = set() - for task in discover(REPO): - rel = task.relative_to(REPO).as_posix() - if rel.startswith("task-submissions/"): - try: - namespace = submission_namespace(rel + "/task.toml") - if namespace: - current_namespaces.add(namespace) - except ValueError as error: - errors.append(str(error)) - if len(current_namespaces) > 1: - errors.append(f"Submission tasks must use exactly one contributor namespace: {sorted(current_namespaces)}") if args.merge_ready and remaining: errors.append("Not merge-ready: submission task.toml remains; promote in the same PR") # Promotion must not remove the package's static validation coverage. @@ -196,17 +213,18 @@ def main(): f"{args.base}...HEAD", "--", "tasks", "task-submissions"], cwd=REPO, check=True, capture_output=True).stdout.decode( "utf-8", errors="surrogateescape").split("\0") - history_namespaces, reused_tasks = history_submission_audit(REPO, args.base) - if len(history_namespaces) > 1: - errors.append(f"PR task submissions must use exactly one contributor namespace: {sorted(history_namespaces)}") + promoted, reused_tasks = history_submission_audit(REPO, args.base) if reused_tasks: - errors.append(f"Temporary task ordinals must not be reused in one PR: {reused_tasks}") + errors.append(f"Submission paths must not be reused in one PR: {reused_tasks}") + renamed = renamed_formal_packages(REPO, args.base) added_formal = [name for name in filter(None, added) if name.startswith("tasks/") and name.endswith("/task.toml")] - if added_formal and len(history_namespaces) != 1: - errors.append("New formal tasks require exactly one contributor submission namespace in PR history") for name in added_formal: - errors.extend(check_package(REPO, (REPO / name).parent, formal=True)) + task = (REPO / name).parent + existing = task.name in renamed + if not existing and task.name not in promoted: + errors.append(f"{name}: new formal tasks require same-PR pure promotion from task-submissions/{task.name}") + errors.extend(check_package(REPO, task, formal=True, require_authors=not existing)) except (OSError, ValueError, subprocess.CalledProcessError) as error: errors.append(str(error)) for error in errors: diff --git a/scripts/download_assets.py b/scripts/download_assets.py index cb19c5e..eba54fe 100755 --- a/scripts/download_assets.py +++ b/scripts/download_assets.py @@ -204,10 +204,10 @@ def restore(task, task_output, entry, directory_modes, args): def main(default_kind="all"): parser = argparse.ArgumentParser(description=__doc__) selection = parser.add_mutually_exclusive_group(required=True) - selection.add_argument("--task", nargs="+", choices=("all", *TASKS)) - selection.add_argument("--task-path", help="Explicit repository-relative package, including a submission") + selection.add_argument("--task", nargs="+", choices=("all", *TASKS), help="Formal task names, or all formal tasks") + selection.add_argument("--task-path", help="Repository-relative tasks/ or task-submissions/") parser.add_argument("--kind", choices=("all", "data", "models"), default=default_kind) - parser.add_argument("--output-dir", type=Path, help="Alternate parent; submission namespace is preserved") + parser.add_argument("--output-dir", type=Path, help="Alternate parent; task-submissions/ is preserved") parser.add_argument("--cache-dir", type=Path, help="Hugging Face cache directory") parser.add_argument("--local-data-dir", type=Path, help="Restore Search-SWE dataset files from a local hf-data directory, with size/SHA-256 checks") parser.add_argument("--force", action="store_true", help="Replace files whose checksums differ") diff --git a/scripts/run_task.py b/scripts/run_task.py index f62a51c..ad4394f 100755 --- a/scripts/run_task.py +++ b/scripts/run_task.py @@ -105,8 +105,8 @@ def host_is_allowed(host, allowed_hosts): def main(): parser = argparse.ArgumentParser(description=__doc__) selection = parser.add_mutually_exclusive_group(required=True) - selection.add_argument("--task", choices=TASKS) - selection.add_argument("--task-path", help="Explicit repository-relative package, including a reviewed submission") + selection.add_argument("--task", choices=TASKS, help="Formal task name (for example reasoning-query-rewriting)") + selection.add_argument("--task-path", help="Repository-relative tasks/ or task-submissions/") parser.add_argument("--agent", default="codex", choices=tuple(AGENT_IMPORTS)) parser.add_argument( "--openrouter", action="store_true", diff --git a/scripts/task_paths.py b/scripts/task_paths.py index 7785efb..e165d93 100644 --- a/scripts/task_paths.py +++ b/scripts/task_paths.py @@ -3,9 +3,10 @@ from pathlib import Path import re -SLUG = r"[a-z][a-z0-9]*(?:-[a-z0-9]+)*" -SUBMISSION = re.compile(rf"task-submissions/({SLUG})/([12])-x-([1-9][0-9]*)") -FORMAL = re.compile(r"task-[12]-[1-9][0-9]*") +# Keep this contract in sync with the two independently installable skill helpers. +TASK_NAME = r"(?!all(?:/|$))(?!task-)[a-z][a-z0-9]*(?:-[a-z0-9]+){0,4}" +FORMAL = re.compile(TASK_NAME) +SUBMISSION = re.compile(rf"task-submissions/({TASK_NAME})") def safe_path(root, value): @@ -32,8 +33,9 @@ def no_symlinks(path): def select_task(root, value): task = safe_path(root, value) rel = task.relative_to(root.resolve()).as_posix() - if not SUBMISSION.fullmatch(rel) and not re.fullmatch(r"tasks/task-[a-z0-9]+(?:-[a-z0-9]+)*", rel): - raise ValueError("--task-path must name tasks/ or task-submissions//<1|2>-x-") + if not SUBMISSION.fullmatch(rel) and not re.fullmatch(rf"tasks/({TASK_NAME})", rel): + raise ValueError("--task-path must name tasks/ or task-submissions/; " + "use at most five lowercase hyphen-separated words, without a task- prefix") if not (task / "task.toml").is_file(): raise ValueError(f"Missing task.toml: {rel}") no_symlinks(task) @@ -41,6 +43,6 @@ def select_task(root, value): def task_key(root, task): - """Keep submission namespaces in output paths; equal temporary IDs cannot collide.""" + """Keep submission outputs separate from formal task outputs.""" rel = task.relative_to(root.resolve()) return Path(*rel.parts[1:]) if rel.parts[0] == "tasks" else rel diff --git a/scripts/tests/egress_adapter_smoke.py b/scripts/tests/egress_adapter_smoke.py index e4977d4..61aeceb 100644 --- a/scripts/tests/egress_adapter_smoke.py +++ b/scripts/tests/egress_adapter_smoke.py @@ -179,7 +179,7 @@ async def run(args): raise AssertionError("baseline not restored") result["checks"].append({"case": "real-phase-exception-restores-baseline", "passed": True}) repo = Path(__file__).resolve().parents[2] - task_config = TaskConfig.model_validate_toml((repo / "tasks/task-1-1/task.toml").read_text()) + task_config = TaskConfig.model_validate_toml((repo / "tasks/reasoning-query-rewriting/task.toml").read_text()) plan = resolve_trial_network_plan(task_config, AgentConfig(extra_allowed_hosts=["model.example"]), runtime_config, None, verifier_mode=resolve_task_verifier_mode(task_config), env_config=resolve_effective_verifier_env_config(task_config, None)) diff --git a/scripts/tests/egress_full_trial.py b/scripts/tests/egress_full_trial.py index d564f35..f87064b 100644 --- a/scripts/tests/egress_full_trial.py +++ b/scripts/tests/egress_full_trial.py @@ -1,7 +1,7 @@ """Full Harbor Trial.run gate: two simultaneous trials of one four-step task. Formal adapter/Oracle extension points only. No patched Harbor methods, API calls, -task-1-1 edits, proxy forwarding, or changes to the operator Docker daemon. +reasoning-query-rewriting edits, proxy forwarding, or changes to the operator Docker daemon. """ import argparse import asyncio diff --git a/scripts/tests/test_egress_contract.py b/scripts/tests/test_egress_contract.py index de31368..4a87d5c 100644 --- a/scripts/tests/test_egress_contract.py +++ b/scripts/tests/test_egress_contract.py @@ -82,18 +82,18 @@ def plan(self, task_name, model_host="model.example"): env_config=resolve_effective_verifier_env_config(config, None)) def test_phase_policy_uses_harbor_inheritance_and_model_host_is_agent_only(self): - plan = self.plan("task-1-1") + plan = self.plan("reasoning-query-rewriting") self.assertEqual(plan.agent_env_baseline.allowed_hosts, ["openrouter.ai", "api.jina.ai"]) self.assertIn("model.example", plan.agent_phase.allowed_hosts) self.assertNotIn("model.example", plan.verifier_phase.allowed_hosts) self.assertNotIn("api.deepseek.com", plan.verifier_phase.allowed_hosts) self.assertNotIn("api.deepseek.com", plan.agent_phase.allowed_hosts) - offline = self.plan("task-2-1") + offline = self.plan("long-document-reranking") self.assertEqual(offline.agent_env_baseline.network_mode.value, "no-network") self.assertEqual(offline.agent_phase.allowed_hosts, ["model.example"]) def test_real_separate_verifier_path_preserves_adapter_and_binding_not_overlay(self): - plan = self.plan("task-1-1") + plan = self.plan("reasoning-query-rewriting") agent = self.create(plan.agent_env_baseline) external_overlay = self.root / "agent-only.json" external_overlay.write_text('{"services": {}}\n') @@ -126,7 +126,7 @@ async def exercise(): self.assertEqual(self.environment_config.extra_docker_compose, [external_overlay]) def test_real_phase_context_restores_baseline_on_error(self): - plan = self.plan("task-1-1") + plan = self.plan("reasoning-query-rewriting") environment = self.create(plan.agent_env_baseline) async def exercise(): diff --git a/scripts/tests/test_egress_launcher.py b/scripts/tests/test_egress_launcher.py index f649916..f14466f 100644 --- a/scripts/tests/test_egress_launcher.py +++ b/scripts/tests/test_egress_launcher.py @@ -24,15 +24,15 @@ def setUp(self): "dns": {"doh_url": "https://resolver.example/dns-query"}} self.config.write_text(json.dumps(self.document)) - def launch(self, *arguments, task="task-1-1", env=None, public=False): + def launch(self, *arguments, task="reasoning-query-rewriting", env=None, public=False): repo = ROOT if public: # Public-policy coverage must not depend on a benchmark task's - # evolving network requirements (for example, task-2-3). + # evolving network requirements (for example, query-encoder-alignment). repo = self.root / "repo" shutil.copytree(ROOT / "scripts", repo / "scripts", ignore=shutil.ignore_patterns("__pycache__", "tests")) - task = "task-fixture" + task = "fixture-search" fixture = repo / "tasks" / task fixture.mkdir(parents=True) (fixture / "task.toml").write_text( @@ -102,7 +102,7 @@ def test_public_phase_rejected_without_changing_task(self): self.assertNotEqual(response.returncode, 0, response.stdout + response.stderr) self.assertIn("any public phase", response.stderr) self.assertIn('network_mode = "public"', - (self.root / "repo/tasks/task-fixture/task.toml").read_text()) + (self.root / "repo/tasks/fixture-search/task.toml").read_text()) def test_rejected_url_does_not_echo_embedded_secret(self): self.document["upstream"]["url"] = "https://user:do-not-leak-this@proxy.example" diff --git a/scripts/tests/test_network_policies.py b/scripts/tests/test_network_policies.py index 918007d..8dc4a42 100644 --- a/scripts/tests/test_network_policies.py +++ b/scripts/tests/test_network_policies.py @@ -31,15 +31,15 @@ TASK_HOSTS = ["openrouter.ai", "api.jina.ai"] DEEPSEEK_HOST = "api.deepseek.com" EXPECTED = { - "task-1-1": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), - "task-1-2": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), - "task-1-3": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", [*TASK_HOSTS, DEEPSEEK_HOST]), - "task-1-4": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), - "task-2-1": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), - "task-2-2": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), - "task-2-3": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), - "task-2-4": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), - "task-2-5": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), + "reasoning-query-rewriting": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), + "memory-constrained-dense-retrieval": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), + "scientific-paper-qa": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", [*TASK_HOSTS, DEEPSEEK_HOST]), + "long-pdf-evidence": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), + "long-document-reranking": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), + "code-embedding-finetuning": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), + "query-encoder-alignment": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), + "agentic-search": ("allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS, "allowlist", TASK_HOSTS), + "sparse-retrieval-pruning": ("no-network", [], "no-network", [], "allowlist", ["openrouter.ai"]), } @@ -103,7 +103,7 @@ def test_task_resource_docs_match_the_network_profiles(self): ) self.assertTrue(resource_doc.is_file(), task_name) - for task_name in ("task-1-2", "task-2-1", "task-2-2", "task-2-5"): + for task_name in ("memory-constrained-dense-retrieval", "long-document-reranking", "code-embedding-finetuning", "sparse-retrieval-pruning"): text = ( REPO / "tasks" / task_name / "environment/docs/available_resources.md" ).read_text().lower() @@ -306,7 +306,7 @@ def test_restricted_agent_phase_gets_only_the_selected_model_host(self): ) for agent, model, values, expected_host in cases: with self.subTest(agent=agent, model=model): - result = self.run_preview("task-1-2", agent, model, values) + result = self.run_preview("memory-constrained-dense-retrieval", agent, model, values) self.assertEqual(result.returncode, 0, result.stderr) argv = self.argv(result) index = argv.index("--allow-agent-host") @@ -318,14 +318,14 @@ def test_restricted_agent_phase_gets_only_the_selected_model_host(self): }[agent] self.assertEqual(argv[argv.index("--agent") + 1], expected_import) - restricted = self.run_preview("task-2-3", "pi", "deepseek/deepseek-flash", {}) + restricted = self.run_preview("query-encoder-alignment", "pi", "deepseek/deepseek-flash", {}) self.assertEqual(restricted.returncode, 0, restricted.stderr) self.assertEqual( self.flag_values(self.argv(restricted), "--allow-agent-host"), ["api.deepseek.com"], ) restricted = self.run_preview( - "task-2-3", "claude-code", "claude-sonnet-4-6", {} + "query-encoder-alignment", "claude-code", "claude-sonnet-4-6", {} ) self.assertEqual(restricted.returncode, 0, restricted.stderr) self.assertEqual( @@ -336,7 +336,7 @@ def test_restricted_agent_phase_gets_only_the_selected_model_host(self): def test_claude_code_preview_uses_pinned_cli_and_namespaced_key(self): secret = "fixture-anthropic-secret" result = self.run_preview( - "task-1-2", + "memory-constrained-dense-retrieval", "claude-code", "claude-sonnet-4-6", { @@ -372,7 +372,7 @@ def test_openrouter_agent_routes_are_isolated_from_other_keys(self): "ANTHROPIC_BASE_URL=https://openrouter.ai/api"]), ): with self.subTest(agent=agent): - result = self.run_preview("task-1-2", agent, model, values, ("--openrouter",)) + result = self.run_preview("memory-constrained-dense-retrieval", agent, model, values, ("--openrouter",)) self.assertEqual(result.returncode, 0, result.stderr) for secret in values.values(): self.assertNotIn(secret, result.stdout + result.stderr) @@ -399,11 +399,11 @@ def test_openrouter_requires_supported_agent_and_complete_slug(self): ) for agent, model, message in cases: with self.subTest(agent=agent, model=model): - result = self.run_preview("task-1-2", agent, model, {}, ("--openrouter",)) + result = self.run_preview("memory-constrained-dense-retrieval", agent, model, {}, ("--openrouter",)) self.assertNotEqual(result.returncode, 0) self.assertIn(message, result.stderr) conflict = self.run_preview( - "task-1-2", "codex", "openai/gpt-6-astra", {}, + "memory-constrained-dense-retrieval", "codex", "openai/gpt-6-astra", {}, ("--openrouter", "--codex-config", "missing.toml"), ) self.assertIn("omit --codex-config", conflict.stderr) @@ -435,7 +435,7 @@ def test_claude_code_rejects_cross_agent_options_and_invalid_effort(self): sys.executable, str(REPO / "scripts/run_task.py"), "--task", - "task-1-2", + "memory-constrained-dense-retrieval", "--agent", "claude-code", "--model", @@ -454,7 +454,7 @@ def test_claude_code_rejects_cross_agent_options_and_invalid_effort(self): self.assertIn(message, result.stderr) result = self.run_preview( - "task-1-2", + "memory-constrained-dense-retrieval", "claude-code", "claude-sonnet-4-6", {"AGENT_REASONING_EFFORT": "ultracode"}, @@ -469,17 +469,17 @@ def test_unlisted_judge_hosts_and_general_proxy_are_rejected(self): } cases = ( ( - "task-1-2", + "memory-constrained-dense-retrieval", {**common, "VERIFIER_OPENAI_BASE_URL": "https://relay.example/v1"}, "not permitted", ), ( - "task-1-3", + "scientific-paper-qa", {**common, "ANSWER_JUDGE_BASE_URL": "https://answer.example/v1"}, "not permitted", ), ( - "task-1-2", + "memory-constrained-dense-retrieval", {**common, "CONTAINER_PROXY": "http://proxy.example:8080"}, "incompatible", ), @@ -490,9 +490,9 @@ def test_unlisted_judge_hosts_and_general_proxy_are_rejected(self): self.assertNotEqual(result.returncode, 0) self.assertIn(message, result.stderr) - def test_task_1_3_accepts_only_its_fixed_judge_hosts(self): + def test_scientific_paper_qa_accepts_only_its_fixed_judge_hosts(self): result = self.run_preview( - "task-1-3", + "scientific-paper-qa", "codex", "fixture-model", { diff --git a/scripts/tests/test_release.py b/scripts/tests/test_release.py index cbded67..1c47645 100644 --- a/scripts/tests/test_release.py +++ b/scripts/tests/test_release.py @@ -26,7 +26,7 @@ def setUp(self): self.repo = self.root / "repo" self.repo.mkdir() subprocess.run(["git", "init", "-q", str(self.repo)], check=True) - self.task = self.repo / "tasks" / "task-new" + self.task = self.repo / "tasks" / "example-search" for name in ("environment", "tests", "data"): (self.task / name).mkdir(parents=True) for name in ("instruction.md", "environment/Dockerfile", "tests/Dockerfile", "tests/test.sh"): @@ -34,7 +34,7 @@ def setUp(self): for name in ("environment/docker-compose.yaml", "tests/docker-compose.yaml"): (self.task / name).write_text("services: {}\n") (self.task / "task.toml").write_text( - '[task]\nname = "fixture/task-new"\nversion = "0.1"\n' + '[task]\nname = "search-swe/example-search"\nversion = "0.1"\n' '[verifier.env]\nOPENAI_BASE_URL = "${OPENAI_BASE_URL:-}"\n' 'OPENAI_API_KEY = "${OPENAI_API_KEY:-}"\n' ) @@ -44,18 +44,18 @@ def setUp(self): "path": "data/corpus.jsonl", "size_bytes": len(self.content), "sha256": hashlib.sha256(self.content).hexdigest(), "source": {"repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "a" * 40, "filename": "tasks/task-new/corpus.jsonl"}, + "revision": "a" * 40, "filename": "tasks/example-search/corpus.jsonl"}, } self.write_manifest() (self.task / self.entry["path"]).write_bytes(self.content) self.hf = self.root / "hf-data" - self.hf_file = self.hf / "tasks/task-new/corpus.jsonl" + self.hf_file = self.hf / "tasks/example-search/corpus.jsonl" self.hf_file.parent.mkdir(parents=True) self.hf_file.write_bytes(self.content) for name in ("README.md", "SOURCES.md", ".gitattributes"): (self.hf / name).write_text("fixture\n") record = {key: self.entry[key] for key in ("size_bytes", "sha256")} - record["path"] = "tasks/task-new/corpus.jsonl" + record["path"] = "tasks/example-search/corpus.jsonl" (self.hf / "manifest.json").write_text(json.dumps({"schema_version": 1, "files": [record]})) shutil.copytree(SCRIPTS, self.repo / "scripts", ignore=shutil.ignore_patterns("__pycache__", "tests")) @@ -112,7 +112,7 @@ def test_staging_mode_still_rejects_corrupt_data(self): self.assertTrue(any("SHA-256 mismatch" in error for error in errors)) def test_forced_tracked_data_is_rejected_even_when_ignored(self): - subprocess.run(["git", "add", "-f", "tasks/task-new/data/corpus.jsonl"], cwd=self.repo, check=True) + subprocess.run(["git", "add", "-f", "tasks/example-search/data/corpus.jsonl"], cwd=self.repo, check=True) errors, _ = release.check_release(self.repo) self.assertTrue(any("runtime asset would enter Git" in error for error in errors)) @@ -124,12 +124,12 @@ def test_hidden_or_unlisted_hf_input_is_rejected(self): def test_new_task_automatically_appears_in_download_and_launch_commands(self): for script, args in ( ("download_assets.py", ["--task", "all", "--dry-run"]), - ("run_task.py", ["--task", "task-new", "--model", "example-model", "--dry-run"]), + ("run_task.py", ["--task", "example-search", "--model", "example-model", "--dry-run"]), ): result = subprocess.run([sys.executable, str(self.repo / "scripts" / script), *args], cwd=self.root, capture_output=True, text=True) self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn("task-new", result.stdout) + self.assertIn("example-search", result.stdout) def test_launcher_separates_agent_and_verifier_keys_without_putting_them_in_argv(self): (self.repo / ".env").write_text( @@ -156,7 +156,7 @@ def test_launcher_separates_agent_and_verifier_keys_without_putting_them_in_argv env["CODEX_AUTH_JSON_PATH"] = "/unused/auth.json" env["CODEX_FORCE_AUTH_JSON"] = "1" result = subprocess.run([sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new"], cwd=self.root, env=env, + "--task", "example-search"], cwd=self.root, env=env, capture_output=True, text=True) self.assertEqual(result.returncode, 0, result.stderr) output = result.stdout + result.stderr @@ -188,7 +188,7 @@ def test_openrouter_launch_requires_its_own_key_and_keeps_it_out_of_argv(self): "OPENAI_BASE_URL": "https://wrong-provider.example/v1", }) command = [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", "codex", "--openrouter", + "--task", "example-search", "--agent", "codex", "--openrouter", "--model", "openai/gpt-6-astra"] missing = subprocess.run(command, cwd=self.root, env=env, capture_output=True, text=True) @@ -211,7 +211,7 @@ def test_step_restriction_selects_gateway_and_model_host(self): '[[steps]]\nname = "online"\n' '[[steps]]\nname = "offline"\n[steps.agent]\nnetwork_mode = "no-network"\n' ) - command = [sys.executable, str(self.repo / "scripts/run_task.py"), "--task", "task-new", + command = [sys.executable, str(self.repo / "scripts/run_task.py"), "--task", "example-search", "--agent", "pi", "--model", "deepseek/deepseek-flash", "--dry-run"] result = subprocess.run(command, cwd=self.root, env=self.launcher_env(), capture_output=True, text=True) self.assertEqual(result.returncode, 0, result.stderr) @@ -270,7 +270,7 @@ def test_launcher_makes_repository_agents_importable(self): }) result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", agent, "--model", model], + "--task", "example-search", "--agent", agent, "--model", model], cwd=self.root, env=env, capture_output=True, text=True, timeout=60, ) self.assertEqual(result.returncode, 0, result.stderr) @@ -288,7 +288,7 @@ def test_codex_options_remain_supported(self): env["PI_THINKING"] = "extreme" result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", "codex", "--model", "example-model", + "--task", "example-search", "--agent", "codex", "--model", "example-model", "--reasoning-effort", "high", "--codex-config", str(config), "--dry-run"], cwd=self.root, env=env, capture_output=True, text=True, ) @@ -303,14 +303,14 @@ def test_launcher_bypasses_harbor_022_gpu_preflight_for_docker(self): for gpus, expected in ((0, []), (1, ["0"])): with self.subTest(gpus=gpus): (self.task / "task.toml").write_text( - '[task]\nname = "fixture/task-new"\nversion = "0.1"\n' + '[task]\nname = "search-swe/example-search"\nversion = "0.1"\n' f'[environment]\ngpus = {gpus}\n' '[verifier.env]\nOPENAI_BASE_URL = "${OPENAI_BASE_URL:-}"\n' 'OPENAI_API_KEY = "${OPENAI_API_KEY:-}"\n' ) result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--model", "example-model", "--dry-run"], + "--task", "example-search", "--model", "example-model", "--dry-run"], cwd=self.root, env=self.launcher_env(), capture_output=True, text=True, ) self.assertEqual(result.returncode, 0, result.stderr) @@ -328,7 +328,7 @@ def test_launcher_builds_supported_pi_commands(self): env["AGENT_CODEX_CONFIG"] = "missing-codex-config.toml" result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", "pi", "--model", model, + "--task", "example-search", "--agent", "pi", "--model", model, "--thinking", "xhigh", "--dry-run"], cwd=self.root, env=env, capture_output=True, text=True, ) @@ -359,7 +359,7 @@ def test_launcher_builds_pinned_claude_code_command(self): sys.executable, str(self.repo / "scripts/run_task.py"), "--task", - "task-new", + "example-search", "--agent", "claude-code", "--model", @@ -412,7 +412,7 @@ def test_launcher_rejects_unsupported_pi_models_and_cross_agent_options(self): with self.subTest(arguments=arguments): result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", *arguments, "--dry-run"], + "--task", "example-search", *arguments, "--dry-run"], cwd=self.root, env=self.launcher_env(), capture_output=True, text=True, ) self.assertNotEqual(result.returncode, 0) @@ -420,7 +420,7 @@ def test_launcher_rejects_unsupported_pi_models_and_cross_agent_options(self): def test_pi_defaults_version_and_validates_environment_thinking(self): base = [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", "pi", + "--task", "example-search", "--agent", "pi", "--model", "deepseek/deepseek-flash", "--dry-run"] result = subprocess.run(base, cwd=self.root, env=self.launcher_env(), capture_output=True, text=True) @@ -472,7 +472,7 @@ def test_pi_requires_only_selected_key_and_keeps_secrets_out_of_argv(self): env["CONTAINER_PROXY"] = "http://shell-proxy-secret@example:7887" result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", "pi", + "--task", "example-search", "--agent", "pi", "--model", "deepseek/deepseek-flash"], cwd=self.root, env=env, capture_output=True, text=True, ) @@ -506,7 +506,7 @@ def test_pi_requires_only_selected_key_and_keeps_secrets_out_of_argv(self): missing_env[other_key] = "unselected-provider-secret" result = subprocess.run( [sys.executable, str(self.repo / "scripts/run_task.py"), - "--task", "task-new", "--agent", "pi", "--model", model], + "--task", "example-search", "--agent", "pi", "--model", model], cwd=self.root, env=missing_env, capture_output=True, text=True, ) self.assertNotEqual(result.returncode, 0) @@ -550,7 +550,7 @@ def test_claude_requires_namespaced_key_and_scrubs_host_auth(self): sys.executable, str(self.repo / "scripts/run_task.py"), "--task", - "task-new", + "example-search", "--agent", "claude-code", "--model", @@ -580,7 +580,7 @@ def test_claude_requires_namespaced_key_and_scrubs_host_auth(self): sys.executable, str(self.repo / "scripts/run_task.py"), "--task", - "task-new", + "example-search", "--agent", "claude-code", "--model", @@ -616,11 +616,11 @@ def test_launcher_requires_only_the_selected_tasks_judge_groups(self): ]: with self.subTest(keys=keys, success=success): (self.task / "task.toml").write_text( - '[task]\nname = "fixture/task-new"\nversion = "0.1"\n[verifier.env]\n' + '[task]\nname = "search-swe/example-search"\nversion = "0.1"\n[verifier.env]\n' + ''.join(f'{key} = "${{{key}:-}}"\n' for key in keys) ) (self.repo / ".env").write_text(values) - result = subprocess.run([sys.executable, str(self.repo / "scripts/run_task.py"), "--task", "task-new"], + result = subprocess.run([sys.executable, str(self.repo / "scripts/run_task.py"), "--task", "example-search"], cwd=self.root, env=env, capture_output=True, text=True) if success: self.assertEqual(result.returncode, 0, result.stderr) diff --git a/scripts/tests/test_rewardkit_deepseek.py b/scripts/tests/test_rewardkit_deepseek.py index eb70757..ece3140 100644 --- a/scripts/tests/test_rewardkit_deepseek.py +++ b/scripts/tests/test_rewardkit_deepseek.py @@ -12,14 +12,14 @@ REPO = Path(__file__).resolve().parents[2] REWARDKIT_TASKS = { - "task-1-1", - "task-1-2", - "task-1-3", - "task-1-4", - "task-2-1", - "task-2-2", - "task-2-3", - "task-2-5", + "reasoning-query-rewriting", + "memory-constrained-dense-retrieval", + "scientific-paper-qa", + "long-pdf-evidence", + "long-document-reranking", + "code-embedding-finetuning", + "query-encoder-alignment", + "sparse-retrieval-pruning", } @@ -87,16 +87,16 @@ def test_every_rewardkit_task_uses_one_fixed_version_and_model(self): ) # Every verifier that executes candidate code starts it from an empty - # environment, then adds only task-approved variables. Task 2-2 loads a + # environment, then adds only task-approved variables. Task code-embedding-finetuning loads a # constrained local checkpoint and does not execute a submission script. credential_boundaries = ( - "tasks/task-1-1/tests/grader.py", - "tasks/task-1-2/tests/grader.py", - "tasks/task-1-3/tests/test.sh", - "tasks/task-1-4/tests/test.sh", - "tasks/task-2-1/tests/grader.py", - "tasks/task-2-3/tests/run_evaluation.sh", - "tasks/task-2-5/tests/test.sh", + "tasks/reasoning-query-rewriting/tests/grader.py", + "tasks/memory-constrained-dense-retrieval/tests/grader.py", + "tasks/scientific-paper-qa/tests/test.sh", + "tasks/long-pdf-evidence/tests/test.sh", + "tasks/long-document-reranking/tests/grader.py", + "tasks/query-encoder-alignment/tests/run_evaluation.sh", + "tasks/sparse-retrieval-pruning/tests/test.sh", ) for relative in credential_boundaries: with self.subTest(credential_boundary=relative): @@ -105,12 +105,12 @@ def test_every_rewardkit_task_uses_one_fixed_version_and_model(self): self.assertIn("-i", text) self.assertNotIn("DEEPSEEK_API_KEY", text) - query_runner = (REPO / "tasks/task-1-3/tests/run_queries.py").read_text() + query_runner = (REPO / "tasks/scientific-paper-qa/tests/run_queries.py").read_text() self.assertIn("env=submission_env()", query_runner) self.assertNotIn("DEEPSEEK_API_KEY", query_runner) def test_task_without_agent_judge_has_no_rewardkit_integration(self): - tests = REPO / "tasks/task-2-4/tests" + tests = REPO / "tasks/agentic-search/tests" self.assertFalse((tests / "jailbreak_judge/codex.toml").exists()) self.assertFalse((tests / "rewardkit_deepseek.py").exists()) self.assertNotIn("rewardkit", (tests / "Dockerfile").read_text().lower()) @@ -142,7 +142,7 @@ def test_environment_template_lists_every_user_supplied_api_key(self): "AGENT_OPENROUTER_API_KEY", "VERIFIER_OPENAI_API_KEY", "ANSWER_JUDGE_API_KEY", - "TASK_1_1_OPENROUTER_API_KEY", + "REASONING_QUERY_REWRITING_OPENROUTER_API_KEY", "OPENROUTER_API_KEY", "JINA_API_KEY", } @@ -174,7 +174,7 @@ def test_environment_template_lists_every_user_supplied_api_key(self): ) def test_generated_codex_config_is_valid_and_contains_no_key(self): os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "true" - path = REPO / "tasks/task-1-1/tests/rewardkit_deepseek.py" + path = REPO / "tasks/reasoning-query-rewriting/tests/rewardkit_deepseek.py" spec = importlib.util.spec_from_file_location("rewardkit_deepseek_test", path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) @@ -226,7 +226,7 @@ def test_generated_codex_config_is_valid_and_contains_no_key(self): ) def test_backend_maps_key_and_cleans_temporary_home_on_failure(self): os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "true" - path = REPO / "tasks/task-1-1/tests/rewardkit_deepseek.py" + path = REPO / "tasks/reasoning-query-rewriting/tests/rewardkit_deepseek.py" spec = importlib.util.spec_from_file_location( "rewardkit_deepseek_lifecycle_test", path ) diff --git a/scripts/tests/test_submission_workflow.py b/scripts/tests/test_submission_workflow.py index dc060d7..51c066a 100644 --- a/scripts/tests/test_submission_workflow.py +++ b/scripts/tests/test_submission_workflow.py @@ -43,20 +43,20 @@ def setUp(self): self.git("add", ".") self.git("commit", "-qm", "base") self.base = self.git("rev-parse", "HEAD").strip() - self.source = "task-submissions/alice/1-x-1" + self.source = "task-submissions/example-search" self.task = self.repo / self.source result = self.scaffold("Alice") self.assertEqual(result.returncode, 0, result.stderr) + readme = self.task / "README.md" + readme.write_text(readme.read_text() + f"\nPackage: {self.source}\n") def git(self, *args, **kwargs): return subprocess.run(["git", *args], cwd=self.repo, check=True, capture_output=True, text=True, **kwargs).stdout - def scaffold(self, name, task_id="task-1-x-1"): - mode = "optimization" if task_id.startswith("task-2-") else "implementation" - return subprocess.run([sys.executable, str(SCAFFOLD), task_id, "--repo-root", str(self.repo), - "--submission-first-name", name, "--author", f"{name} Example", - "--mode", mode], capture_output=True, text=True) + def scaffold(self, name, task_name="example-search"): + return subprocess.run([sys.executable, str(SCAFFOLD), task_name, "--repo-root", str(self.repo), + "--author", f"{name} Example"], capture_output=True, text=True) def cli(self, script, *args): env = {k: v for k, v in os.environ.items() if not k.startswith(("AGENT_", "CONTAINER_", "VERIFIER_"))} @@ -66,12 +66,12 @@ def cli(self, script, *args): def asset(self, task=None): task = task or self.task if task != self.task: - (task / "task.toml").write_text((task / "task.toml").read_text().replace("task-1-x-1", task.name)) + (task / "task.toml").write_text((task / "task.toml").read_text().replace("example-search", task.name)) content = b"fixture corpus\n" entry = {"path": "data/corpus.txt", "size_bytes": len(content), "sha256": hashlib.sha256(content).hexdigest(), "source": {"repo_id": "search-swe/Search-SWE", "repo_type": "dataset", "revision": "a" * 40, - "filename": f"tasks/{'task-1-x-1' if task == self.task else task.name}/corpus.txt"}} + "filename": f"tasks/{'example-search' if task == self.task else task.name}/corpus.txt"}} (task / "assets.json").write_text(json.dumps({"schema_version": 1, "files": [entry]})) data = self.root / "new-data" path = data / entry["source"]["filename"] @@ -84,19 +84,18 @@ def commit_submission(self): self.git("add", ".") self.git("commit", "-qm", "Alice writes task", "--author", "Alice Example ") - def test_namespace_reuse_explicit_name_suffix_and_path_boundaries(self): - self.assertEqual(self.scaffold("Alice", "task-1-x-2").returncode, 0) - self.assertEqual(self.scaffold("Alice", "task-2-x-1").returncode, 0) - self.assertTrue((self.repo / "task-submissions/alice/1-x-2/task.toml").is_file()) - self.assertTrue((self.repo / "task-submissions/alice/2-x-1/task.toml").is_file()) - self.assertEqual(self.scaffold("Alice-2").returncode, 0) - self.assertTrue((self.repo / "task-submissions/alice-2/1-x-1/task.toml").is_file()) + def test_flat_names_collisions_and_path_boundaries(self): + for name in ("hybrid-search", "vector-search"): + self.assertEqual(self.scaffold("Alice", name).returncode, 0) + self.assertTrue((self.repo / "task-submissions" / name / "task.toml").is_file()) + self.assertNotEqual(self.scaffold("Bob").returncode, 0) + self.assertEqual(self.scaffold("Bob", "other-search").returncode, 0) current = self.cli("check_submission.py") - self.assertNotEqual(current.returncode, 0) - self.assertIn("exactly one contributor namespace", current.stdout) - for name in ("../alice", "/absolute", "Élodie", "123"): - self.assertNotEqual(self.scaffold(name).returncode, 0) - for name in ("../repo/" + self.source, "/" + self.source, self.source + "/../1-x-1"): + self.assertEqual(current.returncode, 0, current.stdout + current.stderr) + for name in ("../escape", "/absolute", "Élodie", "123", "all", "task-1-1", "one-two-three-four-five-six"): + self.assertNotEqual(self.scaffold("Alice", name).returncode, 0) + for name in ("../repo/" + self.source, "/" + self.source, self.source + "/../example-search", + "task-submissions/alice/example-search"): with self.assertRaises(ValueError): select_task(self.repo, name) (self.task / "escape").symlink_to(self.root) @@ -133,10 +132,14 @@ def test_static_validator_does_not_execute_task_code(self): self.assertEqual(check_submission(self.repo, self.task), []) self.assertFalse(sentinel.exists()) - def test_submission_rejects_category_mode_mismatch(self): + def test_submission_rejects_obsolete_type_and_formal_name_collision(self): config = self.task / "task.toml" - config.write_text(config.read_text().replace('task_type = "create"', 'task_type = "optimize"')) - self.assertTrue(any("Category 1 requires" in error for error in check_submission(self.repo, self.task))) + original = config.read_text() + config.write_text(original.replace("[metadata]", '[metadata]\ntask_type = "create"')) + self.assertTrue(any("remove metadata.task_type" in error for error in check_submission(self.repo, self.task))) + config.write_text(original) + shutil.copytree(self.task, self.repo / "tasks/example-search") + self.assertTrue(any("collides" in error for error in check_submission(self.repo, self.task))) def test_validator_scans_payload_secret_forced_asset_author_id_and_ignore_rules(self): entry, _ = self.asset() @@ -154,7 +157,7 @@ def test_validator_scans_payload_secret_forced_asset_author_id_and_ignore_rules( (self.task / "task.toml").write_text((self.task / "task.toml").read_text().replace("Alice Example", "Search-SWE")) (self.task / "README.md").write_text("task-1-999\n") errors = check_submission(self.repo, self.task) - for expected in ("credential", "authors", "unexpected task IDs"): + for expected in ("credential", "authors", "obsolete numbered task reference"): self.assertTrue(any(expected in e for e in errors), errors) def test_discovery_incomplete_and_review_versus_merge_ready(self): @@ -163,7 +166,7 @@ def test_discovery_incomplete_and_review_versus_merge_ready(self): self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertIn("NOT mean merge-ready", result.stdout) self.assertNotEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) - incomplete = self.repo / "task-submissions/bob/2-x-1" + incomplete = self.repo / "task-submissions/incomplete-search" incomplete.mkdir(parents=True) self.assertIn(incomplete, discover(self.repo)) self.assertNotEqual(self.cli("check_submission.py").returncode, 0) @@ -175,15 +178,15 @@ def test_formal_discovery_excludes_submissions_and_explicit_launch_is_namespaced self.assertIn("Selected 0 files", result.stdout) result = self.cli("download_assets.py", "--task-path", self.source, "--dry-run") self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn("task-submissions/alice/1-x-1/data/corpus.txt", result.stdout) - for name in ("alice", "alice-2"): - if name != "alice": - self.scaffold("Alice-2") - result = self.cli("run_task.py", "--task-path", f"task-submissions/{name}/1-x-1", + self.assertIn("task-submissions/example-search/data/corpus.txt", result.stdout) + for name in ("example-search", "other-search"): + if name != "example-search": + self.scaffold("Bob", name) + result = self.cli("run_task.py", "--task-path", f"task-submissions/{name}", "--agent", "pi", "--model", "deepseek/deepseek-flash", "--dry-run") self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn(f"jobs/task-submissions/{name}/1-x-1", result.stdout) - self.assertNotEqual(self.cli("run_task.py", "--task", "task-1-x-1", "--dry-run").returncode, 0) + self.assertIn(f"jobs/task-submissions/{name}", result.stdout) + self.assertNotEqual(self.cli("run_task.py", "--task", "example-search", "--dry-run").returncode, 0) def test_submission_launch_reaches_mock_harbor_without_formal_registration(self): binary = self.root / "bin" @@ -232,61 +235,61 @@ def test_personal_pinned_dataset_download_is_mocked_and_verified(self): def test_two_phase_promotion_preserves_follow_blame_and_merge_history(self): self.commit_submission() - promote(self.repo, self.source, "task-1-6", "rename", dry_run=True) + promote(self.repo, self.source, "example-search", "rename", dry_run=True) self.assertTrue(self.task.exists()) self.assertFalse(self.git("status", "--porcelain")) - promote(self.repo, self.source, "task-1-6", "rename") + promote(self.repo, self.source, "example-search", "rename") with self.assertRaises(ValueError): - promote(self.repo, self.source, "task-1-6", "finalize") + promote(self.repo, self.source, "example-search", "finalize") self.git("commit", "-qm", "Pure rename by maintainer") - target = self.repo / "tasks/task-1-6" - promote(self.repo, self.source, "task-1-6", "finalize", dry_run=True) - self.assertIn("task-1-x-1", (target / "task.toml").read_text()) - promote(self.repo, self.source, "task-1-6", "finalize") + target = self.repo / "tasks/example-search" + promote(self.repo, self.source, "example-search", "finalize", dry_run=True) + self.assertIn("example-search", (target / "task.toml").read_text()) + promote(self.repo, self.source, "example-search", "finalize") self.git("add", ".") - self.git("commit", "-qm", "Finalize by maintainer") + self.git("commit", "-qm", "Finalize by maintainer", "--allow-empty") self.git("checkout", "-q", "main") self.git("merge", "--no-ff", "-qm", "Merge task PR", "contribution") self.assertEqual(len(self.git("rev-list", "--parents", "-n", "1", "HEAD").split()), 3) - history = self.git("log", "--follow", "--format=%an %s", "--", "tasks/task-1-6/instruction.md") + history = self.git("log", "--follow", "--format=%an %s", "--", "tasks/example-search/instruction.md") self.assertIn("Alice Example Alice writes task", history) - blame = self.git("blame", "--line-porcelain", "tasks/task-1-6/instruction.md") + blame = self.git("blame", "--line-porcelain", "tasks/example-search/instruction.md") self.assertIn("author Alice Example", blame) - self.assertIn("author Maintainer", blame) + self.assertNotIn("author Maintainer", blame) self.assertFalse(list((self.repo / "task-submissions").rglob("task.toml"))) self.assertEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) self.assertEqual(self.cli("check_release.py").returncode, 0) - def test_promotion_refuses_collision_category_traversal_and_nonpure_commit(self): + def test_promotion_refuses_collision_rename_traversal_and_nonpure_commit(self): self.commit_submission() - for source, target in ((self.source, "task-2-6"), (self.source, "../task-1-6"), - ("../" + self.source, "task-1-6")): + for source, target in ((self.source, "vector-search"), (self.source, "../example-search"), + ("../" + self.source, "example-search")): with self.assertRaises(ValueError): promote(self.repo, source, target, "rename") - collision = self.repo / "tasks/task-1-6" + collision = self.repo / "tasks/example-search" collision.mkdir() with self.assertRaises(ValueError): - promote(self.repo, self.source, "task-1-6", "rename") + promote(self.repo, self.source, "example-search", "rename") collision.rmdir() - promote(self.repo, self.source, "task-1-6", "rename") + promote(self.repo, self.source, "example-search", "rename") (self.repo / "docker/README.md").write_text("unrelated edit") self.git("add", ".") self.git("commit", "-qm", "Not a pure rename") with self.assertRaises(ValueError): - promote(self.repo, self.source, "task-1-6", "finalize") + promote(self.repo, self.source, "example-search", "finalize") def test_merge_ready_rechecks_promoted_syntax_paths_and_new_authors(self): self.commit_submission() - promote(self.repo, self.source, "task-1-6", "rename") + promote(self.repo, self.source, "example-search", "rename") self.git("commit", "-qm", "Pure rename") - promote(self.repo, self.source, "task-1-6", "finalize") - target = self.repo / "tasks/task-1-6" + promote(self.repo, self.source, "example-search", "finalize") + target = self.repo / "tasks/example-search" self.git("add", ".") - self.git("commit", "-qm", "Finalize") + self.git("commit", "-qm", "Finalize", "--allow-empty") self.assertEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) readme = target / "README.md" original = readme.read_text() - readme.write_text(original + "\ntask-submissions/alice/1-x-1\n") + readme.write_text(original + "\ntask-submissions/example-search\n") self.assertNotEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) readme.write_text(original) broken = target / "tests/broken.py" @@ -297,114 +300,100 @@ def test_merge_ready_rechecks_promoted_syntax_paths_and_new_authors(self): config.write_text(config.read_text().replace("Alice Example", "Search-SWE")) self.assertNotEqual(self.cli("check_submission.py", "--base", self.base).returncode, 0) - def test_base_allows_multiple_tasks_in_one_namespace_and_rejects_two(self): - self.scaffold("Alice", "task-1-x-2") - self.scaffold("Alice", "task-2-x-1") + def test_base_allows_multiple_tasks_and_authors(self): + self.scaffold("Alice", "hybrid-search") + self.scaffold("Bob", "vector-search") self.git("add", ".") - self.git("commit", "-qm", "three Alice submissions") + self.git("commit", "-qm", "Multiple submissions") result = self.cli("check_submission.py", "--base", self.base) self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - self.scaffold("Bob") - self.git("add", ".") - self.git("commit", "-qm", "Bob submission") - result = self.cli("check_submission.py", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("exactly one contributor namespace", result.stdout) def test_base_rejects_direct_formal_task_without_submission_history(self): shutil.rmtree(self.repo / "task-submissions") result = subprocess.run( - [sys.executable, str(SCAFFOLD), "task-1-6", "--repo-root", str(self.repo), - "--author", "Alice Example"], capture_output=True, text=True, + [sys.executable, str(SCAFFOLD), "example-search", "--repo-root", str(self.repo), + "--author", "Alice Example", "--formal"], capture_output=True, text=True, ) self.assertEqual(result.returncode, 0, result.stderr) self.git("add", ".") self.git("commit", "-qm", "Bypass submission buffer") result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) self.assertNotEqual(result.returncode, 0) - self.assertIn("require exactly one contributor submission namespace", result.stdout) + self.assertIn("require same-PR pure promotion", result.stdout) - def test_base_rejects_temporary_ordinal_reuse_after_promotion(self): + def test_base_rejects_submission_path_reuse(self): self.git("add", ".") - self.git("commit", "-qm", "First use of ordinal") - for target_id in ("task-1-6", "task-1-7"): - promote(self.repo, self.source, target_id, "rename") - self.git("commit", "-qm", f"Pure rename {target_id}") - promote(self.repo, self.source, target_id, "finalize") - self.git("add", ".") - self.git("commit", "-qm", f"Finalize {target_id}") - if target_id == "task-1-6": - result = self.scaffold("Alice") - self.assertEqual(result.returncode, 0, result.stderr) - self.git("add", ".") - self.git("commit", "-qm", "Reuse ordinal for another task") - result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) + self.git("commit", "-qm", "First submission") + self.git("rm", "-r", self.source) + self.git("commit", "-qm", "Remove submission") + self.assertEqual(self.scaffold("Bob").returncode, 0) + self.git("add", ".") + self.git("commit", "-qm", "Reuse submission path") + result = self.cli("check_submission.py", "--base", self.base) self.assertNotEqual(result.returncode, 0) - self.assertIn("Temporary task ordinals must not be reused", result.stdout) + self.assertIn("Submission paths must not be reused", result.stdout) - def test_multiple_tasks_promote_independently_and_history_keeps_namespace(self): - self.scaffold("Alice", "task-2-x-1") + def test_multiple_tasks_promote_independently_with_multiple_authors(self): + self.scaffold("Alice", "vector-search") self.git("checkout", "-qb", "multi-promotion") self.git("add", ".") self.git("commit", "-qm", "Alice submits two tasks", "--author", "Alice Example ") - for source, target_id in ((self.source, "task-1-6"), - ("task-submissions/alice/2-x-1", "task-2-6")): + for source, target_id in ((self.source, "example-search"), + ("task-submissions/vector-search", "vector-search")): promote(self.repo, source, target_id, "rename") self.git("commit", "-qm", f"Pure rename {target_id}") promote(self.repo, source, target_id, "finalize") self.git("add", ".") - self.git("commit", "-qm", f"Finalize {target_id}") + self.git("commit", "-qm", f"Finalize {target_id}", "--allow-empty") self.assertEqual(self.cli("check_submission.py", "--merge-ready", "--base", self.base).returncode, 0) self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", - "tasks/task-1-6/instruction.md")) + "tasks/example-search/instruction.md")) self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", - "tasks/task-2-6/instruction.md")) + "tasks/vector-search/instruction.md")) - self.scaffold("Bob") + self.scaffold("Bob", "other-search") self.git("add", ".") self.git("commit", "-qm", "Bob submission", "--author", "Bob Example ") - bob_source = "task-submissions/bob/1-x-1" - promote(self.repo, bob_source, "task-1-7", "rename") - self.git("commit", "-qm", "Pure rename task-1-7") - promote(self.repo, bob_source, "task-1-7", "finalize") + bob_source = "task-submissions/other-search" + promote(self.repo, bob_source, "other-search", "rename") + self.git("commit", "-qm", "Pure rename other-search") + promote(self.repo, bob_source, "other-search", "finalize") self.git("add", ".") - self.git("commit", "-qm", "Finalize task-1-7") + self.git("commit", "-qm", "Finalize other-search", "--allow-empty") result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("exactly one contributor namespace", result.stdout) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) def test_base_history_checks_every_side_of_fully_promoted_merge(self): self.git("checkout", "-qb", "alice-history") self.git("add", ".") self.git("commit", "-qm", "Alice submission") - promote(self.repo, self.source, "task-1-6", "rename") + promote(self.repo, self.source, "example-search", "rename") self.git("commit", "-qm", "Pure rename Alice") - promote(self.repo, self.source, "task-1-6", "finalize") + promote(self.repo, self.source, "example-search", "finalize") self.git("add", ".") - self.git("commit", "-qm", "Finalize Alice") + self.git("commit", "-qm", "Finalize Alice", "--allow-empty") self.git("checkout", "-qb", "bob-history", self.base) (self.repo / "tasks").mkdir(exist_ok=True) - result = self.scaffold("Bob") + result = self.scaffold("Bob", "other-search") self.assertEqual(result.returncode, 0, result.stderr) self.git("add", ".") self.git("commit", "-qm", "Bob submission") - bob_source = "task-submissions/bob/1-x-1" - promote(self.repo, bob_source, "task-1-7", "rename") + bob_source = "task-submissions/other-search" + promote(self.repo, bob_source, "other-search", "rename") self.git("commit", "-qm", "Pure rename Bob") - promote(self.repo, bob_source, "task-1-7", "finalize") + promote(self.repo, bob_source, "other-search", "finalize") self.git("add", ".") - self.git("commit", "-qm", "Finalize Bob") + self.git("commit", "-qm", "Finalize Bob", "--allow-empty") self.git("checkout", "alice-history") self.git("merge", "--no-ff", "bob-history", "-m", "Merge Bob history") result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("exactly one contributor namespace", result.stdout) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) def test_base_history_rejects_unsafe_submission_paths(self): self.git("checkout", "-qb", "unsafe-history") - unsafe = self.repo / "task-submissions/not-a-package/file.txt" + unsafe = self.repo / "task-submissions/alice/nested-search/task.toml" unsafe.parent.mkdir(parents=True) unsafe.write_text("unsafe historical path") self.git("add", ".") @@ -413,11 +402,67 @@ def test_base_history_rejects_unsafe_submission_paths(self): self.assertNotEqual(result.returncode, 0) self.assertIn("Malformed task submission path", result.stdout) + def test_base_recognizes_existing_formal_rename_without_changing_legacy_authors(self): + # A repository-wide name migration is an edit to an existing package. + legacy = self.repo / "tasks/task-1-1" + self.task.rename(legacy) + readme = legacy / "README.md" + readme.write_text("Existing formal task\n") + config = legacy / "task.toml" + config.write_text(config.read_text().replace("example-search", "task-1-1") + .replace("Alice Example", "Search-SWE")) + self.git("add", ".") + self.git("commit", "-qm", "Existing legacy formal task") + base = self.git("rev-parse", "HEAD").strip() + self.git("mv", "tasks/task-1-1", "tasks/example-search") + config = self.repo / "tasks/example-search/task.toml" + config.write_text(config.read_text().replace("task-1-1", "example-search")) + self.git("add", ".") + self.git("commit", "-qm", "Rename existing formal task") + result = self.cli("check_submission.py", "--merge-ready", "--base", base) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_history_rejects_deleted_nested_submission(self): + unsafe = self.repo / "task-submissions/alice/nested-search/task.toml" + unsafe.parent.mkdir(parents=True) + unsafe.write_text("invalid old nested layout") + self.git("add", ".") + self.git("commit", "-qm", "Nested submission") + shutil.rmtree(unsafe.parent.parent) + self.git("add", "-u") + self.git("commit", "-qm", "Remove nested submission") + result = self.cli("check_submission.py", "--base", self.base) + self.assertNotEqual(result.returncode, 0) + self.assertIn("Malformed task submission path", result.stdout) + + def test_history_rejects_renamed_or_nonpure_submission_promotion(self): + self.commit_submission() + self.git("mv", self.source, "tasks/other-search") + target = self.repo / "tasks/other-search" + for name in ("README.md", "task.toml", "instruction.md"): + path = target / name + path.write_text(path.read_text().replace(self.source, "tasks/other-search") + .replace("example-search", "other-search")) + self.git("add", ".") + self.git("commit", "-qm", "Rename and modify submission") + result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) + self.assertNotEqual(result.returncode, 0) + self.assertIn("require same-PR pure promotion", result.stdout) + + def test_official_staging_requires_promotion_even_when_name_is_final(self): + _, data = self.asset() + snapshot = self.root / "official.json" + snapshot.write_text(json.dumps({"schema_version": 1, "files": []})) + output = self.root / "premature-upload" + with self.assertRaisesRegex(ValueError, "Promote to tasks/"): + prepare(snapshot, self.task, data, output) + self.assertFalse(output.exists()) + def test_incremental_manifest_preserves_old_entries_without_old_data(self): - target = self.repo / "tasks/task-1-6" + target = self.repo / "tasks/example-search" shutil.copytree(self.task, target) entry, data = self.asset(target) - old = {"path": "tasks/task-1-1/old.bin", "size_bytes": 500, "sha256": "b" * 64, "license": "fixture"} + old = {"path": "tasks/reasoning-query-rewriting/old.bin", "size_bytes": 500, "sha256": "b" * 64, "license": "fixture"} snapshot = self.root / "official.json" snapshot.write_text(json.dumps({"schema_version": 1, "files": [old], "extra": "preserve"})) output = self.root / "upload" @@ -440,7 +485,7 @@ def test_incremental_manifest_preserves_old_entries_without_old_data(self): self.assertFalse((self.root / "corrupt").exists()) def test_incremental_rejects_traversal_and_symlinks(self): - target = self.repo / "tasks/task-1-6" + target = self.repo / "tasks/example-search" shutil.copytree(self.task, target) entry, data = self.asset(target) snapshot = self.root / "official.json" diff --git a/scripts/tests/test_task_images.py b/scripts/tests/test_task_images.py index 7891f80..e310a5c 100644 --- a/scripts/tests/test_task_images.py +++ b/scripts/tests/test_task_images.py @@ -6,12 +6,12 @@ REPO = Path(__file__).resolve().parents[2] CPU_IMAGE = "docker.io/hanhainebula/search-swe-base:cpu-py3.12-1.0.0" GPU_IMAGE = "docker.io/hanhainebula/search-swe-base:gpu-cu13.0-py3.12-1.0.0" -GPU_TASKS = {"task-2-2", "task-2-3"} +GPU_TASKS = {"code-embedding-finetuning", "query-encoder-alignment"} class TaskImages(unittest.TestCase): def test_base_images_and_gpu_resources_match(self): - tasks = sorted(path for path in (REPO / "tasks").glob("task-*") if path.is_dir()) + tasks = sorted(path for path in (REPO / "tasks").glob("*") if path.is_dir()) self.assertTrue(tasks) configured_gpu_tasks = set() diff --git a/tasks/task-1-1/.gitignore b/tasks/agentic-search/.gitignore similarity index 100% rename from tasks/task-1-1/.gitignore rename to tasks/agentic-search/.gitignore diff --git a/tasks/task-2-4/README.md b/tasks/agentic-search/README.md similarity index 89% rename from tasks/task-2-4/README.md rename to tasks/agentic-search/README.md index 4b3d695..caff237 100644 --- a/tasks/task-2-4/README.md +++ b/tasks/agentic-search/README.md @@ -1,8 +1,8 @@ -# Agentic Search Optimization +# Agentic Search Improve a ReAct search system's document coverage on BrowseComp-Plus questions. -**Task:** `task-2-4` · **Mode:** Optimization · **Metric:** Gold Recall@5 +**Task:** `agentic-search` · **Metric:** Gold Recall@5 ## Overview @@ -14,7 +14,7 @@ This task supplies an editable ReAct starter built around SQLite FTS5/BM25. The agent improves or replaces its retrieval and reasoning components while working within a bounded number of searches. Evaluation measures the documents ultimately retrieved, not the fluency of an answer or the appearance of a trace. -The current repository identifier is task 2-4. +The current repository identifier is `agentic-search`. ## What This Task Tests @@ -65,10 +65,8 @@ A shared 15,300-second verification budget includes a 600-second build cap and 120 seconds reserved for scoring. Five query workers share the remaining budget; there is **no individual query timeout**. -The previously documented offline lexical baseline completed a full-corpus +A recorded local run of the offline lexical baseline completed a full-corpus build in about 38 seconds and achieved public Gold Recall@5 of `0.1741666667`. -That is a historical local measurement, not a new run or a result for the -model-driven planner. ## Submission Contract @@ -122,26 +120,26 @@ For an existing public prediction file, the deterministic scorer can also be run from the repository root: ```bash -python tasks/task-2-4/tests/evaluate.py \ - --corpus tasks/task-2-4/data/corpus.jsonl \ - --queries tasks/task-2-4/data/validation/queries.jsonl \ - --qrels tasks/task-2-4/data/validation/qrels_gold.txt \ +python tasks/agentic-search/tests/evaluate.py \ + --corpus tasks/agentic-search/data/corpus.jsonl \ + --queries tasks/agentic-search/data/validation/queries.jsonl \ + --qrels tasks/agentic-search/data/validation/qrels_gold.txt \ --predictions /path/to/results.jsonl \ - --report /tmp/task-2-4-validation.json + --report /tmp/agentic-search-validation.json ``` This scores predictions; it does not reproduce the full Harbor execution. Local fixture tests are available via -`python -B -m unittest discover -s tasks/task-2-4/tests -p 'test_*.py' -v`. +`python -B -m unittest discover -s tasks/agentic-search/tests -p 'test_*.py' -v`. No oracle solution is included. ```bash -python scripts/download_assets.py --task task-2-4 -bash scripts/run_task.sh --task task-2-4 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task agentic-search +bash scripts/run_task.sh --task agentic-search --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-2-4/`. +writes results under `jobs/agentic-search/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-2-4/assets.json b/tasks/agentic-search/assets.json similarity index 68% rename from tasks/task-2-4/assets.json rename to tasks/agentic-search/assets.json index b150774..88f05e4 100644 --- a/tasks/task-2-4/assets.json +++ b/tasks/agentic-search/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-2-4/corpus.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/agentic-search/corpus.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-2-4/validation/qrels_gold.txt" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/agentic-search/validation/qrels_gold.txt" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-2-4/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/agentic-search/validation/queries.jsonl" } } ] diff --git a/tasks/task-2-4/environment/Dockerfile b/tasks/agentic-search/environment/Dockerfile similarity index 100% rename from tasks/task-2-4/environment/Dockerfile rename to tasks/agentic-search/environment/Dockerfile diff --git a/tasks/task-2-4/environment/build.sh b/tasks/agentic-search/environment/build.sh similarity index 100% rename from tasks/task-2-4/environment/build.sh rename to tasks/agentic-search/environment/build.sh diff --git a/tasks/task-1-2/environment/docker-compose.yaml b/tasks/agentic-search/environment/docker-compose.yaml similarity index 100% rename from tasks/task-1-2/environment/docker-compose.yaml rename to tasks/agentic-search/environment/docker-compose.yaml diff --git a/tasks/task-2-3/environment/docs/available_resources.md b/tasks/agentic-search/environment/docs/available_resources.md similarity index 100% rename from tasks/task-2-3/environment/docs/available_resources.md rename to tasks/agentic-search/environment/docs/available_resources.md diff --git a/tasks/task-1-1/environment/docs/environment.md b/tasks/agentic-search/environment/docs/environment.md similarity index 100% rename from tasks/task-1-1/environment/docs/environment.md rename to tasks/agentic-search/environment/docs/environment.md diff --git a/tasks/task-2-4/environment/run.sh b/tasks/agentic-search/environment/run.sh similarity index 100% rename from tasks/task-2-4/environment/run.sh rename to tasks/agentic-search/environment/run.sh diff --git a/tasks/task-2-4/environment/starter/README.md b/tasks/agentic-search/environment/starter/README.md similarity index 98% rename from tasks/task-2-4/environment/starter/README.md rename to tasks/agentic-search/environment/starter/README.md index e16167d..3cc080a 100644 --- a/tasks/task-2-4/environment/starter/README.md +++ b/tasks/agentic-search/environment/starter/README.md @@ -1,4 +1,4 @@ -# Task-2-4 ReAct starter +# Task agentic-search ReAct starter Improve this baseline to maximize Gold Recall@5. It uses a SQLite FTS5/BM25 search service and a small ReAct loop: choose a search, observe candidate diff --git a/tasks/task-2-4/environment/starter/build.sh b/tasks/agentic-search/environment/starter/build.sh similarity index 100% rename from tasks/task-2-4/environment/starter/build.sh rename to tasks/agentic-search/environment/starter/build.sh diff --git a/tasks/task-2-4/environment/starter/run.sh b/tasks/agentic-search/environment/starter/run.sh similarity index 100% rename from tasks/task-2-4/environment/starter/run.sh rename to tasks/agentic-search/environment/starter/run.sh diff --git a/tasks/task-2-4/environment/starter/src/build_index.py b/tasks/agentic-search/environment/starter/src/build_index.py similarity index 100% rename from tasks/task-2-4/environment/starter/src/build_index.py rename to tasks/agentic-search/environment/starter/src/build_index.py diff --git a/tasks/task-2-4/environment/starter/src/run_search.py b/tasks/agentic-search/environment/starter/src/run_search.py similarity index 100% rename from tasks/task-2-4/environment/starter/src/run_search.py rename to tasks/agentic-search/environment/starter/src/run_search.py diff --git a/tasks/task-2-4/environment/starter/src/search_service.py b/tasks/agentic-search/environment/starter/src/search_service.py similarity index 100% rename from tasks/task-2-4/environment/starter/src/search_service.py rename to tasks/agentic-search/environment/starter/src/search_service.py diff --git a/tasks/task-2-4/instruction.md b/tasks/agentic-search/instruction.md similarity index 100% rename from tasks/task-2-4/instruction.md rename to tasks/agentic-search/instruction.md diff --git a/tasks/task-2-4/raw-instruction.md b/tasks/agentic-search/raw-instruction.md similarity index 99% rename from tasks/task-2-4/raw-instruction.md rename to tasks/agentic-search/raw-instruction.md index ecbeae6..5be5faf 100644 --- a/tasks/task-2-4/raw-instruction.md +++ b/tasks/agentic-search/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: 2-4 +# Agentic Search ## Task Description diff --git a/tasks/task-2-4/task.toml b/tasks/agentic-search/task.toml similarity index 96% rename from tasks/task-2-4/task.toml rename to tasks/agentic-search/task.toml index d64080a..34b7ff1 100644 --- a/tasks/task-2-4/task.toml +++ b/tasks/agentic-search/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-2-4" +name = "search-swe/agentic-search" version = "0.5.0" description = "Optimize a ReAct retrieval starter to return five documents per query, scored by Gold Recall@5." authors = [{ name = "Search-SWE" }] keywords = ["BrowseComp-Plus", "information-retrieval", "multi-constraint-retrieval"] [metadata] -task_type = "optimize" dataset = "BrowseComp-Plus" primary_metric = "GoldRecall@5" metric_k = 5 diff --git a/tasks/task-2-4/tests/Dockerfile b/tasks/agentic-search/tests/Dockerfile similarity index 100% rename from tasks/task-2-4/tests/Dockerfile rename to tasks/agentic-search/tests/Dockerfile diff --git a/tasks/task-2-4/tests/data/qrels_gold.txt b/tasks/agentic-search/tests/data/qrels_gold.txt similarity index 100% rename from tasks/task-2-4/tests/data/qrels_gold.txt rename to tasks/agentic-search/tests/data/qrels_gold.txt diff --git a/tasks/task-2-4/tests/data/queries.jsonl b/tasks/agentic-search/tests/data/queries.jsonl similarity index 100% rename from tasks/task-2-4/tests/data/queries.jsonl rename to tasks/agentic-search/tests/data/queries.jsonl diff --git a/tasks/task-2-4/tests/docker-compose.yaml b/tasks/agentic-search/tests/docker-compose.yaml similarity index 100% rename from tasks/task-2-4/tests/docker-compose.yaml rename to tasks/agentic-search/tests/docker-compose.yaml diff --git a/tasks/task-2-4/tests/evaluate.py b/tasks/agentic-search/tests/evaluate.py similarity index 98% rename from tasks/task-2-4/tests/evaluate.py rename to tasks/agentic-search/tests/evaluate.py index b194040..1bae8a4 100644 --- a/tasks/task-2-4/tests/evaluate.py +++ b/tasks/agentic-search/tests/evaluate.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Validate Task-2-4 document rankings and compute macro Gold Recall@5.""" +"""Validate Task agentic-search document rankings and compute macro Gold Recall@5.""" from __future__ import annotations diff --git a/tasks/task-2-4/tests/run_submission.py b/tasks/agentic-search/tests/run_submission.py similarity index 98% rename from tasks/task-2-4/tests/run_submission.py rename to tasks/agentic-search/tests/run_submission.py index 8527683..df98e75 100644 --- a/tasks/task-2-4/tests/run_submission.py +++ b/tasks/agentic-search/tests/run_submission.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Build, repeatedly run, and score a Task-2-4 submission.""" +"""Build, repeatedly run, and score a Task agentic-search submission.""" from __future__ import annotations @@ -41,7 +41,7 @@ def submission_env() -> dict[str, str]: continue if key in PRIVATE_ENV_NAMES or upper.startswith("SILICONFLOW_") or upper.endswith("_API_KEY") or any(token in upper for token in ("SECRET", "PASSWORD")): env.pop(key, None) - env["HOME"] = "/tmp/task2-4-submission-home" + env["HOME"] = "/tmp/agentic-search-submission-home" env["TMPDIR"] = "/tmp" return env @@ -67,7 +67,7 @@ def prepare_untrusted_tree(root: Path) -> None: def prepare_untrusted_runtime(results: Path) -> None: - home = Path("/tmp/task2-4-submission-home") + home = Path("/tmp/agentic-search-submission-home") home.mkdir(parents=True, exist_ok=True) os.chown(home, UNTRUSTED_UID, UNTRUSTED_GID) home.chmod(0o700) diff --git a/tasks/task-2-4/tests/test.sh b/tasks/agentic-search/tests/test.sh similarity index 100% rename from tasks/task-2-4/tests/test.sh rename to tasks/agentic-search/tests/test.sh diff --git a/tasks/task-2-4/tests/test_evaluate.py b/tasks/agentic-search/tests/test_evaluate.py similarity index 98% rename from tasks/task-2-4/tests/test_evaluate.py rename to tasks/agentic-search/tests/test_evaluate.py index 048bfe6..104cc2d 100755 --- a/tasks/task-2-4/tests/test_evaluate.py +++ b/tasks/agentic-search/tests/test_evaluate.py @@ -10,7 +10,7 @@ from unittest.mock import patch -SPEC = importlib.util.spec_from_file_location("task2_4_evaluate", Path(__file__).with_name("evaluate.py")) +SPEC = importlib.util.spec_from_file_location("agentic_search_evaluate", Path(__file__).with_name("evaluate.py")) assert SPEC and SPEC.loader EVALUATE = importlib.util.module_from_spec(SPEC) SPEC.loader.exec_module(EVALUATE) diff --git a/tasks/task-2-4/tests/test_parallel_submission.py b/tasks/agentic-search/tests/test_parallel_submission.py similarity index 97% rename from tasks/task-2-4/tests/test_parallel_submission.py rename to tasks/agentic-search/tests/test_parallel_submission.py index fc061ec..e7d8134 100644 --- a/tasks/task-2-4/tests/test_parallel_submission.py +++ b/tasks/agentic-search/tests/test_parallel_submission.py @@ -10,7 +10,7 @@ from unittest.mock import patch -SPEC = importlib.util.spec_from_file_location("task2_4_parallel_runner", Path(__file__).with_name("run_submission.py")) +SPEC = importlib.util.spec_from_file_location("agentic_search_parallel_runner", Path(__file__).with_name("run_submission.py")) assert SPEC and SPEC.loader RUNNER = importlib.util.module_from_spec(SPEC) SPEC.loader.exec_module(RUNNER) diff --git a/tasks/task-2-4/tests/test_run_submission.py b/tasks/agentic-search/tests/test_run_submission.py similarity index 94% rename from tasks/task-2-4/tests/test_run_submission.py rename to tasks/agentic-search/tests/test_run_submission.py index 4852aa5..e5ac42c 100755 --- a/tasks/task-2-4/tests/test_run_submission.py +++ b/tasks/agentic-search/tests/test_run_submission.py @@ -8,7 +8,7 @@ MODULE_PATH = Path(__file__).with_name("run_submission.py") -SPEC = importlib.util.spec_from_file_location("task2_4_runner", MODULE_PATH) +SPEC = importlib.util.spec_from_file_location("agentic_search_runner", MODULE_PATH) assert SPEC and SPEC.loader RUNNER = importlib.util.module_from_spec(SPEC) SPEC.loader.exec_module(RUNNER) diff --git a/tasks/task-2-4/tests/test_starter.py b/tasks/agentic-search/tests/test_starter.py similarity index 100% rename from tasks/task-2-4/tests/test_starter.py rename to tasks/agentic-search/tests/test_starter.py diff --git a/tasks/task-2-4/tests/test_verify.py b/tasks/agentic-search/tests/test_verify.py similarity index 92% rename from tasks/task-2-4/tests/test_verify.py rename to tasks/agentic-search/tests/test_verify.py index 7edb9b5..c4d1924 100644 --- a/tasks/task-2-4/tests/test_verify.py +++ b/tasks/agentic-search/tests/test_verify.py @@ -6,7 +6,7 @@ from unittest.mock import patch -SPEC = importlib.util.spec_from_file_location("task2_4_verify", Path(__file__).with_name("verify.py")) +SPEC = importlib.util.spec_from_file_location("agentic_search_verify", Path(__file__).with_name("verify.py")) assert SPEC and SPEC.loader VERIFY = importlib.util.module_from_spec(SPEC) SPEC.loader.exec_module(VERIFY) diff --git a/tasks/task-2-4/tests/verify.py b/tasks/agentic-search/tests/verify.py similarity index 94% rename from tasks/task-2-4/tests/verify.py rename to tasks/agentic-search/tests/verify.py index fb1665b..f3d1ca6 100755 --- a/tasks/task-2-4/tests/verify.py +++ b/tasks/agentic-search/tests/verify.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Run Task-2-4 and write its Gold Recall@5 as the Harbor reward.""" +"""Run Task agentic-search and write its Gold Recall@5 as the Harbor reward.""" from __future__ import annotations @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -RESULTS_DIR = REWARD_DIR / "task-2-4-eval" +RESULTS_DIR = REWARD_DIR / "agentic-search-eval" REPORT_PATH = RESULTS_DIR / "evaluation.json" diff --git a/tasks/task-1-2/.gitignore b/tasks/code-embedding-finetuning/.gitignore similarity index 100% rename from tasks/task-1-2/.gitignore rename to tasks/code-embedding-finetuning/.gitignore diff --git a/tasks/task-2-2/README.md b/tasks/code-embedding-finetuning/README.md similarity index 94% rename from tasks/task-2-2/README.md rename to tasks/code-embedding-finetuning/README.md index d5b81d2..2420887 100644 --- a/tasks/task-2-2/README.md +++ b/tasks/code-embedding-finetuning/README.md @@ -1,8 +1,8 @@ -# Domain-specific Embedder Fine-tuning +# Code Embedding Fine-tuning Adapt a fixed embedding backbone to code retrieval and measure the resulting quality gain. -**Task:** `task-2-2` · **Mode:** Optimization · **Metric:** Held-out Accuracy@1 improvement +**Task:** `code-embedding-finetuning` · **Metric:** Held-out Accuracy@1 improvement ## Overview @@ -114,12 +114,12 @@ Dockerfiles. Configure the integrity judge as well as the coding agent. The asset downloader restores both training data and the fixed backbone. ```bash -python scripts/download_assets.py --task task-2-2 -bash scripts/run_task.sh --task task-2-2 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task code-embedding-finetuning +bash scripts/run_task.sh --task code-embedding-finetuning --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-2-2/`. +writes results under `jobs/code-embedding-finetuning/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-2-2/assets.json b/tasks/code-embedding-finetuning/assets.json similarity index 91% rename from tasks/task-2-2/assets.json rename to tasks/code-embedding-finetuning/assets.json index 610752d..c1e6d1a 100644 --- a/tasks/task-2-2/assets.json +++ b/tasks/code-embedding-finetuning/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-2/corpus.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/code-embedding-finetuning/corpus.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-2/train.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/code-embedding-finetuning/train.jsonl" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-2/validation/ground_truth.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/code-embedding-finetuning/validation/ground_truth.jsonl" } }, { @@ -41,8 +41,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-2/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/code-embedding-finetuning/validation/queries.jsonl" } }, { diff --git a/tasks/task-2-2/environment/.dockerignore b/tasks/code-embedding-finetuning/environment/.dockerignore similarity index 100% rename from tasks/task-2-2/environment/.dockerignore rename to tasks/code-embedding-finetuning/environment/.dockerignore diff --git a/tasks/task-2-2/environment/Dockerfile b/tasks/code-embedding-finetuning/environment/Dockerfile similarity index 100% rename from tasks/task-2-2/environment/Dockerfile rename to tasks/code-embedding-finetuning/environment/Dockerfile diff --git a/tasks/task-2-2/environment/docker-compose.yaml b/tasks/code-embedding-finetuning/environment/docker-compose.yaml similarity index 100% rename from tasks/task-2-2/environment/docker-compose.yaml rename to tasks/code-embedding-finetuning/environment/docker-compose.yaml diff --git a/tasks/task-2-2/environment/docs/available_resources.md b/tasks/code-embedding-finetuning/environment/docs/available_resources.md similarity index 100% rename from tasks/task-2-2/environment/docs/available_resources.md rename to tasks/code-embedding-finetuning/environment/docs/available_resources.md diff --git a/tasks/task-2-2/environment/docs/environment.md b/tasks/code-embedding-finetuning/environment/docs/environment.md similarity index 100% rename from tasks/task-2-2/environment/docs/environment.md rename to tasks/code-embedding-finetuning/environment/docs/environment.md diff --git a/tasks/task-2-2/environment/starter/train.sh b/tasks/code-embedding-finetuning/environment/starter/train.sh similarity index 100% rename from tasks/task-2-2/environment/starter/train.sh rename to tasks/code-embedding-finetuning/environment/starter/train.sh diff --git a/tasks/task-2-2/instruction.md b/tasks/code-embedding-finetuning/instruction.md similarity index 100% rename from tasks/task-2-2/instruction.md rename to tasks/code-embedding-finetuning/instruction.md diff --git a/tasks/task-2-1/model-metadata/bge-reranker-large/configuration.json b/tasks/code-embedding-finetuning/model-metadata/bge-base-en-v1.5/configuration.json similarity index 100% rename from tasks/task-2-1/model-metadata/bge-reranker-large/configuration.json rename to tasks/code-embedding-finetuning/model-metadata/bge-base-en-v1.5/configuration.json diff --git a/tasks/task-2-2/raw-instruction.md b/tasks/code-embedding-finetuning/raw-instruction.md similarity index 98% rename from tasks/task-2-2/raw-instruction.md rename to tasks/code-embedding-finetuning/raw-instruction.md index ae83d63..3678c7f 100644 --- a/tasks/task-2-2/raw-instruction.md +++ b/tasks/code-embedding-finetuning/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: Task-2-2 +# Code Embedding Fine-tuning ## Task Description diff --git a/tasks/task-2-2/task.toml b/tasks/code-embedding-finetuning/task.toml similarity index 95% rename from tasks/task-2-2/task.toml rename to tasks/code-embedding-finetuning/task.toml index be85cb6..1b5dbfc 100644 --- a/tasks/task-2-2/task.toml +++ b/tasks/code-embedding-finetuning/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-2-2" +name = "search-swe/code-embedding-finetuning" version = "0.1" description = "Improve a fixed-backbone code-retrieval embedder on supplied training data." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "code-retrieval", "embedding", "fine-tuning", "gpu"] [metadata] -task_type = "optimize" dataset = "code-retrieval" backbone = "BAAI/bge-base-en-v1.5" primary_metric = "held-out Accuracy@1 improvement" diff --git a/tasks/task-2-2/tests/Dockerfile b/tasks/code-embedding-finetuning/tests/Dockerfile similarity index 100% rename from tasks/task-2-2/tests/Dockerfile rename to tasks/code-embedding-finetuning/tests/Dockerfile diff --git a/tasks/task-2-2/tests/data/ground_truth.jsonl b/tasks/code-embedding-finetuning/tests/data/ground_truth.jsonl similarity index 100% rename from tasks/task-2-2/tests/data/ground_truth.jsonl rename to tasks/code-embedding-finetuning/tests/data/ground_truth.jsonl diff --git a/tasks/task-2-2/tests/data/queries.jsonl b/tasks/code-embedding-finetuning/tests/data/queries.jsonl similarity index 100% rename from tasks/task-2-2/tests/data/queries.jsonl rename to tasks/code-embedding-finetuning/tests/data/queries.jsonl diff --git a/tasks/task-2-2/tests/docker-compose.yaml b/tasks/code-embedding-finetuning/tests/docker-compose.yaml similarity index 100% rename from tasks/task-2-2/tests/docker-compose.yaml rename to tasks/code-embedding-finetuning/tests/docker-compose.yaml diff --git a/tasks/task-2-2/tests/finalize_reward.py b/tasks/code-embedding-finetuning/tests/finalize_reward.py similarity index 97% rename from tasks/task-2-2/tests/finalize_reward.py rename to tasks/code-embedding-finetuning/tests/finalize_reward.py index 63fc3dc..556209e 100644 --- a/tasks/task-2-2/tests/finalize_reward.py +++ b/tasks/code-embedding-finetuning/tests/finalize_reward.py @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -EVALUATION_PATH = REWARD_DIR / "task-2-2-eval" / "evaluation.json" +EVALUATION_PATH = REWARD_DIR / "code-embedding-finetuning-eval" / "evaluation.json" JUDGE_REWARD_PATH = REWARD_DIR / "reward.json" DETAILS_PATH = REWARD_DIR / "reward-details.json" TRAJECTORY_PATH = Path("/logs/agent/trajectory.json") diff --git a/tasks/task-2-2/tests/grader.py b/tasks/code-embedding-finetuning/tests/grader.py similarity index 99% rename from tasks/task-2-2/tests/grader.py rename to tasks/code-embedding-finetuning/tests/grader.py index 4d83f66..3a0160c 100644 --- a/tasks/task-2-2/tests/grader.py +++ b/tasks/code-embedding-finetuning/tests/grader.py @@ -19,7 +19,7 @@ CORPUS_PATH = Path("/task/data/corpus.jsonl") QUERIES_PATH = Path("/tests/data/queries.jsonl") GROUND_TRUTH_PATH = Path("/tests/data/ground_truth.jsonl") -REPORT_PATH = Path("/logs/verifier/task-2-2-eval/evaluation.json") +REPORT_PATH = Path("/logs/verifier/code-embedding-finetuning-eval/evaluation.json") MAX_LENGTH = 512 BATCH_SIZE = 32 diff --git a/tasks/task-2-2/tests/jailbreak_judge/codex.toml b/tasks/code-embedding-finetuning/tests/jailbreak_judge/codex.toml similarity index 100% rename from tasks/task-2-2/tests/jailbreak_judge/codex.toml rename to tasks/code-embedding-finetuning/tests/jailbreak_judge/codex.toml diff --git a/tasks/task-1-1/tests/rewardkit_deepseek.py b/tasks/code-embedding-finetuning/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-1-1/tests/rewardkit_deepseek.py rename to tasks/code-embedding-finetuning/tests/rewardkit_deepseek.py diff --git a/tasks/task-2-2/tests/run_evaluation.sh b/tasks/code-embedding-finetuning/tests/run_evaluation.sh similarity index 85% rename from tasks/task-2-2/tests/run_evaluation.sh rename to tasks/code-embedding-finetuning/tests/run_evaluation.sh index 9d07259..3e32475 100644 --- a/tasks/task-2-2/tests/run_evaluation.sh +++ b/tasks/code-embedding-finetuning/tests/run_evaluation.sh @@ -3,7 +3,7 @@ set -u set -o pipefail -LOG_DIR=/logs/verifier/task-2-2-eval +LOG_DIR=/logs/verifier/code-embedding-finetuning-eval mkdir -p /logs/verifier "$LOG_DIR" diff --git a/tasks/task-2-2/tests/test.sh b/tasks/code-embedding-finetuning/tests/test.sh similarity index 99% rename from tasks/task-2-2/tests/test.sh rename to tasks/code-embedding-finetuning/tests/test.sh index 3a090b3..c2710ec 100644 --- a/tasks/task-2-2/tests/test.sh +++ b/tasks/code-embedding-finetuning/tests/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -uo pipefail -TASK_ID=task-2-2 +TASK_ID=code-embedding-finetuning EVALUATION_TIMEOUT_SECONDS=3600 JUDGE_TIMEOUT_SECONDS=1200 FINALIZER_TIMEOUT_SECONDS=60 diff --git a/tasks/task-1-3/.gitignore b/tasks/long-document-reranking/.gitignore similarity index 100% rename from tasks/task-1-3/.gitignore rename to tasks/long-document-reranking/.gitignore diff --git a/tasks/task-2-1/README.md b/tasks/long-document-reranking/README.md similarity index 94% rename from tasks/task-2-1/README.md rename to tasks/long-document-reranking/README.md index bb810c3..b04568a 100644 --- a/tasks/task-2-1/README.md +++ b/tasks/long-document-reranking/README.md @@ -1,8 +1,8 @@ -# Constrained Long-document Reranking +# Long-document Reranking Improve long-document ranking without changing the candidate pool or the relevance model. -**Task:** `task-2-1` · **Mode:** Optimization · **Metric:** Accuracy@5 +**Task:** `long-document-reranking` · **Metric:** Accuracy@5 ## Overview @@ -74,8 +74,7 @@ The index directory is supplied by the caller. Consult A hidden query is a hit if a relevant document is among the five returned candidates. Accuracy@5 is the fraction of hits over all hidden queries. -Unlike the implementation tasks' all-query gates, valid runs receive a -continuous quality score. +Valid runs receive this fraction as a continuous quality score. ### Correctness and Resource Gates @@ -104,12 +103,12 @@ verifier's integrity judge separately from the coding agent. The submission uses the local reranker and does not need external reranking-service credentials. ```bash -python scripts/download_assets.py --task task-2-1 -bash scripts/run_task.sh --task task-2-1 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task long-document-reranking +bash scripts/run_task.sh --task long-document-reranking --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-2-1/`. +writes results under `jobs/long-document-reranking/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-2-1/assets.json b/tasks/long-document-reranking/assets.json similarity index 92% rename from tasks/task-2-1/assets.json rename to tasks/long-document-reranking/assets.json index 8e8bcdd..a8a6f72 100644 --- a/tasks/task-2-1/assets.json +++ b/tasks/long-document-reranking/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-1/corpus.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/long-document-reranking/corpus.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-1/validation/ground_truth.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/long-document-reranking/validation/ground_truth.jsonl" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-1/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/long-document-reranking/validation/queries.jsonl" } }, { diff --git a/tasks/task-2-1/environment/Dockerfile b/tasks/long-document-reranking/environment/Dockerfile similarity index 100% rename from tasks/task-2-1/environment/Dockerfile rename to tasks/long-document-reranking/environment/Dockerfile diff --git a/tasks/task-2-1/environment/docker-compose.yaml b/tasks/long-document-reranking/environment/docker-compose.yaml similarity index 100% rename from tasks/task-2-1/environment/docker-compose.yaml rename to tasks/long-document-reranking/environment/docker-compose.yaml diff --git a/tasks/task-2-1/environment/docs/available_resources.md b/tasks/long-document-reranking/environment/docs/available_resources.md similarity index 100% rename from tasks/task-2-1/environment/docs/available_resources.md rename to tasks/long-document-reranking/environment/docs/available_resources.md diff --git a/tasks/task-1-2/environment/docs/environment.md b/tasks/long-document-reranking/environment/docs/environment.md similarity index 100% rename from tasks/task-1-2/environment/docs/environment.md rename to tasks/long-document-reranking/environment/docs/environment.md diff --git a/tasks/task-2-1/environment/starter/README.md b/tasks/long-document-reranking/environment/starter/README.md similarity index 100% rename from tasks/task-2-1/environment/starter/README.md rename to tasks/long-document-reranking/environment/starter/README.md diff --git a/tasks/task-2-1/environment/starter/build.sh b/tasks/long-document-reranking/environment/starter/build.sh similarity index 100% rename from tasks/task-2-1/environment/starter/build.sh rename to tasks/long-document-reranking/environment/starter/build.sh diff --git a/tasks/task-2-1/environment/starter/run.sh b/tasks/long-document-reranking/environment/starter/run.sh similarity index 100% rename from tasks/task-2-1/environment/starter/run.sh rename to tasks/long-document-reranking/environment/starter/run.sh diff --git a/tasks/task-2-1/environment/starter/src/__init__.py b/tasks/long-document-reranking/environment/starter/src/__init__.py similarity index 100% rename from tasks/task-2-1/environment/starter/src/__init__.py rename to tasks/long-document-reranking/environment/starter/src/__init__.py diff --git a/tasks/task-2-1/environment/starter/src/build_index.py b/tasks/long-document-reranking/environment/starter/src/build_index.py similarity index 100% rename from tasks/task-2-1/environment/starter/src/build_index.py rename to tasks/long-document-reranking/environment/starter/src/build_index.py diff --git a/tasks/task-2-1/environment/starter/src/server.py b/tasks/long-document-reranking/environment/starter/src/server.py similarity index 100% rename from tasks/task-2-1/environment/starter/src/server.py rename to tasks/long-document-reranking/environment/starter/src/server.py diff --git a/tasks/task-2-1/instruction.md b/tasks/long-document-reranking/instruction.md similarity index 100% rename from tasks/task-2-1/instruction.md rename to tasks/long-document-reranking/instruction.md diff --git a/tasks/task-2-1/model-metadata/bge-reranker-large/.gitattributes b/tasks/long-document-reranking/model-metadata/bge-reranker-large/.gitattributes similarity index 100% rename from tasks/task-2-1/model-metadata/bge-reranker-large/.gitattributes rename to tasks/long-document-reranking/model-metadata/bge-reranker-large/.gitattributes diff --git a/tasks/task-2-2/model-metadata/bge-base-en-v1.5/configuration.json b/tasks/long-document-reranking/model-metadata/bge-reranker-large/configuration.json similarity index 100% rename from tasks/task-2-2/model-metadata/bge-base-en-v1.5/configuration.json rename to tasks/long-document-reranking/model-metadata/bge-reranker-large/configuration.json diff --git a/tasks/task-2-1/raw-instruction.md b/tasks/long-document-reranking/raw-instruction.md similarity index 99% rename from tasks/task-2-1/raw-instruction.md rename to tasks/long-document-reranking/raw-instruction.md index 2fc916a..9a2d1ef 100644 --- a/tasks/task-2-1/raw-instruction.md +++ b/tasks/long-document-reranking/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: Task-2-1 +# Long-document Reranking ## Task Description diff --git a/tasks/task-2-1/task.toml b/tasks/long-document-reranking/task.toml similarity index 95% rename from tasks/task-2-1/task.toml rename to tasks/long-document-reranking/task.toml index fece73a..f6e106e 100644 --- a/tasks/task-2-1/task.toml +++ b/tasks/long-document-reranking/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-2-1" +name = "search-swe/long-document-reranking" version = "0.1" description = "Optimize long-document reranking over a frozen BM25 Top-100 candidate pool with a fixed local BGE reranker." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "search", "reranking", "cross-encoder", "long-document", "cpu"] [metadata] -task_type = "optimize" dataset = "MTEB NarrativeQARetrieval" primary_metric = "Accuracy@5" metric_k = 5 diff --git a/tasks/task-2-1/tests/.dockerignore b/tasks/long-document-reranking/tests/.dockerignore similarity index 100% rename from tasks/task-2-1/tests/.dockerignore rename to tasks/long-document-reranking/tests/.dockerignore diff --git a/tasks/task-1-1/tests/Dockerfile b/tasks/long-document-reranking/tests/Dockerfile similarity index 100% rename from tasks/task-1-1/tests/Dockerfile rename to tasks/long-document-reranking/tests/Dockerfile diff --git a/tasks/task-2-1/tests/README.md b/tasks/long-document-reranking/tests/README.md similarity index 92% rename from tasks/task-2-1/tests/README.md rename to tasks/long-document-reranking/tests/README.md index 168be7e..99d2550 100644 --- a/tasks/task-2-1/tests/README.md +++ b/tasks/long-document-reranking/tests/README.md @@ -1,4 +1,4 @@ -# Task-2-1 private verifier +# Task long-document-reranking private verifier The hidden query and ground truth files under `data/` are verifier-only. `test.sh` protects them and invokes `grader.py`, which builds the submitted diff --git a/tasks/task-2-1/tests/data/ground_truth.jsonl b/tasks/long-document-reranking/tests/data/ground_truth.jsonl similarity index 100% rename from tasks/task-2-1/tests/data/ground_truth.jsonl rename to tasks/long-document-reranking/tests/data/ground_truth.jsonl diff --git a/tasks/task-2-1/tests/data/queries.jsonl b/tasks/long-document-reranking/tests/data/queries.jsonl similarity index 100% rename from tasks/task-2-1/tests/data/queries.jsonl rename to tasks/long-document-reranking/tests/data/queries.jsonl diff --git a/tasks/task-2-1/tests/docker-compose.yaml b/tasks/long-document-reranking/tests/docker-compose.yaml similarity index 100% rename from tasks/task-2-1/tests/docker-compose.yaml rename to tasks/long-document-reranking/tests/docker-compose.yaml diff --git a/tasks/task-2-1/tests/finalize_reward.py b/tasks/long-document-reranking/tests/finalize_reward.py similarity index 97% rename from tasks/task-2-1/tests/finalize_reward.py rename to tasks/long-document-reranking/tests/finalize_reward.py index 5d743de..5b9f50f 100755 --- a/tasks/task-2-1/tests/finalize_reward.py +++ b/tasks/long-document-reranking/tests/finalize_reward.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Apply the AgentJudge gate to Task-2-1's Accuracy@5 score.""" +"""Apply the AgentJudge gate to Task long-document-reranking's Accuracy@5 score.""" from __future__ import annotations diff --git a/tasks/task-2-1/tests/grader.py b/tasks/long-document-reranking/tests/grader.py similarity index 98% rename from tasks/task-2-1/tests/grader.py rename to tasks/long-document-reranking/tests/grader.py index cccf26c..6c9a960 100755 --- a/tasks/task-2-1/tests/grader.py +++ b/tasks/long-document-reranking/tests/grader.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Hidden verifier for Task-2-1 long-document reranking.""" +"""Hidden verifier for Task long-document-reranking long-document reranking.""" from __future__ import annotations @@ -17,7 +17,7 @@ CORPUS_PATH = Path("/task/data/corpus.jsonl") QUERIES_PATH = Path("/tests/data/queries.jsonl") GROUND_TRUTH_PATH = Path("/tests/data/ground_truth.jsonl") -WORK_DIR = Path("/tmp/search-swe-task2-1-verifier") +WORK_DIR = Path("/tmp/search-swe-long-document-reranking-verifier") INDEX_DIR = WORK_DIR / "index" INPUT_QUERIES_PATH = WORK_DIR / "hidden-queries.jsonl" OUTPUT_PATH = WORK_DIR / "results.jsonl" @@ -274,7 +274,7 @@ def write_report(report: dict[str, Any]) -> None: def main() -> int: started = time.monotonic() report: dict[str, Any] = { - "task_id": "task-2-1", + "task_id": "long-document-reranking", "valid": False, "reward": 0.0, "primary_metric": "Accuracy@5", diff --git a/tasks/task-2-1/tests/jailbreak_judge/codex.toml b/tasks/long-document-reranking/tests/jailbreak_judge/codex.toml similarity index 100% rename from tasks/task-2-1/tests/jailbreak_judge/codex.toml rename to tasks/long-document-reranking/tests/jailbreak_judge/codex.toml diff --git a/tasks/task-1-2/tests/rewardkit_deepseek.py b/tasks/long-document-reranking/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-1-2/tests/rewardkit_deepseek.py rename to tasks/long-document-reranking/tests/rewardkit_deepseek.py diff --git a/tasks/task-2-1/tests/test.sh b/tasks/long-document-reranking/tests/test.sh similarity index 99% rename from tasks/task-2-1/tests/test.sh rename to tasks/long-document-reranking/tests/test.sh index 5ee61c0..2d33ddf 100755 --- a/tasks/task-2-1/tests/test.sh +++ b/tasks/long-document-reranking/tests/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -u -TASK_ID=task-2-1 +TASK_ID=long-document-reranking RESULTS_DIR=/logs/verifier/${TASK_ID}-eval SUBMISSION_USER=submission VERIFIER_PRIVATE_DIR=/logs/verifier/.private diff --git a/tasks/task-1-4/.gitignore b/tasks/long-pdf-evidence/.gitignore similarity index 100% rename from tasks/task-1-4/.gitignore rename to tasks/long-pdf-evidence/.gitignore diff --git a/tasks/task-1-4/README.md b/tasks/long-pdf-evidence/README.md similarity index 94% rename from tasks/task-1-4/README.md rename to tasks/long-pdf-evidence/README.md index d338710..12e368a 100644 --- a/tasks/task-1-4/README.md +++ b/tasks/long-pdf-evidence/README.md @@ -2,7 +2,7 @@ Find the physical pages in a long PDF that support a natural-language information need. -**Task:** `task-1-4` · **Mode:** Implementation · **Metric:** Page Recall@5 +**Task:** `long-pdf-evidence` · **Metric:** Page Recall@5 ## Overview @@ -69,7 +69,7 @@ text, and numeric scores. Page numbers are **one-based PDF page positions**, which may differ from printed page labels. See [instruction.md](instruction.md) for the exact interface. This task returns -evidence locations; [task 1-3](../task-1-3/README.md) returns answers with +evidence locations; [Scientific-paper QA](../scientific-paper-qa/README.md) returns answers with document-level attribution. ## Evaluation @@ -113,12 +113,12 @@ and Jina keys. Downloading restores the shared PDF and public validation files; Compose keeps private queries and labels out of the agent environment. ```bash -python scripts/download_assets.py --task task-1-4 -bash scripts/run_task.sh --task task-1-4 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task long-pdf-evidence +bash scripts/run_task.sh --task long-pdf-evidence --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-1-4/`. +writes results under `jobs/long-pdf-evidence/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-1-4/assets.json b/tasks/long-pdf-evidence/assets.json similarity index 67% rename from tasks/task-1-4/assets.json rename to tasks/long-pdf-evidence/assets.json index 1234f07..7f1bc96 100644 --- a/tasks/task-1-4/assets.json +++ b/tasks/long-pdf-evidence/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-4/corpus/lp4P6AUxur.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/long-pdf-evidence/corpus/lp4P6AUxur.pdf" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-4/validation/golden_answers.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/long-pdf-evidence/validation/golden_answers.jsonl" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-4/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/long-pdf-evidence/validation/queries.jsonl" } } ] diff --git a/tasks/task-1-1/environment/Dockerfile b/tasks/long-pdf-evidence/environment/Dockerfile similarity index 100% rename from tasks/task-1-1/environment/Dockerfile rename to tasks/long-pdf-evidence/environment/Dockerfile diff --git a/tasks/task-1-4/environment/docker-compose.yaml b/tasks/long-pdf-evidence/environment/docker-compose.yaml similarity index 100% rename from tasks/task-1-4/environment/docker-compose.yaml rename to tasks/long-pdf-evidence/environment/docker-compose.yaml diff --git a/tasks/task-1-4/environment/docs/available_resources.md b/tasks/long-pdf-evidence/environment/docs/available_resources.md similarity index 100% rename from tasks/task-1-4/environment/docs/available_resources.md rename to tasks/long-pdf-evidence/environment/docs/available_resources.md diff --git a/tasks/task-1-3/environment/docs/environment.md b/tasks/long-pdf-evidence/environment/docs/environment.md similarity index 100% rename from tasks/task-1-3/environment/docs/environment.md rename to tasks/long-pdf-evidence/environment/docs/environment.md diff --git a/tasks/task-1-4/instruction.md b/tasks/long-pdf-evidence/instruction.md similarity index 100% rename from tasks/task-1-4/instruction.md rename to tasks/long-pdf-evidence/instruction.md diff --git a/tasks/task-1-4/raw-instruction.md b/tasks/long-pdf-evidence/raw-instruction.md similarity index 99% rename from tasks/task-1-4/raw-instruction.md rename to tasks/long-pdf-evidence/raw-instruction.md index cb85ea6..1bd6414 100644 --- a/tasks/task-1-4/raw-instruction.md +++ b/tasks/long-pdf-evidence/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: Task-1-4 +# Long-PDF Evidence Localization ## Task Description diff --git a/tasks/task-1-4/task.toml b/tasks/long-pdf-evidence/task.toml similarity index 96% rename from tasks/task-1-4/task.toml rename to tasks/long-pdf-evidence/task.toml index ffe9f7c..a770ef8 100644 --- a/tasks/task-1-4/task.toml +++ b/tasks/long-pdf-evidence/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-1-4" +name = "search-swe/long-pdf-evidence" version = "0.3.0" description = "Build a natural-language evidence locator inside a single very long document." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "search", "evidence locator"] [metadata] -task_type = "create" dataset = "single long PDF" primary_metric = "Recall@5" metric_k = 5 diff --git a/tasks/task-1-3/tests/.dockerignore b/tasks/long-pdf-evidence/tests/.dockerignore similarity index 100% rename from tasks/task-1-3/tests/.dockerignore rename to tasks/long-pdf-evidence/tests/.dockerignore diff --git a/tasks/task-1-2/tests/Dockerfile b/tasks/long-pdf-evidence/tests/Dockerfile similarity index 100% rename from tasks/task-1-2/tests/Dockerfile rename to tasks/long-pdf-evidence/tests/Dockerfile diff --git a/tasks/task-1-3/tests/configure_trajectory_judge.py b/tasks/long-pdf-evidence/tests/configure_trajectory_judge.py similarity index 100% rename from tasks/task-1-3/tests/configure_trajectory_judge.py rename to tasks/long-pdf-evidence/tests/configure_trajectory_judge.py diff --git a/tasks/task-1-4/tests/data/golden_answers.jsonl b/tasks/long-pdf-evidence/tests/data/golden_answers.jsonl similarity index 100% rename from tasks/task-1-4/tests/data/golden_answers.jsonl rename to tasks/long-pdf-evidence/tests/data/golden_answers.jsonl diff --git a/tasks/task-1-4/tests/data/queries.jsonl b/tasks/long-pdf-evidence/tests/data/queries.jsonl similarity index 100% rename from tasks/task-1-4/tests/data/queries.jsonl rename to tasks/long-pdf-evidence/tests/data/queries.jsonl diff --git a/tasks/task-1-4/tests/docker-compose.yaml b/tasks/long-pdf-evidence/tests/docker-compose.yaml similarity index 100% rename from tasks/task-1-4/tests/docker-compose.yaml rename to tasks/long-pdf-evidence/tests/docker-compose.yaml diff --git a/tasks/task-1-4/tests/finalize_reward.py b/tasks/long-pdf-evidence/tests/finalize_reward.py similarity index 94% rename from tasks/task-1-4/tests/finalize_reward.py rename to tasks/long-pdf-evidence/tests/finalize_reward.py index 3046189..4920ab7 100644 --- a/tasks/task-1-4/tests/finalize_reward.py +++ b/tasks/long-pdf-evidence/tests/finalize_reward.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Apply the trajectory anti-jailbreak gate to Task-1-4's Recall@5.""" +"""Apply the trajectory anti-jailbreak gate to Task long-pdf-evidence's Recall@5.""" from __future__ import annotations @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -EVALUATION_PATH = Path("/logs/verifier/task-1-4-eval/evaluation.json") +EVALUATION_PATH = Path("/logs/verifier/long-pdf-evidence-eval/evaluation.json") JUDGE_REWARD_PATH = REWARD_DIR / "reward.json" DETAILS_PATH = REWARD_DIR / "reward-details.json" TRAJECTORY_PATH = Path("/logs/agent/trajectory.json") diff --git a/tasks/task-1-4/tests/grader.py b/tasks/long-pdf-evidence/tests/grader.py similarity index 98% rename from tasks/task-1-4/tests/grader.py rename to tasks/long-pdf-evidence/tests/grader.py index 989b255..a9f32b5 100644 --- a/tasks/task-1-4/tests/grader.py +++ b/tasks/long-pdf-evidence/tests/grader.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Validate Task-1-4 evidence locations and compute macro Recall@5.""" +"""Validate Task long-pdf-evidence evidence locations and compute macro Recall@5.""" from __future__ import annotations @@ -13,11 +13,11 @@ import pypdfium2 as pdfium -CORPUS_DIR = Path("/tmp/task-1-4-eval/corpus") +CORPUS_DIR = Path("/tmp/long-pdf-evidence-eval/corpus") QUERIES_PATH = Path("/tests/data/queries.jsonl") GROUND_TRUTH_PATH = Path("/tests/data/golden_answers.jsonl") RESULTS_DIR = Path( - os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/task-1-4-eval") + os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/long-pdf-evidence-eval") ) PREDICTIONS_PATH = RESULTS_DIR / "results.jsonl" REPORT_PATH = RESULTS_DIR / "evaluation.json" @@ -365,7 +365,7 @@ def write_report(report: dict[str, Any]) -> None: def invalid_report(error: Exception) -> dict[str, Any]: return { - "task_id": "task-1-4", + "task_id": "long-pdf-evidence", "valid": False, "score": 0.0, "query_count": 0, @@ -408,7 +408,7 @@ def main() -> int: mean_recall = recall_sum / len(queries) report = { - "task_id": "task-1-4", + "task_id": "long-pdf-evidence", "valid": True, "score": 100.0 * mean_recall, "query_count": len(queries), diff --git a/tasks/task-1-4/tests/jailbreak_judge/codex.toml b/tasks/long-pdf-evidence/tests/jailbreak_judge/codex.toml similarity index 96% rename from tasks/task-1-4/tests/jailbreak_judge/codex.toml rename to tasks/long-pdf-evidence/tests/jailbreak_judge/codex.toml index 9da6794..bc25e44 100644 --- a/tasks/task-1-4/tests/jailbreak_judge/codex.toml +++ b/tasks/long-pdf-evidence/tests/jailbreak_judge/codex.toml @@ -10,7 +10,7 @@ atif-trajectory = "/logs/agent/trajectory.json" name = "jailbreak_judge" type = "binary" description = """ -Review the Task-1-4 Agent's complete execution trajectory and submitted files to +Review the Task long-pdf-evidence Agent's complete execution trajectory and submitted files to decide whether it circumvented this local-PDF evidence-localization task. Return `no` only when there is clear evidence of a violation; otherwise return `yes`. diff --git a/tasks/task-1-3/tests/rewardkit_deepseek.py b/tasks/long-pdf-evidence/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-1-3/tests/rewardkit_deepseek.py rename to tasks/long-pdf-evidence/tests/rewardkit_deepseek.py diff --git a/tasks/task-1-4/tests/run_queries.py b/tasks/long-pdf-evidence/tests/run_queries.py similarity index 100% rename from tasks/task-1-4/tests/run_queries.py rename to tasks/long-pdf-evidence/tests/run_queries.py diff --git a/tasks/task-1-4/tests/test.sh b/tasks/long-pdf-evidence/tests/test.sh similarity index 99% rename from tasks/task-1-4/tests/test.sh rename to tasks/long-pdf-evidence/tests/test.sh index 139ad84..a63a64e 100755 --- a/tasks/task-1-4/tests/test.sh +++ b/tasks/long-pdf-evidence/tests/test.sh @@ -1,13 +1,13 @@ #!/usr/bin/env bash set -u -WORK_DIR=/tmp/task-1-4-eval +WORK_DIR=/tmp/long-pdf-evidence-eval CORPUS_DIR=$WORK_DIR/corpus INPUT_DIR=$WORK_DIR/input INDEX_DIR=$WORK_DIR/index SUBMISSION_OUTPUT_DIR=$WORK_DIR/output STAGED_QUERIES=$INPUT_DIR/queries.jsonl -RESULTS_DIR=/logs/verifier/task-1-4-eval +RESULTS_DIR=/logs/verifier/long-pdf-evidence-eval OUTPUT_PATH=$RESULTS_DIR/results.jsonl BUILD_TIMEOUT_SECONDS=2400 # The runner enforces 1,800 seconds of query execution; allow cleanup afterwards. diff --git a/tasks/task-2-1/.gitignore b/tasks/memory-constrained-dense-retrieval/.gitignore similarity index 100% rename from tasks/task-2-1/.gitignore rename to tasks/memory-constrained-dense-retrieval/.gitignore diff --git a/tasks/task-1-2/README.md b/tasks/memory-constrained-dense-retrieval/README.md similarity index 94% rename from tasks/task-1-2/README.md rename to tasks/memory-constrained-dense-retrieval/README.md index 3c7574d..fa450a8 100644 --- a/tasks/task-1-2/README.md +++ b/tasks/memory-constrained-dense-retrieval/README.md @@ -2,7 +2,7 @@ Search a fixed vector collection larger than available RAM while meeting strict query deadlines. -**Task:** `task-1-2` · **Mode:** Implementation · **Metric:** Mean per-query Accuracy@3 with a latency gate +**Task:** `memory-constrained-dense-retrieval` · **Metric:** Mean per-query Accuracy@3 with a latency gate ## Overview @@ -43,8 +43,7 @@ bit for bit and relevance labels mapped consistently. Labels remain original NQ relevance judgments. Both splits were selected from full-corpus exact Top-3 hits, favoring small relevant-document margins over rank four and balancing those margins between splits. This margin is a difficulty proxy, -not evidence of failure by a measured ANN baseline. Previously public -examples remain public. +not evidence of failure by a measured ANN baseline. ### Fixed Components and Allowed Changes @@ -117,12 +116,12 @@ The candidate search system uses supplied vectors and does not require external model-service credentials. ```bash -python scripts/download_assets.py --task task-1-2 -bash scripts/run_task.sh --task task-1-2 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task memory-constrained-dense-retrieval +bash scripts/run_task.sh --task memory-constrained-dense-retrieval --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-1-2/`. +writes results under `jobs/memory-constrained-dense-retrieval/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-1-2/assets.json b/tasks/memory-constrained-dense-retrieval/assets.json similarity index 64% rename from tasks/task-1-2/assets.json rename to tasks/memory-constrained-dense-retrieval/assets.json index 8420fa8..2002cbb 100644 --- a/tasks/task-1-2/assets.json +++ b/tasks/memory-constrained-dense-retrieval/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/metadata.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/metadata.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/validation/ground_truth.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/validation/ground_truth.jsonl" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/validation/queries.jsonl" } }, { @@ -41,8 +41,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/validation/query_metadata.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/validation/query_metadata.jsonl" } }, { @@ -52,8 +52,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/validation/query_vectors.f32" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/validation/query_vectors.f32" } }, { @@ -63,8 +63,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/validation/vector_config.json" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/validation/vector_config.json" } }, { @@ -74,8 +74,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-1-2/vector_config.json" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/vector_config.json" } }, { @@ -85,8 +85,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "fc3c457243f04b5a23a10cb33ed53bbbcdd30405", - "filename": "tasks/task-1-2/vectors.f32" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/memory-constrained-dense-retrieval/vectors.f32" } } ] diff --git a/tasks/task-1-2/environment/Dockerfile b/tasks/memory-constrained-dense-retrieval/environment/Dockerfile similarity index 100% rename from tasks/task-1-2/environment/Dockerfile rename to tasks/memory-constrained-dense-retrieval/environment/Dockerfile diff --git a/tasks/task-1-3/environment/docker-compose.yaml b/tasks/memory-constrained-dense-retrieval/environment/docker-compose.yaml similarity index 100% rename from tasks/task-1-3/environment/docker-compose.yaml rename to tasks/memory-constrained-dense-retrieval/environment/docker-compose.yaml diff --git a/tasks/task-1-2/environment/docs/available_resources.md b/tasks/memory-constrained-dense-retrieval/environment/docs/available_resources.md similarity index 100% rename from tasks/task-1-2/environment/docs/available_resources.md rename to tasks/memory-constrained-dense-retrieval/environment/docs/available_resources.md diff --git a/tasks/task-1-4/environment/docs/environment.md b/tasks/memory-constrained-dense-retrieval/environment/docs/environment.md similarity index 100% rename from tasks/task-1-4/environment/docs/environment.md rename to tasks/memory-constrained-dense-retrieval/environment/docs/environment.md diff --git a/tasks/task-1-2/instruction.md b/tasks/memory-constrained-dense-retrieval/instruction.md similarity index 100% rename from tasks/task-1-2/instruction.md rename to tasks/memory-constrained-dense-retrieval/instruction.md diff --git a/tasks/task-1-2/raw-instruction.md b/tasks/memory-constrained-dense-retrieval/raw-instruction.md similarity index 99% rename from tasks/task-1-2/raw-instruction.md rename to tasks/memory-constrained-dense-retrieval/raw-instruction.md index 87996f4..a765d27 100644 --- a/tasks/task-1-2/raw-instruction.md +++ b/tasks/memory-constrained-dense-retrieval/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: 1-2 +# Memory-constrained Dense Retrieval ## Task Description diff --git a/tasks/task-1-2/task.toml b/tasks/memory-constrained-dense-retrieval/task.toml similarity index 95% rename from tasks/task-1-2/task.toml rename to tasks/memory-constrained-dense-retrieval/task.toml index ac3c48e..a84d096 100644 --- a/tasks/task-1-2/task.toml +++ b/tasks/memory-constrained-dense-retrieval/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-1-2" +name = "search-swe/memory-constrained-dense-retrieval" version = "0.3.0" description = "Build a memory-constrained dense-vector retrieval system over 1.5 million precomputed NQ corpus vectors." authors = [{ name = "Search-SWE" }] keywords = ["search", "retrieval", "dense-retrieval", "vector-search", "nq", "memory-constrained", "cpu", "indexing"] [metadata] -task_type = "create" dataset = "NaturalQuestions" primary_metric = "Accuracy@3" metric_k = 3 diff --git a/tasks/task-1-2/tests/.dockerignore b/tasks/memory-constrained-dense-retrieval/tests/.dockerignore similarity index 100% rename from tasks/task-1-2/tests/.dockerignore rename to tasks/memory-constrained-dense-retrieval/tests/.dockerignore diff --git a/tasks/task-1-3/tests/Dockerfile b/tasks/memory-constrained-dense-retrieval/tests/Dockerfile similarity index 100% rename from tasks/task-1-3/tests/Dockerfile rename to tasks/memory-constrained-dense-retrieval/tests/Dockerfile diff --git a/tasks/task-1-2/tests/data/ground_truth.jsonl b/tasks/memory-constrained-dense-retrieval/tests/data/ground_truth.jsonl similarity index 100% rename from tasks/task-1-2/tests/data/ground_truth.jsonl rename to tasks/memory-constrained-dense-retrieval/tests/data/ground_truth.jsonl diff --git a/tasks/task-1-2/tests/data/query_metadata.jsonl b/tasks/memory-constrained-dense-retrieval/tests/data/query_metadata.jsonl similarity index 100% rename from tasks/task-1-2/tests/data/query_metadata.jsonl rename to tasks/memory-constrained-dense-retrieval/tests/data/query_metadata.jsonl diff --git a/tasks/task-1-2/tests/data/query_vectors.f32 b/tasks/memory-constrained-dense-retrieval/tests/data/query_vectors.f32 similarity index 100% rename from tasks/task-1-2/tests/data/query_vectors.f32 rename to tasks/memory-constrained-dense-retrieval/tests/data/query_vectors.f32 diff --git a/tasks/task-1-2/tests/data/vector_config.json b/tasks/memory-constrained-dense-retrieval/tests/data/vector_config.json similarity index 100% rename from tasks/task-1-2/tests/data/vector_config.json rename to tasks/memory-constrained-dense-retrieval/tests/data/vector_config.json diff --git a/tasks/task-1-2/tests/docker-compose.yaml b/tasks/memory-constrained-dense-retrieval/tests/docker-compose.yaml similarity index 100% rename from tasks/task-1-2/tests/docker-compose.yaml rename to tasks/memory-constrained-dense-retrieval/tests/docker-compose.yaml diff --git a/tasks/task-1-2/tests/finalize_reward.py b/tasks/memory-constrained-dense-retrieval/tests/finalize_reward.py similarity index 93% rename from tasks/task-1-2/tests/finalize_reward.py rename to tasks/memory-constrained-dense-retrieval/tests/finalize_reward.py index 19e996e..8a3a47e 100644 --- a/tasks/task-1-2/tests/finalize_reward.py +++ b/tasks/memory-constrained-dense-retrieval/tests/finalize_reward.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Apply the trajectory gate to Task-1-2's mean per-query correctness/latency score.""" +"""Apply the trajectory gate to Task memory-constrained-dense-retrieval's mean per-query correctness/latency score.""" from __future__ import annotations @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -EVALUATION_PATH = Path("/logs/verifier/task-1-2-eval/evaluation.json") +EVALUATION_PATH = Path("/logs/verifier/memory-constrained-dense-retrieval-eval/evaluation.json") JUDGE_REWARD_PATH = REWARD_DIR / "reward.json" DETAILS_PATH = REWARD_DIR / "reward-details.json" TRAJECTORY_PATH = Path("/logs/agent/trajectory.json") diff --git a/tasks/task-1-2/tests/grader.py b/tasks/memory-constrained-dense-retrieval/tests/grader.py similarity index 98% rename from tasks/task-1-2/tests/grader.py rename to tasks/memory-constrained-dense-retrieval/tests/grader.py index 7fe9196..8c8cf4c 100644 --- a/tasks/task-1-2/tests/grader.py +++ b/tasks/memory-constrained-dense-retrieval/tests/grader.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Private verifier for Task-1-2's memory-constrained vector retrieval.""" +"""Private verifier for Task memory-constrained-dense-retrieval's memory-constrained vector retrieval.""" from __future__ import annotations @@ -18,12 +18,12 @@ TASK_DATA = Path("/task/data") PRIVATE_DATA = Path("/tests/data") SUBMISSION_DIR = Path("/app") -WORK_DIR = Path("/tmp/task-1-2-eval") +WORK_DIR = Path("/tmp/memory-constrained-dense-retrieval-eval") INPUT_ROOT = WORK_DIR / "input" INDEX_DIR = WORK_DIR / "index" OUTPUT_ROOT = WORK_DIR / "output" RESULTS_DIR = Path( - os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/task-1-2-eval") + os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/memory-constrained-dense-retrieval-eval") ) REPORT_PATH = RESULTS_DIR / "evaluation.json" EXECUTION_ERROR = os.environ.get("SEARCH_SWE_EXECUTION_ERROR", "") @@ -653,7 +653,7 @@ def score_query( def main() -> int: RESULTS_DIR.mkdir(parents=True, exist_ok=True) report: dict[str, Any] = { - "task_id": "task-1-2", "valid": False, "score": 0.0, "reward": 0.0, + "task_id": "memory-constrained-dense-retrieval", "valid": False, "score": 0.0, "reward": 0.0, "errors": [], } records: list[tuple[str, Path, dict[str, Any], str | None]] = [] diff --git a/tasks/task-1-2/tests/jailbreak_judge/codex.toml b/tasks/memory-constrained-dense-retrieval/tests/jailbreak_judge/codex.toml similarity index 100% rename from tasks/task-1-2/tests/jailbreak_judge/codex.toml rename to tasks/memory-constrained-dense-retrieval/tests/jailbreak_judge/codex.toml diff --git a/tasks/task-1-4/tests/rewardkit_deepseek.py b/tasks/memory-constrained-dense-retrieval/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-1-4/tests/rewardkit_deepseek.py rename to tasks/memory-constrained-dense-retrieval/tests/rewardkit_deepseek.py diff --git a/tasks/task-1-2/tests/test.sh b/tasks/memory-constrained-dense-retrieval/tests/test.sh similarity index 98% rename from tasks/task-1-2/tests/test.sh rename to tasks/memory-constrained-dense-retrieval/tests/test.sh index f0fd443..b40a103 100755 --- a/tasks/task-1-2/tests/test.sh +++ b/tasks/memory-constrained-dense-retrieval/tests/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -u -RESULTS_DIR=/logs/verifier/task-1-2-eval +RESULTS_DIR=/logs/verifier/memory-constrained-dense-retrieval-eval SUBMISSION_USER=submission VERIFIER_PRIVATE_DIR=/logs/verifier/.private execution_error= diff --git a/tasks/task-2-2/.gitignore b/tasks/query-encoder-alignment/.gitignore similarity index 100% rename from tasks/task-2-2/.gitignore rename to tasks/query-encoder-alignment/.gitignore diff --git a/tasks/task-2-3/README.md b/tasks/query-encoder-alignment/README.md similarity index 95% rename from tasks/task-2-3/README.md rename to tasks/query-encoder-alignment/README.md index 5fb0aa7..a69ec5a 100644 --- a/tasks/task-2-3/README.md +++ b/tasks/query-encoder-alignment/README.md @@ -2,7 +2,7 @@ Improve a compact query encoder against a document collection whose embeddings cannot change. -**Task:** `task-2-3` · **Mode:** Optimization · **Metric:** Accuracy@1 with a runtime gate +**Task:** `query-encoder-alignment` · **Metric:** Accuracy@1 with a runtime gate ## Overview @@ -114,12 +114,12 @@ as a host user distinct from submission UID 10001, following the asset guide, so the reference model's private file permissions retain their intended effect. ```bash -python scripts/download_assets.py --task task-2-3 -bash scripts/run_task.sh --task task-2-3 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task query-encoder-alignment +bash scripts/run_task.sh --task query-encoder-alignment --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-2-3/`. +writes results under `jobs/query-encoder-alignment/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-2-3/assets.json b/tasks/query-encoder-alignment/assets.json similarity index 95% rename from tasks/task-2-3/assets.json rename to tasks/query-encoder-alignment/assets.json index 46f4f1e..2f938d0 100644 --- a/tasks/task-2-3/assets.json +++ b/tasks/query-encoder-alignment/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-3/corpus.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/query-encoder-alignment/corpus.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-3/doc.npy" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/query-encoder-alignment/doc.npy" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-3/validation/ground_truth.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/query-encoder-alignment/validation/ground_truth.jsonl" } }, { @@ -41,8 +41,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "58394dfd3bcfcbcfb2c3a2617832574ed583b00f", - "filename": "tasks/task-2-3/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/query-encoder-alignment/validation/queries.jsonl" } }, { diff --git a/tasks/task-2-3/environment/.dockerignore b/tasks/query-encoder-alignment/environment/.dockerignore similarity index 100% rename from tasks/task-2-3/environment/.dockerignore rename to tasks/query-encoder-alignment/environment/.dockerignore diff --git a/tasks/task-2-3/environment/Dockerfile b/tasks/query-encoder-alignment/environment/Dockerfile similarity index 100% rename from tasks/task-2-3/environment/Dockerfile rename to tasks/query-encoder-alignment/environment/Dockerfile diff --git a/tasks/task-2-3/environment/docker-compose.yaml b/tasks/query-encoder-alignment/environment/docker-compose.yaml similarity index 100% rename from tasks/task-2-3/environment/docker-compose.yaml rename to tasks/query-encoder-alignment/environment/docker-compose.yaml diff --git a/tasks/task-2-4/environment/docs/available_resources.md b/tasks/query-encoder-alignment/environment/docs/available_resources.md similarity index 100% rename from tasks/task-2-4/environment/docs/available_resources.md rename to tasks/query-encoder-alignment/environment/docs/available_resources.md diff --git a/tasks/task-2-3/environment/docs/environment.md b/tasks/query-encoder-alignment/environment/docs/environment.md similarity index 100% rename from tasks/task-2-3/environment/docs/environment.md rename to tasks/query-encoder-alignment/environment/docs/environment.md diff --git a/tasks/task-2-3/environment/starter/README.md b/tasks/query-encoder-alignment/environment/starter/README.md similarity index 96% rename from tasks/task-2-3/environment/starter/README.md rename to tasks/query-encoder-alignment/environment/starter/README.md index ff78d43..8095c91 100644 --- a/tasks/task-2-3/environment/starter/README.md +++ b/tasks/query-encoder-alignment/environment/starter/README.md @@ -1,4 +1,4 @@ -# Task-2-3 starter +# Task query-encoder-alignment starter This starter is a teacher-side reference baseline. It encodes validation queries with the local Qwen3-Embedding-4B model, receives the frozen 2560-dimensional document vectors through `--doc-vectors`, and searches them with an in-memory FAISS inner-product index. The decoder-only embedder uses its supported `last_token` pooling method, BF16 inference, and the configured retrieval instruction format. diff --git a/tasks/task-2-3/environment/starter/run.sh b/tasks/query-encoder-alignment/environment/starter/run.sh similarity index 100% rename from tasks/task-2-3/environment/starter/run.sh rename to tasks/query-encoder-alignment/environment/starter/run.sh diff --git a/tasks/query-encoder-alignment/environment/starter/src/__init__.py b/tasks/query-encoder-alignment/environment/starter/src/__init__.py new file mode 100644 index 0000000..35181ae --- /dev/null +++ b/tasks/query-encoder-alignment/environment/starter/src/__init__.py @@ -0,0 +1 @@ +"""Task query-encoder-alignment starter implementation.""" diff --git a/tasks/task-2-3/environment/starter/src/common.py b/tasks/query-encoder-alignment/environment/starter/src/common.py similarity index 100% rename from tasks/task-2-3/environment/starter/src/common.py rename to tasks/query-encoder-alignment/environment/starter/src/common.py diff --git a/tasks/task-2-3/environment/starter/src/run_search.py b/tasks/query-encoder-alignment/environment/starter/src/run_search.py similarity index 100% rename from tasks/task-2-3/environment/starter/src/run_search.py rename to tasks/query-encoder-alignment/environment/starter/src/run_search.py diff --git a/tasks/task-2-3/instruction.md b/tasks/query-encoder-alignment/instruction.md similarity index 100% rename from tasks/task-2-3/instruction.md rename to tasks/query-encoder-alignment/instruction.md diff --git a/tasks/task-2-3/model-metadata/Qwen3-Embedding-0.6B/README.md b/tasks/query-encoder-alignment/model-metadata/Qwen3-Embedding-0.6B/README.md similarity index 100% rename from tasks/task-2-3/model-metadata/Qwen3-Embedding-0.6B/README.md rename to tasks/query-encoder-alignment/model-metadata/Qwen3-Embedding-0.6B/README.md diff --git a/tasks/task-2-3/model-metadata/Qwen3-Embedding-0.6B/configuration.json b/tasks/query-encoder-alignment/model-metadata/Qwen3-Embedding-0.6B/configuration.json similarity index 100% rename from tasks/task-2-3/model-metadata/Qwen3-Embedding-0.6B/configuration.json rename to tasks/query-encoder-alignment/model-metadata/Qwen3-Embedding-0.6B/configuration.json diff --git a/tasks/task-2-3/model-metadata/Qwen3-Embedding-4B/configuration.json b/tasks/query-encoder-alignment/model-metadata/Qwen3-Embedding-4B/configuration.json similarity index 100% rename from tasks/task-2-3/model-metadata/Qwen3-Embedding-4B/configuration.json rename to tasks/query-encoder-alignment/model-metadata/Qwen3-Embedding-4B/configuration.json diff --git a/tasks/task-2-3/raw-instruction.md b/tasks/query-encoder-alignment/raw-instruction.md similarity index 98% rename from tasks/task-2-3/raw-instruction.md rename to tasks/query-encoder-alignment/raw-instruction.md index 77098ff..881c746 100644 --- a/tasks/task-2-3/raw-instruction.md +++ b/tasks/query-encoder-alignment/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: Task-2-3 +# Query-side Encoder Alignment ## Task Description diff --git a/tasks/task-2-3/task.toml b/tasks/query-encoder-alignment/task.toml similarity index 96% rename from tasks/task-2-3/task.toml rename to tasks/query-encoder-alignment/task.toml index 181bdca..dc70a99 100644 --- a/tasks/task-2-3/task.toml +++ b/tasks/query-encoder-alignment/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-2-3" +name = "search-swe/query-encoder-alignment" version = "0.1" description = "Optimize a query encoder based on a supplied small backbone against fixed document vectors." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "query-encoder", "distillation", "embedding", "gpu"] [metadata] -task_type = "optimize" dataset = "local frozen retrieval corpus" query_backbone = "Qwen3-Embedding-0.6B" reference_model = "Qwen3-Embedding-4B" diff --git a/tasks/task-2-3/tests/Dockerfile b/tasks/query-encoder-alignment/tests/Dockerfile similarity index 100% rename from tasks/task-2-3/tests/Dockerfile rename to tasks/query-encoder-alignment/tests/Dockerfile diff --git a/tasks/task-2-3/tests/data/ground_truth.jsonl b/tasks/query-encoder-alignment/tests/data/ground_truth.jsonl similarity index 100% rename from tasks/task-2-3/tests/data/ground_truth.jsonl rename to tasks/query-encoder-alignment/tests/data/ground_truth.jsonl diff --git a/tasks/task-2-3/tests/data/queries.jsonl b/tasks/query-encoder-alignment/tests/data/queries.jsonl similarity index 100% rename from tasks/task-2-3/tests/data/queries.jsonl rename to tasks/query-encoder-alignment/tests/data/queries.jsonl diff --git a/tasks/task-2-3/tests/docker-compose.yaml b/tasks/query-encoder-alignment/tests/docker-compose.yaml similarity index 100% rename from tasks/task-2-3/tests/docker-compose.yaml rename to tasks/query-encoder-alignment/tests/docker-compose.yaml diff --git a/tasks/task-2-3/tests/finalize_reward.py b/tasks/query-encoder-alignment/tests/finalize_reward.py similarity index 97% rename from tasks/task-2-3/tests/finalize_reward.py rename to tasks/query-encoder-alignment/tests/finalize_reward.py index e2f4aed..d0bcb6f 100644 --- a/tasks/task-2-3/tests/finalize_reward.py +++ b/tasks/query-encoder-alignment/tests/finalize_reward.py @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -EVALUATION_PATH = REWARD_DIR / "task-2-3-eval" / "evaluation.json" +EVALUATION_PATH = REWARD_DIR / "query-encoder-alignment-eval" / "evaluation.json" JUDGE_REWARD_PATH = REWARD_DIR / "reward.json" DETAILS_PATH = REWARD_DIR / "reward-details.json" TRAJECTORY_PATH = Path("/logs/agent/trajectory.json") diff --git a/tasks/task-2-3/tests/grader.py b/tasks/query-encoder-alignment/tests/grader.py similarity index 95% rename from tasks/task-2-3/tests/grader.py rename to tasks/query-encoder-alignment/tests/grader.py index 20d4e37..18a45d2 100644 --- a/tasks/task-2-3/tests/grader.py +++ b/tasks/query-encoder-alignment/tests/grader.py @@ -18,7 +18,7 @@ VECTOR_PATH = Path("/task/data/doc.npy") QUERIES_PATH = Path("/tests/data/queries.jsonl") GROUND_TRUTH_PATH = Path("/tests/data/ground_truth.jsonl") -REPORT_PATH = Path("/logs/verifier/task-2-3-eval/evaluation.json") +REPORT_PATH = Path("/logs/verifier/query-encoder-alignment-eval/evaluation.json") EXPECTED_VECTOR_SHA256 = "54c2f4a5a83e43d8ab5569f195c7e478a0807c9018e0749a9f62158656c8cd46" TOP_K = 1 TIME_LIMIT_FRACTION = 0.60 @@ -176,15 +176,15 @@ def write_report(report: dict[str, Any]) -> None: def main() -> int: started = time.monotonic() - starter_output = Path(os.environ.get("SEARCH_SWE_STARTER_OUTPUT", "/logs/verifier/task-2-3-eval/starter-results.jsonl")) - candidate_output = Path(os.environ.get("SEARCH_SWE_CANDIDATE_OUTPUT", "/logs/verifier/task-2-3-eval/results.jsonl")) - starter_report_path = Path(os.environ.get("SEARCH_SWE_STARTER_REPORT", "/logs/verifier/task-2-3-eval/starter-report.json")) - starter_time_path = Path(os.environ.get("SEARCH_SWE_STARTER_TIME_FILE", "/logs/verifier/task-2-3-eval/starter-time.txt")) - candidate_time_path = Path(os.environ.get("SEARCH_SWE_CANDIDATE_TIME_FILE", "/logs/verifier/task-2-3-eval/candidate-time.txt")) + starter_output = Path(os.environ.get("SEARCH_SWE_STARTER_OUTPUT", "/logs/verifier/query-encoder-alignment-eval/starter-results.jsonl")) + candidate_output = Path(os.environ.get("SEARCH_SWE_CANDIDATE_OUTPUT", "/logs/verifier/query-encoder-alignment-eval/results.jsonl")) + starter_report_path = Path(os.environ.get("SEARCH_SWE_STARTER_REPORT", "/logs/verifier/query-encoder-alignment-eval/starter-report.json")) + starter_time_path = Path(os.environ.get("SEARCH_SWE_STARTER_TIME_FILE", "/logs/verifier/query-encoder-alignment-eval/starter-time.txt")) + candidate_time_path = Path(os.environ.get("SEARCH_SWE_CANDIDATE_TIME_FILE", "/logs/verifier/query-encoder-alignment-eval/candidate-time.txt")) execution_error = os.environ.get("SEARCH_SWE_EXECUTION_ERROR", "") report: dict[str, Any] = { - "task_id": "task-2-3", + "task_id": "query-encoder-alignment", "valid": False, "reward": 0.0, "primary_metric": {"name": "Accuracy@1", "value": None}, diff --git a/tasks/task-2-3/tests/jailbreak_judge/codex.toml b/tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml similarity index 90% rename from tasks/task-2-3/tests/jailbreak_judge/codex.toml rename to tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml index 6e9c511..4abe8de 100644 --- a/tasks/task-2-3/tests/jailbreak_judge/codex.toml +++ b/tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml @@ -10,7 +10,7 @@ atif-trajectory = "/logs/agent/trajectory.json" name = "jailbreak_judge" type = "binary" description = """ -Review the Agent's complete execution trajectory and `/app/submission` for Task-2-3. Determine whether it genuinely built a query-side retrieval system based on the supplied small backbone and the fixed document vectors. +Review the Agent's complete execution trajectory and `/app/submission` for Task query-encoder-alignment. Determine whether it genuinely built a query-side retrieval system based on the supplied small backbone and the fixed document vectors. A violation occurs when there is clear evidence that the Agent: diff --git a/tasks/task-2-1/tests/rewardkit_deepseek.py b/tasks/query-encoder-alignment/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-2-1/tests/rewardkit_deepseek.py rename to tasks/query-encoder-alignment/tests/rewardkit_deepseek.py diff --git a/tasks/task-2-3/tests/run_evaluation.sh b/tasks/query-encoder-alignment/tests/run_evaluation.sh similarity index 98% rename from tasks/task-2-3/tests/run_evaluation.sh rename to tasks/query-encoder-alignment/tests/run_evaluation.sh index 8432ff9..447aed8 100644 --- a/tasks/task-2-3/tests/run_evaluation.sh +++ b/tasks/query-encoder-alignment/tests/run_evaluation.sh @@ -3,8 +3,8 @@ set -u set -o pipefail -LOG_DIR=/logs/verifier/task-2-3-eval -WORK_DIR=/tmp/task-2-3-verifier +LOG_DIR=/logs/verifier/query-encoder-alignment-eval +WORK_DIR=/tmp/query-encoder-alignment-verifier CANDIDATE_RUNTIME=$WORK_DIR/candidate-runtime CANDIDATE_OUTPUT=$WORK_DIR/candidate-output CANDIDATE_HOME=$WORK_DIR/candidate-home diff --git a/tasks/task-2-3/tests/starter/README.md b/tasks/query-encoder-alignment/tests/starter/README.md similarity index 96% rename from tasks/task-2-3/tests/starter/README.md rename to tasks/query-encoder-alignment/tests/starter/README.md index ff78d43..8095c91 100644 --- a/tasks/task-2-3/tests/starter/README.md +++ b/tasks/query-encoder-alignment/tests/starter/README.md @@ -1,4 +1,4 @@ -# Task-2-3 starter +# Task query-encoder-alignment starter This starter is a teacher-side reference baseline. It encodes validation queries with the local Qwen3-Embedding-4B model, receives the frozen 2560-dimensional document vectors through `--doc-vectors`, and searches them with an in-memory FAISS inner-product index. The decoder-only embedder uses its supported `last_token` pooling method, BF16 inference, and the configured retrieval instruction format. diff --git a/tasks/task-2-3/tests/starter/run.sh b/tasks/query-encoder-alignment/tests/starter/run.sh similarity index 100% rename from tasks/task-2-3/tests/starter/run.sh rename to tasks/query-encoder-alignment/tests/starter/run.sh diff --git a/tasks/query-encoder-alignment/tests/starter/src/__init__.py b/tasks/query-encoder-alignment/tests/starter/src/__init__.py new file mode 100644 index 0000000..35181ae --- /dev/null +++ b/tasks/query-encoder-alignment/tests/starter/src/__init__.py @@ -0,0 +1 @@ +"""Task query-encoder-alignment starter implementation.""" diff --git a/tasks/task-2-3/tests/starter/src/common.py b/tasks/query-encoder-alignment/tests/starter/src/common.py similarity index 100% rename from tasks/task-2-3/tests/starter/src/common.py rename to tasks/query-encoder-alignment/tests/starter/src/common.py diff --git a/tasks/task-2-3/tests/starter/src/run_search.py b/tasks/query-encoder-alignment/tests/starter/src/run_search.py similarity index 100% rename from tasks/task-2-3/tests/starter/src/run_search.py rename to tasks/query-encoder-alignment/tests/starter/src/run_search.py diff --git a/tasks/task-2-3/tests/test.sh b/tasks/query-encoder-alignment/tests/test.sh similarity index 99% rename from tasks/task-2-3/tests/test.sh rename to tasks/query-encoder-alignment/tests/test.sh index 60d194c..8ec20d5 100644 --- a/tasks/task-2-3/tests/test.sh +++ b/tasks/query-encoder-alignment/tests/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -uo pipefail -TASK_ID=task-2-3 +TASK_ID=query-encoder-alignment EVALUATION_TIMEOUT_SECONDS=4200 JUDGE_TIMEOUT_SECONDS=1200 FINALIZER_TIMEOUT_SECONDS=60 diff --git a/tasks/task-2-3/.gitignore b/tasks/reasoning-query-rewriting/.gitignore similarity index 100% rename from tasks/task-2-3/.gitignore rename to tasks/reasoning-query-rewriting/.gitignore diff --git a/tasks/task-1-1/README.md b/tasks/reasoning-query-rewriting/README.md similarity index 92% rename from tasks/task-1-1/README.md rename to tasks/reasoning-query-rewriting/README.md index 75be27e..2b8e3f2 100644 --- a/tasks/task-1-1/README.md +++ b/tasks/reasoning-query-rewriting/README.md @@ -2,7 +2,7 @@ Build a retrieval system that connects difficult biology questions to supporting documents. -**Task:** `task-1-1` · **Mode:** Implementation · **Metric:** Accuracy@3 with per-query partial credit +**Task:** `reasoning-query-rewriting` · **Metric:** Accuracy@3 with per-query partial credit ## Overview @@ -108,16 +108,16 @@ task credential profile. The [asset guide](../../docs/assets.md) covers downloads, checksums, and cache options. Configure the verifier's integrity-judge credentials independently of the coding -agent. If the submission uses permitted APIs, task 1-1 uses -`TASK_1_1_OPENROUTER_API_KEY` and the shared `JINA_API_KEY`. +agent. If the submission uses permitted APIs, `reasoning-query-rewriting` uses +`REASONING_QUERY_REWRITING_OPENROUTER_API_KEY` and the shared `JINA_API_KEY`. ```bash -python scripts/download_assets.py --task task-1-1 -bash scripts/run_task.sh --task task-1-1 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task reasoning-query-rewriting +bash scripts/run_task.sh --task reasoning-query-rewriting --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-1-1/`. +writes results under `jobs/reasoning-query-rewriting/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-1-1/assets.json b/tasks/reasoning-query-rewriting/assets.json similarity index 67% rename from tasks/task-1-1/assets.json rename to tasks/reasoning-query-rewriting/assets.json index a2bab68..282738f 100644 --- a/tasks/task-1-1/assets.json +++ b/tasks/reasoning-query-rewriting/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-1/corpus.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/reasoning-query-rewriting/corpus.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-1/validation/ground_truth.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/reasoning-query-rewriting/validation/ground_truth.jsonl" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-1/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/reasoning-query-rewriting/validation/queries.jsonl" } } ] diff --git a/tasks/task-1-3/environment/Dockerfile b/tasks/reasoning-query-rewriting/environment/Dockerfile similarity index 100% rename from tasks/task-1-3/environment/Dockerfile rename to tasks/reasoning-query-rewriting/environment/Dockerfile diff --git a/tasks/task-1-1/environment/docker-compose.yaml b/tasks/reasoning-query-rewriting/environment/docker-compose.yaml similarity index 100% rename from tasks/task-1-1/environment/docker-compose.yaml rename to tasks/reasoning-query-rewriting/environment/docker-compose.yaml diff --git a/tasks/task-1-1/environment/docs/available_resources.md b/tasks/reasoning-query-rewriting/environment/docs/available_resources.md similarity index 100% rename from tasks/task-1-1/environment/docs/available_resources.md rename to tasks/reasoning-query-rewriting/environment/docs/available_resources.md diff --git a/tasks/task-2-1/environment/docs/environment.md b/tasks/reasoning-query-rewriting/environment/docs/environment.md similarity index 100% rename from tasks/task-2-1/environment/docs/environment.md rename to tasks/reasoning-query-rewriting/environment/docs/environment.md diff --git a/tasks/task-1-1/instruction.md b/tasks/reasoning-query-rewriting/instruction.md similarity index 100% rename from tasks/task-1-1/instruction.md rename to tasks/reasoning-query-rewriting/instruction.md diff --git a/tasks/task-1-1/raw-instruction.md b/tasks/reasoning-query-rewriting/raw-instruction.md similarity index 99% rename from tasks/task-1-1/raw-instruction.md rename to tasks/reasoning-query-rewriting/raw-instruction.md index 63baacd..ad0ce3b 100644 --- a/tasks/task-1-1/raw-instruction.md +++ b/tasks/reasoning-query-rewriting/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: 1-1 +# Reasoning-intensive Query Rewriting ## Task Description diff --git a/tasks/task-1-1/task.toml b/tasks/reasoning-query-rewriting/task.toml similarity index 88% rename from tasks/task-1-1/task.toml rename to tasks/reasoning-query-rewriting/task.toml index b7c78ae..93a1ae9 100644 --- a/tasks/task-1-1/task.toml +++ b/tasks/reasoning-query-rewriting/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-1-1" +name = "search-swe/reasoning-query-rewriting" version = "0.3.1" description = "Build a reasoning-assisted semantic retrieval system over Bright biology short documents." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "search", "biology", "BM25", "query-rewriting", "semantic-retrieval"] [metadata] -task_type = "create" dataset = "Bright biology short-document" primary_metric = "Accuracy@3" metric_k = 3 @@ -57,11 +56,11 @@ gpus = 0 os = "linux" [environment.env] -OPENROUTER_API_KEY = "${TASK_1_1_OPENROUTER_API_KEY:-}" +OPENROUTER_API_KEY = "${REASONING_QUERY_REWRITING_OPENROUTER_API_KEY:-}" JINA_API_KEY = "${JINA_API_KEY:-}" [verifier.env] -OPENROUTER_API_KEY = "${TASK_1_1_OPENROUTER_API_KEY:-}" +OPENROUTER_API_KEY = "${REASONING_QUERY_REWRITING_OPENROUTER_API_KEY:-}" JINA_API_KEY = "${JINA_API_KEY:-}" OPENAI_BASE_URL = "${VERIFIER_OPENAI_BASE_URL:-}" OPENAI_API_KEY = "${VERIFIER_OPENAI_API_KEY:-}" diff --git a/tasks/task-1-1/tests/.dockerignore b/tasks/reasoning-query-rewriting/tests/.dockerignore similarity index 100% rename from tasks/task-1-1/tests/.dockerignore rename to tasks/reasoning-query-rewriting/tests/.dockerignore diff --git a/tasks/task-1-4/tests/Dockerfile b/tasks/reasoning-query-rewriting/tests/Dockerfile similarity index 100% rename from tasks/task-1-4/tests/Dockerfile rename to tasks/reasoning-query-rewriting/tests/Dockerfile diff --git a/tasks/task-1-1/tests/data/ground_truth.jsonl b/tasks/reasoning-query-rewriting/tests/data/ground_truth.jsonl similarity index 100% rename from tasks/task-1-1/tests/data/ground_truth.jsonl rename to tasks/reasoning-query-rewriting/tests/data/ground_truth.jsonl diff --git a/tasks/task-1-1/tests/data/queries.jsonl b/tasks/reasoning-query-rewriting/tests/data/queries.jsonl similarity index 100% rename from tasks/task-1-1/tests/data/queries.jsonl rename to tasks/reasoning-query-rewriting/tests/data/queries.jsonl diff --git a/tasks/task-1-1/tests/docker-compose.yaml b/tasks/reasoning-query-rewriting/tests/docker-compose.yaml similarity index 100% rename from tasks/task-1-1/tests/docker-compose.yaml rename to tasks/reasoning-query-rewriting/tests/docker-compose.yaml diff --git a/tasks/task-1-1/tests/finalize_reward.py b/tasks/reasoning-query-rewriting/tests/finalize_reward.py similarity index 97% rename from tasks/task-1-1/tests/finalize_reward.py rename to tasks/reasoning-query-rewriting/tests/finalize_reward.py index 207ba5d..c746f27 100644 --- a/tasks/task-1-1/tests/finalize_reward.py +++ b/tasks/reasoning-query-rewriting/tests/finalize_reward.py @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -EVALUATION_PATH = REWARD_DIR / "task-1-1-eval" / "evaluation.json" +EVALUATION_PATH = REWARD_DIR / "reasoning-query-rewriting-eval" / "evaluation.json" JUDGE_REWARD_PATH = REWARD_DIR / "reward.json" DETAILS_PATH = REWARD_DIR / "reward-details.json" TRAJECTORY_PATH = Path("/logs/agent/trajectory.json") diff --git a/tasks/task-1-1/tests/grader.py b/tasks/reasoning-query-rewriting/tests/grader.py similarity index 98% rename from tasks/task-1-1/tests/grader.py rename to tasks/reasoning-query-rewriting/tests/grader.py index fd9e096..ed73df2 100644 --- a/tasks/task-1-1/tests/grader.py +++ b/tasks/reasoning-query-rewriting/tests/grader.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Private verifier for Task-1-1 Bright biology retrieval.""" +"""Private verifier for Task reasoning-query-rewriting Bright biology retrieval.""" from __future__ import annotations @@ -14,17 +14,17 @@ from typing import Any -TASK_ID = "task-1-1" +TASK_ID = "reasoning-query-rewriting" TASK_DATA = Path("/task/data") PRIVATE_DATA = Path("/tests/data") SUBMISSION_DIR = Path("/app") -WORK_DIR = Path("/tmp/task-1-1-eval") +WORK_DIR = Path("/tmp/reasoning-query-rewriting-eval") INPUT_DIR = WORK_DIR / "input" INDEX_DIR = WORK_DIR / "index" OUTPUT_DIR = WORK_DIR / "output" OUTPUT_PATH = OUTPUT_DIR / "results.jsonl" RESULTS_DIR = Path( - os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/task-1-1-eval") + os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/reasoning-query-rewriting-eval") ) REPORT_PATH = RESULTS_DIR / "evaluation.json" CORPUS_PATH = TASK_DATA / "corpus.jsonl" diff --git a/tasks/task-1-1/tests/jailbreak_judge/codex.toml b/tasks/reasoning-query-rewriting/tests/jailbreak_judge/codex.toml similarity index 100% rename from tasks/task-1-1/tests/jailbreak_judge/codex.toml rename to tasks/reasoning-query-rewriting/tests/jailbreak_judge/codex.toml diff --git a/tasks/task-2-2/tests/rewardkit_deepseek.py b/tasks/reasoning-query-rewriting/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-2-2/tests/rewardkit_deepseek.py rename to tasks/reasoning-query-rewriting/tests/rewardkit_deepseek.py diff --git a/tasks/task-1-1/tests/test.sh b/tasks/reasoning-query-rewriting/tests/test.sh similarity index 99% rename from tasks/task-1-1/tests/test.sh rename to tasks/reasoning-query-rewriting/tests/test.sh index 7c7977b..aabd86d 100644 --- a/tasks/task-1-1/tests/test.sh +++ b/tasks/reasoning-query-rewriting/tests/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -u -TASK_ID=task-1-1 +TASK_ID=reasoning-query-rewriting REWARD_DIR=/logs/verifier RESULTS_DIR=$REWARD_DIR/${TASK_ID}-eval REWARD_FILE=$REWARD_DIR/reward.txt diff --git a/tasks/task-2-4/.gitignore b/tasks/scientific-paper-qa/.gitignore similarity index 100% rename from tasks/task-2-4/.gitignore rename to tasks/scientific-paper-qa/.gitignore diff --git a/tasks/task-1-3/README.md b/tasks/scientific-paper-qa/README.md similarity index 94% rename from tasks/task-1-3/README.md rename to tasks/scientific-paper-qa/README.md index cf42367..086375b 100644 --- a/tasks/task-1-3/README.md +++ b/tasks/scientific-paper-qa/README.md @@ -2,7 +2,7 @@ Build a RAG system that answers scientific questions and identifies the supporting paper. -**Task:** `task-1-3` · **Mode:** Implementation · **Metric:** LLMJudgeAccuracy +**Task:** `scientific-paper-qa` · **Metric:** LLMJudgeAccuracy ## Overview @@ -34,7 +34,7 @@ the system can share its index and services reliably across concurrent questions Public assets are mounted under `/task/data`. The 10 held-out questions use the same PDF corpus, but their questions and labels are separated from the -agent environment. [Task 1-4](../task-1-4/README.md) instead evaluates page +agent environment. [Long-PDF evidence](../long-pdf-evidence/README.md) instead evaluates page localization within a single long PDF. ### Fixed Components and Allowed Changes @@ -111,12 +111,12 @@ use shared `OPENROUTER_API_KEY` and `JINA_API_KEY`. The evaluation guide explain the separate credential groups and their API requirements. ```bash -python scripts/download_assets.py --task task-1-3 -bash scripts/run_task.sh --task task-1-3 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task scientific-paper-qa +bash scripts/run_task.sh --task scientific-paper-qa --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-1-3/`. +writes results under `jobs/scientific-paper-qa/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-1-3/assets.json b/tasks/scientific-paper-qa/assets.json similarity index 67% rename from tasks/task-1-3/assets.json rename to tasks/scientific-paper-qa/assets.json index 779f27e..852dc0b 100644 --- a/tasks/task-1-3/assets.json +++ b/tasks/scientific-paper-qa/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/08ttQI197D.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/08ttQI197D.pdf" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/0U9YpcM1Bz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/0U9YpcM1Bz.pdf" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/0ppQHeP87n.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/0ppQHeP87n.pdf" } }, { @@ -41,8 +41,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/0v6MoRphvB.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/0v6MoRphvB.pdf" } }, { @@ -52,8 +52,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/0zBZjyfRiD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/0zBZjyfRiD.pdf" } }, { @@ -63,8 +63,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/1N2EdrPMxh.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/1N2EdrPMxh.pdf" } }, { @@ -74,8 +74,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/1gErSq67kj.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/1gErSq67kj.pdf" } }, { @@ -85,8 +85,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/1hG9QVyvW3.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/1hG9QVyvW3.pdf" } }, { @@ -96,8 +96,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/1qMaq7BR5v.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/1qMaq7BR5v.pdf" } }, { @@ -107,8 +107,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/2CFv9XeBPZ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/2CFv9XeBPZ.pdf" } }, { @@ -118,8 +118,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/2GCnI6c0So.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/2GCnI6c0So.pdf" } }, { @@ -129,8 +129,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/2LK7n00ise.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/2LK7n00ise.pdf" } }, { @@ -140,8 +140,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/2lERxQwQW9.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/2lERxQwQW9.pdf" } }, { @@ -151,8 +151,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/3KJjpt7yjf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/3KJjpt7yjf.pdf" } }, { @@ -162,8 +162,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/3rZzRQ1uwp.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/3rZzRQ1uwp.pdf" } }, { @@ -173,8 +173,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/3zASmt7GcA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/3zASmt7GcA.pdf" } }, { @@ -184,8 +184,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/454jks0Vee.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/454jks0Vee.pdf" } }, { @@ -195,8 +195,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/4A3upHp1cu.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/4A3upHp1cu.pdf" } }, { @@ -206,8 +206,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/4RxeiQp43N.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/4RxeiQp43N.pdf" } }, { @@ -217,8 +217,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/4Zb7NyRlH4.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/4Zb7NyRlH4.pdf" } }, { @@ -228,8 +228,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/4b1HwAMcEF.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/4b1HwAMcEF.pdf" } }, { @@ -239,8 +239,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/4qSkq7KRKa.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/4qSkq7KRKa.pdf" } }, { @@ -250,8 +250,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/50KJKv0h2G.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/50KJKv0h2G.pdf" } }, { @@ -261,8 +261,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/5B8Mhcnru5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/5B8Mhcnru5.pdf" } }, { @@ -272,8 +272,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/5L5O4E56Nt.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/5L5O4E56Nt.pdf" } }, { @@ -283,8 +283,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/5WhgjEcjAU.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/5WhgjEcjAU.pdf" } }, { @@ -294,8 +294,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/5gyT0Bl0Xn.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/5gyT0Bl0Xn.pdf" } }, { @@ -305,8 +305,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/5puFRCE2qh.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/5puFRCE2qh.pdf" } }, { @@ -316,8 +316,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/6CEF91fPtc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/6CEF91fPtc.pdf" } }, { @@ -327,8 +327,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/6CsZVuPWQH.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/6CsZVuPWQH.pdf" } }, { @@ -338,8 +338,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/6I6Paw5HXM.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/6I6Paw5HXM.pdf" } }, { @@ -349,8 +349,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/6YW0omrg4W.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/6YW0omrg4W.pdf" } }, { @@ -360,8 +360,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/6tbvTrJ1lX.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/6tbvTrJ1lX.pdf" } }, { @@ -371,8 +371,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/761tzSV3lf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/761tzSV3lf.pdf" } }, { @@ -382,8 +382,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/76nfaoxKa5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/76nfaoxKa5.pdf" } }, { @@ -393,8 +393,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/7UvnNpIPiv.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/7UvnNpIPiv.pdf" } }, { @@ -404,8 +404,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/7aZzsRT9Gu.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/7aZzsRT9Gu.pdf" } }, { @@ -415,8 +415,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/88DNlmVGIe.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/88DNlmVGIe.pdf" } }, { @@ -426,8 +426,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/8EP1ZI1G6q.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/8EP1ZI1G6q.pdf" } }, { @@ -437,8 +437,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/8SllZloSaK.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/8SllZloSaK.pdf" } }, { @@ -448,8 +448,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/8T5zA4I8Bv.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/8T5zA4I8Bv.pdf" } }, { @@ -459,8 +459,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/8r4MxJNYcr.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/8r4MxJNYcr.pdf" } }, { @@ -470,8 +470,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/8uviE6WOmM.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/8uviE6WOmM.pdf" } }, { @@ -481,8 +481,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/96m7AF7YPR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/96m7AF7YPR.pdf" } }, { @@ -492,8 +492,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/9HY69guXmZ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/9HY69guXmZ.pdf" } }, { @@ -503,8 +503,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/9X08emD9XE.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/9X08emD9XE.pdf" } }, { @@ -514,8 +514,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/9ZmHNzIrna.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/9ZmHNzIrna.pdf" } }, { @@ -525,8 +525,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/9kwAIosmTD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/9kwAIosmTD.pdf" } }, { @@ -536,8 +536,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/A8VIjyvHI7.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/A8VIjyvHI7.pdf" } }, { @@ -547,8 +547,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ASGjAnHIkl.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ASGjAnHIkl.pdf" } }, { @@ -558,8 +558,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Al8YqTg7s9.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Al8YqTg7s9.pdf" } }, { @@ -569,8 +569,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/AmMlpbV2ex.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/AmMlpbV2ex.pdf" } }, { @@ -580,8 +580,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/B5WvQYGV6l.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/B5WvQYGV6l.pdf" } }, { @@ -591,8 +591,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/BHWz254vNQ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/BHWz254vNQ.pdf" } }, { @@ -602,8 +602,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/BQFMVtKjuE.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/BQFMVtKjuE.pdf" } }, { @@ -613,8 +613,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/BUnmIrx3Ey.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/BUnmIrx3Ey.pdf" } }, { @@ -624,8 +624,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/BmQPh9dMgc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/BmQPh9dMgc.pdf" } }, { @@ -635,8 +635,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/C8R58rOy0f.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/C8R58rOy0f.pdf" } }, { @@ -646,8 +646,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/CELkQInPsZ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/CELkQInPsZ.pdf" } }, { @@ -657,8 +657,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/CFTy2f5Hjs.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/CFTy2f5Hjs.pdf" } }, { @@ -668,8 +668,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/CQNai524KD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/CQNai524KD.pdf" } }, { @@ -679,8 +679,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/CWqxpKpowp.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/CWqxpKpowp.pdf" } }, { @@ -690,8 +690,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Cb16GWBBDa.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Cb16GWBBDa.pdf" } }, { @@ -701,8 +701,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/CpErLshTyf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/CpErLshTyf.pdf" } }, { @@ -712,8 +712,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/CwXqx0dpB3.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/CwXqx0dpB3.pdf" } }, { @@ -723,8 +723,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/DDCILWcTHz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/DDCILWcTHz.pdf" } }, { @@ -734,8 +734,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/DcE9PakXDq.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/DcE9PakXDq.pdf" } }, { @@ -745,8 +745,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/DfQNJOjqnz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/DfQNJOjqnz.pdf" } }, { @@ -756,8 +756,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Dm6LA1T3VF.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Dm6LA1T3VF.pdf" } }, { @@ -767,8 +767,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/EKOZvAzV2J.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/EKOZvAzV2J.pdf" } }, { @@ -778,8 +778,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/EiM1uwF2Uf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/EiM1uwF2Uf.pdf" } }, { @@ -789,8 +789,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Ety4aaL8KU.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Ety4aaL8KU.pdf" } }, { @@ -800,8 +800,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/EzUz7f65LR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/EzUz7f65LR.pdf" } }, { @@ -811,8 +811,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/FTQzBku1Ix.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/FTQzBku1Ix.pdf" } }, { @@ -822,8 +822,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/FVgXkBWGi1.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/FVgXkBWGi1.pdf" } }, { @@ -833,8 +833,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/FaJhTHM9yw.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/FaJhTHM9yw.pdf" } }, { @@ -844,8 +844,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Fmao3PRvCJ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Fmao3PRvCJ.pdf" } }, { @@ -855,8 +855,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/FtOuhBinFK.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/FtOuhBinFK.pdf" } }, { @@ -866,8 +866,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/G3IuUTtHAX.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/G3IuUTtHAX.pdf" } }, { @@ -877,8 +877,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/G5A7pJTFAN.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/G5A7pJTFAN.pdf" } }, { @@ -888,8 +888,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/G6hhhkkP3b.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/G6hhhkkP3b.pdf" } }, { @@ -899,8 +899,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/GCcecXC4UE.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/GCcecXC4UE.pdf" } }, { @@ -910,8 +910,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/H51ZSLsAKW.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/H51ZSLsAKW.pdf" } }, { @@ -921,8 +921,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/HChQTN0hNN.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/HChQTN0hNN.pdf" } }, { @@ -932,8 +932,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/HJogHSHXgP.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/HJogHSHXgP.pdf" } }, { @@ -943,8 +943,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/He4OslDLxd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/He4OslDLxd.pdf" } }, { @@ -954,8 +954,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Hi3BBRsN2u.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Hi3BBRsN2u.pdf" } }, { @@ -965,8 +965,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Hito7CM2R7.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Hito7CM2R7.pdf" } }, { @@ -976,8 +976,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/HqN5NRxgci.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/HqN5NRxgci.pdf" } }, { @@ -987,8 +987,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/HrL7kzf8NJ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/HrL7kzf8NJ.pdf" } }, { @@ -998,8 +998,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Hz4bSS6oOh.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Hz4bSS6oOh.pdf" } }, { @@ -1009,8 +1009,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/I0EXfDwtN4.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/I0EXfDwtN4.pdf" } }, { @@ -1020,8 +1020,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/IBrC9M0Hkp.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/IBrC9M0Hkp.pdf" } }, { @@ -1031,8 +1031,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/IfDeSIk2Yn.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/IfDeSIk2Yn.pdf" } }, { @@ -1042,8 +1042,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/IfxNbxPJ0F.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/IfxNbxPJ0F.pdf" } }, { @@ -1053,8 +1053,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/IiJ8qS7ggJ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/IiJ8qS7ggJ.pdf" } }, { @@ -1064,8 +1064,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JEpiemBgQD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JEpiemBgQD.pdf" } }, { @@ -1075,8 +1075,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JO1LhroB8w.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JO1LhroB8w.pdf" } }, { @@ -1086,8 +1086,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JX8Zk08F6O.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JX8Zk08F6O.pdf" } }, { @@ -1097,8 +1097,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JZ7BHNdnFW.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JZ7BHNdnFW.pdf" } }, { @@ -1108,8 +1108,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JoK8TNEXtT.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JoK8TNEXtT.pdf" } }, { @@ -1119,8 +1119,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JqZstMzh2V.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JqZstMzh2V.pdf" } }, { @@ -1130,8 +1130,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/JuFMFzr0qF.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/JuFMFzr0qF.pdf" } }, { @@ -1141,8 +1141,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/K2ISwxHQNX.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/K2ISwxHQNX.pdf" } }, { @@ -1152,8 +1152,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/K8m2JUASVe.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/K8m2JUASVe.pdf" } }, { @@ -1163,8 +1163,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/KN1QYMWdqm.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/KN1QYMWdqm.pdf" } }, { @@ -1174,8 +1174,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/KR3hGefyeT.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/KR3hGefyeT.pdf" } }, { @@ -1185,8 +1185,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/KVc4dIz6cX.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/KVc4dIz6cX.pdf" } }, { @@ -1196,8 +1196,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/KzReM0BSKk.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/KzReM0BSKk.pdf" } }, { @@ -1207,8 +1207,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/LI49v3sIKZ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/LI49v3sIKZ.pdf" } }, { @@ -1218,8 +1218,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/LMOzg5Q9Ah.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/LMOzg5Q9Ah.pdf" } }, { @@ -1229,8 +1229,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/LOckV6YiBn.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/LOckV6YiBn.pdf" } }, { @@ -1240,8 +1240,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/LaFg4PZLFr.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/LaFg4PZLFr.pdf" } }, { @@ -1251,8 +1251,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/LcRoTxQZWd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/LcRoTxQZWd.pdf" } }, { @@ -1262,8 +1262,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/LpdOpxdq7w.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/LpdOpxdq7w.pdf" } }, { @@ -1273,8 +1273,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/MLjjNflmAy.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/MLjjNflmAy.pdf" } }, { @@ -1284,8 +1284,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/MpNRZ9chZJ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/MpNRZ9chZJ.pdf" } }, { @@ -1295,8 +1295,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/MtEM5OJwtd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/MtEM5OJwtd.pdf" } }, { @@ -1306,8 +1306,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Mu28391QFI.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Mu28391QFI.pdf" } }, { @@ -1317,8 +1317,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NAea44NKg4.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NAea44NKg4.pdf" } }, { @@ -1328,8 +1328,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NCcaaVQ8YT.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NCcaaVQ8YT.pdf" } }, { @@ -1339,8 +1339,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NN0d652Okt.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NN0d652Okt.pdf" } }, { @@ -1350,8 +1350,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NPUgtIKols.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NPUgtIKols.pdf" } }, { @@ -1361,8 +1361,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NWzi7oPYKR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NWzi7oPYKR.pdf" } }, { @@ -1372,8 +1372,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NczIIqEUM6.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NczIIqEUM6.pdf" } }, { @@ -1383,8 +1383,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NpuJFhOiqp.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NpuJFhOiqp.pdf" } }, { @@ -1394,8 +1394,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NsEgxMD6tt.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NsEgxMD6tt.pdf" } }, { @@ -1405,8 +1405,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Nu34TKeq4p.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Nu34TKeq4p.pdf" } }, { @@ -1416,8 +1416,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/NyzUef8T5Z.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/NyzUef8T5Z.pdf" } }, { @@ -1427,8 +1427,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/OPgW6aRVb5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/OPgW6aRVb5.pdf" } }, { @@ -1438,8 +1438,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/OWo5dTZefO.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/OWo5dTZefO.pdf" } }, { @@ -1449,8 +1449,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/OamNISzz7u.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/OamNISzz7u.pdf" } }, { @@ -1460,8 +1460,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/P1l28Izht6.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/P1l28Izht6.pdf" } }, { @@ -1471,8 +1471,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/PPEVPO9mVl.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/PPEVPO9mVl.pdf" } }, { @@ -1482,8 +1482,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Pb9gVSv6mU.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Pb9gVSv6mU.pdf" } }, { @@ -1493,8 +1493,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/PfMOZsj31C.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/PfMOZsj31C.pdf" } }, { @@ -1504,8 +1504,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/PpFLzBzByt.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/PpFLzBzByt.pdf" } }, { @@ -1515,8 +1515,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Q1Qw1GStXx.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Q1Qw1GStXx.pdf" } }, { @@ -1526,8 +1526,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/QUVxeXJRvA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/QUVxeXJRvA.pdf" } }, { @@ -1537,8 +1537,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/QfcUyqbjD2.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/QfcUyqbjD2.pdf" } }, { @@ -1548,8 +1548,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/QzmixmSdl0.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/QzmixmSdl0.pdf" } }, { @@ -1559,8 +1559,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/RbAyJmDSmx.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/RbAyJmDSmx.pdf" } }, { @@ -1570,8 +1570,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/S4G7yOFFWR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/S4G7yOFFWR.pdf" } }, { @@ -1581,8 +1581,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/S5dojL50DQ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/S5dojL50DQ.pdf" } }, { @@ -1592,8 +1592,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/SCDxFMF8Cq.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/SCDxFMF8Cq.pdf" } }, { @@ -1603,8 +1603,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/SD8TtWeUpR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/SD8TtWeUpR.pdf" } }, { @@ -1614,8 +1614,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/SLwxKZQP2e.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/SLwxKZQP2e.pdf" } }, { @@ -1625,8 +1625,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/SeI1vwRNwv.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/SeI1vwRNwv.pdf" } }, { @@ -1636,8 +1636,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/SfUjvCVEQk.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/SfUjvCVEQk.pdf" } }, { @@ -1647,8 +1647,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/SghmEJsXZA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/SghmEJsXZA.pdf" } }, { @@ -1658,8 +1658,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/TNZPOB6ONM.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/TNZPOB6ONM.pdf" } }, { @@ -1669,8 +1669,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Ttnp4YLQH6.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Ttnp4YLQH6.pdf" } }, { @@ -1680,8 +1680,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/TxmxdfNhJQ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/TxmxdfNhJQ.pdf" } }, { @@ -1691,8 +1691,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/U20zGrFdwQ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/U20zGrFdwQ.pdf" } }, { @@ -1702,8 +1702,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/U6Ai2deODN.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/U6Ai2deODN.pdf" } }, { @@ -1713,8 +1713,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/UMP8UOCPQ4.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/UMP8UOCPQ4.pdf" } }, { @@ -1724,8 +1724,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/UQiskZoVs1.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/UQiskZoVs1.pdf" } }, { @@ -1735,8 +1735,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/UaPwGU9LEf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/UaPwGU9LEf.pdf" } }, { @@ -1746,8 +1746,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/UjPZx32TqX.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/UjPZx32TqX.pdf" } }, { @@ -1757,8 +1757,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/UyQa0SeGN5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/UyQa0SeGN5.pdf" } }, { @@ -1768,8 +1768,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/VCbvUU6Pn2.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/VCbvUU6Pn2.pdf" } }, { @@ -1779,8 +1779,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/VfBQNaq51q.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/VfBQNaq51q.pdf" } }, { @@ -1790,8 +1790,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/VjR1Jzj7Aa.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/VjR1Jzj7Aa.pdf" } }, { @@ -1801,8 +1801,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/VlVfNk0RSM.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/VlVfNk0RSM.pdf" } }, { @@ -1812,8 +1812,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/VxPW6DMdtm.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/VxPW6DMdtm.pdf" } }, { @@ -1823,8 +1823,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/WEilv8VYHz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/WEilv8VYHz.pdf" } }, { @@ -1834,8 +1834,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/WTLqlkm76y.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/WTLqlkm76y.pdf" } }, { @@ -1845,8 +1845,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/WVK6sB7bVG.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/WVK6sB7bVG.pdf" } }, { @@ -1856,8 +1856,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/WiWK74NC4w.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/WiWK74NC4w.pdf" } }, { @@ -1867,8 +1867,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/WrXVv1tGcD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/WrXVv1tGcD.pdf" } }, { @@ -1878,8 +1878,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/XQsdjrmJJq.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/XQsdjrmJJq.pdf" } }, { @@ -1889,8 +1889,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/YbVAMlstmO.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/YbVAMlstmO.pdf" } }, { @@ -1900,8 +1900,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/YoleOBv5g9.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/YoleOBv5g9.pdf" } }, { @@ -1911,8 +1911,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/YqNQPCr2IU.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/YqNQPCr2IU.pdf" } }, { @@ -1922,8 +1922,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Ys1pLbGaUc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Ys1pLbGaUc.pdf" } }, { @@ -1933,8 +1933,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/Z4dTt54oNl.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/Z4dTt54oNl.pdf" } }, { @@ -1944,8 +1944,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ZEf8xs2d2i.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ZEf8xs2d2i.pdf" } }, { @@ -1955,8 +1955,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ZJXncdxUlC.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ZJXncdxUlC.pdf" } }, { @@ -1966,8 +1966,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ZbMEjPOetA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ZbMEjPOetA.pdf" } }, { @@ -1977,8 +1977,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ZdsW1iPozG.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ZdsW1iPozG.pdf" } }, { @@ -1988,8 +1988,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/a0GEDa3wel.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/a0GEDa3wel.pdf" } }, { @@ -1999,8 +1999,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/a5Cp0fdqFK.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/a5Cp0fdqFK.pdf" } }, { @@ -2010,8 +2010,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/aWBh3LGrAC.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/aWBh3LGrAC.pdf" } }, { @@ -2021,8 +2021,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/aYkRVvzoFd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/aYkRVvzoFd.pdf" } }, { @@ -2032,8 +2032,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/b0OrvE0g7s.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/b0OrvE0g7s.pdf" } }, { @@ -2043,8 +2043,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/bCdTlRYS2u.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/bCdTlRYS2u.pdf" } }, { @@ -2054,8 +2054,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/bMVb5dmX5G.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/bMVb5dmX5G.pdf" } }, { @@ -2065,8 +2065,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/bXGg45dCRy.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/bXGg45dCRy.pdf" } }, { @@ -2076,8 +2076,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/bZzI2gmrzm.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/bZzI2gmrzm.pdf" } }, { @@ -2087,8 +2087,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/blGFuC1FW4.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/blGFuC1FW4.pdf" } }, { @@ -2098,8 +2098,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/bt76htNG4E.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/bt76htNG4E.pdf" } }, { @@ -2109,8 +2109,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/c4nL9M9K8G.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/c4nL9M9K8G.pdf" } }, { @@ -2120,8 +2120,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/cC2tL8C8Id.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/cC2tL8C8Id.pdf" } }, { @@ -2131,8 +2131,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/cHh0n94TYl.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/cHh0n94TYl.pdf" } }, { @@ -2142,8 +2142,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/cOdmLpOviW.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/cOdmLpOviW.pdf" } }, { @@ -2153,8 +2153,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/czFtQmV6vN.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/czFtQmV6vN.pdf" } }, { @@ -2164,8 +2164,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/d4DkZ17zKd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/d4DkZ17zKd.pdf" } }, { @@ -2175,8 +2175,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/dN74GJLoPR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/dN74GJLoPR.pdf" } }, { @@ -2186,8 +2186,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/dVNHMjFsRe.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/dVNHMjFsRe.pdf" } }, { @@ -2197,8 +2197,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/diCej7A5RF.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/diCej7A5RF.pdf" } }, { @@ -2208,8 +2208,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/dyfqMULVwL.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/dyfqMULVwL.pdf" } }, { @@ -2219,8 +2219,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/e48zd7dPkp.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/e48zd7dPkp.pdf" } }, { @@ -2230,8 +2230,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/eE0kORov3b.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/eE0kORov3b.pdf" } }, { @@ -2241,8 +2241,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/eGHWPvehAP.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/eGHWPvehAP.pdf" } }, { @@ -2252,8 +2252,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/eK0cFVmH6l.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/eK0cFVmH6l.pdf" } }, { @@ -2263,8 +2263,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/edqxYtw0he.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/edqxYtw0he.pdf" } }, { @@ -2274,8 +2274,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ejAmhF4BXv.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ejAmhF4BXv.pdf" } }, { @@ -2285,8 +2285,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/f1g4vAzwee.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/f1g4vAzwee.pdf" } }, { @@ -2296,8 +2296,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/fU6BJ3UTHf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/fU6BJ3UTHf.pdf" } }, { @@ -2307,8 +2307,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ffvmkhUV3P.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ffvmkhUV3P.pdf" } }, { @@ -2318,8 +2318,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/fruA9kqFXv.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/fruA9kqFXv.pdf" } }, { @@ -2329,8 +2329,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/fvwpL5PARU.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/fvwpL5PARU.pdf" } }, { @@ -2340,8 +2340,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/fzb6bKSiVD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/fzb6bKSiVD.pdf" } }, { @@ -2351,8 +2351,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/gKxfdKmCgJ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/gKxfdKmCgJ.pdf" } }, { @@ -2362,8 +2362,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/gZPw9q08kn.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/gZPw9q08kn.pdf" } }, { @@ -2373,8 +2373,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/goH33JQfVq.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/goH33JQfVq.pdf" } }, { @@ -2384,8 +2384,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/gxiXGkKVPd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/gxiXGkKVPd.pdf" } }, { @@ -2395,8 +2395,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/h82LGBJebf.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/h82LGBJebf.pdf" } }, { @@ -2406,8 +2406,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/hAUiXDubZl.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/hAUiXDubZl.pdf" } }, { @@ -2417,8 +2417,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/hC1gbWXAz8.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/hC1gbWXAz8.pdf" } }, { @@ -2428,8 +2428,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/hCzHZb2UUK.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/hCzHZb2UUK.pdf" } }, { @@ -2439,8 +2439,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/hMUhEgmrTA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/hMUhEgmrTA.pdf" } }, { @@ -2450,8 +2450,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/hpqZUJ53T5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/hpqZUJ53T5.pdf" } }, { @@ -2461,8 +2461,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/htejUiJ6tZ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/htejUiJ6tZ.pdf" } }, { @@ -2472,8 +2472,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/imgCJXHKw4.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/imgCJXHKw4.pdf" } }, { @@ -2483,8 +2483,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/isON29o8IO.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/isON29o8IO.pdf" } }, { @@ -2494,8 +2494,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ixfeqZYdBW.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ixfeqZYdBW.pdf" } }, { @@ -2505,8 +2505,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/izn6l7hEeY.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/izn6l7hEeY.pdf" } }, { @@ -2516,8 +2516,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/jOBU1tXYra.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/jOBU1tXYra.pdf" } }, { @@ -2527,8 +2527,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/jR8tOtUXNc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/jR8tOtUXNc.pdf" } }, { @@ -2538,8 +2538,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/jTV6XPth76.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/jTV6XPth76.pdf" } }, { @@ -2549,8 +2549,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/k6VvnjIICL.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/k6VvnjIICL.pdf" } }, { @@ -2560,8 +2560,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/kNWTLVGMSO.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/kNWTLVGMSO.pdf" } }, { @@ -2571,8 +2571,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/kStCZrQWzE.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/kStCZrQWzE.pdf" } }, { @@ -2582,8 +2582,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/kmq2zfsggI.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/kmq2zfsggI.pdf" } }, { @@ -2593,8 +2593,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/l88gnbsMy2.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/l88gnbsMy2.pdf" } }, { @@ -2604,8 +2604,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/lM2ij0rwqU.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/lM2ij0rwqU.pdf" } }, { @@ -2615,8 +2615,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/llXnJqESjd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/llXnJqESjd.pdf" } }, { @@ -2626,8 +2626,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/lybOwGFK7y.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/lybOwGFK7y.pdf" } }, { @@ -2637,8 +2637,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/m5yptAaVCC.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/m5yptAaVCC.pdf" } }, { @@ -2648,8 +2648,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/m9NdfLpDdr.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/m9NdfLpDdr.pdf" } }, { @@ -2659,8 +2659,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/mBg7Ay7cgA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/mBg7Ay7cgA.pdf" } }, { @@ -2670,8 +2670,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/mTdlkHe3Hz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/mTdlkHe3Hz.pdf" } }, { @@ -2681,8 +2681,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/mjNmE6WWxS.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/mjNmE6WWxS.pdf" } }, { @@ -2692,8 +2692,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/n1uZvGIcko.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/n1uZvGIcko.pdf" } }, { @@ -2703,8 +2703,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/nDWc2eHmC1.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/nDWc2eHmC1.pdf" } }, { @@ -2714,8 +2714,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ndJKvferGA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ndJKvferGA.pdf" } }, { @@ -2725,8 +2725,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ndfdTrESWc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ndfdTrESWc.pdf" } }, { @@ -2736,8 +2736,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/oEBjEuOcrx.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/oEBjEuOcrx.pdf" } }, { @@ -2747,8 +2747,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/oKXWhv8qXm.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/oKXWhv8qXm.pdf" } }, { @@ -2758,8 +2758,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/oXd4jS9kY2.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/oXd4jS9kY2.pdf" } }, { @@ -2769,8 +2769,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/obm4Fp1qRg.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/obm4Fp1qRg.pdf" } }, { @@ -2780,8 +2780,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/ofatIdsz8s.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/ofatIdsz8s.pdf" } }, { @@ -2791,8 +2791,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/otlF21boKP.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/otlF21boKP.pdf" } }, { @@ -2802,8 +2802,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/pC742MC3Jc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/pC742MC3Jc.pdf" } }, { @@ -2813,8 +2813,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/piX2AnZo9J.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/piX2AnZo9J.pdf" } }, { @@ -2824,8 +2824,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/pijEJvoM2Y.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/pijEJvoM2Y.pdf" } }, { @@ -2835,8 +2835,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/pqz5ytFqnQ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/pqz5ytFqnQ.pdf" } }, { @@ -2846,8 +2846,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/q9y3M2wxtk.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/q9y3M2wxtk.pdf" } }, { @@ -2857,8 +2857,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/qkG7zOtJtZ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/qkG7zOtJtZ.pdf" } }, { @@ -2868,8 +2868,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/qmXvEvcWzz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/qmXvEvcWzz.pdf" } }, { @@ -2879,8 +2879,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/qq8QRBAdmP.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/qq8QRBAdmP.pdf" } }, { @@ -2890,8 +2890,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/qxS5WbhqKV.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/qxS5WbhqKV.pdf" } }, { @@ -2901,8 +2901,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/rNTfpHWVIx.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/rNTfpHWVIx.pdf" } }, { @@ -2912,8 +2912,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/s3Ncu4SVjI.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/s3Ncu4SVjI.pdf" } }, { @@ -2923,8 +2923,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/sE3RiEdZ83.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/sE3RiEdZ83.pdf" } }, { @@ -2934,8 +2934,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/sjA95F73NI.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/sjA95F73NI.pdf" } }, { @@ -2945,8 +2945,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/snejaXOT1f.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/snejaXOT1f.pdf" } }, { @@ -2956,8 +2956,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/tLTfYMCz9V.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/tLTfYMCz9V.pdf" } }, { @@ -2967,8 +2967,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/td1MP3YsoL.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/td1MP3YsoL.pdf" } }, { @@ -2978,8 +2978,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/thyJabPFdF.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/thyJabPFdF.pdf" } }, { @@ -2989,8 +2989,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/tklhvozQ5d.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/tklhvozQ5d.pdf" } }, { @@ -3000,8 +3000,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/uDL0Y1cI4D.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/uDL0Y1cI4D.pdf" } }, { @@ -3011,8 +3011,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/uaXMsfaSjW.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/uaXMsfaSjW.pdf" } }, { @@ -3022,8 +3022,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/vBbX6tkbnv.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/vBbX6tkbnv.pdf" } }, { @@ -3033,8 +3033,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/vdpVvU8Xft.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/vdpVvU8Xft.pdf" } }, { @@ -3044,8 +3044,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/viA1sAStQ5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/viA1sAStQ5.pdf" } }, { @@ -3055,8 +3055,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/viFMXjn6IR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/viFMXjn6IR.pdf" } }, { @@ -3066,8 +3066,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/vtII8dVYxR.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/vtII8dVYxR.pdf" } }, { @@ -3077,8 +3077,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/vwpIElLJRO.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/vwpIElLJRO.pdf" } }, { @@ -3088,8 +3088,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/wA1xQ76qI6.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/wA1xQ76qI6.pdf" } }, { @@ -3099,8 +3099,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/wAeQD7s0Wi.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/wAeQD7s0Wi.pdf" } }, { @@ -3110,8 +3110,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/wRVvSwPrtC.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/wRVvSwPrtC.pdf" } }, { @@ -3121,8 +3121,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/wonXbOq9bl.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/wonXbOq9bl.pdf" } }, { @@ -3132,8 +3132,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/wpLzxce3Uz.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/wpLzxce3Uz.pdf" } }, { @@ -3143,8 +3143,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/wvo4dhOmYx.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/wvo4dhOmYx.pdf" } }, { @@ -3154,8 +3154,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/x4H7YrlInt.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/x4H7YrlInt.pdf" } }, { @@ -3165,8 +3165,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/xE5ralu58Q.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/xE5ralu58Q.pdf" } }, { @@ -3176,8 +3176,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/xJosNGAXku.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/xJosNGAXku.pdf" } }, { @@ -3187,8 +3187,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/xQfgWuW5Iw.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/xQfgWuW5Iw.pdf" } }, { @@ -3198,8 +3198,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/xlcBvydUxJ.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/xlcBvydUxJ.pdf" } }, { @@ -3209,8 +3209,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/y9UQgHDJ0h.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/y9UQgHDJ0h.pdf" } }, { @@ -3220,8 +3220,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/yUVfF9gCzA.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/yUVfF9gCzA.pdf" } }, { @@ -3231,8 +3231,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/yZ2MpQ3Pkk.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/yZ2MpQ3Pkk.pdf" } }, { @@ -3242,8 +3242,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/yrKHxB1rRD.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/yrKHxB1rRD.pdf" } }, { @@ -3253,8 +3253,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/yxLx6PXrb5.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/yxLx6PXrb5.pdf" } }, { @@ -3264,8 +3264,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/z5Ypdhntrq.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/z5Ypdhntrq.pdf" } }, { @@ -3275,8 +3275,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/z7rn4ErtFd.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/z7rn4ErtFd.pdf" } }, { @@ -3286,8 +3286,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/zmaxTOkhWr.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/zmaxTOkhWr.pdf" } }, { @@ -3297,8 +3297,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "77386ea1bfcf0a88c0f557d0170efd2dd9850980", - "filename": "tasks/task-1-3/corpus/zrcPkiWZQc.pdf" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/corpus/zrcPkiWZQc.pdf" } }, { @@ -3308,8 +3308,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-3/validation/golden_answers.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/validation/golden_answers.jsonl" } }, { @@ -3319,8 +3319,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "03bcab775c0b733a7115eec019ecbc3aeafac885", - "filename": "tasks/task-1-3/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/scientific-paper-qa/validation/queries.jsonl" } } ] diff --git a/tasks/task-1-4/environment/Dockerfile b/tasks/scientific-paper-qa/environment/Dockerfile similarity index 100% rename from tasks/task-1-4/environment/Dockerfile rename to tasks/scientific-paper-qa/environment/Dockerfile diff --git a/tasks/task-2-4/environment/docker-compose.yaml b/tasks/scientific-paper-qa/environment/docker-compose.yaml similarity index 100% rename from tasks/task-2-4/environment/docker-compose.yaml rename to tasks/scientific-paper-qa/environment/docker-compose.yaml diff --git a/tasks/task-1-3/environment/docs/available_resources.md b/tasks/scientific-paper-qa/environment/docs/available_resources.md similarity index 100% rename from tasks/task-1-3/environment/docs/available_resources.md rename to tasks/scientific-paper-qa/environment/docs/available_resources.md diff --git a/tasks/task-2-4/environment/docs/environment.md b/tasks/scientific-paper-qa/environment/docs/environment.md similarity index 100% rename from tasks/task-2-4/environment/docs/environment.md rename to tasks/scientific-paper-qa/environment/docs/environment.md diff --git a/tasks/task-1-3/instruction.md b/tasks/scientific-paper-qa/instruction.md similarity index 100% rename from tasks/task-1-3/instruction.md rename to tasks/scientific-paper-qa/instruction.md diff --git a/tasks/task-1-3/raw-instruction.md b/tasks/scientific-paper-qa/raw-instruction.md similarity index 99% rename from tasks/task-1-3/raw-instruction.md rename to tasks/scientific-paper-qa/raw-instruction.md index 88d50c4..1f7e0c1 100644 --- a/tasks/task-1-3/raw-instruction.md +++ b/tasks/scientific-paper-qa/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: Task-1-3 +# Scientific-Paper Question Answering ## Task Description diff --git a/tasks/task-1-3/task.toml b/tasks/scientific-paper-qa/task.toml similarity index 96% rename from tasks/task-1-3/task.toml rename to tasks/scientific-paper-qa/task.toml index d5ff07b..279b586 100644 --- a/tasks/task-1-3/task.toml +++ b/tasks/scientific-paper-qa/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-1-3" +name = "search-swe/scientific-paper-qa" version = "0.3.0" description = "Build an evidence-grounded RAG system over a local collection of research-paper PDFs." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "search", "RAG"] [metadata] -task_type = "create" dataset = "research-paper PDFs" primary_metric = "LLMJudgeAccuracy" public_example_count = 10 diff --git a/tasks/task-1-4/tests/.dockerignore b/tasks/scientific-paper-qa/tests/.dockerignore similarity index 100% rename from tasks/task-1-4/tests/.dockerignore rename to tasks/scientific-paper-qa/tests/.dockerignore diff --git a/tasks/task-2-1/tests/Dockerfile b/tasks/scientific-paper-qa/tests/Dockerfile similarity index 100% rename from tasks/task-2-1/tests/Dockerfile rename to tasks/scientific-paper-qa/tests/Dockerfile diff --git a/tasks/task-1-4/tests/configure_trajectory_judge.py b/tasks/scientific-paper-qa/tests/configure_trajectory_judge.py similarity index 100% rename from tasks/task-1-4/tests/configure_trajectory_judge.py rename to tasks/scientific-paper-qa/tests/configure_trajectory_judge.py diff --git a/tasks/task-1-3/tests/data/golden_answers.jsonl b/tasks/scientific-paper-qa/tests/data/golden_answers.jsonl similarity index 100% rename from tasks/task-1-3/tests/data/golden_answers.jsonl rename to tasks/scientific-paper-qa/tests/data/golden_answers.jsonl diff --git a/tasks/task-1-3/tests/data/queries.jsonl b/tasks/scientific-paper-qa/tests/data/queries.jsonl similarity index 100% rename from tasks/task-1-3/tests/data/queries.jsonl rename to tasks/scientific-paper-qa/tests/data/queries.jsonl diff --git a/tasks/task-1-3/tests/docker-compose.yaml b/tasks/scientific-paper-qa/tests/docker-compose.yaml similarity index 100% rename from tasks/task-1-3/tests/docker-compose.yaml rename to tasks/scientific-paper-qa/tests/docker-compose.yaml diff --git a/tasks/task-1-3/tests/finalize_reward.py b/tasks/scientific-paper-qa/tests/finalize_reward.py similarity index 94% rename from tasks/task-1-3/tests/finalize_reward.py rename to tasks/scientific-paper-qa/tests/finalize_reward.py index d6b893a..f7708f8 100644 --- a/tasks/task-1-3/tests/finalize_reward.py +++ b/tasks/scientific-paper-qa/tests/finalize_reward.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Apply the trajectory anti-jailbreak gate to Task-1-3's judge score.""" +"""Apply the trajectory anti-jailbreak gate to Task scientific-paper-qa's judge score.""" from __future__ import annotations @@ -11,7 +11,7 @@ REWARD_DIR = Path("/logs/verifier") -EVALUATION_PATH = Path("/logs/verifier/task-1-3-eval/evaluation.json") +EVALUATION_PATH = Path("/logs/verifier/scientific-paper-qa-eval/evaluation.json") JUDGE_REWARD_PATH = REWARD_DIR / "reward.json" DETAILS_PATH = REWARD_DIR / "reward-details.json" TRAJECTORY_PATH = Path("/logs/agent/trajectory.json") diff --git a/tasks/task-1-3/tests/grader.py b/tasks/scientific-paper-qa/tests/grader.py similarity index 98% rename from tasks/task-1-3/tests/grader.py rename to tasks/scientific-paper-qa/tests/grader.py index d276194..af69dcd 100644 --- a/tasks/task-1-3/tests/grader.py +++ b/tasks/scientific-paper-qa/tests/grader.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Validate Task-1-3 output and compute evidence-gated LLM-judge accuracy.""" +"""Validate Task scientific-paper-qa output and compute evidence-gated LLM-judge accuracy.""" from __future__ import annotations @@ -19,7 +19,7 @@ QUERIES_PATH = Path("/tests/data/queries.jsonl") GROUND_TRUTH_PATH = Path("/tests/data/golden_answers.jsonl") RESULTS_DIR = Path( - os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/task-1-3-eval") + os.environ.get("SEARCH_SWE_RESULTS_DIR", "/logs/verifier/scientific-paper-qa-eval") ) PREDICTIONS_PATH = RESULTS_DIR / "results.jsonl" REPORT_PATH = RESULTS_DIR / "evaluation.json" @@ -309,7 +309,7 @@ def main() -> int: except Exception as error: write_report( { - "task_id": "task-1-3", + "task_id": "scientific-paper-qa", "valid": False, "score": 0.0, "query_count": 0, @@ -332,7 +332,7 @@ def main() -> int: except Exception as error: write_report( { - "task_id": "task-1-3", + "task_id": "scientific-paper-qa", "valid": False, "score": 0.0, "query_count": len(queries), @@ -419,7 +419,7 @@ def judge_one(row: dict[str, str]) -> tuple[str, dict[str, Any]]: accuracy = correct_count / len(per_query) evidence_accuracy = evidence_count / len(per_query) report = { - "task_id": "task-1-3", + "task_id": "scientific-paper-qa", "valid": True, "score": 100.0 * accuracy, "query_count": len(per_query), diff --git a/tasks/task-1-3/tests/jailbreak_judge/codex.toml b/tasks/scientific-paper-qa/tests/jailbreak_judge/codex.toml similarity index 95% rename from tasks/task-1-3/tests/jailbreak_judge/codex.toml rename to tasks/scientific-paper-qa/tests/jailbreak_judge/codex.toml index cb043f6..3179e51 100644 --- a/tasks/task-1-3/tests/jailbreak_judge/codex.toml +++ b/tasks/scientific-paper-qa/tests/jailbreak_judge/codex.toml @@ -10,7 +10,7 @@ atif-trajectory = "/logs/agent/trajectory.json" name = "jailbreak_judge" type = "binary" description = """ -Review the Task-1-3 Agent's complete execution trajectory and submitted files to decide +Review the Task scientific-paper-qa Agent's complete execution trajectory and submitted files to decide whether it circumvented this local-PDF RAG task. Return `no` only when there is clear evidence of a violation; otherwise return `yes`. diff --git a/tasks/task-2-3/tests/rewardkit_deepseek.py b/tasks/scientific-paper-qa/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-2-3/tests/rewardkit_deepseek.py rename to tasks/scientific-paper-qa/tests/rewardkit_deepseek.py diff --git a/tasks/task-1-3/tests/run_queries.py b/tasks/scientific-paper-qa/tests/run_queries.py similarity index 100% rename from tasks/task-1-3/tests/run_queries.py rename to tasks/scientific-paper-qa/tests/run_queries.py diff --git a/tasks/task-1-3/tests/test.sh b/tasks/scientific-paper-qa/tests/test.sh similarity index 99% rename from tasks/task-1-3/tests/test.sh rename to tasks/scientific-paper-qa/tests/test.sh index a36ffd3..5d7ace0 100755 --- a/tasks/task-1-3/tests/test.sh +++ b/tasks/scientific-paper-qa/tests/test.sh @@ -1,13 +1,13 @@ #!/usr/bin/env bash set -u -WORK_DIR=/tmp/task-1-3-eval +WORK_DIR=/tmp/scientific-paper-qa-eval INPUT_DIR=$WORK_DIR/input INDEX_DIR=$WORK_DIR/index SUBMISSION_OUTPUT_DIR=$WORK_DIR/output STAGED_QUERIES=$INPUT_DIR/queries.jsonl SUBMISSION_OUTPUT=$SUBMISSION_OUTPUT_DIR/results.jsonl -RESULTS_DIR=/logs/verifier/task-1-3-eval +RESULTS_DIR=/logs/verifier/scientific-paper-qa-eval OUTPUT_PATH=$RESULTS_DIR/results.jsonl BUILD_TIMEOUT_SECONDS=3600 RUN_TIMEOUT_SECONDS=4800 diff --git a/tasks/task-1-3/tests/test_run_queries.py b/tasks/scientific-paper-qa/tests/test_run_queries.py similarity index 100% rename from tasks/task-1-3/tests/test_run_queries.py rename to tasks/scientific-paper-qa/tests/test_run_queries.py diff --git a/tasks/task-2-5/README.md b/tasks/sparse-retrieval-pruning/README.md similarity index 81% rename from tasks/task-2-5/README.md rename to tasks/sparse-retrieval-pruning/README.md index 91cd3a8..18b3d93 100644 --- a/tasks/task-2-5/README.md +++ b/tasks/sparse-retrieval-pruning/README.md @@ -1,8 +1,8 @@ -# Sparse Retrieval Postings Pruning +# Sparse Retrieval Pruning Optimize a sparse retrieval system while preserving strict quality requirements. -**Task:** `task-2-5` · **Mode:** Optimization · **Metric:** Quality-gated linear latency reward +**Task:** `sparse-retrieval-pruning` · **Metric:** Quality-gated linear latency reward ## Overview @@ -11,9 +11,9 @@ posting lists fit together. A compact index is not useful if remapped terms point to the wrong postings, and aggressive pruning is not useful if it removes relevant results. -The current task packages one million MS MARCO documents as sets of opaque, +The task packages one million MS MARCO documents as sets of opaque, unweighted term IDs derived from frozen sparse representations. It evaluates -a Python implementation against a corrected, unpruned starter. +a Python implementation against the unpruned reference starter. The goal is to preserve strong retrieval quality while reducing total query wall time relative to that reference. Quality thresholds are hard gates; passing submissions receive a latency reward that decreases linearly between @@ -34,17 +34,16 @@ passing submissions receive a latency reward that decreases linearly between | --- | --- | | `data/corpus.jsonl` | One million documents represented by opaque term sets | | `data/validation/` | 200 public queries, qrels, statistics, and reference Top-100 results | -| `environment/starter/` | Corrected retrieval starter | +| `environment/starter/` | Unpruned retrieval starter | | `environment/docs/index_format.md` | Starter dictionary, posting, and metadata layout | The verifier has 1,000 hidden queries, relevance labels, and reference data, -isolated from the agent environment. It runs its own corrected starter when -measuring the runtime baseline. +isolated from the agent environment. It runs its own unpruned reference starter +when measuring the runtime baseline. -The public assets now have an immutable dataset revision in [assets.json](assets.json). -The old README's pending-publication note no longer applies. Downloading uses -the same size and SHA-256 checks as other tasks. Their Hugging Face source -paths are under `tasks/task-2-5/`. +The public assets are pinned to an immutable dataset revision in +[assets.json](assets.json). Downloading verifies file sizes and SHA-256 checksums. +Their Hugging Face source paths are under `tasks/sparse-retrieval-pruning/`. ### Fixed Components and Allowed Changes @@ -91,8 +90,8 @@ minimums are **0.89** and **0.99** respectively. ### Correctness and Latency Reward -The verifier measures the candidate and corrected starter on the same workload -and resource allocation. A latency ratio at or below **0.25** receives reward +The verifier measures the candidate and unpruned reference starter on the same +workload and resource allocation. A latency ratio at or below **0.25** receives reward `1`; a ratio at or above **0.35** receives reward `0`; values strictly between those boundaries receive `(0.35 - ratio) / 0.10`. Successful build/run, valid outputs, and implementation restrictions remain hard requirements. @@ -101,8 +100,7 @@ outputs, and implementation restrictions remain hard requirements. If either quality threshold, output and execution checks, or the trajectory audit fails, final reward is `0`. Otherwise the final reward is the latency -reward described above. The corrected starter is a measured runtime reference, -not a fixed published timing that submissions can assume on every machine. +reward described above. The verifier measures reference wall time afresh for each run. The audit uses `deepseek/deepseek-v4.1-flash` via OpenRouter through pinned RewardKit 0.2.0 with verifier-only DeepSeek endpoint and key settings. @@ -116,16 +114,16 @@ task credential profile. The Configure the verifier's integrity-judge credentials independently of the coding agent. The submission itself does not need task-resource API keys or a model -download. The standard asset command now restores the published corpus and +download. The standard asset command restores the published corpus and validation files directly. ```bash -python scripts/download_assets.py --task task-2-5 -bash scripts/run_task.sh --task task-2-5 --model "YOUR_AGENT_MODEL" +python scripts/download_assets.py --task sparse-retrieval-pruning +bash scripts/run_task.sh --task sparse-retrieval-pruning --model "YOUR_AGENT_MODEL" ``` The shared launcher defaults to Codex, also supports Pi and Claude Code, and -writes results under `jobs/task-2-5/`. +writes results under `jobs/sparse-retrieval-pruning/`. Replace `YOUR_AGENT_MODEL` with your configured model. Add `--dry-run` to inspect command construction without starting an evaluation; this does not validate assets, credentials, or hardware. diff --git a/tasks/task-2-5/assets.json b/tasks/sparse-retrieval-pruning/assets.json similarity index 66% rename from tasks/task-2-5/assets.json rename to tasks/sparse-retrieval-pruning/assets.json index 1e87f87..bb23891 100644 --- a/tasks/task-2-5/assets.json +++ b/tasks/sparse-retrieval-pruning/assets.json @@ -8,8 +8,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "b1bf8b61563f6cf7b0eea0bf0d98b1da6aaea34f", - "filename": "tasks/task-2-5/corpus.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/sparse-retrieval-pruning/corpus.jsonl" } }, { @@ -19,8 +19,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "b1bf8b61563f6cf7b0eea0bf0d98b1da6aaea34f", - "filename": "tasks/task-2-5/validation/queries.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/sparse-retrieval-pruning/validation/queries.jsonl" } }, { @@ -30,8 +30,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "b1bf8b61563f6cf7b0eea0bf0d98b1da6aaea34f", - "filename": "tasks/task-2-5/validation/qrels.tsv" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/sparse-retrieval-pruning/validation/qrels.tsv" } }, { @@ -41,8 +41,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "b1bf8b61563f6cf7b0eea0bf0d98b1da6aaea34f", - "filename": "tasks/task-2-5/validation/stats.json" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/sparse-retrieval-pruning/validation/stats.json" } }, { @@ -52,8 +52,8 @@ "source": { "repo_id": "search-swe/Search-SWE", "repo_type": "dataset", - "revision": "b1bf8b61563f6cf7b0eea0bf0d98b1da6aaea34f", - "filename": "tasks/task-2-5/validation/reference_top100.jsonl" + "revision": "6713b1eb7474e0768e906a0151c76af92f0cfcdd", + "filename": "tasks/sparse-retrieval-pruning/validation/reference_top100.jsonl" } } ] diff --git a/tasks/task-2-5/environment/Dockerfile b/tasks/sparse-retrieval-pruning/environment/Dockerfile similarity index 100% rename from tasks/task-2-5/environment/Dockerfile rename to tasks/sparse-retrieval-pruning/environment/Dockerfile diff --git a/tasks/task-2-5/environment/docker-compose.yaml b/tasks/sparse-retrieval-pruning/environment/docker-compose.yaml similarity index 100% rename from tasks/task-2-5/environment/docker-compose.yaml rename to tasks/sparse-retrieval-pruning/environment/docker-compose.yaml diff --git a/tasks/task-2-5/environment/docs/available_resources.md b/tasks/sparse-retrieval-pruning/environment/docs/available_resources.md similarity index 100% rename from tasks/task-2-5/environment/docs/available_resources.md rename to tasks/sparse-retrieval-pruning/environment/docs/available_resources.md diff --git a/tasks/task-2-5/environment/docs/environment.md b/tasks/sparse-retrieval-pruning/environment/docs/environment.md similarity index 89% rename from tasks/task-2-5/environment/docs/environment.md rename to tasks/sparse-retrieval-pruning/environment/docs/environment.md index f833546..4d9ae01 100644 --- a/tasks/task-2-5/environment/docs/environment.md +++ b/tasks/sparse-retrieval-pruning/environment/docs/environment.md @@ -1,4 +1,4 @@ -# Task-2-5 Environment +# Task sparse-retrieval-pruning Environment The task runs in a prebuilt CPU-only container. The submission works in `/app`; task data and documentation are mounted read-only under `/task/data` and @@ -17,7 +17,7 @@ Submission indexing and retrieval logic must be implemented in Python. Bash is allowed only as a thin entry-point wrapper around Python. Native submission source and compiled native submission executables are not allowed. -The verifier runs the corrected starter as the latency baseline. The starter +The verifier runs the unpruned reference starter as the latency baseline. The starter retains all corpus postings and performs no default DF, prefix, or other posting-list pruning. Candidate wall time is compared with the starter wall time measured by the verifier under the same workload and resource allocation. diff --git a/tasks/task-2-5/environment/docs/index_format.md b/tasks/sparse-retrieval-pruning/environment/docs/index_format.md similarity index 97% rename from tasks/task-2-5/environment/docs/index_format.md rename to tasks/sparse-retrieval-pruning/environment/docs/index_format.md index 72a06d5..98c9e34 100644 --- a/tasks/task-2-5/environment/docs/index_format.md +++ b/tasks/sparse-retrieval-pruning/environment/docs/index_format.md @@ -91,7 +91,7 @@ The starter builds the index with bounded-memory temporary term buckets: 5. append each term's document numbers and metadata; and 6. write the dictionary and manifest. -The corrected starter retains every term and every posting by default. It does +The unpruned reference starter retains every term and every posting by default. It does not perform DF pruning, prefix truncation, upper-bound pruning, candidate pruning, or block skipping when used as the verifier baseline. Its search adds the term IDF contribution for every matching posting and ranks by descending diff --git a/tasks/task-2-5/environment/starter/build.sh b/tasks/sparse-retrieval-pruning/environment/starter/build.sh similarity index 100% rename from tasks/task-2-5/environment/starter/build.sh rename to tasks/sparse-retrieval-pruning/environment/starter/build.sh diff --git a/tasks/task-2-5/environment/starter/run.sh b/tasks/sparse-retrieval-pruning/environment/starter/run.sh similarity index 100% rename from tasks/task-2-5/environment/starter/run.sh rename to tasks/sparse-retrieval-pruning/environment/starter/run.sh diff --git a/tasks/task-2-5/environment/starter/src/sparse_retrieval.py b/tasks/sparse-retrieval-pruning/environment/starter/src/sparse_retrieval.py similarity index 100% rename from tasks/task-2-5/environment/starter/src/sparse_retrieval.py rename to tasks/sparse-retrieval-pruning/environment/starter/src/sparse_retrieval.py diff --git a/tasks/task-2-5/instruction.md b/tasks/sparse-retrieval-pruning/instruction.md similarity index 86% rename from tasks/task-2-5/instruction.md rename to tasks/sparse-retrieval-pruning/instruction.md index 8174266..f8e32e2 100644 --- a/tasks/task-2-5/instruction.md +++ b/tasks/sparse-retrieval-pruning/instruction.md @@ -1,6 +1,6 @@ Optimize the supplied sparse retrieval system over its large corpus. Each document and query is represented by a set of opaque term IDs. The system must return relevant corpus documents while reducing total search wall time for the complete query workload. You may replace the starter index or retrieval algorithm as long as the executable interface and output contract are preserved. -Quality comes first in this comparison. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the corrected, unpruned starter wall time measured by the verifier in the same environment. +Quality comes first in this comparison. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the unpruned reference starter wall time measured by the verifier in the same environment. Please read /task/docs/environment.md for the installed runtime and resource limits, /task/docs/available_resources.md for the local dependency policy, and /task/docs/index_format.md before changing the starter index. That format is an implementation detail and may be replaced. All of /task is read-only, and submission files must only be created or modified under /app. The agent phase allows up to 120 minutes. Do not use precomputed hidden query-to-document mappings, hidden relevance judgments, or external datasets or services containing evaluation answers. The formal build and search run offline and must not need credentials, remote models, package downloads or external services. @@ -14,8 +14,8 @@ Provide executable /app/build.sh and /app/run.sh files in the final submission. Once the build succeeds, the search interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Required runtime state must come from the supplied index directory and query file. Process each supplied query exactly once, preserve input query order, and write one result object per query. A JSONL output line looks like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. query_id must identify its corresponding input query. Return at most 100 results, with unique doc_id values that occur in the supplied corpus and scores that are finite and numeric. Order results by descending score, resolving equal scores by ascending lexicographic order of the opaque document ID. Fewer than 100 results is valid, though it can reduce Recall@100. -The latency baseline used by the verifier is the corrected, unpruned starter. It retains every corpus term and posting, applies no DF, prefix or other posting-list pruning by default, uses the set-valued corpus and query inputs without weights, builds a consistent dictionary, metadata and posting layout, and performs unpruned binary/IDF-overlap search. Any optional pruning flag in starter code is only for explicit local experiments and is not enabled for the verifier's baseline. +The latency baseline used by the verifier is the unpruned reference starter. It retains every corpus term and posting, applies no DF, prefix or other posting-list pruning by default, uses the set-valued corpus and query inputs without weights, builds a consistent dictionary, metadata and posting layout, and performs unpruned binary/IDF-overlap search. Any optional pruning flag in starter code is only for explicit local experiments and is not enabled for the verifier's baseline. -For evaluation, the verifier rebuilds your submission from the full corpus and runs it on a private query split. It checks retrieval integrity, executable behavior, valid output, retrieval quality and search wall time. Hidden results must achieve NDCG@10 of at least 0.89 and qrels Recall@100 of at least 0.99. Candidate and corrected-starter wall time are measured by external process wrappers under the same workload and resource allocation. The latency ratio is candidate wall time divided by that verifier-measured starter wall time. +For evaluation, the verifier rebuilds your submission from the full corpus and runs it on a private query split. It checks retrieval integrity, executable behavior, valid output, retrieval quality and search wall time. Hidden results must achieve NDCG@10 of at least 0.89 and qrels Recall@100 of at least 0.99. Candidate and reference wall time are measured by external process wrappers under the same workload and resource allocation. The latency ratio is candidate wall time divided by that verifier-measured starter wall time. NDCG and Recall are hard gates: failure of either one, build/run behavior, output validation or the trajectory jailbreak judge gives reward 0. Otherwise a latency ratio at or below 0.25 earns reward 1; strictly between 0.25 and 0.35 it earns (0.35 - latency ratio) / 0.10; and at or above 0.35 it earns 0. Submission-reported lookup counters and internal timings are diagnostics only and cannot replace the verifier's wall-time measurement. diff --git a/tasks/task-2-5/raw-instruction.md b/tasks/sparse-retrieval-pruning/raw-instruction.md similarity index 95% rename from tasks/task-2-5/raw-instruction.md rename to tasks/sparse-retrieval-pruning/raw-instruction.md index 22fa856..58c9b27 100644 --- a/tasks/task-2-5/raw-instruction.md +++ b/tasks/sparse-retrieval-pruning/raw-instruction.md @@ -1,4 +1,4 @@ -# Task: Task-2-5 +# Sparse Retrieval Pruning ## Task Description @@ -9,7 +9,7 @@ wall time low. The submission must satisfy the held-out retrieval-quality requirements. Its reward then increases as the wall time for the complete hidden query set falls -from 35% to 25% of the corrected, unpruned starter wall time measured by the +from 35% to 25% of the unpruned reference starter wall time measured by the verifier in the same environment. You may replace the starter index or retrieval algorithm as long as the executable interface and output contract are preserved. @@ -137,9 +137,9 @@ The finalized submission must contain executable `build.sh` and `run.sh` under All required persistent search artifacts must be written beneath the `--index-dir` supplied to `build.sh`. -## Corrected starter baseline +## Unpruned reference baseline -The verifier uses a corrected, unpruned starter as the latency baseline. It: +The verifier uses the unpruned reference starter as the latency baseline. It: - retains every corpus term and every posting; - does not apply DF, prefix, or other posting-list pruning by default; @@ -163,7 +163,7 @@ NDCG@10 >= 0.89 qrels Recall@100 >= 0.99 ``` -The verifier measures candidate and corrected-starter wall time with external +The verifier measures candidate and reference wall time with external process wrappers under the same workload and resource allocation. Define: ```text diff --git a/tasks/task-2-5/task.toml b/tasks/sparse-retrieval-pruning/task.toml similarity index 95% rename from tasks/task-2-5/task.toml rename to tasks/sparse-retrieval-pruning/task.toml index ad3662a..c6ea2e6 100644 --- a/tasks/task-2-5/task.toml +++ b/tasks/sparse-retrieval-pruning/task.toml @@ -6,14 +6,13 @@ artifacts = [ ] [task] -name = "search-swe/task-2-5" +name = "search-swe/sparse-retrieval-pruning" version = "0.4.0" description = "Optimize Python learned-sparse retrieval under quality gates and a continuous starter-relative latency reward." authors = [{ name = "Search-SWE" }] keywords = ["information-retrieval", "search", "ms-marco", "splade", "sparse-retrieval", "postings"] [metadata] -task_type = "optimize" dataset = "MS MARCO Passage v1 with frozen SPLADE sparse vectors" primary_metric = "quality_gated_linear_starter_latency" metric_k = 10 diff --git a/tasks/task-2-5/tests/Dockerfile b/tasks/sparse-retrieval-pruning/tests/Dockerfile similarity index 100% rename from tasks/task-2-5/tests/Dockerfile rename to tasks/sparse-retrieval-pruning/tests/Dockerfile diff --git a/tasks/task-2-5/tests/data/categories.json b/tasks/sparse-retrieval-pruning/tests/data/categories.json similarity index 100% rename from tasks/task-2-5/tests/data/categories.json rename to tasks/sparse-retrieval-pruning/tests/data/categories.json diff --git a/tasks/task-2-5/tests/data/corpus_ids.txt b/tasks/sparse-retrieval-pruning/tests/data/corpus_ids.txt similarity index 100% rename from tasks/task-2-5/tests/data/corpus_ids.txt rename to tasks/sparse-retrieval-pruning/tests/data/corpus_ids.txt diff --git a/tasks/task-2-5/tests/data/hidden_queries.jsonl b/tasks/sparse-retrieval-pruning/tests/data/hidden_queries.jsonl similarity index 100% rename from tasks/task-2-5/tests/data/hidden_queries.jsonl rename to tasks/sparse-retrieval-pruning/tests/data/hidden_queries.jsonl diff --git a/tasks/task-2-5/tests/data/qrels.tsv b/tasks/sparse-retrieval-pruning/tests/data/qrels.tsv similarity index 100% rename from tasks/task-2-5/tests/data/qrels.tsv rename to tasks/sparse-retrieval-pruning/tests/data/qrels.tsv diff --git a/tasks/task-2-5/tests/data/reference_metadata.json b/tasks/sparse-retrieval-pruning/tests/data/reference_metadata.json similarity index 100% rename from tasks/task-2-5/tests/data/reference_metadata.json rename to tasks/sparse-retrieval-pruning/tests/data/reference_metadata.json diff --git a/tasks/task-2-5/tests/data/reference_top100.jsonl b/tasks/sparse-retrieval-pruning/tests/data/reference_top100.jsonl similarity index 100% rename from tasks/task-2-5/tests/data/reference_top100.jsonl rename to tasks/sparse-retrieval-pruning/tests/data/reference_top100.jsonl diff --git a/tasks/task-2-5/tests/docker-compose.yaml b/tasks/sparse-retrieval-pruning/tests/docker-compose.yaml similarity index 100% rename from tasks/task-2-5/tests/docker-compose.yaml rename to tasks/sparse-retrieval-pruning/tests/docker-compose.yaml diff --git a/tasks/task-2-5/tests/finalize_reward.py b/tasks/sparse-retrieval-pruning/tests/finalize_reward.py similarity index 100% rename from tasks/task-2-5/tests/finalize_reward.py rename to tasks/sparse-retrieval-pruning/tests/finalize_reward.py diff --git a/tasks/task-2-5/tests/grader.py b/tasks/sparse-retrieval-pruning/tests/grader.py similarity index 99% rename from tasks/task-2-5/tests/grader.py rename to tasks/sparse-retrieval-pruning/tests/grader.py index d47f45d..073a171 100644 --- a/tasks/task-2-5/tests/grader.py +++ b/tasks/sparse-retrieval-pruning/tests/grader.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Quality and same-host starter-relative latency grader for Task-2-5.""" +"""Quality and same-host starter-relative latency grader for Task sparse-retrieval-pruning.""" from __future__ import annotations diff --git a/tasks/task-2-5/tests/jailbreak_judge/codex.toml b/tasks/sparse-retrieval-pruning/tests/jailbreak_judge/codex.toml similarity index 95% rename from tasks/task-2-5/tests/jailbreak_judge/codex.toml rename to tasks/sparse-retrieval-pruning/tests/jailbreak_judge/codex.toml index 69c1659..2f942c2 100644 --- a/tasks/task-2-5/tests/jailbreak_judge/codex.toml +++ b/tasks/sparse-retrieval-pruning/tests/jailbreak_judge/codex.toml @@ -11,7 +11,7 @@ name = "jailbreak_judge" type = "binary" description = """ Review the Agent's complete execution trajectory and submitted /app files for -Task-2-5. Return yes when the submission is a genuine Python sparse-retrieval +Task sparse-retrieval-pruning. Return yes when the submission is a genuine Python sparse-retrieval implementation for the supplied opaque-term corpus, and return no only when there is clear evidence of circumvention or verifier manipulation. diff --git a/tasks/task-2-5/tests/rewardkit_deepseek.py b/tasks/sparse-retrieval-pruning/tests/rewardkit_deepseek.py similarity index 100% rename from tasks/task-2-5/tests/rewardkit_deepseek.py rename to tasks/sparse-retrieval-pruning/tests/rewardkit_deepseek.py diff --git a/tasks/task-2-5/tests/run_with_metrics.py b/tasks/sparse-retrieval-pruning/tests/run_with_metrics.py similarity index 100% rename from tasks/task-2-5/tests/run_with_metrics.py rename to tasks/sparse-retrieval-pruning/tests/run_with_metrics.py diff --git a/tasks/task-2-5/tests/starter/build.sh b/tasks/sparse-retrieval-pruning/tests/starter/build.sh similarity index 100% rename from tasks/task-2-5/tests/starter/build.sh rename to tasks/sparse-retrieval-pruning/tests/starter/build.sh diff --git a/tasks/task-2-5/tests/starter/run.sh b/tasks/sparse-retrieval-pruning/tests/starter/run.sh similarity index 100% rename from tasks/task-2-5/tests/starter/run.sh rename to tasks/sparse-retrieval-pruning/tests/starter/run.sh diff --git a/tasks/task-2-5/tests/starter/src/sparse_retrieval.py b/tasks/sparse-retrieval-pruning/tests/starter/src/sparse_retrieval.py similarity index 100% rename from tasks/task-2-5/tests/starter/src/sparse_retrieval.py rename to tasks/sparse-retrieval-pruning/tests/starter/src/sparse_retrieval.py diff --git a/tasks/task-2-5/tests/test.sh b/tasks/sparse-retrieval-pruning/tests/test.sh similarity index 99% rename from tasks/task-2-5/tests/test.sh rename to tasks/sparse-retrieval-pruning/tests/test.sh index 7ff6509..26b8bfb 100755 --- a/tasks/task-2-5/tests/test.sh +++ b/tasks/sparse-retrieval-pruning/tests/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -u -RESULTS_DIR=/logs/artifacts/task-2-5-eval +RESULTS_DIR=/logs/artifacts/sparse-retrieval-pruning-eval INDEX_DIR=$RESULTS_DIR/index OUTPUT_PATH=$RESULTS_DIR/results.jsonl BUILD_METRICS=$RESULTS_DIR/build.metrics.json diff --git a/tasks/task-2-3/environment/starter/src/__init__.py b/tasks/task-2-3/environment/starter/src/__init__.py deleted file mode 100644 index 650d857..0000000 --- a/tasks/task-2-3/environment/starter/src/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Task-2-3 starter implementation.""" diff --git a/tasks/task-2-3/tests/starter/src/__init__.py b/tasks/task-2-3/tests/starter/src/__init__.py deleted file mode 100644 index 650d857..0000000 --- a/tasks/task-2-3/tests/starter/src/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Task-2-3 starter implementation.""" From 730b338d7a0b8970665330601b7c5af03ca330ca Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Fri, 9 Oct 2026 20:54:16 +0800 Subject: [PATCH 3/4] Update task authoring and contribution workflows Document same-name submission promotion and asset publication. Require configured model trials, feedback-driven task refinement, and local configuration prompts for missing API keys. Describe the current authoring contract and preserve structured instruction sources during revisions. --- .agents/AGENTS.md | 43 +-- .agents/skills/create-searchswe-task/SKILL.md | 106 +++---- .../create-searchswe-task/agents/openai.yaml | 2 +- .../references/assets-and-resources.md | 20 +- .../references/instruction-rewrite.md | 18 +- .../references/model-evaluation.md | 48 ++- .../references/publication.md | 28 +- .../references/submission-and-pr.md | 107 +++---- .../references/task-authoring.md | 67 ++--- .../references/validation.md | 28 +- .../skills/maintain-searchswe-task/SKILL.md | 15 +- .../references/promotion.md | 82 +++-- .../references/publication.md | 28 +- .../references/task-audit.md | 11 +- .github/pull_request_template.md | 35 +-- CONTRIBUTING.md | 31 +- docs/contributing.md | 281 ++++++++---------- task-submissions/README.md | 57 ++-- 18 files changed, 498 insertions(+), 509 deletions(-) diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index cc1517a..bf623f5 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -1,8 +1,8 @@ # Search-SWE Task Workflow -This is the contributor workflow entry point. Detailed task-authoring knowledge -lives inside the skill, not here. Paths below are relative to the repository -root unless expressed as Markdown links. +This is the contributor workflow entry point. Each skill contains its detailed +guidance. Paths below are relative to the repository root unless expressed as +Markdown links. ## Route the request @@ -10,7 +10,7 @@ root unless expressed as Markdown links. [create-searchswe-task](skills/create-searchswe-task/SKILL.md) and follow its workflow, references, and validation gates. If no native skill loader is available, open that `SKILL.md` directly. It can also be invoked independently - as `$create-searchswe-task`; it does not need this file. + as `$create-searchswe-task`. - **Review, promote or finalize a task PR as maintainer:** load [maintain-searchswe-task](skills/maintain-searchswe-task/SKILL.md), independently invocable as `$maintain-searchswe-task`. Default to read-only gh evidence review; @@ -18,7 +18,8 @@ root unless expressed as Markdown links. - **Run an existing task:** use `docs/quickstart.md`; do not scaffold a task. - **Restore fixed inputs:** use `docs/assets.md` and the existing download tools. - **Edit project documentation:** keep `README.md` and `README_zh.md` aligned. - The current public modes are Implementation and Optimization. + Describe the current contract and workflow directly; keep migration rationale + in the PR description. - **Change shared images:** use `docker/README.md`. An ordinary task contribution should reuse the CPU/GPU images rather than redesign their shared runtime. @@ -28,35 +29,39 @@ root unless expressed as Markdown links. contributors' changes. Ask about uncertainties that affect task design or authorization, not information already supplied by the user. 2. For task contributions, let the skill drive design → structured source → - user-request rewrite → grader checks → configured model evaluation. Preserve + user-request rewrite → grader checks → configured model evaluation → feedback + and task refinement → revalidation of substantive changes. Preserve `raw-instruction.md` and audit all its conditions in the agent-facing `instruction.md`. Describe the task, available inputs, resource constraints, acceptance criteria and optimization objective directly, without invented requester identities or narrative scenes; the skill defines both gates. Before scaffolding, inspect one or two closest reviewed packages under - `tasks/` as read-only structural precedents, selected by mode, grading shape, - resources and hardware. Current repository contracts and validators override - older examples; never inherit task-specific data, thresholds, licenses or - access merely because another task uses them. - A PR may add multiple tasks, all under exactly one first-name namespace, at - `task-submissions//<1|2>-x-`; do not use - usernames, reuse a promoted temporary ordinal, or guess formal IDs. Follow + `tasks/` as read-only structural precedents, selected by engineering objective, + grading shape, resources and hardware. Apply current repository contracts + and validators, and independently establish the new task's data, thresholds, + licenses and access policy. + A PR may add multiple tasks at `task-submissions/`. Choose a + descriptive lowercase name with at most five hyphen-separated words, without + a `task-` prefix; `all` is reserved. Keep that name when promoting to + `tasks/`. Do not reuse a submission path within the same PR. Follow `CONTRIBUTING.md` and `docs/contributing.md`: require actual authors and explicit `--task-path` for local trials, then promote every task with its own - same-PR pure rename and finalization commits before a merge commit (no - squash/rebase). Track which stage is complete and which inputs or approvals + same-PR pure rename and any necessary finalization commits before a merge + commit (no squash/rebase). Track which stage is complete and which inputs or approvals are missing. Do not duplicate the skill's specifications in this file. 3. Keep changes within the selected task plus directly necessary integration changes. Updating a website, publishing a dataset, pushing images, and opening or merging a PR are separate actions, not implied by task creation. 4. Review the resulting diff and validation evidence. Hand off changed paths, - task mode/hardware, commands actually run, observed results, and blockers. + task objective/hardware, commands actually run, observed results, and blockers. Static checks alone are not evidence of a working end-to-end benchmark. Include the exact instruction revision, model/effort, trajectory, score and - judge health from the configured-model validation. Reuse session-authorized - runs and credentials; do not repeat successful runs just because results - were moved or a monitoring pane still looks active. + judge health from the configured-model validation, plus any resulting task + changes and evidence for the final revision. Reuse session-authorized runs + and credentials. If required keys are missing, follow the skill's local + configuration handoff to the contributor. Do not repeat successful runs just + because results were moved or a monitoring pane still looks active. Agents are not guaranteed to auto-load an `AGENTS.md` located under `.agents/` when editing `tasks/`. Open this entry point explicitly or invoke the skill; diff --git a/.agents/skills/create-searchswe-task/SKILL.md b/.agents/skills/create-searchswe-task/SKILL.md index 07b916d..cd15979 100644 --- a/.agents/skills/create-searchswe-task/SKILL.md +++ b/.agents/skills/create-searchswe-task/SKILL.md @@ -7,12 +7,12 @@ description: Create or substantially revise Search-SWE task packages with CPU/GP Deliver a reproducible task whose structured source, natural-language request, environment, submission interface, and grading agree. Validate the setting -with an actual configured coding-model run as well as grader checks. This skill -contains its authoring guidance and helpers; no outer AGENTS.md, other skill, or historical Harbor workspace is -required. It still operates on a **target Search-SWE checkout**: its tasks, -Docker definitions, and release/download/launch tools are project inputs, not -bundled copies of this skill. Missing tools or incompatible versions are -prerequisites to report, not permission to invent a replacement workflow. +with an actual configured coding-model run as well as grader checks, and use +the results to refine the setting where needed. This skill contains its authoring +guidance and helpers. It operates on a **target Search-SWE checkout**, which +supplies the task packages, Docker definitions, and release/download/launch +tools. Report missing tools or incompatible versions as prerequisites for the +affected workflow step. ## 1. Establish the design @@ -21,9 +21,8 @@ target checkout's status and preserve unrelated changes. Resolve its root explicitly; do not infer it from where this skill was installed. Before scaffolding, inspect one or two closest **formal** packages under the -target checkout's `tasks/`. Choose them by engineering mode, evaluation/judge -shape, resources/APIs, hardware and input/artifact layout—not merely by task -number. Record their exact paths and why they are relevant. Treat them as +target checkout's `tasks/`. Choose them by engineering objective, evaluation/judge +shape, resources/APIs, hardware and input/artifact layout. Record their exact paths and why they are relevant. Treat them as read-only structural precedents: the current skill, repository validators, schema and shared-image documentation take priority. Do not copy task-specific IDs, authors, datasets, thresholds, pins, licenses or access policy without @@ -35,8 +34,7 @@ Record a short design summary before writing the package: | Decision | What must be known | | --- | --- | -| Identity | Temporary ID (`task-<1|2>-x-`), one PR-wide first-name slug, actual authors, version | -| Mode | Implementation → `metadata.task_type = "create"`; Optimization → `"optimize"` | +| Identity | Descriptive task name (at most five lowercase hyphen-separated words), actual authors, version | | Goal | What working capability or quality/efficiency improvement is measured | | Interface | Container input paths, commands, output paths/formats, artifact transfer | | Evaluation | Metric, public gates, baseline if needed, failure/timeout behavior | @@ -44,11 +42,10 @@ Record a short design summary before writing the package: | Resources | CPU/GPU, memory/storage/time, network, permitted APIs/models | | Precedent | Formal task path(s), matching dimensions, structural patterns reused, intentional differences | -Only these two engineering modes exist. CPU is the default; choose GPU for -actual GPU execution, not merely because model files are involved. Ask for +CPU is the default; choose GPU when the agent or verifier executes GPU work. Ask for missing facts that change the design; do not invent labels, licenses, service -access, or quality thresholds. An Optimization task needs a meaningful supplied -starting system/model and comparison criterion, not only a different mode label. +access, or quality thresholds. When improvement over a starting system/model is measured, supply that baseline +and a meaningful comparison criterion. ## 2. Create or edit the package @@ -58,45 +55,28 @@ For a new task, run the bundled helper using its **actual installed path**: # Set these to real absolute paths; neither depends on the current directory. SKILL_DIR=/path/to/create-searchswe-task REPO=/path/to/Search-SWE -python "$SKILL_DIR/scripts/scaffold_task.py" task-1-x-1 \ - --repo-root "$REPO" --submission-first-name Alice --author 'Alice Example' \ - --mode implementation --hardware cpu +python "$SKILL_DIR/scripts/scaffold_task.py" example-search \ + --repo-root "$REPO" --author 'Alice Example' --hardware cpu ``` -Category 1 is Implementation (`create`); category 2 is Optimization (`optimize`). -The temporary ID and mode must agree. For optimization: - -```bash -python "$SKILL_DIR/scripts/scaffold_task.py" task-2-x-1 \ - --repo-root "$REPO" --submission-first-name Alice --author 'Alice Example' \ - --mode optimization --hardware cpu -``` - -Hardware is independent of category/mode; use `--hardware gpu` only for GPU work. -New tasks belong in -`task-submissions//<1|2>-x-`, not formal `tasks/` directories. -Use a supplied ASCII first name, not a username; non-Latin names need an explicit -transliteration. Reuse that contributor directory for every task in the PR. -Ordinals are temporary, category-local, positive, and unique in the PR/checkout; -they are not final IDs; never reuse one after promoting its earlier task in the -same PR. If a different contributor with the same first name needs a namespace, -explicitly supply `Alice-2` (then `Alice-3`), including after -checking active PRs; the scaffolder never derives name suffixes from task -collisions. Repeat `--author` for coauthors. -The helper refuses overwrites and symlink roots. Regular mode without -`--submission-first-name` remains compatible for maintainers. Edit an existing -task in place instead; do not delete it to make the helper succeed. - -One PR may add multiple tasks, but every task must use exactly one contributor -first-name namespace. Read the bundled -[submission and PR workflow](references/submission-and-pr.md) for branch, -review gates, maintainer handoff and author identity. A maintainer assigns each final ID close to -merge and performs a separate pure rename then finalization for each task in -**the same PR**, using `scripts/promote_task.py`. All tasks must be promoted -before merge. Preserve every promotion commit and original author commit: -**merge commit only**, never squash/rebase. No submission task package enters -final main. Missing workflow tools in an older checkout are a prerequisite to -report, not a reason to invent a formal ID. +Use `--hardware gpu` only for GPU execution. New tasks belong at +`task-submissions/`, with `task.name = "search-swe/"`. +Names start with an ASCII lowercase letter, contain lowercase letters/digits +and single hyphens, and have at most five words. Use a concise description of +the task; omit the `task-` prefix. `all` is reserved. Check both package roots +and active PRs for collisions. Repeat `--author` for actual coauthors. +The helper refuses overwrites and symlink roots. Its explicit `--formal` option +is for maintainer tooling, not a bypass for new task contributions. Edit an +existing task in place instead of deleting it to make the helper succeed. + +One PR may add multiple uniquely named tasks. Read the bundled +[submission and PR workflow](references/submission-and-pr.md). Promotion keeps +the same name: `task-submissions/` → `tasks/`. Each task +gets its own pure rename commit in **the same PR**, followed by a separate +finalization commit when references, assets or integration need changes. Do not +reuse a submission path in that PR. Preserve original author and promotion +commits: **merge commit only**, never squash/rebase. All tasks must be promoted +before merge; no submission package enters final main. The scaffold is **not a finished task**: its verifier always fails, its asset manifest is empty, data/model mounts are absent, and instructions are writing @@ -130,7 +110,7 @@ Follow the detailed rules in `references/task-authoring.md`: 5. Add author-facing README context and task-local tests, including known-good and invalid submissions. Do not change unrelated tasks or scoring policies. -## 4. Validate the task and run a configured model +## 4. Validate, test with a model, and refine the setting Read [references/validation.md](references/validation.md) and follow its staged checks. Stop advancing to expensive runtime tests when inputs, permissions, @@ -140,9 +120,18 @@ diagnosis; retry after a relevant fix, not in an unbounded loop. After local package, asset and grader checks, follow [references/model-evaluation.md](references/model-evaluation.md). Run the final rewritten instruction with the configured coding model and any required judges; -inspect its trajectory, artifacts, grader result and judge health to validate -the task setting. An infrastructure failure does not measure difficulty, and -one low-scoring model run does not establish that a task is impossible. +inspect its trajectory, artifacts, grader result and judge health. Use this +feedback to identify task-setting issues, correct them, and revalidate the +affected checks. Rerun the model when changes affect its task or scoring, within +the authorized budget, and record the final evaluated revision. An infrastructure +failure does not measure difficulty, and one low-scoring model run does not +establish that a task is impossible or justify weakening it. + +If required API keys are missing, tell the contributor which variables are +needed, what they are for, and where to configure them locally; never ask for +secret values in chat. Wait for that configuration before the dependent run, +and continue independent checks. The model-evaluation reference details this +handoff and the feedback loop. Local task authoring/validation is the default authorization boundary. It does not itself authorize commits, pushes, PR creation/comments, publishing data/images, modifying host @@ -152,11 +141,12 @@ request or prior session. An explicit request to evaluate with a named model authorizes that scoped run; reuse already-authorized local credentials without asking again. Ordinary local checks may proceed. -Report the task path, mode/hardware, commands actually executed and their +Report the task path, objective/hardware, commands actually executed and their results, formal reference tasks consulted (or that none was close), intentional reuse/deviations, the source-to-request equivalence review, known-good/negative-case evidence, the exact evaluated revision/model/effort, observed scores and judge -health, and every unrun validation layer. +health, feedback-driven task changes (or why none were needed), and every unrun +validation layer. Never equate a scaffold, a dry-run, or a static release check with a solvable end-to-end task. diff --git a/.agents/skills/create-searchswe-task/agents/openai.yaml b/.agents/skills/create-searchswe-task/agents/openai.yaml index 863eb5e..ad2d0a9 100644 --- a/.agents/skills/create-searchswe-task/agents/openai.yaml +++ b/.agents/skills/create-searchswe-task/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Contribute Search-SWE Task" short_description: "Author and validate a temporary task submission" - default_prompt: "Use $create-searchswe-task to build and locally validate a first-name task submission, then prepare an authorized PR handoff." + default_prompt: "Use $create-searchswe-task to build and locally validate a task submission with a descriptive name, then prepare an authorized PR handoff." diff --git a/.agents/skills/create-searchswe-task/references/assets-and-resources.md b/.agents/skills/create-searchswe-task/references/assets-and-resources.md index 4b679f6..f73267f 100644 --- a/.agents/skills/create-searchswe-task/references/assets-and-resources.md +++ b/.agents/skills/create-searchswe-task/references/assets-and-resources.md @@ -8,7 +8,7 @@ task. Nothing in this reference grants permission to publish or use paid APIs. Keep downloadable fixed files outside the build contexts, under `data/` or `models/` in the selected package (initially -`task-submissions//<1|2>-x-`, later `tasks/`). +`task-submissions/`, later `tasks/`). Ignore both directories in Git. Hidden queries/labels and grading code go in `tests/`, not the public asset bundle. For a verifier-only corpus stored in `data/verifier/`, mount only public @@ -53,18 +53,18 @@ dataset** with manifest, provenance and license; pin its immutable 40-hex SHA in submission `assets.json`. The downloader accepts arbitrary HF repositories: ```bash -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 --verify-only +python scripts/download_assets.py --task-path task-submissions/example-search +python scripts/download_assets.py --task-path task-submissions/example-search --verify-only ``` -After the maintainer assigns the final ID in the same PR, prepare new official -files at `tasks//...`. The target checkout's +After the maintainer approves promotion in the same PR, prepare new official +files at `tasks//...`. The target checkout's `scripts/prepare_hf_upload.py` accepts a current official manifest snapshot, -the final task package and **only new data**. It verifies hashes, preserves old +the final task package and **only new data**. It verifies hashes, preserves existing entries, refuses collisions and stages new files plus a merged manifest offline. Read the bundled [publication workflow](publication.md) for exact commands and approval boundaries. Append/update -HF `SOURCES.md`, dataset card/license and LFS metadata. Never delete old assets. +HF `SOURCES.md`, dataset card/license and LFS metadata. Preserve existing published assets. Publish through a **HF community PR (`create_pr=True`)** with your personal account, or ask a maintainer to mirror reviewed public files. Never share or request an official token. Refresh a stale manifest snapshot before merging. @@ -73,7 +73,7 @@ The official HF change must merge **before** pinning the resulting official SHA in final `assets.json` and merging the GitHub task PR. Do not pin the community PR SHA or use a mutable branch. Keep original model pins unchanged. Official publication is separately authorized; helpers do not upload. Do not publish -permanent official paths containing temporary task IDs. +official paths that do not use the approved task name. ## Full dataset audit (maintainers) @@ -81,8 +81,8 @@ For unpublished data, prepare an authorized local HF staging directory with `README.md`, `SOURCES.md`, `.gitattributes`, `manifest.json`, and `tasks/` files. Its manifest has `schema_version: 1` and `files` entries containing dataset-relative `path`, `size_bytes`, and `sha256`. It must match the checkout's complete dataset -inventory, not just new files. Contributors normally use the incremental -helper above instead; it does not need all old assets. Full local checks support: +inventory. For individual task contributions, the incremental helper above +needs only the current manifest and the new data files. Full local checks support: ```bash python scripts/check_release.py --hf-data /path/to/hf-data \ diff --git a/.agents/skills/create-searchswe-task/references/instruction-rewrite.md b/.agents/skills/create-searchswe-task/references/instruction-rewrite.md index c8a1227..8cee23a 100644 --- a/.agents/skills/create-searchswe-task/references/instruction-rewrite.md +++ b/.agents/skills/create-searchswe-task/references/instruction-rewrite.md @@ -6,18 +6,20 @@ the task root. Harbor receives `instruction.md`. ## Preserve the source -For an existing task, first copy the original instruction byte for byte to -`raw-instruction.md`; record its Git revision or hash. For a new task, write the -structured specification there first. Keep the source organized enough to -review paths, interfaces, outputs, limits, scoring and restrictions precisely. -For later task changes, update the source deliberately and regenerate the -request; never silently change a requirement during a stylistic rewrite. +Use `raw-instruction.md` as the structured authoring source. For a new task, +write that specification first. For an existing task, preserve its structured +source and record the Git revision or hash before editing. If the structured +source is missing, save the existing instruction byte for byte there before +organizing the specification. Keep the source organized enough to review paths, +interfaces, outputs, limits, scoring and restrictions precisely. For task +changes, update the source deliberately and regenerate the request; never +silently change a requirement during a stylistic rewrite. Read the actual environment, resource documents and verifier before rewriting. If the source conflicts with them, record the discrepancy and resolve the contract explicitly. A typo correction is a semantic change to disclose, not -an opportunity to hide a new requirement in narrative. Keep an original source -snapshot when migrating existing tasks. +an opportunity to hide a new requirement in narrative. Retain the source revision +used for the rewrite so changes can be reviewed. ## Write the request diff --git a/.agents/skills/create-searchswe-task/references/model-evaluation.md b/.agents/skills/create-searchswe-task/references/model-evaluation.md index a02b6f9..2f2d248 100644 --- a/.agents/skills/create-searchswe-task/references/model-evaluation.md +++ b/.agents/skills/create-searchswe-task/references/model-evaluation.md @@ -24,6 +24,24 @@ passing literal secrets in command arguments. Do not alter a shared `.env` just to run one revised task: use an ignored run-specific env file or process overrides. Restrict it to its owner and remove temporary secret copies afterward. +Check only whether required variables are configured; never display their +values. Determine the required groups from the selected agent/provider, task +and judges, using the target checkout's `.env.example`, `docs/evaluation.md` +and task configuration. Optional task APIs need keys only when used. If keys +are missing, explicitly remind the contributor to configure them and provide: + +- The exact missing variable names and the model, task API or judge each serves. +- The local configuration location: the checkout's ignored `.env`, a supported + run-specific env file, or process environment. Give placeholders only; preserve + an existing `.env` instead of overwriting it with the template. +- A request to confirm when local configuration is ready, without sharing key + values in chat, a PR or logs. + +Wait for configuration before the dependent probe/trial, then recheck presence +and authentication without exposing secrets. Continue independent package and +grader checks while waiting. Missing credentials are an unmet prerequisite, +not model failure or permission to skip required validation. + When changing a judge provider/model, first exercise the actual pinned judge backend on a small fixture or an archived trajectory in a disposable verifier. Check authentication, full model ID, Responses/tool support, structured result, @@ -61,7 +79,9 @@ Keep a concurrency cap appropriate to API and hardware limits; start with one validation trial, and ordinarily no more than two while validating a new setup. A 429 calls for lower concurrency/backoff, not a wider launch wave. Honor any user-specified attempt cap; otherwise allow at most three infrastructure attempts -and stop after the first valid completed trial. Do not retry a valid low score. +per unchanged validation revision and stop after its first valid completed trial. +Do not retry a valid low score. Revisions below remain subject to the overall +authorized trial/cost budget; a task edit does not reset a user-specified cap. Poll the specific live process/container and logs. Long indexing, training or verification can produce little output. An unchanged result file or observer @@ -101,6 +121,29 @@ Report the exact observed score and limitations of this sample. Cost fields that are null are unknown, not zero; token-based estimates and provider billing must be distinguished, including judge and failed-attempt costs when available. +## Use feedback to refine the setting + +For each valid trial, record what the trajectory and evaluation reveal about +the task. Distinguish solver mistakes from task-setting defects such as an +ambiguous interface, unavailable inputs, contradictory resource limits or an +incorrect verifier. Use concrete evidence to revise the affected instructions, +environment, budgets or grader. Keep `raw-instruction.md`, `instruction.md`, +resource docs and scoring consistent with the revised contract. + +Preserve the intended engineering objective and held-out evaluation. Do not +lower thresholds just to turn a valid low score into a pass, tailor the task +to one model's solution, or expose hidden answers in the prompt. If the run +supports the current setting, record why no change is needed. + +After a substantive revision, rerun affected package/grader checks and then +the configured model whenever its task or scoring changed. Use a fresh output +directory and record the revised hashes, observations and rationale so evidence +is tied to the final setting. Cosmetic edits or moving completed results do +not require another model run. Stop when the setting is supported by the +checks and a valid interpreted trial, or the agreed budget or a prerequisite +blocks further validation. In the latter case, identify the untested changes +and missing evidence; do not present an earlier trial as validation of them. + ## Archive and hand off Read the destination's `AGENTS.md` before interpreting or moving results. Move @@ -112,6 +155,7 @@ once they have no work; do not stop unrelated user processes. Hand off the exact command (without secrets), revision/hashes, run and trajectory paths, completion and judge status, scores/gate outcomes, resource observations, -cost availability, attempt count and any task changes motivated by the run. +cost availability, attempt count and any task changes motivated by the run +(or why none were needed), with validation evidence for the final setting. Keep unperformed or invalid layers explicit. Do not claim the task validated until the required model run and its interpretation are supported by evidence. diff --git a/.agents/skills/create-searchswe-task/references/publication.md b/.agents/skills/create-searchswe-task/references/publication.md index b65d6a6..6d7b249 100644 --- a/.agents/skills/create-searchswe-task/references/publication.md +++ b/.agents/skills/create-searchswe-task/references/publication.md @@ -34,12 +34,12 @@ immutable SHA in `assets.json`. The explicit submission downloader/launcher then support real pre-promotion validation. Keep the temporary repo accessible through review and migration; only retire it after official pinned downloads are verified. -### Final ID: additive official publication, then GitHub merge +### Task name: additive official publication, then GitHub merge -After ID assignment/finalization, change only the new task's dataset source -repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve +After task-name approval and promotion, change only the new task's dataset source +repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve original model repo pins and bundled metadata. Prepare **only new data files** -in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. +in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. Do not delete, rename or overwrite existing official assets. Fetch the small current official manifest snapshot and metadata, not the whole @@ -60,16 +60,18 @@ print(sha) PY python scripts/prepare_hf_upload.py \ --official-manifest /path/to/official-snapshot/manifest.json \ - --task-path tasks/task-1-6 --new-data /path/to/new-data \ + --task-path tasks/example-search --new-data /path/to/new-data \ --output /path/to/upload-staging ``` -This offline helper verifies new files' hashes, refuses collisions/traversal/ -symlinks, and writes only new files plus a merged manifest preserving every old -record. It does not require old data or upload anything. A trusted current -snapshot is essential: it cannot prove remote freshness offline. Unlike -`check_release.py --hf-data`, which intentionally checks the **full inventory**, -this is the incremental contribution path, not a replacement full release audit. +Official asset staging requires a promoted package at `tasks/`. + +The offline helper verifies hashes and safe paths for new files, rejects +collisions and symlinks, and stages those files with a merged manifest that +preserves all existing records. Supply a trusted current manifest snapshot; +only the new data files are needed locally. Uploading is a separate step. +For a full dataset audit, `check_release.py --hf-data` checks the complete +inventory against a local dataset copy. Copy the snapshot `README.md`, `SOURCES.md`, and `.gitattributes` into staging, then **append** provenance, license/redistribution information, and any required @@ -87,7 +89,7 @@ result = HfApi().upload_folder( repo_id='search-swe/Search-SWE', repo_type='dataset', folder_path='/path/to/upload-staging', parent_commit=Path('/path/to/official-snapshot/BASE_SHA').read_text().strip(), - create_pr=True, commit_message='Add task-1-6 inputs and provenance', + create_pr=True, commit_message='Add example-search inputs and provenance', ) print(result.pr_url) PY @@ -106,4 +108,4 @@ replace a newer manifest with a stale one. GitHub task's `assets.json`. Verify downloads using a fresh output directory or cache and run release checks. Commit finalization and merge the GitHub PR last. No new task with unpublished/unpinned assets or an unfinished submission enters -main. Existing tasks' old pins and assets remain valid. +main. Preserve existing tasks' pinned revisions and published assets. diff --git a/.agents/skills/create-searchswe-task/references/submission-and-pr.md b/.agents/skills/create-searchswe-task/references/submission-and-pr.md index de5cc41..9a466de 100644 --- a/.agents/skills/create-searchswe-task/references/submission-and-pr.md +++ b/.agents/skills/create-searchswe-task/references/submission-and-pr.md @@ -1,39 +1,29 @@ # Submission, branch and PR handoff -Choose `task-1-x-` with `--mode implementation` -(`metadata.task_type = "create"`) or `task-2-x-` with -`--mode optimization` (`"optimize"`). Hardware is independent. The canonical -ID exactly matches the temporary directory with the `task-` prefix. +Choose a descriptive task name of at most five lowercase hyphen-separated words, +starting with a letter and otherwise using letters/digits. Omit `task-`; `all` +is reserved. Choose GPU only when execution requires it. ## Local work first -Resolve the target checkout explicitly and inspect `git status --short`, current -branch, remotes and base before editing. Preserve dirty work and other authors' -commits; never reset, stash or switch their branch to make your task easier. -Agree on local branch/commit scope before changing history. New submissions use -`task-submissions//<1|2>-x-`, with exactly -matching canonical IDs such as `task-1-x-1`/`task-2-x-1`, not usernames or -guessed formal numbers. A scaffold is a deliberately failing starting point, -not a finished task. One PR may add multiple tasks, but all must use exactly one -contributor namespace; ordinals are temporary, category-local, and unique in -that PR/checkout, not final IDs. Do not reuse a temporary path after promoting -its earlier task in the same PR; the base-history CI check rejects reuse and -new formal tasks with no submission history. - -Set `task.toml` authors to the actual author(s). Keep the contributor's original -commits and Git name/email (account-linked or GitHub noreply); do not change -identity configuration on their behalf. Co-authored-by trailers can supplement -this, not replace actual authors or original commits. An ASCII first-name slug -is only a temporary namespace and is reused for all tasks in that PR. Check -authorized read-only PR listings for another contributor with the same first -name and explicitly choose `alice-2`, `alice-3`, etc. if needed; the scaffolder -does not generate those suffixes from task collisions. - -In an approved clean checkout, a local branch could be created with -`git switch -c add-alice-implementation`. Inspect the selected base and branch -name first; do not reuse an existing branch blindly. Local work permission does -not imply permission to commit, fork, push, publish datasets/images, open a PR, -post comments/reviews, change credentials or repository configuration. +Resolve the target checkout explicitly and inspect status, branch, remotes and +base. Preserve other authors' work and commits; never reset, stash or switch +their branch to simplify your task. Agree on branch/commit scope before changing +history. New packages use `task-submissions/` and canonical +`task.name = "search-swe/"`. Check formal tasks, submissions and active +PRs for collisions. A PR may add multiple unique names, but cannot reuse a +submission path. Direct new formal additions without submission history fail CI. +A scaffold is a deliberately failing starting point, not a finished task. + +Set actual `task.toml` authors. Preserve original commits and account-linked Git +name/email; do not change identity configuration on their behalf. Co-authored-by +trailers supplement these records. + +In an approved clean checkout, a local branch could use +`git switch -c add-example-search`. Inspect the base and branch name first. +Local work permission does not itself authorize commits, forks, pushes, dataset +publication, PR creation/comments or remote configuration; reuse any explicit +session authorization already supplied. ## Authorized GitHub handoff @@ -77,38 +67,33 @@ A review-stage CI pass can still contain unfinished submissions. Missing, pending or failed checks are not acceptance. Separately authorize code execution, API/GPU costs and any credentialed runtime tests. -Near merge, the maintainer serially assigns each task's next free category -number from the current base and queued PRs. The contributor must not guess or -reserve final numbers. -Update the PR to the current base without flattening the author's history. -Using the target checkout's offline `scripts/promote_task.py` (missing tools -are a prerequisite blocker), from a clean worktree/index: +Near merge, refresh the base and check name collisions with queued PRs. Preserve +original author history. Use the target checkout's offline promotion helper +from a clean worktree/index; missing tools are a prerequisite blocker: ```bash -# FINAL_ID is assigned by the maintainer, never inferred from this example. -python scripts/promote_task.py rename task-submissions/alice/1-x-1 FINAL_ID --dry-run -python scripts/promote_task.py rename task-submissions/alice/1-x-1 FINAL_ID +python scripts/promote_task.py rename task-submissions/example-search example-search --dry-run +python scripts/promote_task.py rename task-submissions/example-search example-search # Review and, only when authorized, commit this pure git mv alone. -python scripts/promote_task.py finalize task-submissions/alice/1-x-1 FINAL_ID --dry-run -python scripts/promote_task.py finalize task-submissions/alice/1-x-1 FINAL_ID +python scripts/promote_task.py finalize task-submissions/example-search example-search --dry-run +python scripts/promote_task.py finalize task-submissions/example-search example-search ``` -Repeat this pair of commands and separate commits independently for every task. -Finalize requires HEAD to be that task's separate, 100%-identical pure rename -commit of the whole package. Helpers never commit or publish. Review all replacements, image -and external paths, task inventories and authors. Complete the bundled -[publication workflow](publication.md): personal pinned development assets → -new official final-ID paths via HF community PR or authorized maintainer mirror -→ official merge → pin official SHA (not the community PR SHA). Preserve model -pins. Then validate every final package with release checks, full applicable tests -and `python scripts/check_submission.py --merge-ready`; no submission task.toml -may remain. HF migration is per task. Commit each finalization separately in -**the same PR** when authorized. - -Maintainers re-review each head change, recheck the base/ID and exact head before -an authorized **merge commit**; no squash/rebase, admin bypass or unfinished -submission on main. Keep original author commits. Verify the resulting merged -SHA/tree and `git log --follow -- tasks/FINAL_ID/instruction.md` and -`git blame tasks/FINAL_ID/instruction.md` for history/attribution. Report the PR, -final ID, official asset SHA, checks and any outstanding blockers rather than -claiming a merge that was not observed. +Repeat independently for each task. The name stays unchanged. Finalize requires +HEAD to be that task's single-parent, whole-package, 100%-identical pure rename +commit. It rewrites submission paths in tracked text; it never commits or +publishes. Review replacements, mounts, image/external paths, inventories and +authors. Follow [publication](publication.md): personal pinned development data +→ official `tasks//...` paths through a community PR or authorized +mirror → official merge → pin that official SHA. Preserve original model pins. + +Run release and applicable tests plus `python scripts/check_submission.py +--merge-ready`; no submission task.toml may remain. Commit any reference, asset +or integration finalization separately in this same PR. Do not create an empty +commit if no finalization changes are needed. + +Re-review each changed head, recheck base/name collisions and exact head before +an authorized **merge commit**. No squash/rebase or unfinished submissions on +main. Verify the resulting SHA/tree and attribution using +`git log --follow -- tasks/example-search/instruction.md` and `git blame`. +Report observed outcomes, task names, official asset SHAs and remaining blockers. diff --git a/.agents/skills/create-searchswe-task/references/task-authoring.md b/.agents/skills/create-searchswe-task/references/task-authoring.md index 4607d09..9a9ae43 100644 --- a/.agents/skills/create-searchswe-task/references/task-authoring.md +++ b/.agents/skills/create-searchswe-task/references/task-authoring.md @@ -3,24 +3,8 @@ Read this reference before creating or changing a task package. Paths in code blocks and backticks are relative to the target Search-SWE repository root, unless they begin with `/` (container paths). The skill may be installed -elsewhere. No external AGENTS.md or historical workspace is required. - -## Guidance distilled for this repository - -This guide consolidates four historical authoring notes (provenance only, not -required reading or runtime dependencies): - -- the top-level Harbor task contract (`harbor/tasks/AGENTS.md`); -- the Search-SWE task template and CPU/GPU split - (`harbor/tasks/_template/AGENTS.md`); -- the external retrieval/LLM resource profiles - (`harbor/tasks/_resources/AGENTS.md`); -- the shared-image build notes (`harbor/tasks/_docker/AGENTS.md`). - -The old workspace's proxy addresses, local image names, absolute paths, and -dated machine observations are intentionally not copied here. They are not -portable contributor requirements. In this repository, use the published image -tags documented in `docker/README.md`. +elsewhere. Use the target checkout's validators and the published image tags +documented in `docker/README.md`. ## Use current formal tasks as bounded precedents @@ -29,7 +13,7 @@ read-only. Select one or two closest **formal** packages for each proposed task; active `task-submissions/` packages are unreviewed work, not precedent. Prefer similarity in this order when it affects the design: -1. Implementation versus Optimization and the evaluation/baseline shape; +1. engineering objective and the evaluation/baseline shape; 2. deterministic verifier versus model judge, plus artifact-transfer interface; 3. external retrieval/LLM resources and network/credential boundaries; 4. CPU/GPU execution and input/asset layout. @@ -37,7 +21,7 @@ similarity in this order when it affects the design: Use repository search to shortlist candidates rather than reading every task: ```bash -rg -n 'task_type|gpus|environment_mode' tasks -g task.toml +rg -n 'objective|gpus|environment_mode' tasks -g task.toml ``` For selected candidates, inspect `task.toml`, `raw-instruction.md` (if present), @@ -49,12 +33,10 @@ APIs. Record the exact task paths, why they match, which structural patterns are being reused, and intentional differences. If no close analogue exists, record that and follow the current scaffold and contracts directly. -An existing task proves only that a pattern was used before; it is not the -source of truth and may predate current rules. The current skill, repository -validators, Harbor schema, `docker/README.md`, and documented task contract win -on conflict. Reuse structure and reviewed implementation patterns, not task- -specific facts. Independently establish the new task's ID/authors, datasets and -HF paths, hashes/pins, thresholds/baselines, provider/model allowlists, resource +Use the current skill, repository validators, Harbor schema, `docker/README.md`, +and documented task contract to resolve conflicts with reference packages. +Reuse structure and reviewed implementation patterns. Independently establish +the new task's name/authors, datasets and HF paths, hashes/pins, thresholds/baselines, provider/model allowlists, resource budgets, licenses/provenance, instructions, and hidden evaluation design. Do not copy a value merely to make the new package resemble its precedent. @@ -64,8 +46,7 @@ A task must describe one independently buildable, repeatable, and objectively scorable agent job. Before editing files, settle the points that affect its design: -- task ID and engineering mode: Implementation (`task_type = "create"`) or - Optimization (`task_type = "optimize"`); +- descriptive task name, engineering objective and starting system or inputs; - exact submission interface and artifact paths; - primary metric, pass gates, baseline (when applicable), and timeout behavior; - public inputs versus verifier-only inputs, including provenance and license; @@ -83,21 +64,19 @@ or access to a reference solution. ## Required package contract -Create each new package at -`task-submissions//<1|2>-x-/`, with the exactly -matching canonical ID (for example `task-1-x-1` or `task-2-x-1`) and actual -`task.toml` authors. A PR may contain multiple tasks, but all use one supplied -ASCII first-name namespace, not a username. Ordinals are temporary and unique -within category in that PR/checkout, not final IDs. A different contributor -with the same first name explicitly chooses `alice-2` (then `alice-3`); the -scaffolder reuses an existing namespace rather than generating task-based suffixes. -Formal `tasks//` paths below describe the layout after maintainer -promotion in the same PR (pure rename commit, then finalization, merge commit -only). Existing-task edits stay in place. Submission checks reuse +Create each new package at `task-submissions//`, with +`task.name = "search-swe/"` and actual `task.toml` authors. Choose a +concise descriptive name of at most five lowercase hyphen-separated words, +starting with a letter and otherwise using letters/digits. Omit `task-`; `all` +is reserved. Names must be unique across submissions and formal tasks. +A PR may contain multiple tasks; never reuse a submission path in that PR. +Promotion keeps the name at `tasks//` in the same PR: a pure rename +commit, then any necessary finalization in a separate commit, and a merge commit +only. Existing-task edits stay in place. Submission checks reuse `scripts/check_release.py`, which requires these non-empty files: ```text -tasks// +tasks// ├── instruction.md ├── task.toml ├── assets.json @@ -127,15 +106,13 @@ Also provide the repository conventions that apply to the task: ``` Use `schema_version = "1.4"`. Set `task.name` to -`search-swe/`, give the task a version, and keep the objective, +`search-swe/`, give the task a version, and keep the objective, keywords, metadata, artifact paths, timeouts, network modes, and resource budgets mutually consistent. Follow the closest existing task for fields that are not explained by Harbor's schema, subject to the bounded-precedent rules above; do not copy its task-specific values. -Search-SWE has exactly two engineering modes: Implementation maps to -`metadata.task_type = "create"`, and Optimization maps to -`metadata.task_type = "optimize"`. Do not introduce a Repair mode or another -`task_type` without an explicit project-wide taxonomy change. +When measuring improvement against a supplied starting system/model, document +that baseline and the comparison criterion. Search-SWE release packages use a separate verifier: diff --git a/.agents/skills/create-searchswe-task/references/validation.md b/.agents/skills/create-searchswe-task/references/validation.md index 6fd7882..e8236c5 100644 --- a/.agents/skills/create-searchswe-task/references/validation.md +++ b/.agents/skills/create-searchswe-task/references/validation.md @@ -1,8 +1,7 @@ # Validation and completion evidence Run commands from the target repository root, using Python 3.12+. For a new -task set `task_path=task-submissions/alice/1-x-1` (your actual first-name slug and -category); after promotion set `task_path=tasks/`. These checks use +task set `task_path=task-submissions/example-search` (use your actual task name); after promotion set `task_path=tasks/`. These checks use the checkout's tools; the skill does not bundle a second checker or runtime. ## Target prerequisites (stop if missing) @@ -16,6 +15,8 @@ virtual environment. Do not replace missing target tooling with invented command Docker Engine + Compose are needed for builds/trials, published image access for pulls, and NVIDIA driver/Container Toolkit for GPU execution. Check provider, model, credentials, network, disk and runtime budgets before running a trial. +For missing required keys, follow the contributor configuration handoff in +[model-evaluation.md](model-evaluation.md#configure-and-probe) before any dependent run. These are target inputs, not portable skill dependencies. `gh` is needed only for an authorized remote inspection or PR handoff, not local task authoring. @@ -23,7 +24,7 @@ for an authorized remote inspection or PR handoff, not local task authoring. - Revisit the recorded formal reference task(s), or the recorded absence of a close precedent. Compare only the intended structural patterns and explain - meaningful deviations; current contracts override legacy behavior. Check that + meaningful deviations against the current contracts. Check that no task-specific IDs, data/pins, thresholds, allowlists, licenses or resource assumptions were inherited without independent evidence. - Review `raw-instruction.md` against `instruction.md` using @@ -31,7 +32,7 @@ for an authorized remote inspection or PR handoff, not local task authoring. condition in the request and trace graded conditions to the verifier. Confirm the running agent only receives the rewritten instruction. Public limits must not exist only in hidden tests. -- Parse `task.toml` and `assets.json`; verify unique identity, correct mode, +- Parse `task.toml` and `assets.json`; verify unique name, objective and keywords, separate verifier, artifact paths, and all five CPU/GPU configuration points. - Review all scaffold prose, budgets and empty files. An empty `files` manifest is valid only if the task genuinely needs no downloadable fixed inputs. @@ -40,7 +41,7 @@ for an authorized remote inspection or PR handoff, not local task authoring. without actually using that UID when launching code is insufficient. ```bash -task_path=task-submissions/alice/1-x-1 +task_path=task-submissions/example-search python scripts/check_submission.py "$task_path" # Before promotion; requires PyYAML python scripts/check_release.py python -m unittest discover -s scripts/tests -p 'test_*.py' @@ -57,7 +58,7 @@ The current `scripts/tests/test_task_images.py` includes an explicit `GPU_TASKS` inventory. Adding a GPU task may require updating that expected inventory as a direct integration change, while retaining image/resource consistency checks. Do not lower assertions to make a wrongly configured task pass. Download and -launch commands discover formal IDs from `tasks/*/task.toml` automatically; +launch commands discover formal task names from `tasks/*/task.toml` automatically; submissions require explicit `--task-path`. CI review-stage success is not merge readiness: after same-PR pure rename/finalization, run `python scripts/check_submission.py --merge-ready` (no submission `task.toml` @@ -75,7 +76,7 @@ Compose files are Harbor overlays. Direct `docker compose config` needs a temporary base service with an image (this check does not pull that image): ```bash -task_path=task-submissions/alice/1-x-1 +task_path=task-submissions/example-search ( set -eu base=$(mktemp) @@ -122,16 +123,19 @@ equal one: compare to the agreed baseline/gates and explain the expected result. The configured coding-model trial is a required validation layer, separate from known-good and negative-case grading. Follow [model-evaluation.md](model-evaluation.md) for configuration, a small judge -probe, launch, bounded retries, completion checks and interpretation. If the -required run cannot be executed, hand off the missing evidence explicitly; +probe, launch, bounded retries, completion checks and feedback-driven task +refinement. Revalidate affected checks and the model trial after substantive +task/scoring changes; retain evidence identifying the final tested setting. +If the required run cannot be executed, hand off the missing evidence explicitly; do not present static checks or Oracle results as that model run. ## Handoff -Report changed paths, mode, hardware/image selection, input provenance, +Report changed paths, objective, hardware/image selection, input provenance, formal reference task paths (or no close precedent), structural reuse and intentional deviations, commands/results for each completed layer, expected -versus observed scores, and unrun layers with their blockers. Distinguish -static, asset, build, grader, known-good and coding-agent E2E results. Do not +versus observed scores, task changes motivated by model feedback (or why none +were needed), the final evaluated revision, and unrun layers with their blockers. +Distinguish static, asset, build, grader, known-good and coding-agent E2E results. Do not label the task ready for release while placeholders or required runtime checks remain unresolved. diff --git a/.agents/skills/maintain-searchswe-task/SKILL.md b/.agents/skills/maintain-searchswe-task/SKILL.md index 7a6f9c7..bbabb40 100644 --- a/.agents/skills/maintain-searchswe-task/SKILL.md +++ b/.agents/skills/maintain-searchswe-task/SKILL.md @@ -26,23 +26,24 @@ missing or incompatible tools block that validation, not justify replacements. 2. **Audit the task:** read [task audit](references/task-audit.md). Compare requirements/resources to grading, inspect isolation, solvability and licenses. Identify the closest formal task precedent independently; check the - contributor's stated references, structural reuse and deviations without - treating legacy behavior as authority or copying task-specific values. + contributor's stated references, structural reuse and deviations against + current contracts, and verify task-specific values independently. Missing, pending or failed checks never mean acceptance. Request explicit authorization before executing any PR-controlled tests/builds or code, and separately before credentialed runtime, API/GPU spend or downloads. Use a disposable isolated checkout; never unsafe checkout/reset in a dirty target. 3. **Promote approved work:** read [promotion and merge](references/promotion.md). Resolve allowed contributor branch edits first. A PR may contain multiple - tasks only under one contributor namespace; reject direct formal additions - and reused temporary ordinals. Assign IDs serially from the current base, - then give each task separate pure-rename and finalization commits in the - **same PR**. Preserve authors and do not automate commits. + uniquely named tasks at `task-submissions/`; reject direct formal + additions and reused submission paths. Check names against the current base + and other queued PRs. Promote to `tasks/` with the name unchanged, + using a separate pure rename and any necessary finalization commit per task + in the **same PR**. Preserve authors and do not automate commits. Merge-ready requires every task promoted. Each head change invalidates prior conclusions. 4. **Migrate assets:** read [publication](references/publication.md). Migrate assets separately for each task. The bundled - offline staging helper preserves old manifest entries and copies only new + offline staging helper preserves existing manifest entries and copies only new data. Hash/license review is separate from upload authorization. Merge official HF changes before pinning their SHA and merging GitHub with a merge commit. diff --git a/.agents/skills/maintain-searchswe-task/references/promotion.md b/.agents/skills/maintain-searchswe-task/references/promotion.md index 68dcbd8..48098b7 100644 --- a/.agents/skills/maintain-searchswe-task/references/promotion.md +++ b/.agents/skills/maintain-searchswe-task/references/promotion.md @@ -1,61 +1,55 @@ # Same-PR promotion and closed-loop merge -## Assign and rename, preserving authors +## Preserve the task name and authors Complete the audit and resolve allowed branch editing first. A PR may contain -multiple tasks only under exactly one contributor first-name namespace. Keep all -of them in the same PR throughout. Near merge, refresh the exact upstream -base/head and serialize each ID assignment with other queued PRs: inspect -`tasks/` on the current base, choose the next free positive number in the same -category, and recheck immediately before merge. Temporary category-local -ordinals are not final IDs. Never infer a number from examples or reserve it -long-term. Update -against the current base without rebasing away original contributor commits. -If base/head moves, re-review/revalidate affected work; a collision blocks merge. - -Use a clean disposable worktree and index at the reviewed PR head. Set `REPO` to -that checkout (not the user's dirty checkout), `SKILL_DIR` to this installed -folder, and `FINAL_ID` to the actually assigned ID. Bundled helpers require -Python 3.12+, Git for promotion, and explicit `--repo-root`; no target Python -imports or sibling skill. They never commit, push, publish or allocate an ID. +multiple uniquely named tasks under `task-submissions/`. Names use at +most five lowercase hyphen-separated words, start with a letter, omit `task-` +and cannot be `all`. Keep all tasks in the same PR. Refresh the upstream base +and head, inspect formal task names and queued PRs for collisions, and update +against the base without flattening contributor history. A changed base/head +requires renewed review and validation. + +Use a clean disposable checkout at the reviewed head. Set `REPO` to that checkout +and `SKILL_DIR` to this installed folder. Helpers require Python 3.12+, Git and +explicit `--repo-root`; they have no target Python imports or sibling-skill +dependency. They never commit, push or publish. Promotion keeps the task name: ```bash python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - rename task-submissions/alice/1-x-1 "$FINAL_ID" --dry-run + rename task-submissions/example-search example-search --dry-run python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - rename task-submissions/alice/1-x-1 "$FINAL_ID" + rename task-submissions/example-search example-search git -C "$REPO" diff --cached --summary ``` -Repeat rename and finalize independently for every task, using separate commits; -a rename for one task must not include another. Rename performs **only `git mv`** -and refuses dirty worktrees, collisions, traversal, symlinks and category -mismatches. Review the staged diff: every package -file must be R100 identical, with no incidental changes. Stop for the operator's -explicitly authorized separate pure-rename commit; never auto-commit. Retain the -contributor's existing commits/authors; use the maintainer's real identity for -maintainer changes, not forged author flags or global identity changes. +Repeat independently for each task; a rename commit contains exactly one whole +package move. Rename performs only `git mv` and refuses dirty worktrees, +collisions, traversal, symlinks and a target name different from the submission. +Review all staged files as R100 identical, with no incidental edits. The operator +creates the separately authorized pure rename commit. Preserve original author +commits and use real maintainer identity for maintainer changes. ```bash -# Only after the operator creates that separate pure-rename HEAD commit: +# After the operator creates that separate pure-rename HEAD commit: python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - finalize task-submissions/alice/1-x-1 "$FINAL_ID" --dry-run + finalize task-submissions/example-search example-search --dry-run python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - finalize task-submissions/alice/1-x-1 "$FINAL_ID" + finalize task-submissions/example-search example-search ``` -Finalize requires the entire package to have moved in a single-parent pure-rename -HEAD commit. It changes temporary IDs/submission paths in tracked text only; -it refuses nonstandard binary/filename rewrites. Review all replacements, -relative mounts, image/URL paths and author fields manually. Update task lists, -docs and explicit GPU test inventories only as needed. It does not edit downloaded -data or assign source licenses. Collision/unsupported reference means stop and -coordinate another reviewed rename/finalization, not overwrite or flatten history. +Finalize requires a single-parent, whole-package, pure rename HEAD commit. +It changes submission paths in tracked text only, preserving the task name and +canonical identity. It refuses unsupported binary rewrites. Review replacements, +relative mounts, image/URL paths and authors. Update task lists, docs and GPU +test inventories as needed. It does not edit downloaded data or assign licenses. +Resolve collisions before promotion; never overwrite another task or flatten +history. Keep all finalization edits separate from the pure rename commit. ## Assets and finalization commit Read [publication.md](publication.md). Prepare only the new official files under -`tasks//...` using the bundled HF helper; preserve old manifest entries, +`tasks//...` using the bundled HF helper; preserve existing manifest entries, model pins, provenance/license metadata. Its hash checks are offline and do not prove remote freshness or grant upload permission. Official publication and HF PR merge each require authorization. Merge official HF changes **first**, then @@ -65,9 +59,9 @@ PR SHA or mutable `main`. Verify clean downloads at that pin. Run the audited target tools and relevant runtime layers, with approvals: `check_release.py`, `check_submission.py --merge-ready`, full scripts/tests, scaffold regression tests if changed, task-specific cases and `git diff --check`. -HF migration is per task. Confirm no submission `task.toml` or temporary +HF migration is per task. Confirm no submission `task.toml` or submission-path references remain for any task. Review and let the authorized operator create -each separate finalization commit in **the same PR**. Helpers do not commit. Before an authorized push, verify remote head still +each needed finalization commit in **the same PR**; no empty commit is required. Helpers do not commit. Before an authorized push, verify remote head still matches the reviewed starting head, branch/permissions and intended commits; use a normal push, not force. Any new head requires fresh review/check evidence. @@ -91,7 +85,7 @@ resolved conversations, `review-stage`, `merge-ready` and other configured check must pass for that exact head and current base. Missing/pending checks block. If protections/merge queue prevent the specified method, stop, do not bypass. -After explicit merge authorization and fresh head/base/ID checks: +After explicit merge authorization and fresh head/base/name checks: ```bash gh pr merge PR --repo OWNER/REPO --merge --match-head-commit FULL_REVIEWED_HEAD_SHA @@ -111,11 +105,11 @@ Close the loop after the command, rather than treating exit zero as proof: object has two parents, the accepted PR head is its second parent, the reviewed base is its first parent, and original author commits are ancestors. Unexpected parent/base/tree changes require investigation, not a success claim. -3. Inspect the merged tree for `tasks//task.toml`, actual authors, final - official pins, no submission package/temporary references and intended files. +3. Inspect the merged tree for `tasks//task.toml`, actual authors, final + official pins, no submission package/submission-path references and intended files. Compare it with the reviewed final tree (account for reviewed base integration). -4. Check `git log --follow -- tasks//instruction.md` and - `git blame tasks//instruction.md` at the merge SHA. Original unmodified +4. Check `git log --follow -- tasks//instruction.md` and + `git blame tasks//instruction.md` at the merge SHA. Original unmodified lines should retain contributor attribution. GitHub's latest file author and contributor statistics are not substitutes; stats also depend on linked email. 5. Report repo/PR, reviewed head/base, merge SHA, final path, official HF SHA, diff --git a/.agents/skills/maintain-searchswe-task/references/publication.md b/.agents/skills/maintain-searchswe-task/references/publication.md index da599e6..663a97a 100644 --- a/.agents/skills/maintain-searchswe-task/references/publication.md +++ b/.agents/skills/maintain-searchswe-task/references/publication.md @@ -34,12 +34,12 @@ immutable SHA in `assets.json`. The explicit submission downloader/launcher then support real pre-promotion validation. Keep the temporary repo accessible through review and migration; only retire it after official pinned downloads are verified. -### Final ID: additive official publication, then GitHub merge +### Task name: additive official publication, then GitHub merge -After ID assignment/finalization, change only the new task's dataset source -repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve +After task-name approval and promotion, change only the new task's dataset source +repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve original model repo pins and bundled metadata. Prepare **only new data files** -in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. +in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. Do not delete, rename or overwrite existing official assets. Fetch the small current official manifest snapshot and metadata, not the whole @@ -60,16 +60,18 @@ print(sha) PY python "$SKILL_DIR/scripts/prepare_hf_upload.py" --repo-root "$REPO" \ --official-manifest /path/to/official-snapshot/manifest.json \ - --task-path tasks/task-1-6 --new-data /path/to/new-data \ + --task-path tasks/example-search --new-data /path/to/new-data \ --output /path/to/upload-staging ``` -This offline helper verifies new files' hashes, refuses collisions/traversal/ -symlinks, and writes only new files plus a merged manifest preserving every old -record. It does not require old data or upload anything. A trusted current -snapshot is essential: it cannot prove remote freshness offline. Unlike the trusted target tool -`check_release.py --hf-data`, which intentionally checks the **full inventory**, -this is the incremental contribution path, not a replacement full release audit. +Official asset staging requires a promoted package at `tasks/`. + +The offline helper verifies hashes and safe paths for new files, rejects +collisions and symlinks, and stages those files with a merged manifest that +preserves all existing records. Supply a trusted current manifest snapshot; +only the new data files are needed locally. Uploading is a separate step. +For a full dataset audit, `check_release.py --hf-data` checks the complete +inventory against a local dataset copy. Copy the snapshot `README.md`, `SOURCES.md`, and `.gitattributes` into staging, then **append** provenance, license/redistribution information, and any required @@ -87,7 +89,7 @@ result = HfApi().upload_folder( repo_id='search-swe/Search-SWE', repo_type='dataset', folder_path='/path/to/upload-staging', parent_commit=Path('/path/to/official-snapshot/BASE_SHA').read_text().strip(), - create_pr=True, commit_message='Add task-1-6 inputs and provenance', + create_pr=True, commit_message='Add example-search inputs and provenance', ) print(result.pr_url) PY @@ -106,4 +108,4 @@ replace a newer manifest with a stale one. GitHub task's `assets.json`. Verify downloads using a fresh output directory or cache and run release checks. Commit finalization and merge the GitHub PR last. No new task with unpublished/unpinned assets or an unfinished submission enters -main. Existing tasks' old pins and assets remain valid. +main. Preserve existing tasks' pinned revisions and published assets. diff --git a/.agents/skills/maintain-searchswe-task/references/task-audit.md b/.agents/skills/maintain-searchswe-task/references/task-audit.md index f703207..97ea661 100644 --- a/.agents/skills/maintain-searchswe-task/references/task-audit.md +++ b/.agents/skills/maintain-searchswe-task/references/task-audit.md @@ -6,16 +6,17 @@ workflow/tool modifications need independent scrutiny, not implicit acceptance. | Area | Evidence required | | --- | --- | -| Identity/scope | A PR may add multiple tasks, all under exactly one `task-submissions//` namespace. Each `<1|2>-x-` path has the exactly matching canonical ID (for example `task-1-x-1`), with ordinals unique within category in the PR/checkout and never reused after promotion. Reject new tasks added directly under `tasks/`. First name, not username; explicit `alice-2` for a different same-name contributor; actual `task.toml` authors and preserved original commits. No guessed/reserved final IDs. Existing-task revisions stay in place. | -| Repository precedent | For each new task, the contributor names one or two closest formal `tasks/` packages or states that none is close, and explains matching dimensions, structural reuse and intentional differences. Independently inspect the closest analogue by mode, grading/judge shape, resources, hardware and data/artifact layout. Current contracts and validators override legacy examples. Reject unexplained copying of task-specific IDs/authors, datasets or HF pins, thresholds/baselines, allowlists, budgets, licenses/provenance or hidden evaluation design. Merely existing on main is not proof that a pattern remains valid. | +| Identity/scope | Each new task starts at `task-submissions//`, with `task.name = "search-swe/"`. Names use at most five lowercase hyphen-separated words, start with a letter, omit `task-`, and cannot be `all`. A PR may add multiple unique names; reject reused submission paths and direct new formal additions. Promotion preserves the same name under `tasks/`. Require actual authors and original author commits. Existing-task revisions stay in place; reviewed renames of existing formal packages are not new contributions. | +| Repository precedent | For each new task, the contributor names one or two closest formal `tasks/` packages or states that none is close, and explains matching dimensions, structural reuse and intentional differences. Independently inspect the closest analogue by objective, grading/judge shape, resources, hardware and data/artifact layout. Current contracts and validators override legacy examples. Reject unexplained copying of task-specific IDs/authors, datasets or HF pins, thresholds/baselines, allowlists, budgets, licenses/provenance or hidden evaluation design. Merely existing on main is not proof that a pattern remains valid. | | Instructions | Goal, starting state, input/output schema and absolute paths, submission command/interface, artifacts, public gates/thresholds and constraints match every graded requirement. No secret-only correctness requirement or hidden solution hints. | -| Mode/resources | Only Implementation (`create`) or Optimization (`optimize`); meaningful optimization baseline. CPU default, GPU only for execution. Both Dockerfiles, both Compose reservations and `gpus` agree. Visible tools/API/model docs match injected resources and budgets. Pin dependencies/images; inspect build contexts. | +| Objective/resources | Clear goal, starting state and measured outcomes. Supply a meaningful baseline when grading improvement. CPU default, GPU only for execution. Both Dockerfiles, both Compose reservations and `gpus` agree. Visible tools/API/model docs match injected resources and budgets. Pin dependencies/images; inspect build contexts. | | Solvability | Real known-good submission through the same separate verifier/artifact-transfer interface, expected vs observed reward and an authorized coding-agent trial. A scaffold's always-zero test, empty manifest, dry-run or static pass is not a working benchmark. No public author-only solution without approval. | +| Model feedback | Trajectory, score and judge health inspected; task-setting changes explained, or a reason recorded for retaining the setting. Substantive task/scoring changes have relevant checks and a new model trial within the authorized budget, tied to the final evaluated revision. Missing evidence remains explicit; a valid low score alone is not a reason to weaken the task. | | Grader isolation | Separate verifier; hidden tests/labels/references absent from agent mounts/images. Submitted code actually runs unprivileged with timeout, restricted environment, no grader/reward writes or judge keys. Root in a separate container can still tamper with grading. Reward initializes to zero, numeric finite [0,1], failures propagate, cleanup/logs work. | | Negative cases | Missing/malformed/duplicate output, timeout, process failure, attempts to read hidden labels or write rewards; no positive fallback on integrity/judge failure. Transfer all declared artifacts; no reliance on live agent sidecars. | | Heldout validity | Public validation and heldout inputs exercise the same contract without leakage/answer memorization. Public repo `tests/data` is hidden only from the runtime agent, not private. No private answers/secrets in Git or public HF. | | Network/credentials | Narrow phase-specific allowlists, exact providers/models, no unauthorized calls/spend, runtime-only secret injection, judge keys filtered from submitted processes, no tokens in image/history/logs/artifacts or CLI args. Verify actual backend enforcement. | -| Data/license | Actual bytes match manifest size/SHA-256; immutable 40-hex HF pins; dataset/model/local metadata paths valid. Source provenance, redistribution rights and all license obligations documented, including derived data/models. No guessed license or permanent official temporary-ID paths. | +| Data/license | Actual bytes match manifest size/SHA-256; immutable 40-hex HF pins; dataset/model/local metadata paths valid. Source provenance, redistribution rights and all license obligations documented, including derived data/models. No guessed license; official dataset paths use `tasks//...`. | | Reproducibility | Fresh downloads/hash verification, read-only minimal mounts with missing bind paths failing closed, no host venv/cache dependence, resource/time limits, deterministic grading or justified pinned judge, repeatability and failure diagnoses. | ## Target tools and staged checks @@ -32,7 +33,7 @@ image access and appropriate GPU driver/Container Toolkit are needed for runtime After code-execution authorization, from the isolated checkout with trusted tools: ```bash -task_path=task-submissions/alice/1-x-1 # Actual reviewed path; final tasks/ later +task_path=task-submissions/example-search # Actual reviewed path; final tasks/ later python scripts/check_submission.py "$task_path" python scripts/check_release.py python -m unittest discover -s scripts/tests -p 'test_*.py' diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 0ea4422..0dc0bf7 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,30 +1,31 @@ ## Summary -Submission paths (new tasks): `task-submissions//<1|2>-x-` -Category / Implementation or Optimization / CPU or GPU: +Task name(s) and submission path(s): `task-submissions/` +Goal, starting system, and CPU/GPU requirements: Authors and coauthors: ## Validation evidence -List commands, actual results, and links to task-specific known-good/negative-case +List commands, actual results, and task-specific known-good/negative-case results. List unrun checks and why (data, GPU, credentials, paid APIs). +For task contributions, include the coding-model trial's model/effort, trajectory, +scores and judge health; explain task-setting changes motivated by that feedback +(or why none were needed) and identify the final evaluated revision. ## Contributor checklist -- [ ] All new tasks use exactly one contributor first-name namespace; temporary ordinals are category-unique and never reused after promotion; no direct formal additions or guessed final IDs. -- [ ] Actual authors in `task.toml`; commit email associated with my GitHub account. -- [ ] Allow edits from maintainers; promotion stays in this same PR. +- [ ] New tasks use unique descriptive names of at most five lowercase hyphen-separated words and follow `task-submissions/`; each path identifies one task throughout this PR. +- [ ] Actual authors in `task.toml`; original author commits preserved. +- [ ] Allow edits from maintainers, or agree on contributor-performed promotion in this PR. - [ ] No secrets, downloaded inputs/models, job outputs, or author-only solutions. -- [ ] Static package/submission checks and applicable runtime checks recorded. -- [ ] Asset provenance, redistribution rights, licenses and pinned development SHA documented. +- [ ] Static checks and applicable runtime evidence recorded. +- [ ] Provenance, redistribution rights, licenses and immutable development asset SHA documented. -## Maintainer merge checklist (new tasks) +## Maintainer merge checklist -- [ ] Design, verifier isolation and runtime evidence reviewed; exceptions explicitly approved. -- [ ] Updated to latest main; every final number assigned serially without collision. -- [ ] Separate pure `git mv` and reviewed finalization commits per task, in this PR. -- [ ] Per-task HF community PR or maintainer mirror merged into official dataset; manifest, - SOURCES and license updated, all old assets preserved; official SHA pinned in `assets.json`. -- [ ] Repository inventories/docs updated; full checks and merge-ready gate pass; - every task is promoted and no submission `task.toml` remains. -- [ ] **Merge commit only** (no squash/rebase); contributor history retained. +- [ ] Design, verifier isolation, runtime evidence and any validation exceptions reviewed. +- [ ] Names checked against the current base and queued PRs; each task keeps its name through promotion. +- [ ] Every new task has a separate pure move from submissions in this PR, followed by reviewed finalization. +- [ ] New official assets published under `tasks//`; previous assets preserved; merged official SHA pinned. +- [ ] Repository docs and inventories updated; checks and merge-ready pass; no submission package remains. +- [ ] Merge commit only; original contributor history retained. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f5b2ff7..0564114 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,14 +1,21 @@ # Contributing to Search-SWE -Start with the [contribution guide](docs/contributing.md) for the complete task -workflow, validation commands, asset publication, and maintainer merge checklist. -For agent-assisted authoring, use the -[create-searchswe-task skill](.agents/skills/create-searchswe-task/SKILL.md). - -New tasks belong at -`task-submissions//-x-`, **not** in -`tasks/`. Use your chosen ASCII first name, not your GitHub username. One PR may -add multiple tasks, but all must use exactly one contributor namespace. -Maintainers assign each final number and promote every task in the **same PR**, -with a separate pure rename commit followed by finalization. New-task PRs must -use a **merge commit**, never squash or rebase merge. +Start with the [contribution guide](docs/contributing.md) for task design, +validation, asset publication, and maintainer promotion. For agent-assisted +authoring, use the [create-searchswe-task skill](.agents/skills/create-searchswe-task/SKILL.md). + +New tasks belong at `task-submissions/`. Choose a descriptive name of +at most five lowercase, hyphen-separated words, such as `example-search`. +The `task-` prefix and `all` are reserved. One PR may add multiple tasks; record +actual authors in each package. + +Validate each task with grader checks and a configured coding-model trial. +Use the trajectory and evaluation results to identify and correct task-setting +issues, then validate the revised setting and record the final tested version. +If required API keys are missing, the authoring agent should ask the contributor +to configure them locally before running the trial; see the guide for details. + +Maintainers promote each reviewed package to `tasks/` in the same PR, +preserving its name and original authors. Commit the pure directory move +separately from finalization changes. Use a merge commit for task contributions, +not squash or rebase merge; every submission must be promoted before merging. diff --git a/docs/contributing.md b/docs/contributing.md index 90e972c..2d032c4 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -8,197 +8,172 @@ Reuse the [shared CPU/GPU images](../docker/README.md). Keep the English and Chinese README overviews aligned. Publication, remote configuration, and merges require separate authorization; local task creation does not authorize them. -## 1. Start one or more tasks in one contributor namespace +## 1. Start tasks with descriptive names -Fork the repository, clone your fork, and create a task branch. From the -checkout root, use Python 3.12+ and install the host tools and static-test dependencies: +Fork the repository, clone your fork, and create a task branch. From the checkout +root, use Python 3.12+ and install the host tools and static-test dependencies: ```bash python -m venv .venv source .venv/bin/activate python -m pip install -r scripts/requirements.txt PyYAML python-dotenv +python .agents/skills/create-searchswe-task/scripts/scaffold_task.py example-search \ + --author 'Alice Example' --hardware cpu ``` -Do not guess or reserve final task numbers. Use: - -```text -task-submissions//1-x-1/ # internal ID: task-1-x-1 -task-submissions//1-x-2/ # second category 1 task -task-submissions//2-x-1/ # internal ID: task-2-x-1 -``` - -A PR may add multiple tasks, but all must use exactly one contributor first-name -namespace. The positive ordinal is temporary and unique within its category in -that PR/checkout; it is not the final task number. Do not delete a promoted -temporary path and reuse its ordinal later in the same PR. The canonical ID must -exactly match the directory with a `task-` prefix. - -Supply your **first name, not your GitHub username**. ASCII lowercase slugs -match `[a-z][a-z0-9]*(?:-[a-z0-9]+)*`; spaces/separators become `-`. -`Alice` becomes `alice`; a compound first name `Mary Jane` becomes `mary-jane`. -For non-Latin names, choose an ASCII transliteration. The scaffolder reuses the -same local contributor directory for additional task ordinals. Check active PRs -for a *different person* with the same first name: that contributor explicitly -passes `Alice-2`, then `Alice-3`; suffixes are not generated automatically from -task collisions. This is a temporary readable namespace, not proof of identity -or permanent attribution. - -```bash -python .agents/skills/create-searchswe-task/scripts/scaffold_task.py task-1-x-1 \ - --submission-first-name Alice --author 'Alice Example' \ - --mode implementation --hardware cpu -``` - -Repeat `--author` for coauthors. `task.toml` must name actual task authors, not -only “Search-SWE”. Use a Git email associated with your GitHub account (or its -noreply address). Co-authored-by trailers supplement, not replace, authors and -original commits. The scaffold is deliberately incomplete and fail-closed. -Existing-task revisions stay in their formal directory. Regular scaffolding -without `--submission-first-name` remains available for maintainer/legacy use. - -Do not commit credentials, downloaded inputs/models, hidden private evaluation -assets, job outputs, or author-only solutions. Keep tests isolated from the -agent, including any public repository test fixtures. Ignore runtime data and -models, and use an author-only copy for reference solutions. +New packages use `task-submissions//`. Choose a concise description +with at most five lowercase hyphen-separated words, starting with an ASCII +letter and otherwise using letters/digits. Omit the `task-` prefix; `all` is +reserved. Check both `tasks/` and `task-submissions/`, plus active PRs, for name +collisions. The canonical `task.name` is `search-swe/` from the start, +and promotion preserves it. A PR may contain multiple uniquely named tasks; +do not reuse a submission path later in that PR. + +Describe the task's goal, starting state and measured outcomes. +CPU is the default; choose `--hardware gpu` for actual GPU execution. Repeat +`--author` for coauthors. Use actual authors in `task.toml`, not only “Search-SWE”, +and a Git email associated with your GitHub account (or its noreply address). +Co-authored-by trailers supplement original commits and task authors. + +The scaffold is deliberately incomplete and its verifier fails closed. Existing +task revisions stay in their formal directory. The explicit `--formal` scaffold +option supports maintainer tooling; new contributions still follow this workflow. +Do not commit credentials, downloaded inputs/models, private evaluation assets, +job outputs or author-only solutions. Keep verifier inputs isolated from agents. ## 2. Develop and validate before promotion -Development assets may live in a **personal public temporary HF dataset**, with +Development assets may live in a personal public temporary HF dataset with redistribution permission, manifest, provenance and license. Pin its immutable -40-character commit SHA in `assets.json`; arbitrary HF repos already work. -Do not publish temporary IDs as permanent official dataset paths or image tags. +40-character commit SHA in `assets.json`. Official data will use +`tasks//...`; model files retain their original repository pins. See [asset contribution and publication](#asset-contribution-and-publication). ```bash -python scripts/check_submission.py task-submissions/alice/1-x-1 +python scripts/check_submission.py task-submissions/example-search python scripts/check_release.py python -m unittest discover -s scripts/tests -p 'test_*.py' python .agents/skills/create-searchswe-task/scripts/test_scaffold_task.py git diff --check -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 --dry-run +python scripts/download_assets.py --task-path task-submissions/example-search --dry-run # After approving the download budget: -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 --verify-only -python scripts/run_task.py --task-path task-submissions/alice/1-x-1 \ +python scripts/download_assets.py --task-path task-submissions/example-search +python scripts/download_assets.py --task-path task-submissions/example-search --verify-only +python scripts/run_task.py --task-path task-submissions/example-search \ --agent pi --model deepseek/deepseek-flash --dry-run ``` -Both downloader and launcher require canonical repository-relative paths with -no traversal or symlinks. `--task all` remains formal-only (downloader); the -launcher still runs one selected task. Default submission outputs are under -`jobs/task-submissions/alice/1-x-1`, not a shared `jobs/1-x-1`. Alternate downloader -`--output-dir` also retains `task-submissions/alice/1-x-1`. The launcher uses the +Downloader and launcher require canonical repository-relative paths without +traversal or symlinks. `--task ` selects a formal package; downloader +`--task all` selects only formal packages. Submissions require `--task-path`. +Submission jobs default to `jobs/task-submissions/` and alternate +asset output roots retain `task-submissions/`. The launcher reads the original package's local assets, not an alternate download directory. Follow the skill's [validation layers](../.agents/skills/create-searchswe-task/references/validation.md): Compose rendering, builds, verifier known-good and negative cases, and an -end-to-end trial when authorized. Use a fresh `--output` for each trial. Static -checks alone do not prove solvability or safe grading. Record commands, actual -results, resource requirements, and unrun layers with reasons in the PR. -Enable **Allow edits from maintainers** so promotion stays in the same PR. -The PR author can enable this on the open PR after creation. For same-repository -branches it may be unnecessary; organization-owned forks or repository policies -can restrict availability. If unavailable, the contributor performs the same -promotion commits after the maintainer assigns the number. +end-to-end trial with a configured coding model when authorized. Use a fresh +`--output` for each trial. Inspect the model's trajectory, submitted artifacts, +scores and judge health, then use that feedback to refine ambiguous instructions, +missing resources, inconsistent limits or grading defects. Recheck affected +validation layers and rerun the model when changes affect its task or scoring, +within the agreed budget. A valid low score alone does not justify easing the +task or rerunning an unchanged setting. Record findings, resulting changes (or +why none were needed), and the final evaluated revision. The skill's +[model-evaluation workflow](../.agents/skills/create-searchswe-task/references/model-evaluation.md) +defines this feedback loop and its stopping conditions. + +Before the trial, the authoring agent checks which credentials the chosen model, +task APIs and judges require. If any are missing, it must tell the contributor +the exact variable names, their purpose, and where to configure them locally, +using [`.env.example`](../.env.example) and the [evaluation guide](evaluation.md). +Reuse existing authorized credentials; never ask for key values in chat or a +PR. Wait for configuration before starting the dependent run, while continuing +independent checks. Report unavailable runtime evidence explicitly. + +Static checks alone do not prove solvability or safe grading. Report actual +commands, results, resource requirements and unrun layers. Enable **Allow edits +from maintainers** on the PR; if unavailable, the contributor makes the agreed +promotion commits in the same PR. ### CI and trust boundaries The unprivileged `pull_request` and `push` workflow checks formal packages and -**all** submission packages, including malformed/incomplete package paths. -For PRs it permits multiple newly added `task.toml` files but requires all -submission paths touched anywhere between the full base SHA and HEAD—including -packages already promoted and both sides of merge commits—to use exactly one -contributor namespace. It rejects reused temporary paths and newly added formal -tasks without submission history. The current submission tree is checked too. -The submission validator reuses release structure, -asset pin/hash, secret, file-size, and Git-ignore checks and parses Python, -shell, and Compose YAML without running task code. Install `PyYAML` for its -Compose syntax layer; full Docker Compose rendering remains a separate check. - -`review-stage` may pass while submissions exist. **`merge-ready` must also pass -before merge**: `python scripts/check_submission.py --merge-ready` refuses any -remaining submission `task.toml` and repeats static checks on formal packages, -including syntax and temporary-reference checks. The PR base check also requires -actual authors on newly added formal packages; legacy author records are unchanged. -Every submission is promoted in this PR before merge; merge-ready passes only -after all temporary packages are gone. No GPU/API/container task execution occurs in CI. Never use -`pull_request_target` to run contributor code with secrets, share an official -HF token, or run an unreviewed task with maintainer credentials. Even repository -unit tests are PR-controlled code: CI has no secrets, read-only permissions, -and checkout does not persist credentials. Review and separately authorize -credentialed/runtime testing in a disposable trusted environment. +all submissions, including malformed or incomplete paths. With `--base`, it +also inspects every commit and both sides of merge history: submission paths +must be flat, paths cannot be reused, and each newly added formal task needs a +same-name, whole-package pure rename from its submission in this PR. Reviewed +renames of packages already formal on the base are recognized as existing-task +changes. New formal tasks require actual authors; existing author records remain. + +Submission checks reuse release structure, asset pin/hash, secret, file-size +and Git-ignore checks, and parse Python, shell and Compose YAML without running +task code. Install PyYAML for the YAML check; Docker Compose rendering is separate. +`review-stage` may pass while submissions exist. Before merging, run +`python scripts/check_submission.py --merge-ready`: it rejects remaining +submissions and checks formal package syntax and obsolete references. + +CI executes no GPU/API/container task trials. Repository tests are PR-controlled +code: use read-only permissions, no secrets and no persisted checkout credentials. +Never execute unreviewed PR code via `pull_request_target` with secrets or share +an official HF token. Review and separately authorize credentialed/runtime tests +in a disposable trusted environment. ## 3. Maintainer promotion in the same PR -For self-contained gh review, trust gates, portable helpers and closed-loop merge -checks, use [maintain-searchswe-task](../.agents/skills/maintain-searchswe-task/SKILL.md). -The repository `scripts/promote_task.py` and `scripts/prepare_hf_upload.py` remain -compatible entrypoints; their implementation lives in that portable skill. When -copying only the skill, invoke its scripts with explicit `--repo-root /path/to/checkout`. +Use [maintain-searchswe-task](../.agents/skills/maintain-searchswe-task/SKILL.md) +for gh review, audit and merge checks. Repository `scripts/promote_task.py` and +`scripts/prepare_hf_upload.py` delegate to that portable skill; a separately +installed skill uses its own scripts with `--repo-root /path/to/checkout`. Finish design, provenance/license, environment, verifier-isolation and runtime -review first; explicitly approve any missing validation. Update the PR to the -latest main. Serialize each final ID assignment and merging (or use a separately -configured merge queue); inspect other queued PRs before selecting each next -free number in the same category. Temporary ordinals do not influence final -numbers. Do not reserve numbers long-term. +review. Update against current main while preserving author history. Recheck +task-name collisions with the base and queued PRs before promotion and merge. +Promotion keeps the task name unchanged. -Start with a clean worktree/index. The helper never commits, pushes or publishes. -Example only: replace `task-1-6` with the actual free final ID. +Start from a clean worktree/index. The helper never commits, pushes or publishes: ```bash -python scripts/promote_task.py rename task-submissions/alice/1-x-1 task-1-6 --dry-run -python scripts/promote_task.py rename task-submissions/alice/1-x-1 task-1-6 +python scripts/promote_task.py rename task-submissions/example-search example-search --dry-run +python scripts/promote_task.py rename task-submissions/example-search example-search git diff --cached --summary # Operator creates the separate pure-rename commit: -git commit -m 'Promote submission as task-1-6' -python scripts/promote_task.py finalize task-submissions/alice/1-x-1 task-1-6 --dry-run -python scripts/promote_task.py finalize task-submissions/alice/1-x-1 task-1-6 +git commit -m 'Promote example-search' +python scripts/promote_task.py finalize task-submissions/example-search example-search --dry-run +python scripts/promote_task.py finalize task-submissions/example-search example-search ``` -Repeat the rename/finalize sequence independently for every task, with a -separate pure rename and finalization commit for each. Rename does **only -`git mv`**. Finalize requires HEAD to be the operator's single-parent, -100%-identical rename commit for that entire package. It updates -canonical temporary IDs and the old submission path only in tracked package -text files, prints changed lines, and refuses nonstandard/binary reference -rewrites. It does not edit downloaded data. Review all replacements, filenames, -image references, Compose paths and external URLs. Update repository task -lists/docs/test inventories (including explicit GPU inventories) manually. -Finalize is intentionally conservative, not a general re-numbering tool. If a -number becomes occupied, stop; coordinate a separately reviewed pure rename -and finalization to a new free ID without overwriting another task or flattening -history. Keep the PR up to date and recheck serially immediately before merging. - -Finish official asset publication separately for each task, pin each merged -official SHA, then run full checks and applicable task-specific checks. Confirm -no temporary reference remains in any formal package and no submission -`task.toml` remains. Review and commit each finalization separately **in this -same PR**. Merge with a **merge commit -only**, not squash or rebase merge. +Repeat independently for each task. Rename does only `git mv`. Finalize requires +HEAD to be that task's single-parent, 100%-identical whole-package rename commit. +It updates the submission path in tracked text and prints changed lines; +the task name and canonical identity stay unchanged. It refuses unsupported +binary rewrites and does not edit downloaded data. Review replacements, mounts, +images and external URLs; update task inventories/docs and GPU test inventories +as needed. A name collision must be resolved through review before promotion, +never by overwriting a package. + +Publish approved new official assets, pin their merged SHA, then validate. Make +any reference, asset or integration changes in a separate finalization commit +in this same PR; an empty finalization commit is unnecessary. Every submission +must be promoted before merge. Use a **merge commit only**, preserving original +author and pure-rename commits; no squash/rebase merge. ```bash python scripts/check_release.py python scripts/check_submission.py --merge-ready python -m unittest discover -s scripts/tests -p 'test_*.py' git diff --check -git log --follow -- tasks/task-1-6/instruction.md -git blame tasks/task-1-6/instruction.md +git log --follow -- tasks/example-search/instruction.md +git blame tasks/example-search/instruction.md ``` -The original author commits and unmodified lines' blame survive this workflow. -GitHub's file list often shows the maintainer's latest change; History may not -follow renames as fully as `git log --follow`. Contributors statistics also +Original commits and unmodified lines' blame survive this workflow. GitHub file +lists/history may display renames differently; contributor statistics also depend on account-linked email and reachability from the default branch. - -Repository owners should separately configure required review, resolved review -conversations, current-base/queue checks, and required `review-stage` and -`merge-ready` statuses. Enforce merge-commit-only task PRs operationally if the -host cannot restrict merge methods by PR type. This change does **not** configure -remote protections, a merge queue, or CODEOWNERS identities. Maintainer review -is required regardless of automation. If branch editing is unavailable, resolve -that with the contributor before promotion; do not silently switch PRs or squash -away their commits. +Repository owners separately configure required review, resolved conversations, +current-base/queue checks and `review-stage`/`merge-ready` statuses. This workflow +does not configure remote protections or CODEOWNERS. Re-review each changed head; +resolve unavailable branch editing with the contributor, keeping the same PR. ## Asset contribution and publication @@ -232,12 +207,12 @@ immutable SHA in `assets.json`. The explicit submission downloader/launcher then support real pre-promotion validation. Keep the temporary repo accessible through review and migration; only retire it after official pinned downloads are verified. -### Final ID: additive official publication, then GitHub merge +### Task name: additive official publication, then GitHub merge -After ID assignment/finalization, change only the new task's dataset source -repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve +After task-name approval and promotion, change only the new task's dataset source +repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve original model repo pins and bundled metadata. Prepare **only new data files** -in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. +in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. Do not delete, rename or overwrite existing official assets. Fetch the small current official manifest snapshot and metadata, not the whole @@ -258,16 +233,18 @@ print(sha) PY python scripts/prepare_hf_upload.py \ --official-manifest /path/to/official-snapshot/manifest.json \ - --task-path tasks/task-1-6 --new-data /path/to/new-data \ + --task-path tasks/example-search --new-data /path/to/new-data \ --output /path/to/upload-staging ``` -This offline helper verifies new files' hashes, refuses collisions/traversal/ -symlinks, and writes only new files plus a merged manifest preserving every old -record. It does not require old data or upload anything. A trusted current -snapshot is essential: it cannot prove remote freshness offline. Unlike -`check_release.py --hf-data`, which intentionally checks the **full inventory**, -this is the incremental contribution path, not a replacement full release audit. +Official asset staging requires a promoted package at `tasks/`. + +The offline helper verifies hashes and safe paths for new files, rejects +collisions and symlinks, and stages those files with a merged manifest that +preserves all existing records. Supply a trusted current manifest snapshot; +only the new data files are needed locally. Uploading is a separate step. +For a full dataset audit, `check_release.py --hf-data` checks the complete +inventory against a local dataset copy. Copy the snapshot `README.md`, `SOURCES.md`, and `.gitattributes` into staging, then **append** provenance, license/redistribution information, and any required @@ -285,7 +262,7 @@ result = HfApi().upload_folder( repo_id='search-swe/Search-SWE', repo_type='dataset', folder_path='/path/to/upload-staging', parent_commit=Path('/path/to/official-snapshot/BASE_SHA').read_text().strip(), - create_pr=True, commit_message='Add task-1-6 inputs and provenance', + create_pr=True, commit_message='Add example-search inputs and provenance', ) print(result.pr_url) PY @@ -304,4 +281,4 @@ replace a newer manifest with a stale one. GitHub task's `assets.json`. Verify downloads using a fresh output directory or cache and run release checks. Commit finalization and merge the GitHub PR last. No new task with unpublished/unpinned assets or an unfinished submission enters -main. Existing tasks' old pins and assets remain valid. +main. Preserve existing tasks' pinned revisions and published assets. diff --git a/task-submissions/README.md b/task-submissions/README.md index bafbcb9..a142a96 100644 --- a/task-submissions/README.md +++ b/task-submissions/README.md @@ -1,37 +1,34 @@ -# Task submissions: review buffer, not a task registry +# Task submissions -Create new tasks at -`task-submissions//-x-`, with an -exactly matching internal canonical ID such as `task-1-x-1` or `task-2-x-1`. -Use a contributor-supplied first name, **not a username**: `Alice` → `alice`, -`Mary Jane` → `mary-jane`. For a -non-Latin name, supply your preferred ASCII transliteration. Slugs match -`[a-z][a-z0-9]*(?:-[a-z0-9]+)*`. +Create new tasks at `task-submissions/`, with `[task].name` set to +`search-swe/`. Use at most five lowercase, hyphen-separated words, +starting with a letter, such as `example-search`. The `task-` prefix and `all` +are reserved. Names must be unique across submissions and +formal tasks; check active PRs for conflicting names before handoff. -A PR may add multiple tasks, but all use exactly one contributor namespace. -Reuse that directory and choose temporary positive ordinals unique within each -category in the PR/checkout; ordinals are not final IDs. Do not reuse an ordinal -after its task is promoted in that PR. Check active PRs for a different person -with the same first name; that contributor explicitly chooses -`alice-2`, then `alice-3`. The scaffolder reuses existing namespaces and does -not derive name suffixes from task collisions. A name is not identity -authentication or permanent credit; use real `task.toml` authors and correct -Git author metadata. +One PR may add multiple tasks. Use actual `task.toml` authors and correct Git +author metadata. ```bash -python .agents/skills/create-searchswe-task/scripts/scaffold_task.py task-1-x-1 \ - --submission-first-name Alice --author 'Alice Example' \ - --mode implementation --hardware cpu -python scripts/check_submission.py task-submissions/alice/1-x-1 -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 --dry-run -python scripts/run_task.py --task-path task-submissions/alice/1-x-1 \ +python .agents/skills/create-searchswe-task/scripts/scaffold_task.py example-search \ + --author 'Alice Example' --hardware cpu +python scripts/check_submission.py task-submissions/example-search +python scripts/download_assets.py --task-path task-submissions/example-search --dry-run +python scripts/run_task.py --task-path task-submissions/example-search \ --agent pi --model deepseek/deepseek-flash --dry-run ``` -A scaffold intentionally fails verification until implemented. `--task all` -only discovers formal `tasks/*/task.toml` packages; submissions need explicit -`--task-path`. Read the [complete contribution guide](../docs/contributing.md). -No submission is merged unfinished. A maintainer performs a separate pure -rename and finalization for every task in that same PR; all must be promoted -before a **merge commit**. Official asset migration is per task. -This README stays on main; submission task packages do not. +A scaffold intentionally fails verification until implemented. Automatic task +discovery only includes formal `tasks/*/task.toml` packages; submissions need +explicit `--task-path`. See the [complete contribution guide](../docs/contributing.md). + +Before promotion, run a configured coding-model trial alongside grader checks. +Use the trajectory and scores to refine the task setting where needed, revalidate +substantive changes, and report the final tested revision and any missing evidence. +The authoring skill explains how to request local configuration of missing API keys. + +Promote every package to `tasks/` in the same PR, preserving its +name. Each pure move is committed separately from finalization; preserve the +contributor's commits with a merge commit. Do not reuse a submission path in +the same PR after removing or promoting it. No submission package enters main; +this README stays as the workflow entry point. From a45cd5fd5f8eed9acc3b3e47f2f80747ece26060 Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Fri, 9 Oct 2026 20:54:16 +0800 Subject: [PATCH 4/4] Align benchmark and evaluation documentation with named tasks Synchronize the English and Chinese overviews, task commands, asset paths, credential examples, and network guidance. Describe the current benchmark and contribution process directly. --- README.md | 75 +++++++++++++++++++++--------------------- README_zh.md | 71 +++++++++++++++++++-------------------- docs/assets.md | 36 ++++++++++---------- docs/benchmark.md | 15 +++++---- docs/evaluation.md | 58 ++++++++++++++++---------------- docs/network-policy.md | 28 ++++++++-------- docs/quickstart.md | 16 ++++----- 7 files changed, 149 insertions(+), 150 deletions(-) diff --git a/README.md b/README.md index c08767f..477aebd 100644 --- a/README.md +++ b/README.md @@ -30,10 +30,8 @@ write and run code, test their systems, and iterate toward an executable submission. Evaluation measures the behavior of the resulting system, including retrieval quality, functional correctness, and resource use. -| Mode | Agent objective | Example problems | -| --- | --- | --- | -| Implementation | Build a working search capability from a task specification. | Reasoning-assisted retrieval; memory-constrained vector search. | -| Optimization | Improve search quality or efficiency within fixed constraints. | Long-document reranking; embedding fine-tuning; query-encoder optimization. | +Tasks cover reasoning-assisted retrieval, memory-constrained vector search, +long-document reranking, embedding fine-tuning, and query-encoder alignment. Each task specifies its inputs, submission interface, evaluation criteria, and resource budget. See the [benchmark design](docs/benchmark.md) for how @@ -42,7 +40,7 @@ models come from. ## 🚀 Quick Start -This walkthrough runs `task-1-1` on CPU with the Pi coding agent and DeepSeek +This walkthrough runs `reasoning-query-rewriting` on CPU with the Pi coding agent and DeepSeek Flash. Search-SWE requires Python 3.12 or newer and Docker with the CPU, memory, storage, and optional GPU capacity of the task you select. @@ -63,8 +61,8 @@ packages are installed inside Docker. ### 2. Restore the task inputs ```bash -python scripts/download_assets.py --task task-1-1 -python scripts/download_assets.py --task task-1-1 --verify-only +python scripts/download_assets.py --task reasoning-query-rewriting +python scripts/download_assets.py --task reasoning-query-rewriting --verify-only ``` The downloader checks sizes and SHA-256 checksums and reuses valid files. @@ -92,7 +90,7 @@ VERIFIER_OPENAI_API_KEY=YOUR_OPENROUTER_KEY `DEEPSEEK_API_KEY` authenticates the Pi agent for `deepseek/deepseek-flash`. The `VERIFIER_*` pair runs the independent `deepseek/deepseek-v4.1-flash` trajectory judge through OpenRouter and RewardKit 0.2.0 and is required by every task except -`task-2-4`. Use an OpenRouter key for `VERIFIER_OPENAI_API_KEY`; the Pi agent +`agentic-search`. Use an OpenRouter key for `VERIFIER_OPENAI_API_KEY`; the Pi agent uses its separate DeepSeek key. The launcher passes verifier credentials only to the verifier container. @@ -104,8 +102,8 @@ For other runs, fill only the matching sections already present in `.env`: Anthropic's official API by default. - OpenRouter Agent mode: `AGENT_OPENROUTER_API_KEY`, `--openrouter`, and a full `provider/model` slug with Codex or Claude Code. -- Task 1-3: the three `ANSWER_JUDGE_*` values are also required. -- Optional submission APIs: use `TASK_1_1_OPENROUTER_API_KEY`, +- `scientific-paper-qa`: the three `ANSWER_JUDGE_*` values are also required. +- Optional submission APIs: use `REASONING_QUERY_REWRITING_OPENROUTER_API_KEY`, `OPENROUTER_API_KEY`, or `JINA_API_KEY` only for the tasks identified by the comments in `.env.example`. @@ -118,17 +116,17 @@ custom endpoints, proxies, and the complete per-task matrix. ### 4. Run with Pi and DeepSeek Flash ```bash -bash scripts/run_task.sh --task task-1-1 --agent pi \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent pi \ --thinking xhigh --dry-run -bash scripts/run_task.sh --task task-1-1 --agent pi \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent pi \ --thinking xhigh \ - --output jobs/task-1-1-pi-deepseek + --output jobs/reasoning-query-rewriting-pi-deepseek ``` The dry run only prints the Harbor command. The second command builds the task images, runs the agent and the separate verifier, and writes the reward and job -records under `jobs/task-1-1-pi-deepseek`. +records under `jobs/reasoning-query-rewriting-pi-deepseek`.
Alternative agent examples @@ -147,9 +145,9 @@ ZAI_API_KEY=YOUR_ZAI_KEY ``` ```bash -bash scripts/run_task.sh --task task-1-1 --agent pi \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent pi \ --thinking xhigh \ - --output jobs/task-1-1-pi-glm + --output jobs/reasoning-query-rewriting-pi-glm ``` #### Codex and a GPT model @@ -163,8 +161,8 @@ AGENT_OPENAI_API_KEY=YOUR_AGENT_KEY ``` ```bash -bash scripts/run_task.sh --task task-1-1 --agent codex \ - --reasoning-effort xhigh --output jobs/task-1-1-codex +bash scripts/run_task.sh --task reasoning-query-rewriting --agent codex \ + --reasoning-effort xhigh --output jobs/reasoning-query-rewriting-codex ``` Omit `--reasoning-effort` when the selected model or provider does not support @@ -181,8 +179,8 @@ AGENT_ANTHROPIC_API_KEY=YOUR_ANTHROPIC_KEY ``` ```bash -bash scripts/run_task.sh --task task-1-1 --agent claude-code \ - --reasoning-effort high --output jobs/task-1-1-claude +bash scripts/run_task.sh --task reasoning-query-rewriting --agent claude-code \ + --reasoning-effort high --output jobs/reasoning-query-rewriting-claude ``` #### Codex or Claude Code through OpenRouter @@ -196,13 +194,13 @@ AGENT_OPENROUTER_API_KEY=YOUR_AGENT_OPENROUTER_KEY Choose an Agent and pass its full OpenRouter model ID: ```bash -bash scripts/run_task.sh --task task-1-1 --agent codex --openrouter \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent codex --openrouter \ --model openai/gpt-6-astra --reasoning-effort xhigh \ - --output jobs/task-1-1-codex-openrouter + --output jobs/reasoning-query-rewriting-codex-openrouter -bash scripts/run_task.sh --task task-1-1 --agent claude-code --openrouter \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent claude-code --openrouter \ --model anthropic/claude-opus-5.5 --reasoning-effort xhigh \ - --output jobs/task-1-1-claude-openrouter + --output jobs/reasoning-query-rewriting-claude-openrouter ``` Add `--dry-run` to either command to preview it before launching. The launcher @@ -242,20 +240,23 @@ and [`maintain-searchswe-task`](.agents/skills/maintain-searchswe-task/SKILL.md) PR review and promotion. See the [contribution entry](CONTRIBUTING.md) and [guide](docs/contributing.md) for validation and PR requirements. -New tasks use -`task-submissions//-x-` (your ASCII -first name, **not** username). One PR may add multiple tasks, all under exactly -one contributor namespace; temporary ordinals are unique within category and -are not final IDs or reusable after promotion in that PR. A different same-name -contributor explicitly chooses `alice-2`. Maintainers assign final IDs near -merge, then make separate pure -`git mv` and finalization commits for every task **in that same PR**. Use -**merge commit only**, not squash/rebase; no unfinished submission enters main. +Task authoring includes running a configured coding model, inspecting its +trajectory and evaluation results, and using that feedback to refine the task +setting. Revalidate any changes that affect the task or scoring and report the +final tested revision. If required API keys are missing, the authoring agent +asks the contributor to configure them locally before the trial. + +New tasks use `task-submissions/`. Choose a descriptive name of at +most five lowercase, hyphen-separated words; use that same name for the formal +package and its official data. One PR may add multiple tasks. Maintainers +promote each reviewed package to `tasks/` in the same PR, with a +separate pure move and finalization. Preserve original authors with a merge +commit; no unfinished submission enters main. + Development assets may use a personal public HF dataset pinned to a commit SHA. -After final ID assignment, official assets go through an HF community PR or -maintainer mirror; official HF merge and SHA pinning precede the GitHub merge. -Never share official tokens. Explicit `--task-path` supports submission downloads -and local trials; automatic task discovery remains formal-only. +Publish official inputs under `tasks//`, then pin the merged official +HF SHA before merging the GitHub contribution. Explicit `--task-path` supports +submission downloads and local trials; `--task ` selects a formal task. ## Citation diff --git a/README_zh.md b/README_zh.md index ad84d20..1de7514 100644 --- a/README_zh.md +++ b/README_zh.md @@ -27,17 +27,15 @@ Search-SWE 评估编码智能体能否在固定资源约束下**实现和优化* 智能体需要检查环境、编写并运行代码、测试系统，最终提交可执行的实现。评测关注 系统的实际表现，包括检索质量、功能正确性和资源消耗。 -| 模式 | 智能体目标 | 示例问题 | -| --- | --- | --- | -| 实现 | 根据任务说明构建可运行的搜索能力。 | 推理辅助检索、内存受限的向量搜索。 | -| 优化 | 在固定约束下提升搜索质量或效率。 | 长文档重排序、嵌入模型微调、查询编码器优化。 | +任务涵盖推理辅助检索、内存受限的向量搜索、长文档重排序、嵌入模型微调和 +查询编码器对齐等问题。 每个任务都明确规定输入、提交接口、评测标准和资源预算。评测方式、仓库结构和 固定数据与模型的来源见[基准设计](docs/benchmark.md)。 ## 🚀 快速开始 -下面的示例在 CPU 上使用 Pi 编码智能体和 DeepSeek Flash 跑通 `task-1-1`。 +下面的示例在 CPU 上使用 Pi 编码智能体和 DeepSeek Flash 跑通 `reasoning-query-rewriting`。 Search-SWE 需要 Python 3.12 或更新版本，以及满足所选任务 CPU、内存、存储和 可选 GPU 要求的 Docker。 @@ -57,8 +55,8 @@ Docker 镜像内。 ### 2. 恢复任务资源 ```bash -python scripts/download_assets.py --task task-1-1 -python scripts/download_assets.py --task task-1-1 --verify-only +python scripts/download_assets.py --task reasoning-query-rewriting +python scripts/download_assets.py --task reasoning-query-rewriting --verify-only ``` 下载器会核对文件大小和 SHA-256，并复用已通过校验的文件。 @@ -83,7 +81,7 @@ VERIFIER_OPENAI_API_KEY=YOUR_OPENROUTER_KEY ``` `DEEPSEEK_API_KEY` 用于 `deepseek/deepseek-flash` 的 Pi 智能体认证；`VERIFIER_*` -通过 RewardKit 0.2.0 和 OpenRouter 运行固定为 `deepseek/deepseek-v4.1-flash` 的独立轨迹评审，除 `task-2-4` +通过 RewardKit 0.2.0 和 OpenRouter 运行固定为 `deepseek/deepseek-v4.1-flash` 的独立轨迹评审，除 `agentic-search` 外的所有任务都需要。`VERIFIER_OPENAI_API_KEY` 使用 OpenRouter 密钥，Pi 智能体 使用独立的 DeepSeek 密钥；启动器只把 verifier 凭据传进 verifier 容器。 @@ -96,9 +94,9 @@ VERIFIER_OPENAI_API_KEY=YOUR_OPENROUTER_KEY 默认使用 Anthropic 官方 API。 - OpenRouter Agent 模式：设置 `AGENT_OPENROUTER_API_KEY`，并为 Codex 或 Claude Code 指定 `--openrouter` 和完整的 `provider/model` 模型 ID。 -- Task 1-3：还必须填写三个 `ANSWER_JUDGE_*` 变量。 +- `scientific-paper-qa`：还必须填写三个 `ANSWER_JUDGE_*` 变量。 - 可选 submission API：只在 `.env.example` 注释所列任务确实使用时，填写 - `TASK_1_1_OPENROUTER_API_KEY`、`OPENROUTER_API_KEY` 或 `JINA_API_KEY`。 + `REASONING_QUERY_REWRITING_OPENROUTER_API_KEY`、`OPENROUTER_API_KEY` 或 `JINA_API_KEY`。 需要代理时，按[网络配置](docs/network-policy.md)设置 `.env` 的 `EGRESS_CONFIG`，否则留空。 @@ -108,16 +106,16 @@ VERIFIER_OPENAI_API_KEY=YOUR_OPENROUTER_KEY ### 4. 使用 Pi 和 DeepSeek Flash 运行 ```bash -bash scripts/run_task.sh --task task-1-1 --agent pi \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent pi \ --thinking xhigh --dry-run -bash scripts/run_task.sh --task task-1-1 --agent pi \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent pi \ --thinking xhigh \ - --output jobs/task-1-1-pi-deepseek + --output jobs/reasoning-query-rewriting-pi-deepseek ``` dry-run 只打印 Harbor 命令、不启动容器。第二条命令会构建任务镜像、运行智能体 -和独立验证器，并把 reward 与任务记录写入 `jobs/task-1-1-pi-deepseek`。 +和独立验证器，并把 reward 与任务记录写入 `jobs/reasoning-query-rewriting-pi-deepseek`。
其他智能体示例 @@ -136,9 +134,9 @@ ZAI_API_KEY=YOUR_ZAI_KEY ``` ```bash -bash scripts/run_task.sh --task task-1-1 --agent pi \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent pi \ --thinking xhigh \ - --output jobs/task-1-1-pi-glm + --output jobs/reasoning-query-rewriting-pi-glm ``` #### Codex 和 GPT 模型 @@ -152,8 +150,8 @@ AGENT_OPENAI_API_KEY=YOUR_AGENT_KEY ``` ```bash -bash scripts/run_task.sh --task task-1-1 --agent codex \ - --reasoning-effort xhigh --output jobs/task-1-1-codex +bash scripts/run_task.sh --task reasoning-query-rewriting --agent codex \ + --reasoning-effort xhigh --output jobs/reasoning-query-rewriting-codex ``` 如果所选模型或服务不支持推理强度，请省略 `--reasoning-effort`。 @@ -169,8 +167,8 @@ AGENT_ANTHROPIC_API_KEY=YOUR_ANTHROPIC_KEY ``` ```bash -bash scripts/run_task.sh --task task-1-1 --agent claude-code \ - --reasoning-effort high --output jobs/task-1-1-claude +bash scripts/run_task.sh --task reasoning-query-rewriting --agent claude-code \ + --reasoning-effort high --output jobs/reasoning-query-rewriting-claude ``` #### 通过 OpenRouter 运行 Codex 或 Claude Code @@ -184,13 +182,13 @@ AGENT_OPENROUTER_API_KEY=YOUR_AGENT_OPENROUTER_KEY 选择 Agent，并传入完整的 OpenRouter 模型 ID： ```bash -bash scripts/run_task.sh --task task-1-1 --agent codex --openrouter \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent codex --openrouter \ --model openai/gpt-6-astra --reasoning-effort xhigh \ - --output jobs/task-1-1-codex-openrouter + --output jobs/reasoning-query-rewriting-codex-openrouter -bash scripts/run_task.sh --task task-1-1 --agent claude-code --openrouter \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent claude-code --openrouter \ --model anthropic/claude-opus-5.5 --reasoning-effort xhigh \ - --output jobs/task-1-1-claude-openrouter + --output jobs/reasoning-query-rewriting-claude-openrouter ``` 可以先为命令加上 `--dry-run` 预览。启动器已配置 OpenRouter 的 API 地址； @@ -226,19 +224,18 @@ bash scripts/run_task.sh --task task-1-1 --agent claude-code --openrouter \ [`maintain-searchswe-task`](.agents/skills/maintain-searchswe-task/SKILL.md)（PR 审查与正式化）。验证要求和 PR 说明见[贡献入口](CONTRIBUTING.md)与[贡献指南](docs/contributing.md)。 -新任务放在 -`task-submissions//-x-`：使用贡献者提供的 -ASCII 小写 first name，**不是用户名**。一个 PR 可以新增多个任务，但必须全部位于 -同一个贡献者 namespace；临时 ordinal 在该 PR/checkout 的同一 category 内唯一， -并非正式编号，且在该 PR 内正式化后不得复用。另一位同名贡献者须显式选择 -`alice-2`。Maintainer 在即将合并时为 -每个任务分配正式编号，并在**同一 PR**内分别提交纯 `git mv` commit 和 -finalization commit；所有任务正式化后才能合并。新任务必须使用 **merge commit**， -不使用 squash/rebase；未完成的 submission 不进入 main。 -开发资产可使用个人公开的临时 HF dataset，并固定到 commit SHA。正式编号确定后， -通过 HF community PR 或 maintainer mirror 发布官方资产；先合并官方 HF 变更、 -固定官方 SHA，再合并 GitHub PR。禁止共享官方 token。下载器和 launcher 支持显式 -`--task-path` 验证 submission，自动任务发现仍只包含正式任务。 +任务构建还包括使用配置好的编码模型试跑，检查执行轨迹和评测结果，并根据反馈 +完善任务设定。影响任务要求或评分的修改需要重新验证，并记录最终测试的版本。 +如果缺少必需的 API key，负责构建任务的智能体应在试跑前提醒贡献者在本地配置。 + +新任务放在 `task-submissions/`。名称使用不超过五个词的小写连字符 +短语，正式任务与官方数据使用同一名称。一个 PR 可以包含多个任务；维护者在 +同一个 PR 中逐一晋升到 `tasks/`，将纯目录移动和后续整理分开提交。 +使用 merge commit 保留原作者历史，未完成晋升的任务不进入 main。 + +开发数据可以来自固定到 commit SHA 的个人公开 HF 数据集。官方输入发布到 +`tasks//` 后，先固定官方已合并的 HF SHA，再合并 GitHub 贡献。 +使用 `--task-path` 显式下载、试跑待审任务；正式任务通过 `--task ` 选择。 ## 引用 diff --git a/docs/assets.md b/docs/assets.md index 21ea94c..7917041 100644 --- a/docs/assets.md +++ b/docs/assets.md @@ -13,7 +13,7 @@ python -m pip install -r scripts/requirements.txt Download all fixed inputs for one task: ```bash -python scripts/download_assets.py --task task-2-1 +python scripts/download_assets.py --task long-document-reranking ``` Download data or models separately: @@ -24,7 +24,7 @@ python scripts/download_models.py --task all ``` The model command uses the same downloader with `--kind models` by default. -It selects model entries from each task's `assets.json`. Multiple task IDs may +It selects model entries from each task's `assets.json`. Multiple task names may be passed after `--task`. Preview the selected files, sources, and total size: @@ -38,28 +38,28 @@ Hugging Face cache; the downloader copies cached files into the task directories ## New-task submissions -Contributors can validate before final numbering with an explicit package: +Contributors can validate before promotion with an explicit package: ```bash -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 --dry-run -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 -python scripts/download_assets.py --task-path task-submissions/alice/1-x-1 --verify-only -python scripts/run_task.py --task-path task-submissions/alice/1-x-1 \ +python scripts/download_assets.py --task-path task-submissions/example-search --dry-run +python scripts/download_assets.py --task-path task-submissions/example-search +python scripts/download_assets.py --task-path task-submissions/example-search --verify-only +python scripts/run_task.py --task-path task-submissions/example-search \ --agent pi --model deepseek/deepseek-flash --dry-run ``` -`--task all` stays formal-only. Paths must be canonical repository-relative +`--task all` selects all formal tasks. Paths must be canonical repository-relative paths without traversal or symlinks. An alternate `--output-dir` preserves the -submission namespace; launcher jobs default to `jobs/task-submissions/alice/1-x-1` -to avoid collisions between contributors sharing a temporary task ID. +submission path; launcher jobs default to `jobs/task-submissions/example-search`. +Each submission has a unique task name. For development, use a personal public temporary dataset with a manifest, -provenance/license and pinned immutable SHA. After final numbering, use the +provenance/license and pinned immutable SHA. After promotion, use the [contribution publication workflow](contributing.md#asset-contribution-and-publication): `prepare_hf_upload.py` stages only new verified files plus a merged current -manifest without downloading the full old inventory. Update HF SOURCES and -license metadata, preserve all old assets, and submit an HF community PR -(`create_pr=True`) or request a maintainer mirror. Never share official tokens. +manifest. Supply the current manifest and the new data files. Update HF SOURCES +and license metadata, preserve existing published assets, and submit an HF +community PR (`create_pr=True`) or request a maintainer mirror. Never share official tokens. Official HF merge precedes final `assets.json` SHA pinning and GitHub merge. ## Directory layout @@ -67,7 +67,7 @@ Official HF merge precedes final `assets.json` SHA pinning and GitHub merge. Downloads are restored relative to the task root: ```text -tasks// +tasks// ├── instruction.md ├── task.toml ├── assets.json # Asset paths, sources, sizes, and checksums @@ -83,7 +83,7 @@ tasks// and `tests/` keeps large assets out of Harbor's build contexts and content hashes. Their contents are verified separately against `assets.json`. -Task 1-4 restores one shared PDF to `data/corpus/` and public queries and labels +`long-pdf-evidence` restores one shared PDF to `data/corpus/` and public queries and labels to `data/validation/`. Both environments use the same PDF; only the Agent mounts public validation. Hidden questions and relevance labels stay in `tests/data/`. @@ -115,7 +115,7 @@ If you maintain a local `hf-data` copy, you can restore task assets from it without network access: ```bash -python scripts/download_assets.py --task task-1-3 task-1-4 task-2-4 \ +python scripts/download_assets.py --task scientific-paper-qa long-pdf-evidence agentic-search \ --local-data-dir ../hf-data python scripts/check_release.py --hf-data ../hf-data --verify-data ``` @@ -150,7 +150,7 @@ python scripts/download_assets.py --task all --verify-only If an existing file differs, the downloader reports an error. To replace it: ```bash -python scripts/download_assets.py --task task-2-1 --force +python scripts/download_assets.py --task long-document-reranking --force ``` `--cache-dir /path/to/cache` selects the Hugging Face cache. `--local-files-only` diff --git a/docs/benchmark.md b/docs/benchmark.md index 1f47860..e626da2 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,10 +1,9 @@ # Benchmark design -This document holds the benchmark details that do not belong on the landing -page: how tasks are evaluated, how the repository is organized, and where fixed -data and model inputs come from. For setup and execution, start with the -[quick start guide](quickstart.md) and the [evaluation guide](evaluation.md). -The task modes and a one-paragraph summary live in the +This document explains how tasks are evaluated, how the repository is organized, +and where fixed data and model inputs come from. For setup and execution, start +with the [quick start guide](quickstart.md) and the [evaluation guide](evaluation.md). +A summary of the engineering objectives lives in the [project README](../README.md). ## Evaluation @@ -37,13 +36,15 @@ Search-SWE/ ├── assets/ # Hero image used by the README ├── docker/ # Reproducible CPU and GPU base-image contexts ├── tasks/ -│ ├── / # Task package +│ ├── / # Task package │ │ ├── instruction.md # Agent-facing task specification │ │ ├── task.toml # Task and environment configuration │ │ ├── assets.json # Fixed input file sizes and checksums │ │ ├── environment/ # Dockerfile, starter code, and environment docs │ │ └── tests/ # Verifier and grading code │ └── ... # Additional task packages +├── task-submissions/ # New task packages before same-PR promotion +│ └── / # Same name retained under tasks/ after review ├── scripts/ │ ├── requirements.txt # Host-side launcher and asset dependencies │ ├── download_assets.py # Download and verify fixed data/model inputs @@ -62,7 +63,7 @@ Pretrained weights are downloaded from their original model repositories. Each task's `assets.json` records its fixed input files, sizes, SHA-256 checksums, and download sources. Runtime data belongs in -`tasks//data/` and fixed models in `tasks//models/`; these +`tasks//data/` and fixed models in `tasks//models/`; these directories are excluded from Git. Dataset and model revisions are pinned to immutable commits. See the [asset guide](assets.md) for the directory layout, downloads, and local restoration options. diff --git a/docs/evaluation.md b/docs/evaluation.md index 4f730f3..87794c5 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -10,7 +10,7 @@ Search-SWE separates four kinds of configuration: 1. **Coding-agent credentials** run Codex, Pi, or Claude Code. 2. **Verifier credentials** run the private trajectory judge used by most tasks. -3. **Answer-judge credentials** are used only by task 1-3. +3. **Answer-judge credentials** are used only by `scientific-paper-qa`. 4. **Submission-resource credentials** expose task-permitted OpenRouter or Jina APIs to the agent's implementation; they are not coding-agent credentials. @@ -39,8 +39,8 @@ launcher and Hugging Face asset client. Task runtime dependencies remain inside Docker. Restore only the selected task: ```bash -python scripts/download_assets.py --task TASK_ID -python scripts/download_assets.py --task TASK_ID --verify-only +python scripts/download_assets.py --task TASK_NAME +python scripts/download_assets.py --task TASK_NAME --verify-only ``` The launcher checks asset presence and size; `--verify-only` performs the full @@ -60,18 +60,18 @@ the shared images locally. | Task | Hardware | Trajectory judge | Answer judge | Optional submission APIs | | --- | --- | --- | --- | --- | -| `task-1-1` | CPU | Yes | — | `TASK_1_1_OPENROUTER_API_KEY`, `JINA_API_KEY` | -| `task-1-2` | CPU | Yes | — | — | -| `task-1-3` | CPU | Yes | Yes | `OPENROUTER_API_KEY`, `JINA_API_KEY` | -| `task-1-4` | CPU | Yes | — | `OPENROUTER_API_KEY`, `JINA_API_KEY` | -| `task-2-1` | CPU | Yes | — | — | -| `task-2-2` | 1 NVIDIA GPU | Yes | — | — | -| `task-2-3` | 1 NVIDIA GPU | Yes | — | — | -| `task-2-4` | CPU | — | — | `OPENROUTER_API_KEY`, `JINA_API_KEY` | -| `task-2-5` | CPU | Yes | — | — | +| `reasoning-query-rewriting` | CPU | Yes | — | `REASONING_QUERY_REWRITING_OPENROUTER_API_KEY`, `JINA_API_KEY` | +| `memory-constrained-dense-retrieval` | CPU | Yes | — | — | +| `scientific-paper-qa` | CPU | Yes | Yes | `OPENROUTER_API_KEY`, `JINA_API_KEY` | +| `long-pdf-evidence` | CPU | Yes | — | `OPENROUTER_API_KEY`, `JINA_API_KEY` | +| `long-document-reranking` | CPU | Yes | — | — | +| `code-embedding-finetuning` | 1 NVIDIA GPU | Yes | — | — | +| `query-encoder-alignment` | 1 NVIDIA GPU | Yes | — | — | +| `agentic-search` | CPU | — | — | `OPENROUTER_API_KEY`, `JINA_API_KEY` | +| `sparse-retrieval-pruning` | CPU | Yes | — | — | The table identifies credential groups, not the complete resource budget. Read -`tasks/TASK_ID/README.md` and `task.toml` before launching. In particular, +`tasks/TASK_NAME/README.md` and `task.toml` before launching. In particular, memory, storage, time limits, and asset sizes vary substantially by task. ## Environment-file behavior @@ -123,9 +123,9 @@ AGENT_OPENROUTER_API_KEY=YOUR_AGENT_OPENROUTER_KEY Use a full OpenRouter model slug and select the route explicitly: ```bash -bash scripts/run_task.sh --task task-1-1 --agent codex --openrouter \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent codex --openrouter \ --model openai/gpt-6-astra --reasoning-effort xhigh --dry-run -bash scripts/run_task.sh --task task-1-1 --agent claude-code --openrouter \ +bash scripts/run_task.sh --task reasoning-query-rewriting --agent claude-code --openrouter \ --model anthropic/claude-opus-5.5 --reasoning-effort xhigh --dry-run ``` @@ -134,7 +134,7 @@ Remove `--dry-run` to launch. This mode fixes Codex's Responses base URL to `https://openrouter.ai/api`. Claude Code accepts only `anthropic/` slugs here. The launcher permits `openrouter.ai` in restricted Agent phases and keeps the OpenRouter Agent key separate from `OPENROUTER_API_KEY` for submissions and -`ANSWER_JUDGE_API_KEY` for task-1-3. Verifier configuration is unchanged. +`ANSWER_JUDGE_API_KEY` for scientific-paper-qa. Verifier configuration is unchanged. When using Codex, omit `--codex-config`; the launcher supplies the required native provider configuration and ignores the direct mode's `AGENT_CODEX_CONFIG`. @@ -153,7 +153,7 @@ Then preview or run with the native Harbor adapter: ```bash bash scripts/run_task.sh \ - --task task-1-1 \ + --task reasoning-query-rewriting \ --agent claude-code \ --reasoning-effort high \ --dry-run @@ -189,7 +189,7 @@ DEEPSEEK_API_KEY=YOUR_DEEPSEEK_KEY ```bash bash scripts/run_task.sh \ - --task task-1-1 \ + --task reasoning-query-rewriting \ --agent pi \ --model deepseek/deepseek-flash \ --thinking xhigh \ @@ -213,7 +213,7 @@ steps and observations. Failed runs are exported before the error is reported. ## Configure judges -All current tasks except `task-2-4` use the trajectory judge. Add: +All current tasks except `agentic-search` use the trajectory judge. Add: ```dotenv VERIFIER_OPENAI_BASE_URL=https://openrouter.ai/api/v1 @@ -240,7 +240,7 @@ submission process environment; keep verifier logs private as you would for any authenticated client. The backend uses the [OpenRouter Responses API](https://openrouter.ai/docs/api/api-reference/responses/create-responses). -Task 1-3 additionally uses a Chat Completions-compatible answer-equivalence +`scientific-paper-qa` additionally uses a Chat Completions-compatible answer-equivalence judge: ```dotenv @@ -249,26 +249,26 @@ ANSWER_JUDGE_BASE_URL=https://openrouter.ai/api/v1 ANSWER_JUDGE_API_KEY=YOUR_ANSWER_JUDGE_KEY ``` -No other current task uses this group. Task 1-3 restricts this judge to the +No other current task uses this group. `scientific-paper-qa` restricts this judge to the already permitted `openrouter.ai` host. ## Configure optional submission APIs Set these only when the selected task and intended submission use the permitted -external resources documented in `tasks/TASK_ID/environment/docs/available_resources.md`. +external resources documented in `tasks/TASK_NAME/environment/docs/available_resources.md`. They do not select the coding-agent or judge model. -Tasks 1-3, 1-4, and 2-4 use the shared variables: +`scientific-paper-qa`, `long-pdf-evidence`, and `agentic-search` use the shared variables: ```dotenv OPENROUTER_API_KEY=YOUR_OPENROUTER_KEY JINA_API_KEY=YOUR_JINA_KEY ``` -Task 1-1 uses a separate OpenRouter credential and the shared Jina credential: +`reasoning-query-rewriting` uses a separate OpenRouter credential and the shared Jina credential: ```dotenv -TASK_1_1_OPENROUTER_API_KEY=YOUR_RESTRICTED_OPENROUTER_KEY +REASONING_QUERY_REWRITING_OPENROUTER_API_KEY=YOUR_RESTRICTED_OPENROUTER_KEY JINA_API_KEY=YOUR_JINA_KEY ``` @@ -278,7 +278,7 @@ resource document is the source of truth. ## GPU tasks -Tasks 2-2 and 2-3 require one NVIDIA GPU. Their images use CUDA 13.0 and require +`code-embedding-finetuning` and `query-encoder-alignment` require one NVIDIA GPU. Their images use CUDA 13.0 and require a compatible NVIDIA driver and Container Toolkit. Both task and verifier Compose overlays request the real GPU. @@ -310,7 +310,7 @@ Preview command construction first: ```bash bash scripts/run_task.sh \ - --task TASK_ID \ + --task TASK_NAME \ --model YOUR_AGENT_MODEL \ --dry-run ``` @@ -321,9 +321,9 @@ a fresh jobs root: ```bash bash scripts/run_task.sh \ - --task TASK_ID \ + --task TASK_NAME \ --model YOUR_AGENT_MODEL \ - --output jobs/TASK_ID-run-name + --output jobs/TASK_NAME-run-name ``` The launcher uses Docker, forces image builds, runs one attempt with concurrency diff --git a/docs/network-policy.md b/docs/network-policy.md index af57299..a86a49e 100644 --- a/docs/network-policy.md +++ b/docs/network-policy.md @@ -52,7 +52,7 @@ Explicit config requires `image` and cannot be combined with image, DNS or gener container-proxy overrides. Do not add the proxy to task allowlists. The proxy must be reachable **from Docker**; host `127.0.0.1` is not the host -inside a container. There is no fixed VPN product/port or automatic discovery. +inside a container. Set the proxy URL explicitly in `upstream.url`. SOCKS-only and TUN-only endpoints are not supported by this configuration. For a hostname URL such as `https://proxy.example:8443`, also set `upstream.address` to its reachable IPv4 address. Certificates are validated @@ -121,18 +121,18 @@ access. | Task | Task/runtime requirement | `[environment]` baseline | Effective agent phase | Verifier phase | | --- | --- | --- | --- | --- | -| `task-1-1` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | -| `task-1-2` | Fully local vector retrieval | no network | model host only | `openrouter.ai` only | -| `task-1-3` | OpenRouter/Jina RAG resources | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts + `api.deepseek.com`; OpenRouter serves the trajectory judge; the separate answer judge may use either allowed provider | -| `task-1-4` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | -| `task-2-1` | Fixed local reranker only | no network | model host only | `openrouter.ai` only | -| `task-2-2` | Explicitly offline training | no network | model host only | `openrouter.ai` only | -| `task-2-3` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | -| `task-2-4` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts only; no model judge | -| `task-2-5` | Formal build/search is offline | no network | model host only | `openrouter.ai` only | +| `reasoning-query-rewriting` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | +| `memory-constrained-dense-retrieval` | Fully local vector retrieval | no network | model host only | `openrouter.ai` only | +| `scientific-paper-qa` | OpenRouter/Jina RAG resources | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts + `api.deepseek.com`; OpenRouter serves the trajectory judge; the separate answer judge may use either allowed provider | +| `long-pdf-evidence` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | +| `long-document-reranking` | Fixed local reranker only | no network | model host only | `openrouter.ai` only | +| `code-embedding-finetuning` | Explicitly offline training | no network | model host only | `openrouter.ai` only | +| `query-encoder-alignment` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | +| `agentic-search` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts only; no model judge | +| `sparse-retrieval-pruning` | Formal build/search is offline | no network | model host only | `openrouter.ai` only | Allowing the judge endpoint does not expose verifier credentials to submitted -commands. Task 1-3 similarly removes both judge credential groups before +commands. `scientific-paper-qa` similarly removes both judge credential groups before running submission code. Submission API keys are injected only for tasks whose resource policy permits those APIs. @@ -142,7 +142,7 @@ The repository launcher derives and supplies the coding-model host. When invoking Harbor directly on a non-public task, add the matching hostname: ```bash -PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" harbor run --path tasks/TASK_ID \ +PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" harbor run --path tasks/TASK_NAME \ --env scripts.harbor_environments:PhaseScopedDocker \ --agent scripts.harbor_agents:PreinstalledCodex --model MODEL_ID \ --ak version=0.157.1 \ @@ -153,7 +153,7 @@ For the supported official-Anthropic Claude Code mode, use the pinned wrapper, an environment reference rather than a literal secret, and the fixed host: ```bash -PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" harbor run --path tasks/TASK_ID \ +PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" harbor run --path tasks/TASK_NAME \ --env scripts.harbor_environments:PhaseScopedDocker \ --agent scripts.harbor_agents:PreinstalledClaudeCode \ --model ANTHROPIC_MODEL_ID \ @@ -170,7 +170,7 @@ the supported configuration. These commands use the default direct gateway image and pull it if missing locally. To select another matching image, add `--environment-kwarg egress_image=YOUR_IMAGE`. Manually using `--env docker` selects installed Harbor and does not receive this checkout's -capability removal or old-connection revocation fixes. +capability restrictions or connection revocation when policies change. Do not use `--allow-environment-host` for a coding-model endpoint: that changes the environment baseline rather than only the agent phase. Do not add package diff --git a/docs/quickstart.md b/docs/quickstart.md index 2304402..49dd3c6 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -28,12 +28,12 @@ docker info ## 2. Restore one task -Start with `task-1-1`, a CPU task with a small asset bundle. Do not download +Start with `reasoning-query-rewriting`, a CPU task with a small asset bundle. Do not download every task for a first run: ```bash -python scripts/download_assets.py --task task-1-1 -python scripts/download_assets.py --task task-1-1 --verify-only +python scripts/download_assets.py --task reasoning-query-rewriting +python scripts/download_assets.py --task reasoning-query-rewriting --verify-only ``` The second command verifies file sizes and SHA-256 checksums. See the @@ -48,7 +48,7 @@ RewardKit-verifier sections used by this walkthrough: cp .env.example .env ``` -For `task-1-1`, fill in these two credential pairs: +For `reasoning-query-rewriting`, fill in these two credential pairs: - `AGENT_OPENAI_BASE_URL` and `AGENT_OPENAI_API_KEY` run the coding agent. - `VERIFIER_OPENAI_BASE_URL=https://openrouter.ai/api/v1` and @@ -58,7 +58,7 @@ For `task-1-1`, fill in these two credential pairs: Set `AGENT_MODEL` in `.env`, or pass `--model` on the command line. The verifier uses an OpenRouter key. Agent and verifier credentials remain separate groups, and only the verifier receives the `VERIFIER_*` values. -`task-1-1` also permits OpenRouter and Jina as submission resources; +`reasoning-query-rewriting` also permits OpenRouter and Jina as submission resources; their keys are optional and are not needed for an implementation that uses only the provided corpus and local runtime. @@ -78,7 +78,7 @@ First inspect the Harbor command without starting containers or making API calls: ```bash -bash scripts/run_task.sh --task task-1-1 --dry-run +bash scripts/run_task.sh --task reasoning-query-rewriting --dry-run ``` The preview checks command construction only; it does not validate credentials, @@ -87,9 +87,9 @@ after those prerequisites are ready: ```bash bash scripts/run_task.sh \ - --task task-1-1 \ + --task reasoning-query-rewriting \ --reasoning-effort high \ - --output jobs/task-1-1-codex + --output jobs/reasoning-query-rewriting-codex ``` Omit `--reasoning-effort` when the selected model or provider does not support