From cf7e277009acc0b2973423b3fc879b1306443f3b Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Sat, 10 Oct 2026 21:16:30 +0800 Subject: [PATCH 1/3] Accept task contributions directly under tasks/ --- .agents/AGENTS.md | 25 +- .agents/skills/create-searchswe-task/SKILL.md | 42 +- .../create-searchswe-task/agents/openai.yaml | 4 +- .../references/assets-and-resources.md | 13 +- .../references/model-evaluation.md | 7 +- .../references/publication.md | 20 +- .../references/submission-and-pr.md | 69 +-- .../references/task-authoring.md | 24 +- .../references/validation.md | 27 +- .../scripts/scaffold_task.py | 15 +- .../scripts/test_scaffold_task.py | 23 +- .../skills/maintain-searchswe-task/SKILL.md | 27 +- .../agents/openai.yaml | 4 +- .../references/finalization.md | 87 +++ .../references/pr-review.md | 2 +- .../references/promotion.md | 117 ---- .../references/publication.md | 21 +- .../references/task-audit.md | 22 +- .../scripts/maintainer_helpers.py | 127 +---- .../scripts/promote_task.py | 6 - .../scripts/test_maintainer_skill.py | 85 +-- .github/pull_request_template.md | 13 +- .github/workflows/task-checks.yml | 10 +- .gitignore | 6 +- CONTRIBUTING.md | 11 +- README.md | 19 +- README_zh.md | 11 +- docs/assets.md | 33 +- docs/benchmark.md | 2 - docs/contributing.md | 174 +++--- docs/network-policy.md | 2 +- scripts/_maintainer_skill.py | 2 +- scripts/check_release.py | 10 +- scripts/check_submission.py | 238 --------- scripts/check_tasks.py | 154 ++++++ scripts/download_assets.py | 6 +- scripts/prepare_hf_upload.py | 4 +- scripts/promote_task.py | 16 - scripts/run_task.py | 4 +- scripts/task_paths.py | 12 +- scripts/tests/test_submission_workflow.py | 503 ------------------ scripts/tests/test_task_workflow.py | 354 ++++++++++++ task-submissions/README.md | 34 -- 43 files changed, 912 insertions(+), 1473 deletions(-) create mode 100644 .agents/skills/maintain-searchswe-task/references/finalization.md delete mode 100644 .agents/skills/maintain-searchswe-task/references/promotion.md delete mode 100644 .agents/skills/maintain-searchswe-task/scripts/promote_task.py delete mode 100644 scripts/check_submission.py create mode 100755 scripts/check_tasks.py delete mode 100644 scripts/promote_task.py delete mode 100644 scripts/tests/test_submission_workflow.py create mode 100644 scripts/tests/test_task_workflow.py delete mode 100644 task-submissions/README.md diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index 346579c..e64ceaa 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -11,10 +11,10 @@ Markdown links. workflow, references, and validation gates. If no native skill loader is available, open that `SKILL.md` directly. It can also be invoked independently as `$create-searchswe-task`. -- **Review, promote or finalize a task PR as maintainer:** load +- **Review or finalize a task PR as maintainer:** load [maintain-searchswe-task](skills/maintain-searchswe-task/SKILL.md), independently invocable as `$maintain-searchswe-task`. Default to read-only gh evidence review; - use its bundled offline promotion/HF staging helpers only at approved stages. + use its bundled offline HF staging helper only at approved stages. - **Run an existing task:** use `docs/quickstart.md`; do not scaffold a task. - **Restore fixed inputs:** use `docs/assets.md` and the existing download tools. - **Edit project documentation:** keep `README.md` and `README_zh.md` aligned. @@ -37,20 +37,19 @@ Markdown links. acceptance criteria and optimization objective in natural prose. Preserve practical context without self-introductions or invented identities; the skill defines the rewrite and validation gates. - Before scaffolding, inspect one or two closest reviewed packages under - `tasks/` as read-only structural precedents, selected by engineering objective, + Before scaffolding, inspect one or two closest reviewed packages already merged + into the base branch under `tasks/` as read-only structural precedents, selected + by engineering objective, grading shape, resources and hardware. Apply current repository contracts and validators, and independently establish the new task's data, thresholds, licenses and access policy. - A PR may add multiple tasks at `task-submissions/`. Choose a - descriptive lowercase name with at most five hyphen-separated words, without - a `task-` prefix; `all` is reserved. Keep that name when promoting to - `tasks/`. Do not reuse a submission path within the same PR. Follow - `CONTRIBUTING.md` and `docs/contributing.md`: require actual authors and - explicit `--task-path` for local trials, then promote every task with its own - same-PR pure rename and any necessary finalization commits before a merge - commit (no squash/rebase). Track which stage is complete and which inputs or approvals - are missing. + A PR may add multiple tasks at `tasks/` in its contribution branch. + Choose a descriptive lowercase name with at most five hyphen-separated words; + the `task-` prefix and `all` are reserved. Check names against the base and + active PRs. Follow `CONTRIBUTING.md` and `docs/contributing.md`: record actual + authors, use `--task ` for local trials, and finish task review and + official asset publication before merging with a merge commit. Track completed + stages, validation evidence and missing inputs or approvals. Do not duplicate the skill's specifications in this file. 3. Keep changes within the selected task plus directly necessary integration changes. Updating a website, publishing a dataset, pushing images, and opening diff --git a/.agents/skills/create-searchswe-task/SKILL.md b/.agents/skills/create-searchswe-task/SKILL.md index 4b7d143..918a05e 100644 --- a/.agents/skills/create-searchswe-task/SKILL.md +++ b/.agents/skills/create-searchswe-task/SKILL.md @@ -20,9 +20,10 @@ Read [references/task-authoring.md](references/task-authoring.md). Inspect the target checkout's status and preserve unrelated changes. Resolve its root explicitly; do not infer it from where this skill was installed. -Before scaffolding, inspect one or two closest **formal** packages under the -target checkout's `tasks/`. Choose them by engineering objective, evaluation/judge -shape, resources/APIs, hardware and input/artifact layout. Record their exact paths and why they are relevant. Treat them as +Before scaffolding, inspect one or two closest reviewed packages already merged +into the base branch under `tasks/`. Choose them by engineering objective, +evaluation/judge shape, resources/APIs, hardware and input/artifact layout. +Record their exact paths and why they are relevant. Treat them as read-only structural precedents: the current skill, repository validators, schema and shared-image documentation take priority. Do not copy task-specific IDs, authors, datasets, thresholds, pins, licenses or access policy without @@ -40,7 +41,7 @@ Record a short design summary before writing the package: | Evaluation | Metric, public gates, baseline if needed, failure/timeout behavior | | Inputs | Public/hidden split, actual files, provenance, redistribution rights | | Resources | CPU/GPU, memory/storage/time, network, permitted APIs/models | -| Precedent | Formal task path(s), matching dimensions, structural patterns reused, intentional differences | +| Precedent | Reviewed task path(s), matching dimensions, structural patterns reused, intentional differences | CPU is the default; choose GPU when the agent or verifier executes GPU work. Ask for missing facts that change the design; do not invent labels, licenses, service @@ -59,24 +60,19 @@ python "$SKILL_DIR/scripts/scaffold_task.py" example-search \ --repo-root "$REPO" --author 'Alice Example' --hardware cpu ``` -Use `--hardware gpu` only for GPU execution. New tasks belong at -`task-submissions/`, with `task.name = "search-swe/"`. -Names start with an ASCII lowercase letter, contain lowercase letters/digits -and single hyphens, and have at most five words. Use a concise description of -the task; omit the `task-` prefix. `all` is reserved. Check both package roots -and active PRs for collisions. Repeat `--author` for actual coauthors. -The helper refuses overwrites and symlink roots. Its explicit `--formal` option -is for maintainer tooling, not a bypass for new task contributions. Edit an -existing task in place instead of deleting it to make the helper succeed. - -One PR may add multiple uniquely named tasks. Read the bundled -[submission and PR workflow](references/submission-and-pr.md). Promotion keeps -the same name: `task-submissions/` → `tasks/`. Each task -gets its own pure rename commit in **the same PR**, followed by a separate -finalization commit when references, assets or integration need changes. Do not -reuse a submission path in that PR. Preserve original author and promotion -commits: **merge commit only**, never squash/rebase. All tasks must be promoted -before merge; no submission package enters final main. +Use `--hardware gpu` only for GPU execution. Create new tasks directly at +`tasks/` in the contribution branch, with +`task.name = "search-swe/"`. Names start with an ASCII lowercase letter, +contain lowercase letters/digits and single hyphens, and have at most five words. +Choose a concise description; the `task-` prefix and `all` are reserved. Check +names against `tasks/` on the base branch and active PRs. Repeat `--author` for +actual coauthors. The helper protects existing files and rejects symlink roots. +Revise existing packages in place. + +One PR may add multiple uniquely named tasks. Follow the +[branch and PR workflow](references/submission-and-pr.md): review and validate +each package, publish official inputs and pin their merged SHA, then merge the +PR with a **merge commit** to preserve original author history. The scaffold is **not a finished task**: its verifier always fails, its asset manifest is empty, data/model mounts are absent, and instructions are writing @@ -144,7 +140,7 @@ authorizes that scoped run; reuse already-authorized local credentials without asking again. Ordinary local checks may proceed. Report the task path, objective/hardware, commands actually executed and their -results, formal reference tasks consulted (or that none was close), intentional +results, reviewed reference tasks consulted (or that none was close), intentional reuse/deviations, the source-to-request equivalence review, known-good/negative-case evidence, the exact evaluated revision/model/effort, observed scores and judge health, feedback-driven task changes (or why none were needed), and every unrun diff --git a/.agents/skills/create-searchswe-task/agents/openai.yaml b/.agents/skills/create-searchswe-task/agents/openai.yaml index ad2d0a9..69398a5 100644 --- a/.agents/skills/create-searchswe-task/agents/openai.yaml +++ b/.agents/skills/create-searchswe-task/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Contribute Search-SWE Task" - short_description: "Author and validate a temporary task submission" - default_prompt: "Use $create-searchswe-task to build and locally validate a task submission with a descriptive name, then prepare an authorized PR handoff." + short_description: "Create and validate tasks directly under tasks/" + default_prompt: "Use $create-searchswe-task to build and locally validate a task at tasks/, then prepare an authorized PR handoff." diff --git a/.agents/skills/create-searchswe-task/references/assets-and-resources.md b/.agents/skills/create-searchswe-task/references/assets-and-resources.md index f73267f..452a30f 100644 --- a/.agents/skills/create-searchswe-task/references/assets-and-resources.md +++ b/.agents/skills/create-searchswe-task/references/assets-and-resources.md @@ -7,8 +7,7 @@ task. Nothing in this reference grants permission to publish or use paid APIs. ## Manifest construction Keep downloadable fixed files outside the build contexts, under -`data/` or `models/` in the selected package (initially -`task-submissions/`, later `tasks/`). +`data/` or `models/` in the package at `tasks/`. Ignore both directories in Git. Hidden queries/labels and grading code go in `tests/`, not the public asset bundle. For a verifier-only corpus stored in `data/verifier/`, mount only public @@ -48,16 +47,16 @@ Do not attempt `chmod` on read-only mounted files during verification. ## Development and incremental publication -For pre-promotion development, use an authorized **personal public temporary HF +During development, use an authorized **personal public temporary HF dataset** with manifest, provenance and license; pin its immutable 40-hex SHA in -submission `assets.json`. The downloader accepts arbitrary HF repositories: +the task's `assets.json`. The downloader accepts arbitrary HF repositories: ```bash -python scripts/download_assets.py --task-path task-submissions/example-search -python scripts/download_assets.py --task-path task-submissions/example-search --verify-only +python scripts/download_assets.py --task example-search +python scripts/download_assets.py --task example-search --verify-only ``` -After the maintainer approves promotion in the same PR, prepare new official +After task and asset review, prepare new official files at `tasks//...`. The target checkout's `scripts/prepare_hf_upload.py` accepts a current official manifest snapshot, the final task package and **only new data**. It verifies hashes, preserves existing diff --git a/.agents/skills/create-searchswe-task/references/model-evaluation.md b/.agents/skills/create-searchswe-task/references/model-evaluation.md index 2f2d248..bc35e5e 100644 --- a/.agents/skills/create-searchswe-task/references/model-evaluation.md +++ b/.agents/skills/create-searchswe-task/references/model-evaluation.md @@ -54,16 +54,17 @@ host Codex settings. The answer-equivalence judge, if present, remains separate. ## Launch and observe -Use `--task-path` for a contribution and `--task` for a formal package. Preview +Use `--task ` to select the task from `tasks/`. Preview first, then run in a new output directory once inputs, credentials, hardware and scoped API/GPU authorization are available. An explicit model-evaluation request supplies authorization for that evaluation; do not ask for it again. For example, replacing placeholders with the actual agreed task and effort: ```bash -bash scripts/run_task.sh --task-path "$task_path" --agent codex \ +task_name=example-search # Use the task being evaluated. +bash scripts/run_task.sh --task "$task_name" --agent codex \ --openrouter --model "$model_id" --reasoning-effort "$effort" --dry-run -bash scripts/run_task.sh --task-path "$task_path" --agent codex \ +bash scripts/run_task.sh --task "$task_name" --agent codex \ --openrouter --model "$model_id" --reasoning-effort "$effort" \ --output "$fresh_output" ``` diff --git a/.agents/skills/create-searchswe-task/references/publication.md b/.agents/skills/create-searchswe-task/references/publication.md index 6d7b249..870b817 100644 --- a/.agents/skills/create-searchswe-task/references/publication.md +++ b/.agents/skills/create-searchswe-task/references/publication.md @@ -25,18 +25,18 @@ repo = 'YOUR_ACCOUNT/search-swe-development' api.create_repo(repo_id=repo, repo_type='dataset', private=False, exist_ok=True) commit = api.upload_folder(repo_id=repo, repo_type='dataset', folder_path='/path/to/dev-data', commit_message='Add temporary task development inputs') -print(commit.oid) # Pin this full SHA in submission assets.json. +print(commit.oid) # Pin this full SHA in task assets.json. PY ``` Use this personal repo, its dataset-relative filenames, size/hash, and printed -immutable SHA in `assets.json`. The explicit submission downloader/launcher then -support real pre-promotion validation. Keep the temporary repo accessible through -review and migration; only retire it after official pinned downloads are verified. +immutable SHA in `assets.json`. Select the task by name with the downloader and +launcher for development validation. Keep the temporary repo accessible through +review and publication; retire it after official pinned downloads are verified. -### Task name: additive official publication, then GitHub merge +### Official publication, then GitHub merge -After task-name approval and promotion, change only the new task's dataset source +After task and asset review, change only the new task's dataset source repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve original model repo pins and bundled metadata. Prepare **only new data files** in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. @@ -64,7 +64,7 @@ python scripts/prepare_hf_upload.py \ --output /path/to/upload-staging ``` -Official asset staging requires a promoted package at `tasks/`. +Stage assets from the reviewed package at `tasks/`. The offline helper verifies hashes and safe paths for new files, rejects collisions and symlinks, and stages those files with a merged manifest that @@ -106,6 +106,6 @@ replace a newer manifest with a stale one. **Merge the official HF change first.** Only then pin the resulting official 40-hex commit SHA (not the community PR/head SHA or `main`) in the finalized GitHub task's `assets.json`. Verify downloads using a fresh output directory or -cache and run release checks. Commit finalization and merge the GitHub PR last. -No new task with unpublished/unpinned assets or an unfinished submission enters -main. Preserve existing tasks' pinned revisions and published assets. +cache and run release and merge-ready checks. Commit the verified asset pins +in the task PR, then merge the GitHub contribution. Preserve existing tasks' +pinned revisions and published assets. diff --git a/.agents/skills/create-searchswe-task/references/submission-and-pr.md b/.agents/skills/create-searchswe-task/references/submission-and-pr.md index 9a466de..3a38388 100644 --- a/.agents/skills/create-searchswe-task/references/submission-and-pr.md +++ b/.agents/skills/create-searchswe-task/references/submission-and-pr.md @@ -1,4 +1,4 @@ -# Submission, branch and PR handoff +# Contribution branch and PR handoff Choose a descriptive task name of at most five lowercase hyphen-separated words, starting with a letter and otherwise using letters/digits. Omit `task-`; `all` @@ -9,11 +9,10 @@ is reserved. Choose GPU only when execution requires it. Resolve the target checkout explicitly and inspect status, branch, remotes and base. Preserve other authors' work and commits; never reset, stash or switch their branch to simplify your task. Agree on branch/commit scope before changing -history. New packages use `task-submissions/` and canonical -`task.name = "search-swe/"`. Check formal tasks, submissions and active -PRs for collisions. A PR may add multiple unique names, but cannot reuse a -submission path. Direct new formal additions without submission history fail CI. -A scaffold is a deliberately failing starting point, not a finished task. +history. New packages use `tasks/` in the contribution branch, with +`task.name = "search-swe/"`. Check names against the current base and +active PRs. A PR may add multiple uniquely named tasks. Complete the scaffold's +instructions, assets, environment and grader before runtime validation. Set actual `task.toml` authors. Preserve original commits and account-linked Git name/email; do not change identity configuration on their behalf. Co-authored-by @@ -42,12 +41,12 @@ all uppercase values with verified values): ```bash git push FORK_REMOTE CONTRIBUTION_BRANCH gh pr create --repo OWNER/REPO --base BASE_BRANCH \ - --head FORK_OWNER:CONTRIBUTION_BRANCH --title 'Add task submission' \ + --head FORK_OWNER:CONTRIBUTION_BRANCH --title 'Add example-search task' \ --body-file /path/to/reviewed-pr-body.md ``` Use a trusted body file, not shell interpolation of task/PR text. Report actual -validation commands/results and unrun layers. For each task, name the formal +validation commands/results and unrun layers. For each task, name the reviewed reference packages consulted (or state that none was close), the structural patterns reused, and intentional differences; do not claim precedent as proof that a task-specific value is correct. Link any authorized personal HF repo at @@ -56,44 +55,28 @@ the **PR author** enables **Allow edits from maintainers** on the PR. Verify the setting (`maintainerCanModify` in `gh pr view --json maintainerCanModify`), rather than assuming PR creation enabled it. Same-repository branches may not need it; organization-owned forks/policies may make it unavailable. In that case agree -that the contributor performs the promotion commits in the same PR. Never +that the contributor applies review fixes in the same PR. Never request broad credentials or silently replace the PR. -## Review and finalization in that same PR +## Review and finalization in the same PR The maintainer audits instructions/resources, heldout split, known-good and -negative cases, grader isolation, networking/credentials, sources and licenses. -A review-stage CI pass can still contain unfinished submissions. Missing, -pending or failed checks are not acceptance. Separately authorize code execution, -API/GPU costs and any credentialed runtime tests. - -Near merge, refresh the base and check name collisions with queued PRs. Preserve -original author history. Use the target checkout's offline promotion helper -from a clean worktree/index; missing tools are a prerequisite blocker: - -```bash -python scripts/promote_task.py rename task-submissions/example-search example-search --dry-run -python scripts/promote_task.py rename task-submissions/example-search example-search -# Review and, only when authorized, commit this pure git mv alone. -python scripts/promote_task.py finalize task-submissions/example-search example-search --dry-run -python scripts/promote_task.py finalize task-submissions/example-search example-search -``` - -Repeat independently for each task. The name stays unchanged. Finalize requires -HEAD to be that task's single-parent, whole-package, 100%-identical pure rename -commit. It rewrites submission paths in tracked text; it never commits or -publishes. Review replacements, mounts, image/external paths, inventories and -authors. Follow [publication](publication.md): personal pinned development data -→ official `tasks//...` paths through a community PR or authorized -mirror → official merge → pin that official SHA. Preserve original model pins. - -Run release and applicable tests plus `python scripts/check_submission.py ---merge-ready`; no submission task.toml may remain. Commit any reference, asset -or integration finalization separately in this same PR. Do not create an empty -commit if no finalization changes are needed. - -Re-review each changed head, recheck base/name collisions and exact head before -an authorized **merge commit**. No squash/rebase or unfinished submissions on -main. Verify the resulting SHA/tree and attribution using +negative cases, model-trial feedback, grader isolation, networking/credentials, +sources and licenses. Separately authorize code execution, API/GPU costs and +credentialed runtime tests. Track each required check for the reviewed head. + +Refresh the base and check names against queued PRs. Apply review fixes and +necessary inventory or documentation changes in the contribution branch, +preserving original author commits. Follow [publication](publication.md): +publish approved dataset inputs under official `tasks//...` paths, +merge the HF change, then pin its official SHA in `assets.json` and verify clean +downloads. Preserve original model pins. + +Run release checks, applicable tests and `python scripts/check_tasks.py +--merge-ready --base FULL_PR_BASE_SHA`. Commit the reviewed changes in the same +PR. Re-review each changed head and verify required checks against the current +base before an authorized **merge commit**. + +Verify the resulting SHA/tree and attribution using `git log --follow -- tasks/example-search/instruction.md` and `git blame`. Report observed outcomes, task names, official asset SHAs and remaining blockers. diff --git a/.agents/skills/create-searchswe-task/references/task-authoring.md b/.agents/skills/create-searchswe-task/references/task-authoring.md index 9a9ae43..0d16326 100644 --- a/.agents/skills/create-searchswe-task/references/task-authoring.md +++ b/.agents/skills/create-searchswe-task/references/task-authoring.md @@ -6,12 +6,11 @@ unless they begin with `/` (container paths). The skill may be installed elsewhere. Use the target checkout's validators and the published image tags documented in `docker/README.md`. -## Use current formal tasks as bounded precedents +## Use reviewed tasks as bounded precedents -Before scaffolding, inspect the target checkout's current `tasks/` tree -read-only. Select one or two closest **formal** packages for each proposed task; -active `task-submissions/` packages are unreviewed work, not precedent. Prefer -similarity in this order when it affects the design: +Before scaffolding, inspect one or two closest reviewed packages already merged +into the base branch under `tasks/`. Record that base revision alongside the +selected paths. Prefer similarity in this order when it affects the design: 1. engineering objective and the evaluation/baseline shape; 2. deterministic verifier versus model judge, plus artifact-transfer interface; @@ -64,15 +63,14 @@ or access to a reference solution. ## Required package contract -Create each new package at `task-submissions//`, with +Create each new package at `tasks//` in the contribution branch, with `task.name = "search-swe/"` and actual `task.toml` authors. Choose a concise descriptive name of at most five lowercase hyphen-separated words, -starting with a letter and otherwise using letters/digits. Omit `task-`; `all` -is reserved. Names must be unique across submissions and formal tasks. -A PR may contain multiple tasks; never reuse a submission path in that PR. -Promotion keeps the name at `tasks//` in the same PR: a pure rename -commit, then any necessary finalization in a separate commit, and a merge commit -only. Existing-task edits stay in place. Submission checks reuse +starting with a letter and otherwise using letters/digits. The `task-` prefix +and `all` are reserved. Check uniqueness against the current base and active PRs. +A PR may contain multiple tasks. Review each package, complete validation and +official asset publication, then merge with a merge commit to preserve authors. +Revise existing tasks in place. `scripts/check_tasks.py` reuses `scripts/check_release.py`, which requires these non-empty files: ```text @@ -248,7 +246,7 @@ task uses a less restricted profile. Prefer deterministic programmatic checks for facts that code can measure. Use a model judge only for a clearly subjective dimension, and pin its tooling -and configuration following a selected formal precedent with the same judging +and configuration following a selected reviewed precedent with the same judging mode, while independently verifying that its judge and access policy apply. The separate verifier must be self-contained. Its Dockerfile copies `tests/` diff --git a/.agents/skills/create-searchswe-task/references/validation.md b/.agents/skills/create-searchswe-task/references/validation.md index e8236c5..099c74d 100644 --- a/.agents/skills/create-searchswe-task/references/validation.md +++ b/.agents/skills/create-searchswe-task/references/validation.md @@ -1,12 +1,12 @@ # Validation and completion evidence Run commands from the target repository root, using Python 3.12+. For a new -task set `task_path=task-submissions/example-search` (use your actual task name); after promotion set `task_path=tasks/`. These checks use +task set `task_path=tasks/example-search`, using your actual task name. These checks use the checkout's tools; the skill does not bundle a second checker or runtime. ## Target prerequisites (stop if missing) -Verify the explicit checkout contains `scripts/check_submission.py`, +Verify the explicit checkout contains `scripts/check_tasks.py`, `check_release.py`, `download_assets.py`, `run_task.py`/`run_task.sh`, and their imports. Use Python 3.12+, host dependencies from `scripts/requirements.txt` (Harbor 0.22.0 and huggingface_hub 1.x), PyYAML for static Compose parsing, and @@ -22,7 +22,7 @@ for an authorized remote inspection or PR handoff, not local task authoring. ## 1. Inspect the package before execution -- Revisit the recorded formal reference task(s), or the recorded absence of a +- Revisit the recorded reviewed reference task(s), or the recorded absence of a close precedent. Compare only the intended structural patterns and explain meaningful deviations against the current contracts. Check that no task-specific IDs, data/pins, thresholds, allowlists, licenses or resource @@ -41,11 +41,11 @@ for an authorized remote inspection or PR handoff, not local task authoring. without actually using that UID when launching code is insufficient. ```bash -task_path=task-submissions/example-search -python scripts/check_submission.py "$task_path" # Before promotion; requires PyYAML +task_path=tasks/example-search +python scripts/check_tasks.py "$task_path" # Requires PyYAML python scripts/check_release.py python -m unittest discover -s scripts/tests -p 'test_*.py' -python scripts/download_assets.py --task-path "$task_path" --dry-run +python scripts/download_assets.py --task "${task_path#tasks/}" --dry-run bash -n "$task_path/tests/test.sh" git diff --check ``` @@ -58,12 +58,11 @@ The current `scripts/tests/test_task_images.py` includes an explicit `GPU_TASKS` inventory. Adding a GPU task may require updating that expected inventory as a direct integration change, while retaining image/resource consistency checks. Do not lower assertions to make a wrongly configured task pass. Download and -launch commands discover formal task names from `tasks/*/task.toml` automatically; -submissions require explicit `--task-path`. CI review-stage success is not -merge readiness: after same-PR pure rename/finalization, run -`python scripts/check_submission.py --merge-ready` (no submission `task.toml` -may remain). Do not run unreviewed contributor code with secrets; credentialed -runtime testing is separate from unprivileged PR CI. +launch commands discover task names from `tasks/*/task.toml` automatically. +Use `--task ` for each trial. Before merge, run +`python scripts/check_tasks.py --merge-ready --base FULL_PR_BASE_SHA` to check +package syntax, new-task authors and pinned official dataset sources. Review +and authorize credentialed runtime testing separately from unprivileged PR CI. ## 2. Assets, Compose, and images @@ -76,7 +75,7 @@ Compose files are Harbor overlays. Direct `docker compose config` needs a temporary base service with an image (this check does not pull that image): ```bash -task_path=task-submissions/example-search +task_path=tasks/example-search ( set -eu base=$(mktemp) @@ -132,7 +131,7 @@ do not present static checks or Oracle results as that model run. ## Handoff Report changed paths, objective, hardware/image selection, input provenance, -formal reference task paths (or no close precedent), structural reuse and +reviewed reference task paths (or no close precedent), structural reuse and intentional deviations, commands/results for each completed layer, expected versus observed scores, task changes motivated by model feedback (or why none were needed), the final evaluated revision, and unrun layers with their blockers. diff --git a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py index e0bc366..e6a6d9f 100755 --- a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py @@ -260,7 +260,6 @@ def parse_args() -> argparse.Namespace: default=Path.cwd(), help="Search-SWE repository root; defaults to the current directory", ) - parser.add_argument("--formal", action="store_true", help="Maintainer-only scaffold under tasks/; default is task-submissions/") parser.add_argument("--author", action="append", required=True, help="Actual author name; repeat for coauthors") return parser.parse_args() @@ -279,21 +278,17 @@ def main() -> int: print(f"error: not a Search-SWE repository root: {root}", file=sys.stderr) return 2 - if (root / "tasks").is_symlink() or (root / "task-submissions").is_symlink(): + if (root / "tasks").is_symlink(): print("error: task roots must not be symlinks", file=sys.stderr) return 2 if any(not name.strip() or name.strip().lower() in ("search-swe", "your name", "author") for name in args.author): print("error: actual --author names are required", file=sys.stderr) return 2 - for directory in ("tasks", "task-submissions"): - existing = root / directory / args.task_name - if existing.exists() or existing.is_symlink(): - print(f"error: task name already exists; refusing overwrite: {existing}", file=sys.stderr) - return 1 - parent = root / ("tasks" if args.formal else "task-submissions") - parent.mkdir(exist_ok=True) - destination = parent / args.task_name + destination = root / "tasks" / args.task_name + if destination.exists() or destination.is_symlink(): + print(f"error: task name already exists; refusing overwrite: {destination}", file=sys.stderr) + return 1 generated = files_for(args.task_name, args.hardware) if args.author: diff --git a/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py index fc8d867..64d2862 100644 --- a/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/test_scaffold_task.py @@ -39,7 +39,7 @@ def test_cpu_gpu_generation_and_fail_closed_entrypoint(self): name = f"example-{hardware}-search" result = self.scaffold(name, "--hardware", hardware, "--author", "Coauthor Example") self.assertEqual(result.returncode, 0, result.stderr) - task = self.repo / "task-submissions" / name + task = self.repo / "tasks" / name self.assertNotEqual((task / "raw-instruction.md").read_text(), (task / "instruction.md").read_text()) config = tomllib.loads((task / "task.toml").read_text()) @@ -75,29 +75,20 @@ def test_name_rules_authors_and_removed_options(self): for name in ("retrieval", "one-two-three-four-five", "bm25-search"): self.assertEqual(self.scaffold(name).returncode, 0) - def test_names_cannot_collide_across_submission_and_formal_roots(self): + def test_existing_task_files_are_preserved(self): self.assertEqual(self.scaffold("example-search").returncode, 0) - marker = self.repo / "task-submissions/example-search/keep.txt" + marker = self.repo / "tasks/example-search/keep.txt" marker.write_text("keep") self.assertNotEqual(self.scaffold("example-search").returncode, 0) - self.assertNotEqual(self.scaffold("example-search", "--formal").returncode, 0) self.assertEqual(marker.read_text(), "keep") - self.assertEqual(self.scaffold("formal-search", "--formal").returncode, 0) - self.assertTrue((self.repo / "tasks/formal-search/task.toml").is_file()) - self.assertNotEqual(self.scaffold("formal-search").returncode, 0) def test_symlink_roots_and_destinations_are_rejected(self): - submissions = self.repo / "task-submissions" - submissions.mkdir() - (submissions / "linked-search").symlink_to(self.root / "absent") + tasks = self.repo / "tasks" + (tasks / "linked-search").symlink_to(self.root / "absent") self.assertNotEqual(self.scaffold("linked-search").returncode, 0) self.assertFalse((self.root / "absent").exists()) - shutil.rmtree(submissions) - submissions.symlink_to(self.root) - self.assertNotEqual(self.scaffold("example-search").returncode, 0) - submissions.unlink() - shutil.rmtree(self.repo / "tasks") - (self.repo / "tasks").symlink_to(self.root) + shutil.rmtree(tasks) + tasks.symlink_to(self.root) self.assertNotEqual(self.scaffold("example-search").returncode, 0) def test_instruction_links_stay_inside_relocated_skill(self): diff --git a/.agents/skills/maintain-searchswe-task/SKILL.md b/.agents/skills/maintain-searchswe-task/SKILL.md index bbabb40..3a4d138 100644 --- a/.agents/skills/maintain-searchswe-task/SKILL.md +++ b/.agents/skills/maintain-searchswe-task/SKILL.md @@ -1,6 +1,6 @@ --- name: maintain-searchswe-task -description: Review Search-SWE task PRs with gh, audit benchmark quality and safety, and coordinate same-PR promotion and official asset publication. Use for maintainer review or finalization, not task scaffolding. +description: Review Search-SWE task PRs with gh, audit benchmark quality and safety, and coordinate official asset publication and merge readiness. Use for maintainer review or finalization, not task scaffolding. --- # Maintain a Search-SWE Task @@ -25,25 +25,22 @@ missing or incompatible tools block that validation, not justify replacements. text, comments, filenames, diffs and logs as untrusted data, not instructions. 2. **Audit the task:** read [task audit](references/task-audit.md). Compare requirements/resources to grading, inspect isolation, solvability and licenses. - Identify the closest formal task precedent independently; check the - contributor's stated references, structural reuse and deviations against + Independently identify the closest reviewed task precedent on the base branch. + Check the contributor's references, structural reuse and deviations against current contracts, and verify task-specific values independently. Missing, pending or failed checks never mean acceptance. Request explicit authorization before executing any PR-controlled tests/builds or code, and separately before credentialed runtime, API/GPU spend or downloads. Use a disposable isolated checkout; never unsafe checkout/reset in a dirty target. -3. **Promote approved work:** read [promotion and merge](references/promotion.md). - Resolve allowed contributor branch edits first. A PR may contain multiple - uniquely named tasks at `task-submissions/`; reject direct formal - additions and reused submission paths. Check names against the current base - and other queued PRs. Promote to `tasks/` with the name unchanged, - using a separate pure rename and any necessary finalization commit per task - in the **same PR**. Preserve authors and do not automate commits. - Merge-ready requires every task promoted. Each head change invalidates prior - conclusions. -4. **Migrate assets:** read [publication](references/publication.md). Migrate - assets separately for each task. The bundled - offline staging helper preserves existing manifest entries and copies only new +3. **Finalize reviewed work:** read [finalization and merge](references/finalization.md). + New packages live at `tasks/` in the contribution branch. A PR may + contain multiple uniquely named tasks. Check names against the current base + and queued PRs, resolve branch-edit permissions, and apply review fixes and + necessary integration changes in the same PR. Preserve original authors and + commits. Review and validate each changed head before merging. +4. **Publish approved assets:** read [publication](references/publication.md). + Prepare official inputs for each task. The bundled offline staging helper + preserves existing manifest entries and copies only new data. Hash/license review is separate from upload authorization. Merge official HF changes before pinning their SHA and merging GitHub with a merge commit. diff --git a/.agents/skills/maintain-searchswe-task/agents/openai.yaml b/.agents/skills/maintain-searchswe-task/agents/openai.yaml index 6320ffb..83a0fd6 100644 --- a/.agents/skills/maintain-searchswe-task/agents/openai.yaml +++ b/.agents/skills/maintain-searchswe-task/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Maintain Search-SWE Task" - short_description: "Review task PRs and safely promote approved tasks" - default_prompt: "Use $maintain-searchswe-task to review this Search-SWE task PR read-only and report validation and promotion blockers." + short_description: "Review task PRs and prepare approved tasks for merge" + default_prompt: "Use $maintain-searchswe-task to review this Search-SWE task PR read-only and report validation and merge blockers." diff --git a/.agents/skills/maintain-searchswe-task/references/finalization.md b/.agents/skills/maintain-searchswe-task/references/finalization.md new file mode 100644 index 0000000..10a64f1 --- /dev/null +++ b/.agents/skills/maintain-searchswe-task/references/finalization.md @@ -0,0 +1,87 @@ +# Finalization and merge + +## Complete the reviewed package + +Each task lives at `tasks/` in the contribution branch, with canonical +`task.name = "search-swe/"` and actual authors. A PR may contain multiple +uniquely named tasks. Check names against the current base and queued PRs, and +update against the base while preserving original author commits. Record the +reviewed head and base; renew review and validation when either changes. + +Resolve contributor branch-edit permissions, then apply review fixes and directly +necessary integration changes in the same PR. Check instructions, relative +mounts, images, external URLs, task inventories and GPU test inventories. Use +actual maintainer identity for maintainer commits and preserve task authors. + +## Publish assets and validate the final revision + +Read [publication.md](publication.md). Use the bundled offline HF helper to stage +the reviewed task's new files under `tasks//...`, preserving existing +manifest entries, model pins and provenance/license metadata. Set `REPO` to the +trusted checkout and `SKILL_DIR` to this installed skill. The helper requires +Python 3.12+ and an explicit `--repo-root`; it validates paths and hashes locally. +Official publication and HF PR merge each require authorization. + +Merge the official HF change first, then pin its resulting 40-hex commit in +`assets.json` and verify clean downloads. For each task, record the final +instruction revision, configured model/effort, trajectory, score, judge health +and feedback-driven corrections. Revalidate substantive task or scoring changes +within the authorized runtime budget. + +Run the audited target tools and applicable runtime layers: +`check_release.py`, `check_tasks.py --merge-ready --base FULL_PR_BASE_SHA`, +repository tests, changed skill regression tests, task-specific cases and +`git diff --check`. Review and commit the final asset pins and integration +changes in the same PR when authorized. Before an authorized push, verify that +the remote branch still matches the reviewed starting head and use a normal +push. Collect fresh review and check evidence for the resulting head. + +## Review, merge and verify + +Prepare a review summary locally by default. Only post a comment/review after +explicit authorization for its content, repo and PR. Example commands (replace +verified values and use a trusted body file): + +```bash +gh pr comment PR --repo OWNER/REPO --body-file /path/to/reviewed-comment.md +gh pr review PR --repo OWNER/REPO --request-changes --body-file /path/to/review.md +# Or --approve only when the head is actually accepted and posting is authorized. +``` + +Re-query the full head/base immediately before posting or merging; new commits +invalidate prior approval. The gh review command has no match-head flag: bind +review evidence to its full SHA, check again after posting, and resolve any race +rather than claiming acceptance of the changed head. All required human reviews, +resolved conversations, `review-stage`, `merge-ready` and other configured checks +must pass for that exact head and current base. Missing/pending checks block. +If protections/merge queue prevent the specified method, stop, do not bypass. + +After explicit merge authorization and fresh head/base/name checks: + +```bash +gh pr merge PR --repo OWNER/REPO --merge --match-head-commit FULL_REVIEWED_HEAD_SHA +``` + +**Merge commit only**: no `--squash`, `--rebase`, `--admin`, auto-merge shortcuts +or branch deletion unless separately requested. `--match-head-commit` guards the +head; coordinate against the current base as well. Never silently configure +protections, a queue or CODEOWNERS identities. + +Close the loop after the command, rather than treating exit zero as proof: + +1. Query `gh pr view PR --repo OWNER/REPO --json state,mergedAt,mergeCommit,headRefOid,baseRefName`. + Confirm MERGED and record the full merge SHA; if queued/not merged, report that. +2. Fetch the resulting base/merge into the disposable checkout. Verify the merge + object has two parents, the accepted PR head is its second parent, the reviewed + base is its first parent, and original author commits are ancestors. Unexpected + parent/base/tree changes require investigation, not a success claim. +3. Inspect the merged tree for `tasks//task.toml`, actual authors, final + official pins and intended files. + Compare it with the reviewed final tree (account for reviewed base integration). +4. Check `git log --follow -- tasks//instruction.md` and + `git blame tasks//instruction.md` at the merge SHA. Original unmodified + lines should retain contributor attribution. GitHub's latest file author and + contributor statistics are not substitutes; stats also depend on linked email. +5. Report repo/PR, reviewed head/base, merge SHA, final path, official HF SHA, + check/runtime evidence and author/history outcome. Do not retire personal + development assets until verified official downloads work. diff --git a/.agents/skills/maintain-searchswe-task/references/pr-review.md b/.agents/skills/maintain-searchswe-task/references/pr-review.md index 93ef138..eeb09ad 100644 --- a/.agents/skills/maintain-searchswe-task/references/pr-review.md +++ b/.agents/skills/maintain-searchswe-task/references/pr-review.md @@ -91,6 +91,6 @@ fork PRs default to editable. Same-repository branches depend on repository permission; organization-owned forks and policies can disallow maintainer edits. This flag is not sufficient proof of push rights and may allow workflow changes: review that exposure with the author. If unavailable, ask the contributor to make -the agreed pure-rename/finalization commits in the same PR. Do not replace their +the agreed review fixes and asset updates in the same PR. Do not replace their PR, impersonate them, alter their Git identity or request tokens. Pushing still requires explicit authorization for the exact branch and reviewed head. diff --git a/.agents/skills/maintain-searchswe-task/references/promotion.md b/.agents/skills/maintain-searchswe-task/references/promotion.md deleted file mode 100644 index 48098b7..0000000 --- a/.agents/skills/maintain-searchswe-task/references/promotion.md +++ /dev/null @@ -1,117 +0,0 @@ -# Same-PR promotion and closed-loop merge - -## Preserve the task name and authors - -Complete the audit and resolve allowed branch editing first. A PR may contain -multiple uniquely named tasks under `task-submissions/`. Names use at -most five lowercase hyphen-separated words, start with a letter, omit `task-` -and cannot be `all`. Keep all tasks in the same PR. Refresh the upstream base -and head, inspect formal task names and queued PRs for collisions, and update -against the base without flattening contributor history. A changed base/head -requires renewed review and validation. - -Use a clean disposable checkout at the reviewed head. Set `REPO` to that checkout -and `SKILL_DIR` to this installed folder. Helpers require Python 3.12+, Git and -explicit `--repo-root`; they have no target Python imports or sibling-skill -dependency. They never commit, push or publish. Promotion keeps the task name: - -```bash -python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - rename task-submissions/example-search example-search --dry-run -python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - rename task-submissions/example-search example-search -git -C "$REPO" diff --cached --summary -``` - -Repeat independently for each task; a rename commit contains exactly one whole -package move. Rename performs only `git mv` and refuses dirty worktrees, -collisions, traversal, symlinks and a target name different from the submission. -Review all staged files as R100 identical, with no incidental edits. The operator -creates the separately authorized pure rename commit. Preserve original author -commits and use real maintainer identity for maintainer changes. - -```bash -# After the operator creates that separate pure-rename HEAD commit: -python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - finalize task-submissions/example-search example-search --dry-run -python "$SKILL_DIR/scripts/promote_task.py" --repo-root "$REPO" \ - finalize task-submissions/example-search example-search -``` - -Finalize requires a single-parent, whole-package, pure rename HEAD commit. -It changes submission paths in tracked text only, preserving the task name and -canonical identity. It refuses unsupported binary rewrites. Review replacements, -relative mounts, image/URL paths and authors. Update task lists, docs and GPU -test inventories as needed. It does not edit downloaded data or assign licenses. -Resolve collisions before promotion; never overwrite another task or flatten -history. Keep all finalization edits separate from the pure rename commit. - -## Assets and finalization commit - -Read [publication.md](publication.md). Prepare only the new official files under -`tasks//...` using the bundled HF helper; preserve existing manifest entries, -model pins, provenance/license metadata. Its hash checks are offline and do not -prove remote freshness or grant upload permission. Official publication and HF -PR merge each require authorization. Merge official HF changes **first**, then -pin the resulting official 40-hex commit in GitHub `assets.json`, not community -PR SHA or mutable `main`. Verify clean downloads at that pin. - -Run the audited target tools and relevant runtime layers, with approvals: -`check_release.py`, `check_submission.py --merge-ready`, full scripts/tests, -scaffold regression tests if changed, task-specific cases and `git diff --check`. -HF migration is per task. Confirm no submission `task.toml` or submission-path -references remain for any task. Review and let the authorized operator create -each needed finalization commit in **the same PR**; no empty commit is required. Helpers do not commit. Before an authorized push, verify remote head still -matches the reviewed starting head, branch/permissions and intended commits; -use a normal push, not force. Any new head requires fresh review/check evidence. - -## Review, merge and verify - -Prepare a review summary locally by default. Only post a comment/review after -explicit authorization for its content, repo and PR. Example commands (replace -verified values and use a trusted body file): - -```bash -gh pr comment PR --repo OWNER/REPO --body-file /path/to/reviewed-comment.md -gh pr review PR --repo OWNER/REPO --request-changes --body-file /path/to/review.md -# Or --approve only when the head is actually accepted and posting is authorized. -``` - -Re-query the full head/base immediately before posting or merging; new commits -invalidate prior approval. The gh review command has no match-head flag: bind -review evidence to its full SHA, check again after posting, and resolve any race -rather than claiming acceptance of the changed head. All required human reviews, -resolved conversations, `review-stage`, `merge-ready` and other configured checks -must pass for that exact head and current base. Missing/pending checks block. -If protections/merge queue prevent the specified method, stop, do not bypass. - -After explicit merge authorization and fresh head/base/name checks: - -```bash -gh pr merge PR --repo OWNER/REPO --merge --match-head-commit FULL_REVIEWED_HEAD_SHA -``` - -**Merge commit only**: no `--squash`, `--rebase`, `--admin`, auto-merge shortcuts -or branch deletion unless separately requested. No unfinished submission reaches -main. `--match-head-commit` guards the head, not a concurrent base change: serial -coordination/current-base policy is still required. Never silently configure -protections, a queue or CODEOWNERS identities. - -Close the loop after the command, rather than treating exit zero as proof: - -1. Query `gh pr view PR --repo OWNER/REPO --json state,mergedAt,mergeCommit,headRefOid,baseRefName`. - Confirm MERGED and record the full merge SHA; if queued/not merged, report that. -2. Fetch the resulting base/merge into the disposable checkout. Verify the merge - object has two parents, the accepted PR head is its second parent, the reviewed - base is its first parent, and original author commits are ancestors. Unexpected - parent/base/tree changes require investigation, not a success claim. -3. Inspect the merged tree for `tasks//task.toml`, actual authors, final - official pins, no submission package/submission-path references and intended files. - Compare it with the reviewed final tree (account for reviewed base integration). -4. Check `git log --follow -- tasks//instruction.md` and - `git blame tasks//instruction.md` at the merge SHA. Original unmodified - lines should retain contributor attribution. GitHub's latest file author and - contributor statistics are not substitutes; stats also depend on linked email. -5. Report repo/PR, reviewed head/base, merge SHA, final path, official HF SHA, - check/runtime evidence and author/history outcome. Do not retire personal - development assets until verified official downloads work. diff --git a/.agents/skills/maintain-searchswe-task/references/publication.md b/.agents/skills/maintain-searchswe-task/references/publication.md index 663a97a..1054bae 100644 --- a/.agents/skills/maintain-searchswe-task/references/publication.md +++ b/.agents/skills/maintain-searchswe-task/references/publication.md @@ -25,18 +25,18 @@ repo = 'YOUR_ACCOUNT/search-swe-development' api.create_repo(repo_id=repo, repo_type='dataset', private=False, exist_ok=True) commit = api.upload_folder(repo_id=repo, repo_type='dataset', folder_path='/path/to/dev-data', commit_message='Add temporary task development inputs') -print(commit.oid) # Pin this full SHA in submission assets.json. +print(commit.oid) # Pin this full SHA in task assets.json. PY ``` Use this personal repo, its dataset-relative filenames, size/hash, and printed -immutable SHA in `assets.json`. The explicit submission downloader/launcher then -support real pre-promotion validation. Keep the temporary repo accessible through -review and migration; only retire it after official pinned downloads are verified. +immutable SHA in `assets.json`. Select the task by name with the downloader and +launcher for development validation. Keep the temporary repo accessible through +review and publication; retire it after official pinned downloads are verified. -### Task name: additive official publication, then GitHub merge +### Official publication, then GitHub merge -After task-name approval and promotion, change only the new task's dataset source +After task and asset review, change only the new task's dataset source repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve original model repo pins and bundled metadata. Prepare **only new data files** in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. @@ -64,7 +64,8 @@ python "$SKILL_DIR/scripts/prepare_hf_upload.py" --repo-root "$REPO" \ --output /path/to/upload-staging ``` -Official asset staging requires a promoted package at `tasks/`. +Set `SKILL_DIR` to this installed skill and `REPO` to the trusted checkout. +Stage assets from its reviewed package at `tasks/`. The offline helper verifies hashes and safe paths for new files, rejects collisions and symlinks, and stages those files with a merged manifest that @@ -106,6 +107,6 @@ replace a newer manifest with a stale one. **Merge the official HF change first.** Only then pin the resulting official 40-hex commit SHA (not the community PR/head SHA or `main`) in the finalized GitHub task's `assets.json`. Verify downloads using a fresh output directory or -cache and run release checks. Commit finalization and merge the GitHub PR last. -No new task with unpublished/unpinned assets or an unfinished submission enters -main. Preserve existing tasks' pinned revisions and published assets. +cache and run release and merge-ready checks. Commit the verified asset pins +in the task PR, then merge the GitHub contribution. Preserve existing tasks' +pinned revisions and published assets. diff --git a/.agents/skills/maintain-searchswe-task/references/task-audit.md b/.agents/skills/maintain-searchswe-task/references/task-audit.md index 97ea661..cd0240d 100644 --- a/.agents/skills/maintain-searchswe-task/references/task-audit.md +++ b/.agents/skills/maintain-searchswe-task/references/task-audit.md @@ -6,8 +6,8 @@ workflow/tool modifications need independent scrutiny, not implicit acceptance. | Area | Evidence required | | --- | --- | -| Identity/scope | Each new task starts at `task-submissions//`, with `task.name = "search-swe/"`. Names use at most five lowercase hyphen-separated words, start with a letter, omit `task-`, and cannot be `all`. A PR may add multiple unique names; reject reused submission paths and direct new formal additions. Promotion preserves the same name under `tasks/`. Require actual authors and original author commits. Existing-task revisions stay in place; reviewed renames of existing formal packages are not new contributions. | -| Repository precedent | For each new task, the contributor names one or two closest formal `tasks/` packages or states that none is close, and explains matching dimensions, structural reuse and intentional differences. Independently inspect the closest analogue by objective, grading/judge shape, resources, hardware and data/artifact layout. Current contracts and validators override legacy examples. Reject unexplained copying of task-specific IDs/authors, datasets or HF pins, thresholds/baselines, allowlists, budgets, licenses/provenance or hidden evaluation design. Merely existing on main is not proof that a pattern remains valid. | +| Identity/scope | Each new task lives at `tasks//` in the contribution branch, with `task.name = "search-swe/"`. Names use at most five lowercase hyphen-separated words, start with a letter, and are unique against the base and active PRs; `task-` and `all` are reserved. A PR may add multiple tasks. Require actual authors and preserve original author commits. Revise existing tasks in place. | +| Repository precedent | For each new task, the contributor names one or two closest reviewed `tasks/` packages already merged into the base branch or states that none is close, and explains matching dimensions, structural reuse and intentional differences. Independently inspect the closest analogue by objective, grading/judge shape, resources, hardware and data/artifact layout. Current contracts and validators override legacy examples. Reject unexplained copying of task-specific IDs/authors, datasets or HF pins, thresholds/baselines, allowlists, budgets, licenses/provenance or hidden evaluation design. Merely existing on main is not proof that a pattern remains valid. | | Instructions | Goal, starting state, input/output schema and absolute paths, submission command/interface, artifacts, public gates/thresholds and constraints match every graded requirement. No secret-only correctness requirement or hidden solution hints. | | Objective/resources | Clear goal, starting state and measured outcomes. Supply a meaningful baseline when grading improvement. CPU default, GPU only for execution. Both Dockerfiles, both Compose reservations and `gpus` agree. Visible tools/API/model docs match injected resources and budgets. Pin dependencies/images; inspect build contexts. | | Solvability | Real known-good submission through the same separate verifier/artifact-transfer interface, expected vs observed reward and an authorized coding-agent trial. A scaffold's always-zero test, empty manifest, dry-run or static pass is not a working benchmark. No public author-only solution without approval. | @@ -23,7 +23,7 @@ workflow/tool modifications need independent scrutiny, not implicit acceptance. The portable skill deliberately does not bundle Search-SWE's universal validator, launcher or Harbor runtime. Inspect an explicitly selected **trusted** target -checkout/version providing `scripts/check_submission.py`, `check_release.py`, +checkout/version providing `scripts/check_tasks.py`, `check_release.py`, `download_assets.py`, `run_task.py` and their dependencies. Missing tools or an older workflow version stop that layer; do not improvise acceptance commands. Python 3.12+, PyYAML, python-dotenv and host `scripts/requirements.txt` (Harbor @@ -33,18 +33,18 @@ image access and appropriate GPU driver/Container Toolkit are needed for runtime After code-execution authorization, from the isolated checkout with trusted tools: ```bash -task_path=task-submissions/example-search # Actual reviewed path; final tasks/ later -python scripts/check_submission.py "$task_path" +task_path=tasks/example-search # Actual package under review +python scripts/check_tasks.py "$task_path" python scripts/check_release.py python -m unittest discover -s scripts/tests -p 'test_*.py' -python scripts/download_assets.py --task-path "$task_path" --dry-run +python scripts/download_assets.py --task "${task_path#tasks/}" --dry-run bash -n "$task_path/tests/test.sh" git diff --check # After separate input-download authorization: -python scripts/download_assets.py --task-path "$task_path" -python scripts/download_assets.py --task-path "$task_path" --verify-only +python scripts/download_assets.py --task "${task_path#tasks/}" +python scripts/download_assets.py --task "${task_path#tasks/}" --verify-only # Preview a configured trial, not evidence of successful runtime: -python scripts/run_task.py --task-path "$task_path" --agent codex --model MODEL_ID --dry-run +python scripts/run_task.py --task "${task_path#tasks/}" --agent codex --model MODEL_ID --dry-run ``` These commands are target contracts, not permission to execute PR-modified tools. @@ -64,7 +64,7 @@ verifier logs and artifacts; record exact commands, head SHA and outcomes. Stop on missing hardware/auth/data or budget exhaustion; retry only after a relevant fix. Explicitly report unrun layers, not a blanket 'validated'. -Final readiness additionally requires `python scripts/check_submission.py ---merge-ready` (no submission task.toml), updated inventories including GPU tests, +Final readiness additionally requires `python scripts/check_tasks.py +--merge-ready --base FULL_PR_BASE_SHA`, updated inventories including GPU tests, immutable official asset pins and head-specific required checks. Static CI alone cannot establish solvability, fair heldout evaluation, licensing or isolation. diff --git a/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py b/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py index bfbc15a..9bb4f62 100644 --- a/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py +++ b/.agents/skills/maintain-searchswe-task/scripts/maintainer_helpers.py @@ -1,4 +1,4 @@ -"""Offline promotion and additive HF staging; no network or automatic commits. +"""Offline additive HF staging; no network or automatic commits. This is the implementation shared by the portable skill and repository CLIs. """ @@ -9,27 +9,10 @@ from pathlib import Path, PurePosixPath import re import shutil -import subprocess import tomllib TASK_NAME = r"(?!all(?:/|$))(?!task-)[a-z][a-z0-9]*(?:-[a-z0-9]+){0,4}" -SUBMISSION = re.compile(rf"task-submissions/({TASK_NAME})") -FORMAL = re.compile(TASK_NAME) - -def safe_path(root, value): - """Accept a canonical repo-relative path, never symlinks (even internal ones).""" - root = root.resolve() - value = str(value) - path = Path(value) - if (not value or path.is_absolute() or "\\" in value - or any(part in ("", ".", "..") for part in value.split("/"))): - raise ValueError(f"Expected canonical repository-relative path: {value}") - current = root - for part in path.parts: - current /= part - if current.is_symlink(): - raise ValueError(f"Symlink is not allowed: {current}") - return current +TASK_NAME_RE = re.compile(TASK_NAME) def no_symlinks(path): if path.is_symlink() or any(item.is_symlink() for item in path.rglob("*")): @@ -75,81 +58,6 @@ def read_manifest(task): return manifest -def git(repo, *args): - return subprocess.run(["git", *args], cwd=repo, check=True, capture_output=True, text=True).stdout - -def promote(repo, source, target_name, phase, dry_run=False): - repo = checked_path(repo) - if phase not in ("rename", "finalize"): - raise ValueError("Unknown promotion phase") - match = SUBMISSION.fullmatch(source) - if not match or not FORMAL.fullmatch(target_name): - raise ValueError("Use task-submissions/ and the same task-name (at most five lowercase words)") - if target_name != match[1]: - raise ValueError("Promotion preserves the task name; rename the submission before promotion") - src = safe_path(repo, source) - target = safe_path(repo, f"tasks/{target_name}") - if git(repo, "status", "--porcelain"): - raise ValueError("Promotion requires a clean worktree and index; commit reviewed work first") - if phase == "rename": - if not src.is_dir() or target.exists(): - raise ValueError("Source must exist and target must not exist; refusing overwrite") - no_symlinks(src) - config = tomllib.loads((src / "task.toml").read_text()) - if config.get("task", {}).get("name") != f"search-swe/{target_name}": - raise ValueError("Source task name does not match its directory") - print(f"git mv {source} tasks/{target_name}") - if not dry_run: - target.parent.mkdir(exist_ok=True) - git(repo, "mv", "--", source, f"tasks/{target_name}") - print("STOP: review and commit ONLY this pure rename; then run finalize with the same source and target.") - return - if src.exists() or not target.is_dir(): - raise ValueError("Finalize requires the source to have been renamed") - no_symlinks(target) - # HEAD itself must be the operator's separate pure rename commit. No state file. - parents = git(repo, "rev-list", "--parents", "-n", "1", "HEAD").split() - if len(parents) != 2: - raise ValueError("Finalize requires a single-parent pure rename HEAD commit") - fields = git(repo, "diff", "--name-status", "-z", "-M100%", "HEAD^", "HEAD").strip("\0").split("\0") - renamed = set() - for index in range(0, len(fields), 3): - chunk = fields[index:index + 3] - if (len(chunk) != 3 or chunk[0] != "R100" or not chunk[1].startswith(source + "/") - or chunk[2] != f"tasks/{target_name}/" + chunk[1][len(source) + 1:]): - raise ValueError("HEAD must contain only 100% identical source-to-target renames") - renamed.add(chunk[2]) - tracked = set(filter(None, git(repo, "ls-files", "-z", "--", f"tasks/{target_name}").split("\0"))) - if not tracked or renamed != tracked: - raise ValueError("Pure rename commit must include every tracked package file") - config = tomllib.loads((target / "task.toml").read_text()) - if config.get("task", {}).get("name") != f"search-swe/{target_name}": - raise ValueError("Expected the same task name after promotion") - changes = [] - text_suffixes = {".md", ".toml", ".json", ".yaml", ".yml", ".sh", ".py", ".txt", ".cfg", ".ini"} - for name in sorted(tracked): - path = repo / name - raw = path.read_bytes() - if source.encode() not in raw: - continue - if path.suffix not in text_suffixes and path.name not in ("Dockerfile", ".gitignore", ".dockerignore"): - raise ValueError(f"Manual review required for nonstandard text file: {name}") - text = raw.decode("utf-8") - updated = re.sub(re.escape(source) + r"(?![a-z0-9-])", f"tasks/{target_name}", text) - if re.search(re.escape(source) + r"(?![a-z0-9-])", updated): - raise ValueError(f"Submission reference remains: {name}") - changes.append((path, updated)) - for number, (before, after) in enumerate(zip(text.splitlines(), updated.splitlines()), 1): - if before != after: - print(f"{name}:{number}: {before} -> {after}") - if not dry_run: - for path, updated in changes: - path.write_text(updated) - print("Submission package references remaining after planned changes: 0.") - print("Review every change, external asset paths/images and repository task inventories. " - "Publish approved assets, pin the official SHA, validate, then commit finalization in this SAME PR.") - - def checked_path(path): path = Path(path).absolute() if ".." in path.parts: @@ -164,11 +72,11 @@ def prepare(snapshot, task, data, output): no_symlinks(tree) if output.exists(): raise ValueError("Upload output must not exist; refusing overwrite") - if task.parent.name != "tasks" or not FORMAL.fullmatch(task.name): - raise ValueError("Promote to tasks/ before preparing official publication") + if task.parent.name != "tasks" or not TASK_NAME_RE.fullmatch(task.name): + raise ValueError("Task package must be at tasks/") config = tomllib.loads((task / "task.toml").read_text()) if config.get("task", {}).get("name") != f"search-swe/{task.name}": - raise ValueError("Finalize task.toml before preparing official publication") + raise ValueError("task.toml name must match the task directory") if not data.is_dir() or output.is_relative_to(data) or output.is_relative_to(task): raise ValueError("Use a separate output directory and an existing new-data directory") manifest = json.loads(snapshot.read_text()) @@ -214,31 +122,16 @@ def prepare(snapshot, task, data, output): return len(additions) -def promotion_main(default_repo=None): - parser = argparse.ArgumentParser(description="Two-phase offline promotion; never commits or publishes") - parser.add_argument("phase", choices=("rename", "finalize")) - parser.add_argument("source") - parser.add_argument("target_name") - parser.add_argument("--repo-root", type=Path, default=default_repo, required=default_repo is None) - parser.add_argument("--dry-run", action="store_true") - args = parser.parse_args() - try: - repo = checked_path(args.repo_root) - promote(repo, args.source, args.target_name, args.phase, args.dry_run) - except (OSError, ValueError, subprocess.CalledProcessError) as error: - parser.error(str(error)) - return 0 - -def upload_main(legacy=False): +def upload_main(repository_cli=False): parser = argparse.ArgumentParser(description="Offline incremental HF staging; never uploads") - parser.add_argument("--repo-root", type=Path, required=not legacy) + parser.add_argument("--repo-root", type=Path, required=not repository_cli) parser.add_argument("--official-manifest", required=True, type=Path) parser.add_argument("--task-path", required=True, type=Path) parser.add_argument("--new-data", required=True, type=Path) parser.add_argument("--output", required=True, type=Path) args = parser.parse_args() try: - # Legacy repository CLI accepted paths relative to cwd, or any absolute task. + # The repository CLI accepts paths relative to cwd, or absolute task paths. # Portable CLI always requires an explicit target root. task = args.task_path if args.repo_root is not None: @@ -252,6 +145,6 @@ def upload_main(legacy=False): count = prepare(args.official_manifest, task, args.new_data, args.output) except (OSError, ValueError, KeyError, TypeError) as error: parser.error(str(error)) - print(f"Prepared {count} new files and merged manifest; preserved old entries without downloading old data.") - print("No upload performed. Review SOURCES.md, dataset card/license and .gitattributes; never delete old assets.") + print(f"Prepared {count} new files and merged manifest; preserved existing manifest entries.") + print("Review SOURCES.md, dataset card/license and .gitattributes before the authorized upload.") return 0 diff --git a/.agents/skills/maintain-searchswe-task/scripts/promote_task.py b/.agents/skills/maintain-searchswe-task/scripts/promote_task.py deleted file mode 100644 index fdc870b..0000000 --- a/.agents/skills/maintain-searchswe-task/scripts/promote_task.py +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env python3 -"""Portable offline helper; --repo-root is required.""" -from maintainer_helpers import promotion_main - -if __name__ == "__main__": - raise SystemExit(promotion_main()) diff --git a/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py b/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py index c16a914..0b074cf 100644 --- a/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py +++ b/.agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Offline relocation, fake-gh, history and staging acceptance tests.""" +"""Offline skill relocation, PR capture and asset staging acceptance tests.""" import hashlib import json @@ -92,31 +92,6 @@ def capture(self, mode="pass", output=None): "--repo-root", self.repo, "--output", output or self.out, env=dict(self.env, FIXTURE=mode)) - def git(self, *args): - return subprocess.run(["git", *args], cwd=self.repo, check=True, - capture_output=True, text=True).stdout - - def task(self): - self.git("init", "-q", "-b", "main") - self.git("config", "user.name", "Maintainer") - self.git("config", "user.email", "maintainer@example.test") - (self.repo / "tasks").mkdir() - (self.repo / ".gitignore").write_text("__pycache__/\n*.py[cod]\n") - (self.repo / "base.txt").write_text("base\n") - self.git("add", "."); self.git("commit", "-qm", "base") - self.source = "task-submissions/example-search" - task = self.repo / self.source - task.mkdir(parents=True) - (task / "task.toml").write_text('[task]\nname = "search-swe/example-search"\nauthors = ["Alice Example", "Bob Example"]\n') - (task / "instruction.md").write_text("# example-search\n\ntask-submissions/example-search\nOriginal contributor content\n") - (task / "assets.json").write_text('{"schema_version":1,"files":[]}\n') - self.git("add", ".") - self.git("commit", "-qm", "Write submission", "--author", "Alice Example ") - return task - - def promote(self, phase, *extra): - return self.cli("promote_task.py", "--repo-root", self.repo, phase, self.source, "example-search", *extra) - def test_capture_relocated_paginated_untrusted_and_read_only(self): # Dirty local content is untouched; dangerous PR text remains inert. dirty = self.repo / "dirty.py" @@ -189,29 +164,6 @@ def test_partial_check_set_and_intrafile_diff_are_not_verified(self): self.assertIn("raw command collection only", snapshot["completeness_scope"]) self.assertIn("unverified", result.stdout) - def test_promotion_relocation_collision_history_and_authors(self): - task = self.task() - target = self.repo / "tasks/example-search" - target.mkdir() - self.assertNotEqual(self.promote("rename").returncode, 0) - target.rmdir() - self.assertEqual(self.promote("rename", "--dry-run").returncode, 0) - self.assertTrue(task.exists()) - original = self.git("rev-parse", "HEAD") - self.assertEqual(self.promote("rename").returncode, 0) - self.assertEqual(self.git("rev-parse", "HEAD"), original) # never commits - self.assertNotEqual(self.promote("finalize").returncode, 0) - self.git("commit", "-qm", "Pure rename") - result = self.promote("finalize") - self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn('authors = ["Alice Example", "Bob Example"]', (target / "task.toml").read_text()) - self.assertIn("search-swe/example-search", (target / "task.toml").read_text()) - self.git("add", "."); self.git("commit", "-qm", "Finalize") - self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", "tasks/example-search/instruction.md")) - self.assertIn("author Alice Example", self.git("blame", "--line-porcelain", "tasks/example-search/instruction.md")) - # Explicit target is mandatory even when cwd is a Search-SWE checkout. - self.assertNotEqual(self.cli("promote_task.py", "rename", self.source, "task-1-9").returncode, 0) - def staging(self): target = self.repo / "tasks/example-search" target.mkdir(parents=True) @@ -229,19 +181,19 @@ def staging(self): snapshot.write_text(json.dumps({"schema_version": 1, "files": [old], "extra": "preserve"})) return target, data, file, snapshot, old - def test_official_staging_rejects_submissions_and_nested_formal_paths(self): + def test_official_staging_requires_canonical_task_paths(self): target, data, _, snapshot, _ = self.staging() - submission = self.repo / "task-submissions/example-search" - submission.parent.mkdir() - target.rename(submission) + misplaced = self.repo / "drafts/example-search" + misplaced.parent.mkdir() + target.rename(misplaced) args = ("--repo-root", self.repo, "--official-manifest", snapshot, "--new-data", data, "--output", self.out) - result = self.cli("prepare_hf_upload.py", *args, "--task-path", submission.relative_to(self.repo)) + result = self.cli("prepare_hf_upload.py", *args, "--task-path", misplaced.relative_to(self.repo)) self.assertNotEqual(result.returncode, 0) self.assertIn("requires tasks/", result.stderr) nested = self.repo / "nested/tasks/example-search" nested.parent.mkdir(parents=True) - submission.rename(nested) + misplaced.rename(nested) result = self.cli("prepare_hf_upload.py", *args, "--task-path", nested.relative_to(self.repo)) self.assertNotEqual(result.returncode, 0) self.assertFalse(self.out.exists()) @@ -265,33 +217,28 @@ def test_incremental_staging_relocated_hashes_and_manifest_preservation(self): self.assertNotEqual(self.cli("prepare_hf_upload.py", *args, "--output", self.root / "corrupt").returncode, 0) self.assertFalse((self.root / "corrupt").exists()) - def test_legacy_wrapper_cli_compatibility(self): - # In the source repository also exercise the old CLI syntax, no --repo-root. + def test_repository_wrapper_paths(self): + # Exercise the repository CLI with absolute and cwd-relative paths. repository = SKILL.parents[2] if not (repository / "scripts/_maintainer_skill.py").is_file(): - self.skipTest("Compatibility wrappers are repository inputs, absent in standalone skill copy") + self.skipTest("Repository wrappers are inputs, absent in standalone skill copy") scripts = self.repo / "scripts" scripts.mkdir() - for name in ("_maintainer_skill.py", "promote_task.py", "prepare_hf_upload.py"): + for name in ("_maintainer_skill.py", "prepare_hf_upload.py"): shutil.copyfile(repository / "scripts" / name, scripts / name) shutil.copytree(self.skill, self.repo / ".agents/skills/maintain-searchswe-task") - self.task() - def legacy(name, *args): + def repository_cli(name, *args): return subprocess.run([sys.executable, str(scripts / name), *map(str, args)], cwd=self.root, capture_output=True, text=True) - self.assertEqual(legacy("promote_task.py", "rename", self.source, "example-search").returncode, 0) - self.git("commit", "-qm", "Pure rename") - self.assertEqual(legacy("promote_task.py", "finalize", self.source, "example-search").returncode, 0) - shutil.rmtree(self.repo / "tasks/example-search") target, data, file, snapshot, old = self.staging() - result = legacy("prepare_hf_upload.py", "--task-path", target, "--official-manifest", snapshot, + result = repository_cli("prepare_hf_upload.py", "--task-path", target, "--official-manifest", snapshot, "--new-data", data, "--output", self.out) self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(json.loads((self.out / "manifest.json").read_text())["files"][0], old) - # Original HF CLI resolves relative paths from cwd, not wrapper location. - result = legacy("prepare_hf_upload.py", "--task-path", target.relative_to(self.root), + # Repository HF CLI resolves relative paths from cwd, not wrapper location. + result = repository_cli("prepare_hf_upload.py", "--task-path", target.relative_to(self.root), "--official-manifest", snapshot, "--new-data", data, - "--output", self.root / "legacy-relative") + "--output", self.root / "relative-output") self.assertEqual(result.returncode, 0, result.stderr) def test_staging_rejects_symlinks_traversal_extra_files_and_external_target(self): diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 0dc0bf7..145b4f6 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,6 +1,6 @@ ## Summary -Task name(s) and submission path(s): `task-submissions/` +Task name(s) and package path(s): `tasks/` Goal, starting system, and CPU/GPU requirements: Authors and coauthors: @@ -14,9 +14,9 @@ scores and judge health; explain task-setting changes motivated by that feedback ## Contributor checklist -- [ ] New tasks use unique descriptive names of at most five lowercase hyphen-separated words and follow `task-submissions/`; each path identifies one task throughout this PR. +- [ ] New tasks use unique descriptive names of at most five lowercase hyphen-separated words and live at `tasks/`. - [ ] Actual authors in `task.toml`; original author commits preserved. -- [ ] Allow edits from maintainers, or agree on contributor-performed promotion in this PR. +- [ ] Allow edits from maintainers, or agree on contributor-applied review fixes in this PR. - [ ] No secrets, downloaded inputs/models, job outputs, or author-only solutions. - [ ] Static checks and applicable runtime evidence recorded. - [ ] Provenance, redistribution rights, licenses and immutable development asset SHA documented. @@ -24,8 +24,7 @@ scores and judge health; explain task-setting changes motivated by that feedback ## Maintainer merge checklist - [ ] Design, verifier isolation, runtime evidence and any validation exceptions reviewed. -- [ ] Names checked against the current base and queued PRs; each task keeps its name through promotion. -- [ ] Every new task has a separate pure move from submissions in this PR, followed by reviewed finalization. +- [ ] Names checked against the current base and queued PRs. - [ ] New official assets published under `tasks//`; previous assets preserved; merged official SHA pinned. -- [ ] Repository docs and inventories updated; checks and merge-ready pass; no submission package remains. -- [ ] Merge commit only; original contributor history retained. +- [ ] Repository docs and inventories updated; checks and merge-ready pass. +- [ ] Merge commit; original contributor history retained. diff --git a/.github/workflows/task-checks.yml b/.github/workflows/task-checks.yml index 9ab1038..9487816 100644 --- a/.github/workflows/task-checks.yml +++ b/.github/workflows/task-checks.yml @@ -21,18 +21,18 @@ jobs: python-version: '3.12' - run: python -m pip install -r scripts/requirements.txt PyYAML python-dotenv - run: python scripts/check_release.py - - run: python scripts/check_submission.py - - name: Submission history and same-PR promotion + - run: python scripts/check_tasks.py + - name: Check authors of new tasks if: github.event_name == 'pull_request' env: BASE_SHA: ${{ github.event.pull_request.base.sha }} - run: python scripts/check_submission.py --base "$BASE_SHA" + run: python scripts/check_tasks.py --base "$BASE_SHA" - run: python -m unittest discover -s scripts/tests -p 'test_*.py' - run: python .agents/skills/create-searchswe-task/scripts/test_scaffold_task.py - run: python .agents/skills/maintain-searchswe-task/scripts/test_maintainer_skill.py - run: git diff --check merge-ready: - # Intentionally fails during submission review. Require this before merging; + # Requires pinned official dataset sources before merging; # repository branch-protection settings are configured separately by owners. runs-on: ubuntu-latest steps: @@ -43,4 +43,4 @@ jobs: with: python-version: '3.12' - run: python -m pip install PyYAML - - run: python scripts/check_submission.py --merge-ready + - run: python scripts/check_tasks.py --merge-ready diff --git a/.gitignore b/.gitignore index eeb3fed..89838f3 100644 --- a/.gitignore +++ b/.gitignore @@ -6,9 +6,7 @@ /.venv/ tasks/*/data/ tasks/*/models/ -task-submissions/*/data/ -task-submissions/*/models/ -task-submissions/*/jobs/ -task-submissions/*/solution/ +tasks/*/jobs/ +tasks/*/solution/ __pycache__/ *.py[cod] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0564114..5e6e17f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,10 +1,10 @@ # Contributing to Search-SWE Start with the [contribution guide](docs/contributing.md) for task design, -validation, asset publication, and maintainer promotion. For agent-assisted +validation, asset publication, and maintainer review. For agent-assisted authoring, use the [create-searchswe-task skill](.agents/skills/create-searchswe-task/SKILL.md). -New tasks belong at `task-submissions/`. Choose a descriptive name of +New tasks belong at `tasks/`. Choose a descriptive name of at most five lowercase, hyphen-separated words, such as `example-search`. The `task-` prefix and `all` are reserved. One PR may add multiple tasks; record actual authors in each package. @@ -15,7 +15,6 @@ issues, then validate the revised setting and record the final tested version. If required API keys are missing, the authoring agent should ask the contributor to configure them locally before running the trial; see the guide for details. -Maintainers promote each reviewed package to `tasks/` in the same PR, -preserving its name and original authors. Commit the pure directory move -separately from finalization changes. Use a merge commit for task contributions, -not squash or rebase merge; every submission must be promoted before merging. +Maintainers review task quality, validation evidence and asset provenance in the +PR. Publish official inputs and pin their merged HF commit before merging the +GitHub contribution. Use a merge commit to preserve original author history. diff --git a/README.md b/README.md index 477aebd..b614dbe 100644 --- a/README.md +++ b/README.md @@ -235,9 +235,9 @@ For task descriptions and results, visit the Contributions are welcome. For new or substantially revised tasks, start with the workflow and skills in [`.agents/`](.agents/AGENTS.md), including -[`create-searchswe-task`](.agents/skills/create-searchswe-task/SKILL.md) for submissions +[`create-searchswe-task`](.agents/skills/create-searchswe-task/SKILL.md) for task authoring and [`maintain-searchswe-task`](.agents/skills/maintain-searchswe-task/SKILL.md) for -PR review and promotion. See the +PR review and publication. See the [contribution entry](CONTRIBUTING.md) and [guide](docs/contributing.md) for validation and PR requirements. Task authoring includes running a configured coding model, inspecting its @@ -246,17 +246,16 @@ setting. Revalidate any changes that affect the task or scoring and report the final tested revision. If required API keys are missing, the authoring agent asks the contributor to configure them locally before the trial. -New tasks use `task-submissions/`. Choose a descriptive name of at -most five lowercase, hyphen-separated words; use that same name for the formal -package and its official data. One PR may add multiple tasks. Maintainers -promote each reviewed package to `tasks/` in the same PR, with a -separate pure move and finalization. Preserve original authors with a merge -commit; no unfinished submission enters main. +Create tasks directly at `tasks/` in your contribution branch. Choose +a descriptive name of at most five lowercase, hyphen-separated words, and use +it for the package and its official data. One PR may add multiple tasks. Review +and validate each package, then merge with a merge commit to preserve original +authors. Development assets may use a personal public HF dataset pinned to a commit SHA. Publish official inputs under `tasks//`, then pin the merged official -HF SHA before merging the GitHub contribution. Explicit `--task-path` supports -submission downloads and local trials; `--task ` selects a formal task. +HF SHA before merging the GitHub contribution. Use `--task ` for +downloads and local trials. ## Citation diff --git a/README_zh.md b/README_zh.md index 1de7514..f572c29 100644 --- a/README_zh.md +++ b/README_zh.md @@ -221,21 +221,20 @@ bash scripts/run_task.sh --task reasoning-query-rewriting --agent claude-code -- 欢迎参与贡献。创建新任务或大幅修改已有任务时，请先使用 [`.agents/`](.agents/AGENTS.md) 中的工作流和 skills，包括 [`create-searchswe-task`](.agents/skills/create-searchswe-task/SKILL.md)（任务投稿）和 -[`maintain-searchswe-task`](.agents/skills/maintain-searchswe-task/SKILL.md)（PR 审查与正式化）。验证要求和 +[`maintain-searchswe-task`](.agents/skills/maintain-searchswe-task/SKILL.md)（PR 审查与发布）。验证要求和 PR 说明见[贡献入口](CONTRIBUTING.md)与[贡献指南](docs/contributing.md)。 任务构建还包括使用配置好的编码模型试跑，检查执行轨迹和评测结果，并根据反馈 完善任务设定。影响任务要求或评分的修改需要重新验证，并记录最终测试的版本。 如果缺少必需的 API key，负责构建任务的智能体应在试跑前提醒贡献者在本地配置。 -新任务放在 `task-submissions/`。名称使用不超过五个词的小写连字符 -短语，正式任务与官方数据使用同一名称。一个 PR 可以包含多个任务；维护者在 -同一个 PR 中逐一晋升到 `tasks/`，将纯目录移动和后续整理分开提交。 -使用 merge commit 保留原作者历史，未完成晋升的任务不进入 main。 +在贡献分支中直接将新任务放在 `tasks/`。名称使用不超过五个词的 +小写连字符短语，任务包与官方数据使用同一名称。一个 PR 可以包含多个任务； +逐一完成审查与验证后，使用 merge commit 保留原作者历史。 开发数据可以来自固定到 commit SHA 的个人公开 HF 数据集。官方输入发布到 `tasks//` 后，先固定官方已合并的 HF SHA，再合并 GitHub 贡献。 -使用 `--task-path` 显式下载、试跑待审任务；正式任务通过 `--task ` 选择。 +使用 `--task ` 下载数据和进行本地试跑。 ## 引用 diff --git a/docs/assets.md b/docs/assets.md index 7917041..0edb2d0 100644 --- a/docs/assets.md +++ b/docs/assets.md @@ -36,31 +36,30 @@ python scripts/download_assets.py --task all --dry-run Use the preview to estimate disk requirements. Allow additional space for the Hugging Face cache; the downloader copies cached files into the task directories. -## New-task submissions +## Developing a task -Contributors can validate before promotion with an explicit package: +Create the package at `tasks/`, then select it by name: ```bash -python scripts/download_assets.py --task-path task-submissions/example-search --dry-run -python scripts/download_assets.py --task-path task-submissions/example-search -python scripts/download_assets.py --task-path task-submissions/example-search --verify-only -python scripts/run_task.py --task-path task-submissions/example-search \ +python scripts/download_assets.py --task example-search --dry-run +python scripts/download_assets.py --task example-search +python scripts/download_assets.py --task example-search --verify-only +python scripts/run_task.py --task example-search \ --agent pi --model deepseek/deepseek-flash --dry-run ``` -`--task all` selects all formal tasks. Paths must be canonical repository-relative -paths without traversal or symlinks. An alternate `--output-dir` preserves the -submission path; launcher jobs default to `jobs/task-submissions/example-search`. -Each submission has a unique task name. +`--task all` selects every package under `tasks/` in the current checkout. +`--task-path tasks/` accepts a canonical repository-relative path. +An alternate `--output-dir` contains `/`; launcher jobs default to +`jobs/`. For development, use a personal public temporary dataset with a manifest, -provenance/license and pinned immutable SHA. After promotion, use the -[contribution publication workflow](contributing.md#asset-contribution-and-publication): -`prepare_hf_upload.py` stages only new verified files plus a merged current -manifest. Supply the current manifest and the new data files. Update HF SOURCES -and license metadata, preserve existing published assets, and submit an HF -community PR (`create_pr=True`) or request a maintainer mirror. Never share official tokens. -Official HF merge precedes final `assets.json` SHA pinning and GitHub merge. +provenance/license and pinned immutable SHA. After task and asset review, follow +the [publication workflow](contributing.md#asset-contribution-and-publication): +`prepare_hf_upload.py` stages new verified files with a merged current manifest. +Update HF SOURCES and license metadata, preserve published assets, and submit +an authorized HF community PR or request a maintainer mirror. Pin the merged +official HF SHA in `assets.json`, verify downloads, then merge the GitHub task PR. ## Directory layout diff --git a/docs/benchmark.md b/docs/benchmark.md index e626da2..1369c27 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -43,8 +43,6 @@ Search-SWE/ │ │ ├── environment/ # Dockerfile, starter code, and environment docs │ │ └── tests/ # Verifier and grading code │ └── ... # Additional task packages -├── task-submissions/ # New task packages before same-PR promotion -│ └── / # Same name retained under tasks/ after review ├── scripts/ │ ├── requirements.txt # Host-side launcher and asset dependencies │ ├── download_assets.py # Download and verify fixed data/model inputs diff --git a/docs/contributing.md b/docs/contributing.md index 2d032c4..6a07030 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -21,13 +21,12 @@ python .agents/skills/create-searchswe-task/scripts/scaffold_task.py example-sea --author 'Alice Example' --hardware cpu ``` -New packages use `task-submissions//`. Choose a concise description -with at most five lowercase hyphen-separated words, starting with an ASCII -letter and otherwise using letters/digits. Omit the `task-` prefix; `all` is -reserved. Check both `tasks/` and `task-submissions/`, plus active PRs, for name -collisions. The canonical `task.name` is `search-swe/` from the start, -and promotion preserves it. A PR may contain multiple uniquely named tasks; -do not reuse a submission path later in that PR. +Create each package directly at `tasks//` in your contribution branch. +Choose a concise description with at most five lowercase hyphen-separated words, +starting with an ASCII letter and otherwise using letters/digits. The `task-` +prefix and `all` are reserved. Check `tasks/` on the current base and active PRs +for name collisions. Set `task.name` to `search-swe/`. A PR may contain +multiple uniquely named tasks. Describe the task's goal, starting state and measured outcomes. CPU is the default; choose `--hardware gpu` for actual GPU execution. Repeat @@ -35,13 +34,13 @@ CPU is the default; choose `--hardware gpu` for actual GPU execution. Repeat and a Git email associated with your GitHub account (or its noreply address). Co-authored-by trailers supplement original commits and task authors. -The scaffold is deliberately incomplete and its verifier fails closed. Existing -task revisions stay in their formal directory. The explicit `--formal` scaffold -option supports maintainer tooling; new contributions still follow this workflow. -Do not commit credentials, downloaded inputs/models, private evaluation assets, -job outputs or author-only solutions. Keep verifier inputs isolated from agents. +The scaffold provides the package structure, writing prompts and a verifier that +fails closed. Complete the instructions, assets, environment and grader before +runtime validation. Revise existing tasks in place. Keep credentials, downloaded +inputs/models, job outputs and author-only solutions outside Git, and isolate +verifier inputs from agents. -## 2. Develop and validate before promotion +## 2. Develop, evaluate and refine Development assets may live in a personal public temporary HF dataset with redistribution permission, manifest, provenance and license. Pin its immutable @@ -50,25 +49,24 @@ redistribution permission, manifest, provenance and license. Pin its immutable See [asset contribution and publication](#asset-contribution-and-publication). ```bash -python scripts/check_submission.py task-submissions/example-search +python scripts/check_tasks.py tasks/example-search python scripts/check_release.py python -m unittest discover -s scripts/tests -p 'test_*.py' python .agents/skills/create-searchswe-task/scripts/test_scaffold_task.py git diff --check -python scripts/download_assets.py --task-path task-submissions/example-search --dry-run +python scripts/download_assets.py --task example-search --dry-run # After approving the download budget: -python scripts/download_assets.py --task-path task-submissions/example-search -python scripts/download_assets.py --task-path task-submissions/example-search --verify-only -python scripts/run_task.py --task-path task-submissions/example-search \ +python scripts/download_assets.py --task example-search +python scripts/download_assets.py --task example-search --verify-only +python scripts/run_task.py --task example-search \ --agent pi --model deepseek/deepseek-flash --dry-run ``` -Downloader and launcher require canonical repository-relative paths without -traversal or symlinks. `--task ` selects a formal package; downloader -`--task all` selects only formal packages. Submissions require `--task-path`. -Submission jobs default to `jobs/task-submissions/` and alternate -asset output roots retain `task-submissions/`. The launcher reads the -original package's local assets, not an alternate download directory. +Use `--task ` for downloads and local trials. The downloader's +`--task all` selects every package under `tasks/` in the current checkout. +`--task-path tasks/` also accepts a canonical repository-relative +path. Jobs default to `jobs/`; alternate asset output roots contain +`/`. The launcher reads the package's local assets. Follow the skill's [validation layers](../.agents/skills/create-searchswe-task/references/validation.md): Compose rendering, builds, verifier known-good and negative cases, and an @@ -91,89 +89,63 @@ Reuse existing authorized credentials; never ask for key values in chat or a PR. Wait for configuration before starting the dependent run, while continuing independent checks. Report unavailable runtime evidence explicitly. -Static checks alone do not prove solvability or safe grading. Report actual -commands, results, resource requirements and unrun layers. Enable **Allow edits -from maintainers** on the PR; if unavailable, the contributor makes the agreed -promotion commits in the same PR. +Static checks cover package structure and source syntax. Record runtime +commands, results, resource requirements and any unrun validation layers. Enable +**Allow edits from maintainers** on the PR where available; otherwise agree on +contributor-applied review fixes in the same PR. ### CI and trust boundaries -The unprivileged `pull_request` and `push` workflow checks formal packages and -all submissions, including malformed or incomplete paths. With `--base`, it -also inspects every commit and both sides of merge history: submission paths -must be flat, paths cannot be reused, and each newly added formal task needs a -same-name, whole-package pure rename from its submission in this PR. Reviewed -renames of packages already formal on the base are recognized as existing-task -changes. New formal tasks require actual authors; existing author records remain. - -Submission checks reuse release structure, asset pin/hash, secret, file-size -and Git-ignore checks, and parse Python, shell and Compose YAML without running -task code. Install PyYAML for the YAML check; Docker Compose rendering is separate. -`review-stage` may pass while submissions exist. Before merging, run -`python scripts/check_submission.py --merge-ready`: it rejects remaining -submissions and checks formal package syntax and obsolete references. - -CI executes no GPU/API/container task trials. Repository tests are PR-controlled -code: use read-only permissions, no secrets and no persisted checkout credentials. -Never execute unreviewed PR code via `pull_request_target` with secrets or share -an official HF token. Review and separately authorize credentialed/runtime tests -in a disposable trusted environment. - -## 3. Maintainer promotion in the same PR +The unprivileged `pull_request` and `push` workflow checks every package under +`tasks/`, including incomplete or malformed directories. `check_tasks.py` reuses +release structure, asset pin/hash, secret, file-size and Git-ignore checks, and +parses Python, shell and Compose YAML without running task code. With `--base`, +it additionally requires actual authors for newly added tasks while preserving +existing author records, including reviewed package renames. -Use [maintain-searchswe-task](../.agents/skills/maintain-searchswe-task/SKILL.md) -for gh review, audit and merge checks. Repository `scripts/promote_task.py` and -`scripts/prepare_hf_upload.py` delegate to that portable skill; a separately -installed skill uses its own scripts with `--repo-root /path/to/checkout`. +`review-stage` accepts pinned personal development datasets. Before merging, +`python scripts/check_tasks.py --merge-ready` requires dataset sources in +`search-swe/Search-SWE`, under `tasks//`, at immutable commit SHAs. +Model files retain their original repository pins. Maintainers also verify that +the official HF changes have merged and that runtime evidence covers the final +revision. Install PyYAML for static Compose parsing; render Compose separately. -Finish design, provenance/license, environment, verifier-isolation and runtime -review. Update against current main while preserving author history. Recheck -task-name collisions with the base and queued PRs before promotion and merge. -Promotion keeps the task name unchanged. +CI runs static checks and repository tests with read-only permissions and no +secrets or persisted checkout credentials. Repository tests are PR-controlled +code. Review and separately authorize credentialed runtime tests in a disposable +trusted environment; run GPU/API/container trials there. -Start from a clean worktree/index. The helper never commits, pushes or publishes: +## 3. Review, publish and merge -```bash -python scripts/promote_task.py rename task-submissions/example-search example-search --dry-run -python scripts/promote_task.py rename task-submissions/example-search example-search -git diff --cached --summary -# Operator creates the separate pure-rename commit: -git commit -m 'Promote example-search' -python scripts/promote_task.py finalize task-submissions/example-search example-search --dry-run -python scripts/promote_task.py finalize task-submissions/example-search example-search -``` - -Repeat independently for each task. Rename does only `git mv`. Finalize requires -HEAD to be that task's single-parent, 100%-identical whole-package rename commit. -It updates the submission path in tracked text and prints changed lines; -the task name and canonical identity stay unchanged. It refuses unsupported -binary rewrites and does not edit downloaded data. Review replacements, mounts, -images and external URLs; update task inventories/docs and GPU test inventories -as needed. A name collision must be resolved through review before promotion, -never by overwriting a package. - -Publish approved new official assets, pin their merged SHA, then validate. Make -any reference, asset or integration changes in a separate finalization commit -in this same PR; an empty finalization commit is unnecessary. Every submission -must be promoted before merge. Use a **merge commit only**, preserving original -author and pure-rename commits; no squash/rebase merge. +Use [maintain-searchswe-task](../.agents/skills/maintain-searchswe-task/SKILL.md) +for gh review, task audit and merge checks. Finish design, provenance/license, +environment, verifier-isolation and runtime review. Update against current main +while preserving author history, and check task names against the base and queued +PRs. Apply review fixes, asset pins and directly necessary documentation or +inventory changes to the task package in the same PR. + +Publish approved official inputs using the +[asset workflow](#asset-contribution-and-publication), then pin the merged HF SHA +and verify downloads. `scripts/prepare_hf_upload.py` delegates to the portable +maintainer skill; a separately installed skill uses its own helper with +`--repo-root /path/to/checkout`. ```bash python scripts/check_release.py -python scripts/check_submission.py --merge-ready +python scripts/check_tasks.py --merge-ready --base FULL_PR_BASE_SHA python -m unittest discover -s scripts/tests -p 'test_*.py' git diff --check git log --follow -- tasks/example-search/instruction.md git blame tasks/example-search/instruction.md ``` -Original commits and unmodified lines' blame survive this workflow. GitHub file -lists/history may display renames differently; contributor statistics also -depend on account-linked email and reachability from the default branch. -Repository owners separately configure required review, resolved conversations, -current-base/queue checks and `review-stage`/`merge-ready` statuses. This workflow -does not configure remote protections or CODEOWNERS. Re-review each changed head; -resolve unavailable branch editing with the contributor, keeping the same PR. +Use a **merge commit** to preserve original author commits and unmodified lines' +attribution. Re-review each changed head and verify required checks for the +current head and base. Repository owners configure required reviews, resolved +conversations, current-base/queue checks and `review-stage`/`merge-ready` statuses. +After the merge, verify the resulting tree, official asset pins and author +history. GitHub contributor statistics also depend on account-linked email and +reachability from the default branch. ## Asset contribution and publication @@ -198,18 +170,18 @@ repo = 'YOUR_ACCOUNT/search-swe-development' api.create_repo(repo_id=repo, repo_type='dataset', private=False, exist_ok=True) commit = api.upload_folder(repo_id=repo, repo_type='dataset', folder_path='/path/to/dev-data', commit_message='Add temporary task development inputs') -print(commit.oid) # Pin this full SHA in submission assets.json. +print(commit.oid) # Pin this full SHA in task assets.json. PY ``` Use this personal repo, its dataset-relative filenames, size/hash, and printed -immutable SHA in `assets.json`. The explicit submission downloader/launcher then -support real pre-promotion validation. Keep the temporary repo accessible through -review and migration; only retire it after official pinned downloads are verified. +immutable SHA in `assets.json`. Select the task by name with the downloader and +launcher for development validation. Keep the temporary repo accessible through +review and publication; retire it after official pinned downloads are verified. -### Task name: additive official publication, then GitHub merge +### Official publication, then GitHub merge -After task-name approval and promotion, change only the new task's dataset source +After task and asset review, change only the new task's dataset source repo/filename to `search-swe/Search-SWE` and `tasks//...`. Preserve original model repo pins and bundled metadata. Prepare **only new data files** in `/path/to/new-data/tasks//...`, at the exact asset sizes/hashes. @@ -237,7 +209,7 @@ python scripts/prepare_hf_upload.py \ --output /path/to/upload-staging ``` -Official asset staging requires a promoted package at `tasks/`. +Stage assets from the reviewed package at `tasks/`. The offline helper verifies hashes and safe paths for new files, rejects collisions and symlinks, and stages those files with a merged manifest that @@ -279,6 +251,6 @@ replace a newer manifest with a stale one. **Merge the official HF change first.** Only then pin the resulting official 40-hex commit SHA (not the community PR/head SHA or `main`) in the finalized GitHub task's `assets.json`. Verify downloads using a fresh output directory or -cache and run release checks. Commit finalization and merge the GitHub PR last. -No new task with unpublished/unpinned assets or an unfinished submission enters -main. Preserve existing tasks' pinned revisions and published assets. +cache and run release and merge-ready checks. Commit the verified asset pins +in the task PR, then merge the GitHub contribution. Preserve existing tasks' +pinned revisions and published assets. diff --git a/docs/network-policy.md b/docs/network-policy.md index a86a49e..8bb7153 100644 --- a/docs/network-policy.md +++ b/docs/network-policy.md @@ -129,7 +129,7 @@ access. | `code-embedding-finetuning` | Explicitly offline training | no network | model host only | `openrouter.ai` only | | `query-encoder-alignment` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts (OpenRouter also serves the trajectory judge) | | `agentic-search` | OpenRouter/Jina resources only | allowlist: `openrouter.ai`, `api.jina.ai` | task hosts + model host | task hosts only; no model judge | -| `sparse-retrieval-pruning` | Formal build/search is offline | no network | model host only | `openrouter.ai` only | +| `sparse-retrieval-pruning` | Offline index building and search | no network | model host only | `openrouter.ai` only | Allowing the judge endpoint does not expose verifier credentials to submitted commands. `scientific-paper-qa` similarly removes both judge credential groups before diff --git a/scripts/_maintainer_skill.py b/scripts/_maintainer_skill.py index 4dfe4cc..d1e3601 100644 --- a/scripts/_maintainer_skill.py +++ b/scripts/_maintainer_skill.py @@ -1,4 +1,4 @@ -"""Load the portable skill's single-source offline helpers for legacy CLIs.""" +"""Load the portable skill's single-source offline helpers for the repository CLI.""" import importlib.util from pathlib import Path diff --git a/scripts/check_release.py b/scripts/check_release.py index 5e8fd67..d69a471 100644 --- a/scripts/check_release.py +++ b/scripts/check_release.py @@ -10,7 +10,7 @@ import tomllib from download_assets import REPO, matches, read_manifest, relative_path -from task_paths import FORMAL, no_symlinks, safe_path +from task_paths import TASK_NAME_RE, no_symlinks, safe_path def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False, *, packages=None): @@ -45,12 +45,12 @@ def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False errors.append(f"{task.name}: missing or empty {name}") try: config = tomllib.loads((task / "task.toml").read_text()) - if not FORMAL.fullmatch(task.name): + if not TASK_NAME_RE.fullmatch(task.name): errors.append(f"Invalid task name: {task.name}; use at most five lowercase hyphen-separated words without a task- prefix") if config.get("task", {}).get("name") != f"search-swe/{task.name}": errors.append(f"{task.name}: expected task name search-swe/{task.name}") if "task_type" in config.get("metadata", {}): - errors.append(f"{task.name}: remove metadata.task_type; describe the task through its objective and keywords") + errors.append(f"{task.name}: unsupported metadata field: task_type; describe the task through its objective and keywords") if not config.get("task", {}).get("version"): errors.append(f"{task.name}: missing task version") manifest = read_manifest(task) @@ -94,7 +94,7 @@ def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False if not path.is_file(): continue # A tracked deletion is absent from the next commit. parts = Path(name).parts - task_payload = (len(parts) > 3 and parts[0] in ("tasks", "task-submissions") + task_payload = (len(parts) > 3 and parts[0] == "tasks" and parts[2] in ("data", "models", "jobs", "solution")) if name in runtime_paths or task_payload: errors.append(f"Downloaded runtime asset would enter Git: {name}") @@ -107,8 +107,6 @@ def check_release(repo, hf_data=None, verify_data=False, allow_unpublished=False text = path.read_text() except UnicodeError: continue - if name.startswith("tasks/") and re.search(r"task-submissions/[a-z0-9-]+", text): - errors.append(f"Unfinalized submission path in formal package: {name}") if secret.search(text): errors.append(f"Possible credential in Git file: {name} (value omitted)") if runtime_paths: diff --git a/scripts/check_submission.py b/scripts/check_submission.py deleted file mode 100644 index e14a5b9..0000000 --- a/scripts/check_submission.py +++ /dev/null @@ -1,238 +0,0 @@ -#!/usr/bin/env python3 -"""Static, credential-free submission checks; never import or run task code.""" - -import argparse -import ast -from pathlib import Path -import re -import subprocess -import sys -import tomllib - -from check_release import check_release -from download_assets import REPO -from task_paths import FORMAL, SUBMISSION, no_symlinks, safe_path, select_task - - -def discover(repo): - root = safe_path(repo, "task-submissions") - no_symlinks(root) - # Include malformed paths and incomplete package directories, not just valid TOMLs. - return sorted(set(root.glob("*/")) | {p.parent for p in root.rglob("task.toml")}) - - -def submission_root(path): - """Validate a historical filename and return its flat submission package path.""" - if "\\" in path or any(part in ("", ".", "..") for part in path.split("/")): - raise ValueError(f"Unsafe task-submissions path: {path!r}") - parts = path.split("/") - if path == "task-submissions/README.md": - return None - if (len(parts) < 3 or not SUBMISSION.fullmatch("/".join(parts[:2])) - or (parts[-1] == "task.toml" and len(parts) != 3)): - raise ValueError(f"Malformed task submission path in PR history: {path!r}") - return "/".join(parts[:2]) - - -def git_names(repo, *args): - """Run a NUL-delimited Git pathname query without shell interpolation.""" - return [name for name in subprocess.run( - ["git", *args], cwd=repo, check=True, capture_output=True, - ).stdout.decode("utf-8", errors="surrogateescape").split("\0") if name] - - -def git_object_exists(repo, commit, path): - return subprocess.run( - ["git", "cat-file", "-e", f"{commit}:{path}"], cwd=repo, - stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, - ).returncode == 0 - - -def history_submission_audit(repo, base): - """Track path reuse and unchanged, separate promotion commits across every parent.""" - commits = subprocess.run( - ["git", "rev-list", "--reverse", "--topo-order", f"{base}..HEAD"], - cwd=repo, check=True, capture_output=True, text=True, - ).stdout.splitlines() - additions, promoted = {}, set() - for commit in commits: - parents = subprocess.run( - ["git", "show", "-s", "--format=%P", commit], cwd=repo, - check=True, capture_output=True, text=True, - ).stdout.split() - changed = set() - for parent in parents: - names = git_names(repo, "diff-tree", "--no-commit-id", "-r", "--name-only", - "-z", "--no-renames", parent, commit) - changed.update(name for name in names if name.startswith("task-submissions/")) - if len(parents) != 1: - continue # A promotion must be its own single-parent commit. - for name in names: - parts = name.split("/") - if len(parts) != 3 or parts[0] != "tasks" or parts[2] != "task.toml": - continue - task_name = parts[1] - if not FORMAL.fullmatch(task_name): - continue - source, target = f"task-submissions/{task_name}", f"tasks/{task_name}" - if (git_object_exists(repo, parent, target) - or git_object_exists(repo, commit, source) - or not git_object_exists(repo, parent, source) - or not git_object_exists(repo, commit, target)): - continue - if any(not (p.startswith(source + "/") or p.startswith(target + "/")) for p in names): - continue - trees = subprocess.run( - ["git", "rev-parse", f"{parent}:{source}", f"{commit}:{target}"], - cwd=repo, check=True, capture_output=True, text=True, - ).stdout.splitlines() - if len(trees) == 2 and trees[0] == trees[1]: - promoted.add(task_name) - if not parents: - changed.update(git_names(repo, "diff-tree", "--root", "--no-commit-id", "-r", - "--name-only", "-z", "--no-renames", commit, "--", "task-submissions")) - for name in changed: - submission_root(name) - if (name.endswith("/task.toml") and git_object_exists(repo, commit, name) - and all(not git_object_exists(repo, parent, name) for parent in parents)): - additions[name] = additions.get(name, 0) + 1 - return promoted, sorted(name.rsplit("/", 1)[0] for name, count in additions.items() if count > 1) - - -def renamed_formal_packages(repo, base): - """Recognize existing formal packages renamed in this PR, including legacy names.""" - fields = git_names(repo, "diff", "--name-status", "-z", "--find-renames=50%", - f"{base}...HEAD", "--", "tasks") - renamed, index = set(), 0 - while index < len(fields): - status = fields[index] - index += 1 - source = fields[index] - index += 1 - if not status.startswith(("R", "C")): - continue - target = fields[index] - index += 1 - old, new = Path(source), Path(target) - if (status.startswith("R") and len(old.parts) == len(new.parts) == 3 - and old.name == new.name == "task.toml" - and git_object_exists(repo, base, source) - and not git_object_exists(repo, "HEAD", str(old.parent))): - renamed.add(new.parent.name) - return renamed - - -def check_package(repo, task, *, formal=False, require_authors=True): - errors = [] - try: - task = select_task(repo, task.relative_to(repo).as_posix()) - match = SUBMISSION.fullmatch(task.relative_to(repo).as_posix()) - if not formal and not match: - raise ValueError("Not a submission path") - config = tomllib.loads((task / "task.toml").read_text()) - expected = task.name - if not formal and (repo / "tasks" / expected).exists(): - errors.append(f"Task name collides with an existing formal package: {expected}") - if config.get("task", {}).get("name") != f"search-swe/{expected}": - errors.append(f"Expected task name search-swe/{expected}") - authors = config.get("task", {}).get("authors", []) - if require_authors and (not authors or any(not isinstance(a, dict) or not isinstance(a.get("name"), str) - or a["name"].strip().lower() in ("", "search-swe", "your name", "author") for a in authors)): - errors.append("Actual task authors are required") - shared, _ = check_release(repo, packages=[task]) - errors.extend(shared) - # Only parse source files; do not execute Python, shell, Docker or verifiers. - candidates = subprocess.run( - ["git", "ls-files", "--cached", "--others", "--exclude-standard", "-z", "--", str(task.relative_to(repo))], - cwd=repo, check=True, capture_output=True, text=True, - ).stdout.split("\0") - for name in filter(None, candidates): - path = repo / name - if not path.is_file(): - continue - if path.stat().st_size >= 100 * 1024**2: - continue # Shared checks already reject large files. - try: - text = path.read_text() - except UnicodeError: - continue - if re.search(r"\btask-[12]-(?:[1-9][0-9]*|x-[1-9][0-9]*)\b", text): - errors.append(f"{name}: obsolete numbered task reference") - for ref in re.findall(r"task-submissions/[a-z0-9-]+", text): - if formal or ref != task.relative_to(repo).as_posix(): - errors.append(f"{name}: another or unfinalized submission path: {ref}") - if path.suffix == ".py": - ast.parse(text, filename=name) - if path.suffix == ".sh": - result = subprocess.run(["bash", "--noprofile", "--norc", "-n", str(path)], - capture_output=True, text=True, env={"PATH": "/usr/bin:/bin"}) - if result.returncode: - errors.append(f"{name}: shell syntax error: {result.stderr.strip()}") - import yaml - for phase in ("environment", "tests"): - overlay = yaml.safe_load((task / phase / "docker-compose.yaml").read_text()) - if not isinstance(overlay, dict) or not isinstance(overlay.get("services"), dict) or "main" not in overlay["services"]: - errors.append(f"{phase}: Compose overlay must define services.main") - except (OSError, ValueError, KeyError, TypeError, SyntaxError, subprocess.CalledProcessError) as error: - errors.append(str(error)) - except ImportError: - errors.append("Compose syntax checks require PyYAML: python -m pip install PyYAML") - except Exception as error: - # Safe YAML parse errors should be a failed check, not an omitted layer. - errors.append(f"Static parse failed: {error}") - return errors - - -def check_submission(repo, task): - return check_package(repo, task) - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("task_path", nargs="?", help="Repository-relative submission; otherwise check all") - parser.add_argument("--merge-ready", action="store_true", help="Fail if any submission task.toml remains") - parser.add_argument("--base", help="Full PR base commit for submission history and formal-package checks") - args = parser.parse_args() - errors = [] - try: - tasks = [safe_path(REPO, args.task_path)] if args.task_path else discover(REPO) - for task in tasks: - errors.extend(f"{task.relative_to(REPO)}: {error}" for error in check_submission(REPO, task)) - remaining = list((REPO / "task-submissions").rglob("task.toml")) - if args.merge_ready and remaining: - errors.append("Not merge-ready: submission task.toml remains; promote in the same PR") - # Promotion must not remove the package's static validation coverage. - if args.merge_ready: - for config in sorted((REPO / "tasks").glob("*/task.toml")): - errors.extend(f"{config.parent.relative_to(REPO)}: {error}" for error in - check_package(REPO, config.parent, formal=True, require_authors=False)) - if args.base: - if not re.fullmatch(r"[0-9a-f]{40}", args.base): - raise ValueError("--base requires a full commit SHA") - added = subprocess.run(["git", "diff", "--no-renames", "--diff-filter=A", "--name-only", "-z", - f"{args.base}...HEAD", "--", "tasks", "task-submissions"], - cwd=REPO, check=True, capture_output=True).stdout.decode( - "utf-8", errors="surrogateescape").split("\0") - promoted, reused_tasks = history_submission_audit(REPO, args.base) - if reused_tasks: - errors.append(f"Submission paths must not be reused in one PR: {reused_tasks}") - renamed = renamed_formal_packages(REPO, args.base) - added_formal = [name for name in filter(None, added) - if name.startswith("tasks/") and name.endswith("/task.toml")] - for name in added_formal: - task = (REPO / name).parent - existing = task.name in renamed - if not existing and task.name not in promoted: - errors.append(f"{name}: new formal tasks require same-PR pure promotion from task-submissions/{task.name}") - errors.extend(check_package(REPO, task, formal=True, require_authors=not existing)) - except (OSError, ValueError, subprocess.CalledProcessError) as error: - errors.append(str(error)) - for error in errors: - print(f"ERROR: {error}") - print(f"Submission static check: {len(errors)} errors. " - + ("Merge-ready tree required." if args.merge_ready else "Review-stage only; success does NOT mean merge-ready.")) - return bool(errors) - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/check_tasks.py b/scripts/check_tasks.py new file mode 100755 index 0000000..cdc4f76 --- /dev/null +++ b/scripts/check_tasks.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +"""Static, credential-free task checks; never import or run task code.""" + +import argparse +import ast +import re +import subprocess +import sys +import tomllib +from pathlib import Path + +from check_release import check_release +from download_assets import REPO, read_manifest +from task_paths import no_symlinks, safe_path, select_task + + +def discover(repo): + root = safe_path(repo, "tasks") + no_symlinks(root) + # Include incomplete directories and nested packages so both fail validation. + return sorted(set(root.glob("*/")) | {p.parent for p in root.rglob("task.toml")}) + + +def git_names(repo, *args): + """Run a NUL-delimited Git pathname query without shell interpolation.""" + return [name for name in subprocess.run( + ["git", *args], cwd=repo, check=True, capture_output=True, + ).stdout.decode("utf-8", errors="surrogateescape").split("\0") if name] + + +def git_object_exists(repo, commit, path): + return subprocess.run( + ["git", "cat-file", "-e", f"{commit}:{path}"], cwd=repo, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ).returncode == 0 + + +def renamed_packages(repo, base): + """Recognize existing packages renamed in this PR.""" + fields = git_names(repo, "diff", "--name-status", "-z", "--find-renames=50%", + f"{base}...HEAD", "--", "tasks") + renamed, index = set(), 0 + while index < len(fields): + status = fields[index] + index += 1 + source = fields[index] + index += 1 + if not status.startswith(("R", "C")): + continue + target = fields[index] + index += 1 + old, new = Path(source), Path(target) + if (status.startswith("R") and len(old.parts) == len(new.parts) == 3 + and old.name == new.name == "task.toml" + and git_object_exists(repo, base, source) + and not git_object_exists(repo, "HEAD", str(old.parent))): + renamed.add(new.parent.name) + return renamed + + +def check_package(repo, task, *, require_authors=True, merge_ready=False): + errors = [] + try: + task = select_task(repo, task.relative_to(repo).as_posix()) + config = tomllib.loads((task / "task.toml").read_text()) + expected = task.name + if config.get("task", {}).get("name") != f"search-swe/{expected}": + errors.append(f"Expected task name search-swe/{expected}") + authors = config.get("task", {}).get("authors", []) + if require_authors and (not authors or any(not isinstance(a, dict) or not isinstance(a.get("name"), str) + or a["name"].strip().lower() in ("", "search-swe", "your name", "author") for a in authors)): + errors.append("Actual task authors are required") + shared, _ = check_release(repo, packages=[task]) + errors.extend(shared) + if merge_ready: + for entry in read_manifest(task)["files"]: + source = entry.get("source", {}) + if source.get("repo_type") == "dataset" and "local_path" not in source: + if source.get("repo_id") != "search-swe/Search-SWE": + errors.append(f"{entry['path']}: merge-ready data must use search-swe/Search-SWE") + if not source.get("filename", "").startswith(f"tasks/{task.name}/"): + errors.append(f"{entry['path']}: official data must use tasks/{task.name}/") + # Only parse source files; do not execute Python, shell, Docker or verifiers. + candidates = subprocess.run( + ["git", "ls-files", "--cached", "--others", "--exclude-standard", "-z", "--", str(task.relative_to(repo))], + cwd=repo, check=True, capture_output=True, text=True, + ).stdout.split("\0") + for name in filter(None, candidates): + path = repo / name + if not path.is_file(): + continue + if path.stat().st_size >= 100 * 1024**2: + continue # Shared checks already reject large files. + try: + text = path.read_text() + except UnicodeError: + continue + if path.suffix == ".py": + ast.parse(text, filename=name) + if path.suffix == ".sh": + result = subprocess.run(["bash", "--noprofile", "--norc", "-n", str(path)], + capture_output=True, text=True, env={"PATH": "/usr/bin:/bin"}) + if result.returncode: + errors.append(f"{name}: shell syntax error: {result.stderr.strip()}") + import yaml + for phase in ("environment", "tests"): + overlay = yaml.safe_load((task / phase / "docker-compose.yaml").read_text()) + if not isinstance(overlay, dict) or not isinstance(overlay.get("services"), dict) or "main" not in overlay["services"]: + errors.append(f"{phase}: Compose overlay must define services.main") + except (OSError, ValueError, KeyError, TypeError, SyntaxError, subprocess.CalledProcessError) as error: + errors.append(str(error)) + except ImportError: + errors.append("Compose syntax checks require PyYAML: python -m pip install PyYAML") + except Exception as error: + # Safe YAML parse errors should be a failed check, not an omitted layer. + errors.append(f"Static parse failed: {error}") + return errors + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("task_path", nargs="?", help="Repository-relative tasks/; otherwise check all") + parser.add_argument("--merge-ready", action="store_true", help="Require official dataset sources and pinned revisions") + parser.add_argument("--base", help="Full PR base commit; require actual authors for new tasks") + args = parser.parse_args() + errors = [] + try: + tasks = [safe_path(REPO, args.task_path)] if args.task_path else discover(REPO) + if not tasks: + errors.append("No task packages found") + base, renamed = None, set() + if args.base: + if not re.fullmatch(r"[0-9a-f]{40}", args.base): + raise ValueError("--base requires a full commit SHA") + base = subprocess.run(["git", "merge-base", args.base, "HEAD"], cwd=REPO, + check=True, capture_output=True, text=True).stdout.strip() + renamed = renamed_packages(REPO, base) + for task in tasks: + relative = task.relative_to(REPO).as_posix() + require_authors = (task.name not in renamed and not git_object_exists( + REPO, base or "HEAD", f"{relative}/task.toml")) + errors.extend(f"{relative}: {error}" for error in check_package( + REPO, task, require_authors=require_authors, merge_ready=args.merge_ready)) + except (OSError, ValueError, subprocess.CalledProcessError) as error: + errors.append(str(error)) + for error in errors: + print(f"ERROR: {error}") + stage = "Merge-ready" if args.merge_ready else "Review-stage" + print(f"{stage} task static check: {len(errors)} errors.") + return bool(errors) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/download_assets.py b/scripts/download_assets.py index eba54fe..b5e8975 100755 --- a/scripts/download_assets.py +++ b/scripts/download_assets.py @@ -204,10 +204,10 @@ def restore(task, task_output, entry, directory_modes, args): def main(default_kind="all"): parser = argparse.ArgumentParser(description=__doc__) selection = parser.add_mutually_exclusive_group(required=True) - selection.add_argument("--task", nargs="+", choices=("all", *TASKS), help="Formal task names, or all formal tasks") - selection.add_argument("--task-path", help="Repository-relative tasks/ or task-submissions/") + selection.add_argument("--task", nargs="+", choices=("all", *TASKS), help="Task names, or all tasks in this checkout") + selection.add_argument("--task-path", help="Repository-relative tasks/") parser.add_argument("--kind", choices=("all", "data", "models"), default=default_kind) - parser.add_argument("--output-dir", type=Path, help="Alternate parent; task-submissions/ is preserved") + parser.add_argument("--output-dir", type=Path, help="Alternate parent for directories") parser.add_argument("--cache-dir", type=Path, help="Hugging Face cache directory") parser.add_argument("--local-data-dir", type=Path, help="Restore Search-SWE dataset files from a local hf-data directory, with size/SHA-256 checks") parser.add_argument("--force", action="store_true", help="Replace files whose checksums differ") diff --git a/scripts/prepare_hf_upload.py b/scripts/prepare_hf_upload.py index af06409..ca3f68d 100644 --- a/scripts/prepare_hf_upload.py +++ b/scripts/prepare_hf_upload.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Compatibility entrypoint for the portable offline HF staging helper.""" +"""Repository entrypoint for the portable offline HF staging helper.""" try: from _maintainer_skill import helpers except ModuleNotFoundError: @@ -9,7 +9,7 @@ def main(): - return helpers.upload_main(legacy=True) + return helpers.upload_main(repository_cli=True) if __name__ == "__main__": diff --git a/scripts/promote_task.py b/scripts/promote_task.py deleted file mode 100644 index 9606cc5..0000000 --- a/scripts/promote_task.py +++ /dev/null @@ -1,16 +0,0 @@ -#!/usr/bin/env python3 -"""Compatibility entrypoint for the portable two-phase promotion helper.""" -try: - from _maintainer_skill import REPO, helpers -except ModuleNotFoundError: - from scripts._maintainer_skill import REPO, helpers - -promote = helpers.promote - - -def main(): - return helpers.promotion_main(REPO) - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/scripts/run_task.py b/scripts/run_task.py index ad4394f..015a2c8 100755 --- a/scripts/run_task.py +++ b/scripts/run_task.py @@ -105,8 +105,8 @@ def host_is_allowed(host, allowed_hosts): def main(): parser = argparse.ArgumentParser(description=__doc__) selection = parser.add_mutually_exclusive_group(required=True) - selection.add_argument("--task", choices=TASKS, help="Formal task name (for example reasoning-query-rewriting)") - selection.add_argument("--task-path", help="Repository-relative tasks/ or task-submissions/") + selection.add_argument("--task", choices=TASKS, help="Task name (for example reasoning-query-rewriting)") + selection.add_argument("--task-path", help="Repository-relative tasks/") parser.add_argument("--agent", default="codex", choices=tuple(AGENT_IMPORTS)) parser.add_argument( "--openrouter", action="store_true", diff --git a/scripts/task_paths.py b/scripts/task_paths.py index e165d93..4aaed8f 100644 --- a/scripts/task_paths.py +++ b/scripts/task_paths.py @@ -5,8 +5,7 @@ # Keep this contract in sync with the two independently installable skill helpers. TASK_NAME = r"(?!all(?:/|$))(?!task-)[a-z][a-z0-9]*(?:-[a-z0-9]+){0,4}" -FORMAL = re.compile(TASK_NAME) -SUBMISSION = re.compile(rf"task-submissions/({TASK_NAME})") +TASK_NAME_RE = re.compile(TASK_NAME) def safe_path(root, value): @@ -33,8 +32,8 @@ def no_symlinks(path): def select_task(root, value): task = safe_path(root, value) rel = task.relative_to(root.resolve()).as_posix() - if not SUBMISSION.fullmatch(rel) and not re.fullmatch(rf"tasks/({TASK_NAME})", rel): - raise ValueError("--task-path must name tasks/ or task-submissions/; " + if not re.fullmatch(rf"tasks/({TASK_NAME})", rel): + raise ValueError("--task-path must name tasks/; " "use at most five lowercase hyphen-separated words, without a task- prefix") if not (task / "task.toml").is_file(): raise ValueError(f"Missing task.toml: {rel}") @@ -43,6 +42,5 @@ def select_task(root, value): def task_key(root, task): - """Keep submission outputs separate from formal task outputs.""" - rel = task.relative_to(root.resolve()) - return Path(*rel.parts[1:]) if rel.parts[0] == "tasks" else rel + """Use the task name for local output directories.""" + return Path(task.relative_to(root.resolve()).name) diff --git a/scripts/tests/test_submission_workflow.py b/scripts/tests/test_submission_workflow.py deleted file mode 100644 index 51c066a..0000000 --- a/scripts/tests/test_submission_workflow.py +++ /dev/null @@ -1,503 +0,0 @@ -"""Offline submission, promotion/history, and additive asset publication regressions.""" - -import hashlib -import json -import os -from pathlib import Path -import shutil -import subprocess -import sys -import tempfile -from types import SimpleNamespace -import unittest -from unittest.mock import patch - -SCRIPTS = Path(__file__).resolve().parents[1] -sys.path.insert(0, str(SCRIPTS)) -from check_submission import check_submission, discover -from download_assets import destination_path, restore -from prepare_hf_upload import prepare -from promote_task import promote -from task_paths import select_task - -SCAFFOLD = SCRIPTS.parent / ".agents/skills/create-searchswe-task/scripts/scaffold_task.py" - - -class SubmissionWorkflow(unittest.TestCase): - def setUp(self): - temp = tempfile.TemporaryDirectory() - self.addCleanup(temp.cleanup) - self.root = Path(temp.name) - self.repo = self.root / "repo" - (self.repo / "tasks").mkdir(parents=True) - (self.repo / "docker").mkdir() - (self.repo / "docker/README.md").write_text("fixture") - self.git("init", "-q", "-b", "main") - self.git("config", "user.name", "Maintainer") - self.git("config", "user.email", "maintainer@example.test") - (self.repo / ".gitignore").write_text("__pycache__/\n/jobs/\n") - shutil.copytree(SCRIPTS, self.repo / "scripts", ignore=shutil.ignore_patterns("tests", "__pycache__")) - shutil.copytree(SCRIPTS.parent / ".agents/skills/maintain-searchswe-task", - self.repo / ".agents/skills/maintain-searchswe-task", - ignore=shutil.ignore_patterns("__pycache__", "test_*.py")) - self.git("add", ".") - self.git("commit", "-qm", "base") - self.base = self.git("rev-parse", "HEAD").strip() - self.source = "task-submissions/example-search" - self.task = self.repo / self.source - result = self.scaffold("Alice") - self.assertEqual(result.returncode, 0, result.stderr) - readme = self.task / "README.md" - readme.write_text(readme.read_text() + f"\nPackage: {self.source}\n") - - def git(self, *args, **kwargs): - return subprocess.run(["git", *args], cwd=self.repo, check=True, - capture_output=True, text=True, **kwargs).stdout - - def scaffold(self, name, task_name="example-search"): - return subprocess.run([sys.executable, str(SCAFFOLD), task_name, "--repo-root", str(self.repo), - "--author", f"{name} Example"], capture_output=True, text=True) - - def cli(self, script, *args): - env = {k: v for k, v in os.environ.items() if not k.startswith(("AGENT_", "CONTAINER_", "VERIFIER_"))} - return subprocess.run([sys.executable, str(self.repo / "scripts" / script), *args], - cwd=self.root, env=env, capture_output=True, text=True) - - def asset(self, task=None): - task = task or self.task - if task != self.task: - (task / "task.toml").write_text((task / "task.toml").read_text().replace("example-search", task.name)) - content = b"fixture corpus\n" - entry = {"path": "data/corpus.txt", "size_bytes": len(content), - "sha256": hashlib.sha256(content).hexdigest(), - "source": {"repo_id": "search-swe/Search-SWE", "repo_type": "dataset", "revision": "a" * 40, - "filename": f"tasks/{'example-search' if task == self.task else task.name}/corpus.txt"}} - (task / "assets.json").write_text(json.dumps({"schema_version": 1, "files": [entry]})) - data = self.root / "new-data" - path = data / entry["source"]["filename"] - path.parent.mkdir(parents=True, exist_ok=True) - path.write_bytes(content) - return entry, data - - def commit_submission(self): - self.git("checkout", "-qb", "contribution") - self.git("add", ".") - self.git("commit", "-qm", "Alice writes task", "--author", "Alice Example ") - - def test_flat_names_collisions_and_path_boundaries(self): - for name in ("hybrid-search", "vector-search"): - self.assertEqual(self.scaffold("Alice", name).returncode, 0) - self.assertTrue((self.repo / "task-submissions" / name / "task.toml").is_file()) - self.assertNotEqual(self.scaffold("Bob").returncode, 0) - self.assertEqual(self.scaffold("Bob", "other-search").returncode, 0) - current = self.cli("check_submission.py") - self.assertEqual(current.returncode, 0, current.stdout + current.stderr) - for name in ("../escape", "/absolute", "Élodie", "123", "all", "task-1-1", "one-two-three-four-five-six"): - self.assertNotEqual(self.scaffold("Alice", name).returncode, 0) - for name in ("../repo/" + self.source, "/" + self.source, self.source + "/../example-search", - "task-submissions/alice/example-search"): - with self.assertRaises(ValueError): - select_task(self.repo, name) - (self.task / "escape").symlink_to(self.root) - with self.assertRaises(ValueError): - select_task(self.repo, self.source) - - def test_symlink_roots_and_download_parent_rejected(self): - shutil.rmtree(self.repo / "task-submissions") - (self.repo / "task-submissions").symlink_to(self.root) - self.assertNotEqual(self.scaffold("Alice").returncode, 0) - with self.assertRaises(ValueError): - discover(self.repo) - link = self.root / "linked" - link.symlink_to(self.repo) - with self.assertRaises(ValueError): - destination_path(link / "output", Path("data/file")) - - def test_validator_reuses_release_checks_and_static_parsing(self): - self.assertEqual(check_submission(self.repo, self.task), []) - self.asset() - self.assertEqual(check_submission(self.repo, self.task), []) - manifest = json.loads((self.task / "assets.json").read_text()) - manifest["files"][0]["source"]["revision"] = "main" - (self.task / "assets.json").write_text(json.dumps(manifest)) - self.assertTrue(any("immutable" in e for e in check_submission(self.repo, self.task))) - (self.task / "tests/broken.py").write_text("def broken(\n") - self.assertTrue(check_submission(self.repo, self.task)) - - def test_static_validator_does_not_execute_task_code(self): - sentinel = self.root / "must-not-exist" - (self.task / "tests/untrusted.py").write_text( - f"from pathlib import Path\nPath({str(sentinel)!r}).touch()\n") - (self.task / "tests/untrusted.sh").write_text(f"#!/bin/bash\ntouch {sentinel}\n") - self.assertEqual(check_submission(self.repo, self.task), []) - self.assertFalse(sentinel.exists()) - - def test_submission_rejects_obsolete_type_and_formal_name_collision(self): - config = self.task / "task.toml" - original = config.read_text() - config.write_text(original.replace("[metadata]", '[metadata]\ntask_type = "create"')) - self.assertTrue(any("remove metadata.task_type" in error for error in check_submission(self.repo, self.task))) - config.write_text(original) - shutil.copytree(self.task, self.repo / "tasks/example-search") - self.assertTrue(any("collides" in error for error in check_submission(self.repo, self.task))) - - def test_validator_scans_payload_secret_forced_asset_author_id_and_ignore_rules(self): - entry, _ = self.asset() - local = self.task / entry["path"] - local.parent.mkdir() - local.write_text("fixture corpus\n") - self.git("add", "-f", str(local.relative_to(self.repo))) - errors = check_submission(self.repo, self.task) - self.assertTrue(any("runtime asset would enter Git" in e for e in errors)) - self.git("reset", "-q") - (self.task / ".gitignore").write_text("") - errors = check_submission(self.repo, self.task) - self.assertTrue(any("not ignored" in e for e in errors)) - (self.task / "leak.txt").write_text("hf_" + "a" * 30) - (self.task / "task.toml").write_text((self.task / "task.toml").read_text().replace("Alice Example", "Search-SWE")) - (self.task / "README.md").write_text("task-1-999\n") - errors = check_submission(self.repo, self.task) - for expected in ("credential", "authors", "obsolete numbered task reference"): - self.assertTrue(any(expected in e for e in errors), errors) - - def test_discovery_incomplete_and_review_versus_merge_ready(self): - self.assertEqual(discover(self.repo), [self.task]) - result = self.cli("check_submission.py") - self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - self.assertIn("NOT mean merge-ready", result.stdout) - self.assertNotEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) - incomplete = self.repo / "task-submissions/incomplete-search" - incomplete.mkdir(parents=True) - self.assertIn(incomplete, discover(self.repo)) - self.assertNotEqual(self.cli("check_submission.py").returncode, 0) - - def test_formal_discovery_excludes_submissions_and_explicit_launch_is_namespaced(self): - self.asset() - result = self.cli("download_assets.py", "--task", "all", "--dry-run") - self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn("Selected 0 files", result.stdout) - result = self.cli("download_assets.py", "--task-path", self.source, "--dry-run") - self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn("task-submissions/example-search/data/corpus.txt", result.stdout) - for name in ("example-search", "other-search"): - if name != "example-search": - self.scaffold("Bob", name) - result = self.cli("run_task.py", "--task-path", f"task-submissions/{name}", - "--agent", "pi", "--model", "deepseek/deepseek-flash", "--dry-run") - self.assertEqual(result.returncode, 0, result.stderr) - self.assertIn(f"jobs/task-submissions/{name}", result.stdout) - self.assertNotEqual(self.cli("run_task.py", "--task", "example-search", "--dry-run").returncode, 0) - - def test_submission_launch_reaches_mock_harbor_without_formal_registration(self): - binary = self.root / "bin" - binary.mkdir() - harbor = binary / "harbor" - harbor.write_text(f"#!{sys.executable}\nimport json, sys\nprint(json.dumps(sys.argv[1:]))\n") - harbor.chmod(0o755) - with patch.dict(os.environ, {"PATH": str(binary) + os.pathsep + os.environ.get("PATH", ""), - "DEEPSEEK_API_KEY": "fixture-only"}): - result = self.cli("run_task.py", "--task-path", self.source, "--agent", "pi", - "--model", "deepseek/deepseek-flash") - self.assertEqual(result.returncode, 0, result.stderr) - command = json.loads(result.stdout.splitlines()[-1]) - self.assertEqual(command[command.index("--path") + 1], str(self.task)) - self.assertEqual(command[command.index("-o") + 1], str(self.repo / "jobs" / self.source)) - self.assertNotIn("fixture-only", result.stdout) - - def test_offline_download_into_submission_and_alternate_output(self): - entry, data = self.asset() - result = self.cli("download_assets.py", "--task-path", self.source, "--local-data-dir", str(data)) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertTrue((self.task / entry["path"]).is_file()) - output = self.root / "alternate" - result = self.cli("download_assets.py", "--task-path", self.source, "--local-data-dir", str(data), - "--output-dir", str(output)) - self.assertEqual(result.returncode, 0, result.stderr) - self.assertTrue((output / self.source / entry["path"]).is_file()) - self.assertEqual(self.cli("download_assets.py", "--task-path", self.source, "--verify-only").returncode, 0) - - def test_personal_pinned_dataset_download_is_mocked_and_verified(self): - entry, data = self.asset() - source_file = data / entry["source"]["filename"] - entry["source"]["repo_id"] = "alice/development" - args = SimpleNamespace(cache_dir=None, local_files_only=True, force=False, verify_only=False) - with patch("huggingface_hub.hf_hub_download", return_value=str(source_file)) as download: - self.assertEqual(restore(self.task, self.task, entry, {}, args), "downloaded") - self.assertEqual(download.call_args.kwargs["repo_id"], "alice/development") - self.assertEqual(download.call_args.kwargs["revision"], "a" * 40) - self.assertTrue(download.call_args.kwargs["local_files_only"]) - (self.task / entry["path"]).unlink() - entry["source"]["revision"] = "main" - with patch("huggingface_hub.hf_hub_download") as download: - with self.assertRaises(ValueError): - restore(self.task, self.task, entry, {}, args) - download.assert_not_called() - - def test_two_phase_promotion_preserves_follow_blame_and_merge_history(self): - self.commit_submission() - promote(self.repo, self.source, "example-search", "rename", dry_run=True) - self.assertTrue(self.task.exists()) - self.assertFalse(self.git("status", "--porcelain")) - promote(self.repo, self.source, "example-search", "rename") - with self.assertRaises(ValueError): - promote(self.repo, self.source, "example-search", "finalize") - self.git("commit", "-qm", "Pure rename by maintainer") - target = self.repo / "tasks/example-search" - promote(self.repo, self.source, "example-search", "finalize", dry_run=True) - self.assertIn("example-search", (target / "task.toml").read_text()) - promote(self.repo, self.source, "example-search", "finalize") - self.git("add", ".") - self.git("commit", "-qm", "Finalize by maintainer", "--allow-empty") - self.git("checkout", "-q", "main") - self.git("merge", "--no-ff", "-qm", "Merge task PR", "contribution") - self.assertEqual(len(self.git("rev-list", "--parents", "-n", "1", "HEAD").split()), 3) - history = self.git("log", "--follow", "--format=%an %s", "--", "tasks/example-search/instruction.md") - self.assertIn("Alice Example Alice writes task", history) - blame = self.git("blame", "--line-porcelain", "tasks/example-search/instruction.md") - self.assertIn("author Alice Example", blame) - self.assertNotIn("author Maintainer", blame) - self.assertFalse(list((self.repo / "task-submissions").rglob("task.toml"))) - self.assertEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) - self.assertEqual(self.cli("check_release.py").returncode, 0) - - def test_promotion_refuses_collision_rename_traversal_and_nonpure_commit(self): - self.commit_submission() - for source, target in ((self.source, "vector-search"), (self.source, "../example-search"), - ("../" + self.source, "example-search")): - with self.assertRaises(ValueError): - promote(self.repo, source, target, "rename") - collision = self.repo / "tasks/example-search" - collision.mkdir() - with self.assertRaises(ValueError): - promote(self.repo, self.source, "example-search", "rename") - collision.rmdir() - promote(self.repo, self.source, "example-search", "rename") - (self.repo / "docker/README.md").write_text("unrelated edit") - self.git("add", ".") - self.git("commit", "-qm", "Not a pure rename") - with self.assertRaises(ValueError): - promote(self.repo, self.source, "example-search", "finalize") - - def test_merge_ready_rechecks_promoted_syntax_paths_and_new_authors(self): - self.commit_submission() - promote(self.repo, self.source, "example-search", "rename") - self.git("commit", "-qm", "Pure rename") - promote(self.repo, self.source, "example-search", "finalize") - target = self.repo / "tasks/example-search" - self.git("add", ".") - self.git("commit", "-qm", "Finalize", "--allow-empty") - self.assertEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) - readme = target / "README.md" - original = readme.read_text() - readme.write_text(original + "\ntask-submissions/example-search\n") - self.assertNotEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) - readme.write_text(original) - broken = target / "tests/broken.py" - broken.write_text("def broken(\n") - self.assertNotEqual(self.cli("check_submission.py", "--merge-ready").returncode, 0) - broken.unlink() - config = target / "task.toml" - config.write_text(config.read_text().replace("Alice Example", "Search-SWE")) - self.assertNotEqual(self.cli("check_submission.py", "--base", self.base).returncode, 0) - - def test_base_allows_multiple_tasks_and_authors(self): - self.scaffold("Alice", "hybrid-search") - self.scaffold("Bob", "vector-search") - self.git("add", ".") - self.git("commit", "-qm", "Multiple submissions") - result = self.cli("check_submission.py", "--base", self.base) - self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - - def test_base_rejects_direct_formal_task_without_submission_history(self): - shutil.rmtree(self.repo / "task-submissions") - result = subprocess.run( - [sys.executable, str(SCAFFOLD), "example-search", "--repo-root", str(self.repo), - "--author", "Alice Example", "--formal"], capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 0, result.stderr) - self.git("add", ".") - self.git("commit", "-qm", "Bypass submission buffer") - result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("require same-PR pure promotion", result.stdout) - - def test_base_rejects_submission_path_reuse(self): - self.git("add", ".") - self.git("commit", "-qm", "First submission") - self.git("rm", "-r", self.source) - self.git("commit", "-qm", "Remove submission") - self.assertEqual(self.scaffold("Bob").returncode, 0) - self.git("add", ".") - self.git("commit", "-qm", "Reuse submission path") - result = self.cli("check_submission.py", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("Submission paths must not be reused", result.stdout) - - def test_multiple_tasks_promote_independently_with_multiple_authors(self): - self.scaffold("Alice", "vector-search") - self.git("checkout", "-qb", "multi-promotion") - self.git("add", ".") - self.git("commit", "-qm", "Alice submits two tasks", "--author", "Alice Example ") - for source, target_id in ((self.source, "example-search"), - ("task-submissions/vector-search", "vector-search")): - promote(self.repo, source, target_id, "rename") - self.git("commit", "-qm", f"Pure rename {target_id}") - promote(self.repo, source, target_id, "finalize") - self.git("add", ".") - self.git("commit", "-qm", f"Finalize {target_id}", "--allow-empty") - self.assertEqual(self.cli("check_submission.py", "--merge-ready", "--base", self.base).returncode, 0) - self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", - "tasks/example-search/instruction.md")) - self.assertIn("Alice Example", self.git("log", "--follow", "--format=%an", "--", - "tasks/vector-search/instruction.md")) - - self.scaffold("Bob", "other-search") - self.git("add", ".") - self.git("commit", "-qm", "Bob submission", "--author", "Bob Example ") - bob_source = "task-submissions/other-search" - promote(self.repo, bob_source, "other-search", "rename") - self.git("commit", "-qm", "Pure rename other-search") - promote(self.repo, bob_source, "other-search", "finalize") - self.git("add", ".") - self.git("commit", "-qm", "Finalize other-search", "--allow-empty") - result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) - self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - - def test_base_history_checks_every_side_of_fully_promoted_merge(self): - self.git("checkout", "-qb", "alice-history") - self.git("add", ".") - self.git("commit", "-qm", "Alice submission") - promote(self.repo, self.source, "example-search", "rename") - self.git("commit", "-qm", "Pure rename Alice") - promote(self.repo, self.source, "example-search", "finalize") - self.git("add", ".") - self.git("commit", "-qm", "Finalize Alice", "--allow-empty") - - self.git("checkout", "-qb", "bob-history", self.base) - (self.repo / "tasks").mkdir(exist_ok=True) - result = self.scaffold("Bob", "other-search") - self.assertEqual(result.returncode, 0, result.stderr) - self.git("add", ".") - self.git("commit", "-qm", "Bob submission") - bob_source = "task-submissions/other-search" - promote(self.repo, bob_source, "other-search", "rename") - self.git("commit", "-qm", "Pure rename Bob") - promote(self.repo, bob_source, "other-search", "finalize") - self.git("add", ".") - self.git("commit", "-qm", "Finalize Bob", "--allow-empty") - - self.git("checkout", "alice-history") - self.git("merge", "--no-ff", "bob-history", "-m", "Merge Bob history") - result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) - self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - - def test_base_history_rejects_unsafe_submission_paths(self): - self.git("checkout", "-qb", "unsafe-history") - unsafe = self.repo / "task-submissions/alice/nested-search/task.toml" - unsafe.parent.mkdir(parents=True) - unsafe.write_text("unsafe historical path") - self.git("add", ".") - self.git("commit", "-qm", "Malformed submission path") - result = self.cli("check_submission.py", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("Malformed task submission path", result.stdout) - - def test_base_recognizes_existing_formal_rename_without_changing_legacy_authors(self): - # A repository-wide name migration is an edit to an existing package. - legacy = self.repo / "tasks/task-1-1" - self.task.rename(legacy) - readme = legacy / "README.md" - readme.write_text("Existing formal task\n") - config = legacy / "task.toml" - config.write_text(config.read_text().replace("example-search", "task-1-1") - .replace("Alice Example", "Search-SWE")) - self.git("add", ".") - self.git("commit", "-qm", "Existing legacy formal task") - base = self.git("rev-parse", "HEAD").strip() - self.git("mv", "tasks/task-1-1", "tasks/example-search") - config = self.repo / "tasks/example-search/task.toml" - config.write_text(config.read_text().replace("task-1-1", "example-search")) - self.git("add", ".") - self.git("commit", "-qm", "Rename existing formal task") - result = self.cli("check_submission.py", "--merge-ready", "--base", base) - self.assertEqual(result.returncode, 0, result.stdout + result.stderr) - - def test_history_rejects_deleted_nested_submission(self): - unsafe = self.repo / "task-submissions/alice/nested-search/task.toml" - unsafe.parent.mkdir(parents=True) - unsafe.write_text("invalid old nested layout") - self.git("add", ".") - self.git("commit", "-qm", "Nested submission") - shutil.rmtree(unsafe.parent.parent) - self.git("add", "-u") - self.git("commit", "-qm", "Remove nested submission") - result = self.cli("check_submission.py", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("Malformed task submission path", result.stdout) - - def test_history_rejects_renamed_or_nonpure_submission_promotion(self): - self.commit_submission() - self.git("mv", self.source, "tasks/other-search") - target = self.repo / "tasks/other-search" - for name in ("README.md", "task.toml", "instruction.md"): - path = target / name - path.write_text(path.read_text().replace(self.source, "tasks/other-search") - .replace("example-search", "other-search")) - self.git("add", ".") - self.git("commit", "-qm", "Rename and modify submission") - result = self.cli("check_submission.py", "--merge-ready", "--base", self.base) - self.assertNotEqual(result.returncode, 0) - self.assertIn("require same-PR pure promotion", result.stdout) - - def test_official_staging_requires_promotion_even_when_name_is_final(self): - _, data = self.asset() - snapshot = self.root / "official.json" - snapshot.write_text(json.dumps({"schema_version": 1, "files": []})) - output = self.root / "premature-upload" - with self.assertRaisesRegex(ValueError, "Promote to tasks/"): - prepare(snapshot, self.task, data, output) - self.assertFalse(output.exists()) - - def test_incremental_manifest_preserves_old_entries_without_old_data(self): - target = self.repo / "tasks/example-search" - shutil.copytree(self.task, target) - entry, data = self.asset(target) - old = {"path": "tasks/reasoning-query-rewriting/old.bin", "size_bytes": 500, "sha256": "b" * 64, "license": "fixture"} - snapshot = self.root / "official.json" - snapshot.write_text(json.dumps({"schema_version": 1, "files": [old], "extra": "preserve"})) - output = self.root / "upload" - self.assertEqual(prepare(snapshot, target, data, output), 1) - merged = json.loads((output / "manifest.json").read_text()) - self.assertEqual(merged["files"][0], old) - self.assertEqual(merged["extra"], "preserve") - self.assertFalse((output / old["path"]).exists()) - self.assertTrue((output / entry["source"]["filename"]).is_file()) - with self.assertRaises(ValueError): - prepare(snapshot, target, data, output) - # Existing official paths are never replaced, even if bytes match. - snapshot.write_text(json.dumps(merged)) - with self.assertRaises(ValueError): - prepare(snapshot, target, data, self.root / "collision") - snapshot.write_text(json.dumps({"schema_version": 1, "files": [old]})) - (data / entry["source"]["filename"]).write_text("corrupt") - with self.assertRaises(ValueError): - prepare(snapshot, target, data, self.root / "corrupt") - self.assertFalse((self.root / "corrupt").exists()) - - def test_incremental_rejects_traversal_and_symlinks(self): - target = self.repo / "tasks/example-search" - shutil.copytree(self.task, target) - entry, data = self.asset(target) - snapshot = self.root / "official.json" - snapshot.write_text(json.dumps({"schema_version": 1, "files": [ - {"path": "../escape", "size_bytes": 1, "sha256": "a" * 64}]})) - with self.assertRaises(ValueError): - prepare(snapshot, target, data, self.root / "upload") - snapshot.write_text(json.dumps({"schema_version": 1, "files": []})) - (data / "link").symlink_to(self.root) - with self.assertRaises(ValueError): - prepare(snapshot, target, data, self.root / "upload") - - -if __name__ == "__main__": - unittest.main() diff --git a/scripts/tests/test_task_workflow.py b/scripts/tests/test_task_workflow.py new file mode 100644 index 0000000..d247628 --- /dev/null +++ b/scripts/tests/test_task_workflow.py @@ -0,0 +1,354 @@ +"""Offline task contribution, validation, and additive asset publication regressions.""" + +import hashlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import patch + +SCRIPTS = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(SCRIPTS)) +from check_tasks import check_package, discover +from download_assets import destination_path, restore +from prepare_hf_upload import prepare +from task_paths import select_task + +SCAFFOLD = SCRIPTS.parent / ".agents/skills/create-searchswe-task/scripts/scaffold_task.py" + + +class TaskWorkflow(unittest.TestCase): + def setUp(self): + temp = tempfile.TemporaryDirectory() + self.addCleanup(temp.cleanup) + self.root = Path(temp.name) + self.repo = self.root / "repo" + (self.repo / "tasks").mkdir(parents=True) + (self.repo / "docker").mkdir() + (self.repo / "docker/README.md").write_text("fixture") + self.git("init", "-q", "-b", "main") + self.git("config", "user.name", "Maintainer") + self.git("config", "user.email", "maintainer@example.test") + (self.repo / ".gitignore").write_text("__pycache__/\n/jobs/\n") + shutil.copytree(SCRIPTS, self.repo / "scripts", ignore=shutil.ignore_patterns("tests", "__pycache__")) + shutil.copytree(SCRIPTS.parent / ".agents/skills/maintain-searchswe-task", + self.repo / ".agents/skills/maintain-searchswe-task", + ignore=shutil.ignore_patterns("__pycache__", "test_*.py")) + self.git("add", ".") + self.git("commit", "-qm", "base") + self.base = self.git("rev-parse", "HEAD").strip() + self.source = "tasks/example-search" + self.task = self.repo / self.source + result = self.scaffold("Alice") + self.assertEqual(result.returncode, 0, result.stderr) + readme = self.task / "README.md" + readme.write_text(readme.read_text() + f"\nPackage: {self.source}\n") + + def git(self, *args, **kwargs): + return subprocess.run(["git", *args], cwd=self.repo, check=True, + capture_output=True, text=True, **kwargs).stdout + + def scaffold(self, name, task_name="example-search"): + return subprocess.run([sys.executable, str(SCAFFOLD), task_name, "--repo-root", str(self.repo), + "--author", f"{name} Example"], capture_output=True, text=True) + + def cli(self, script, *args): + env = {k: v for k, v in os.environ.items() if not k.startswith(("AGENT_", "CONTAINER_", "VERIFIER_"))} + return subprocess.run([sys.executable, str(self.repo / "scripts" / script), *args], + cwd=self.root, env=env, capture_output=True, text=True) + + def asset(self, task=None): + task = task or self.task + if task != self.task: + (task / "task.toml").write_text((task / "task.toml").read_text().replace("example-search", task.name)) + content = b"fixture corpus\n" + entry = {"path": "data/corpus.txt", "size_bytes": len(content), + "sha256": hashlib.sha256(content).hexdigest(), + "source": {"repo_id": "search-swe/Search-SWE", "repo_type": "dataset", "revision": "a" * 40, + "filename": f"tasks/{'example-search' if task == self.task else task.name}/corpus.txt"}} + (task / "assets.json").write_text(json.dumps({"schema_version": 1, "files": [entry]})) + data = self.root / "new-data" + path = data / entry["source"]["filename"] + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content) + return entry, data + + def commit_task(self): + self.git("checkout", "-qb", "contribution") + self.git("add", ".") + self.git("commit", "-qm", "Alice writes task", "--author", "Alice Example ") + + def test_flat_names_collisions_and_path_boundaries(self): + for name in ("hybrid-search", "vector-search"): + self.assertEqual(self.scaffold("Alice", name).returncode, 0) + self.assertTrue((self.repo / "tasks" / name / "task.toml").is_file()) + self.assertNotEqual(self.scaffold("Bob").returncode, 0) + self.assertEqual(self.scaffold("Bob", "other-search").returncode, 0) + current = self.cli("check_tasks.py") + self.assertEqual(current.returncode, 0, current.stdout + current.stderr) + for name in ("../escape", "/absolute", "Élodie", "123", "all", "task-1-1", "one-two-three-four-five-six"): + self.assertNotEqual(self.scaffold("Alice", name).returncode, 0) + for name in ("../repo/" + self.source, "/" + self.source, self.source + "/../example-search", + "tasks/alice/example-search"): + with self.assertRaises(ValueError): + select_task(self.repo, name) + (self.task / "escape").symlink_to(self.root) + with self.assertRaises(ValueError): + select_task(self.repo, self.source) + + def test_symlink_roots_and_download_parent_rejected(self): + shutil.rmtree(self.repo / "tasks") + (self.repo / "tasks").symlink_to(self.root) + self.assertNotEqual(self.scaffold("Alice").returncode, 0) + with self.assertRaises(ValueError): + discover(self.repo) + link = self.root / "linked" + link.symlink_to(self.repo) + with self.assertRaises(ValueError): + destination_path(link / "output", Path("data/file")) + + def test_validator_reuses_release_checks_and_static_parsing(self): + self.assertEqual(check_package(self.repo, self.task), []) + self.asset() + self.assertEqual(check_package(self.repo, self.task), []) + manifest = json.loads((self.task / "assets.json").read_text()) + manifest["files"][0]["source"]["revision"] = "main" + (self.task / "assets.json").write_text(json.dumps(manifest)) + self.assertTrue(any("immutable" in e for e in check_package(self.repo, self.task))) + (self.task / "tests/broken.py").write_text("def broken(\n") + self.assertTrue(check_package(self.repo, self.task)) + + def test_static_validator_does_not_execute_task_code(self): + sentinel = self.root / "must-not-exist" + (self.task / "tests/untrusted.py").write_text( + f"from pathlib import Path\nPath({str(sentinel)!r}).touch()\n") + (self.task / "tests/untrusted.sh").write_text(f"#!/bin/bash\ntouch {sentinel}\n") + self.assertEqual(check_package(self.repo, self.task), []) + self.assertFalse(sentinel.exists()) + + def test_task_rejects_type_metadata_and_name_mismatch(self): + config = self.task / "task.toml" + original = config.read_text() + config.write_text(original.replace("[metadata]", '[metadata]\ntask_type = "create"')) + self.assertTrue(any("unsupported metadata field: task_type" in error for error in check_package(self.repo, self.task))) + config.write_text(original) + config.write_text(original.replace("search-swe/example-search", "search-swe/different-search")) + self.assertTrue(any("Expected task name" in error for error in check_package(self.repo, self.task))) + + def test_validator_scans_payload_secret_forced_asset_author_id_and_ignore_rules(self): + entry, _ = self.asset() + local = self.task / entry["path"] + local.parent.mkdir() + local.write_text("fixture corpus\n") + self.git("add", "-f", str(local.relative_to(self.repo))) + errors = check_package(self.repo, self.task) + self.assertTrue(any("runtime asset would enter Git" in e for e in errors)) + self.git("reset", "-q") + (self.task / ".gitignore").write_text("") + errors = check_package(self.repo, self.task) + self.assertTrue(any("not ignored" in e for e in errors)) + (self.task / "leak.txt").write_text("hf_" + "a" * 30) + (self.task / "task.toml").write_text((self.task / "task.toml").read_text().replace("Alice Example", "Search-SWE")) + errors = check_package(self.repo, self.task) + for expected in ("credential", "authors"): + self.assertTrue(any(expected in e for e in errors), errors) + + def test_discovery_incomplete_and_review_versus_merge_ready(self): + self.assertEqual(discover(self.repo), [self.task]) + result = self.cli("check_tasks.py") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("Review-stage task static check", result.stdout) + self.assertEqual(self.cli("check_tasks.py", "--merge-ready").returncode, 0) + incomplete = self.repo / "tasks/incomplete-search" + incomplete.mkdir(parents=True) + self.assertIn(incomplete, discover(self.repo)) + self.assertNotEqual(self.cli("check_tasks.py").returncode, 0) + + def test_discovery_downloads_and_launches_tasks_by_name(self): + self.asset() + result = self.cli("download_assets.py", "--task", "all", "--dry-run") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("Selected 1 files", result.stdout) + result = self.cli("download_assets.py", "--task-path", self.source, "--dry-run") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("example-search/data/corpus.txt", result.stdout) + for name in ("example-search", "other-search"): + if name != "example-search": + self.scaffold("Bob", name) + result = self.cli("run_task.py", "--task", name, + "--agent", "pi", "--model", "deepseek/deepseek-flash", "--dry-run") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn(f"jobs/{name}", result.stdout) + + def test_task_launch_reaches_mock_harbor(self): + binary = self.root / "bin" + binary.mkdir() + harbor = binary / "harbor" + harbor.write_text(f"#!{sys.executable}\nimport json, sys\nprint(json.dumps(sys.argv[1:]))\n") + harbor.chmod(0o755) + with patch.dict(os.environ, {"PATH": str(binary) + os.pathsep + os.environ.get("PATH", ""), + "DEEPSEEK_API_KEY": "fixture-only"}): + result = self.cli("run_task.py", "--task-path", self.source, "--agent", "pi", + "--model", "deepseek/deepseek-flash") + self.assertEqual(result.returncode, 0, result.stderr) + command = json.loads(result.stdout.splitlines()[-1]) + self.assertEqual(command[command.index("--path") + 1], str(self.task)) + self.assertEqual(command[command.index("-o") + 1], str(self.repo / "jobs" / self.task.name)) + self.assertNotIn("fixture-only", result.stdout) + + def test_offline_download_into_task_and_alternate_output(self): + entry, data = self.asset() + result = self.cli("download_assets.py", "--task-path", self.source, "--local-data-dir", str(data)) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue((self.task / entry["path"]).is_file()) + output = self.root / "alternate" + result = self.cli("download_assets.py", "--task-path", self.source, "--local-data-dir", str(data), + "--output-dir", str(output)) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue((output / self.task.name / entry["path"]).is_file()) + self.assertEqual(self.cli("download_assets.py", "--task-path", self.source, "--verify-only").returncode, 0) + + def test_personal_pinned_dataset_download_is_mocked_and_verified(self): + entry, data = self.asset() + source_file = data / entry["source"]["filename"] + entry["source"]["repo_id"] = "alice/development" + args = SimpleNamespace(cache_dir=None, local_files_only=True, force=False, verify_only=False) + with patch("huggingface_hub.hf_hub_download", return_value=str(source_file)) as download: + self.assertEqual(restore(self.task, self.task, entry, {}, args), "downloaded") + self.assertEqual(download.call_args.kwargs["repo_id"], "alice/development") + self.assertEqual(download.call_args.kwargs["revision"], "a" * 40) + self.assertTrue(download.call_args.kwargs["local_files_only"]) + (self.task / entry["path"]).unlink() + entry["source"]["revision"] = "main" + with patch("huggingface_hub.hf_hub_download") as download: + with self.assertRaises(ValueError): + restore(self.task, self.task, entry, {}, args) + download.assert_not_called() + + def test_direct_contribution_preserves_authors_and_merge_history(self): + self.asset() + self.commit_task() + for name, author in (("hybrid-search", "Alice"), ("vector-search", "Bob")): + self.assertEqual(self.scaffold(author, name).returncode, 0) + self.asset(self.repo / "tasks" / name) + self.git("add", ".") + self.git("commit", "-qm", f"Add {name}", "--author", f"{author} Example <{author.lower()}@example.test>") + result = self.cli("check_tasks.py", "--merge-ready", "--base", self.base) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.git("checkout", "-q", "main") + self.git("merge", "--no-ff", "-qm", "Merge task PR", "contribution") + self.assertEqual(len(self.git("rev-list", "--parents", "-n", "1", "HEAD").split()), 3) + for name, author in (("example-search", "Alice"), ("vector-search", "Bob")): + self.assertIn(f"author {author} Example", self.git("blame", "--line-porcelain", f"tasks/{name}/instruction.md")) + self.assertEqual(self.cli("check_release.py").returncode, 0) + + def test_merge_ready_checks_official_sources_namespace_and_revision(self): + entry, _ = self.asset() + for field, value, message in (("repo_id", "alice/development", "search-swe/Search-SWE"), + ("filename", "tasks/other-search/corpus.txt", "tasks/example-search/"), + ("revision", "main", "immutable")): + with self.subTest(field=field): + manifest = {"schema_version": 1, "files": [entry]} + original = entry["source"][field] + entry["source"][field] = value + (self.task / "assets.json").write_text(json.dumps(manifest)) + if field == "repo_id": + result = self.cli("check_tasks.py") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + result = self.cli("check_tasks.py", "--merge-ready") + self.assertNotEqual(result.returncode, 0) + self.assertIn(message, result.stdout) + entry["source"][field] = original + (self.task / "assets.json").write_text(json.dumps({"schema_version": 1, "files": [entry]})) + self.assertEqual(self.cli("check_tasks.py", "--merge-ready").returncode, 0) + + def test_all_tasks_receive_syntax_checks_and_new_tasks_require_authors(self): + config = self.task / "task.toml" + original = config.read_text() + config.write_text(original.replace("Alice Example", "Search-SWE")) + for args in ((), (self.source,)): + self.assertNotEqual(self.cli("check_tasks.py", *args).returncode, 0) + config.write_text(original) + self.commit_task() + broken = self.task / "tests/broken.py" + broken.write_text("def broken(\n") + for args in ((), ("--merge-ready",)): + self.assertNotEqual(self.cli("check_tasks.py", *args).returncode, 0) + broken.unlink() + config = self.task / "task.toml" + config.write_text(config.read_text().replace("Alice Example", "Search-SWE")) + self.assertNotEqual(self.cli("check_tasks.py", "--base", self.base).returncode, 0) + self.assertNotEqual(self.cli("check_tasks.py", "--base", "main").returncode, 0) + + def test_existing_task_authors_survive_edits_and_renames(self): + config = self.task / "task.toml" + config.write_text(config.read_text().replace("Alice Example", "Search-SWE")) + self.commit_task() + base = self.git("rev-parse", "HEAD").strip() + self.assertEqual(self.cli("check_tasks.py", self.source).returncode, 0) + (self.task / "README.md").write_text("Updated task documentation\n") + self.git("add", ".") + self.git("commit", "-qm", "Document task") + self.assertEqual(self.cli("check_tasks.py", "--base", base).returncode, 0) + self.git("mv", self.source, "tasks/renamed-search") + config = self.repo / "tasks/renamed-search/task.toml" + config.write_text(config.read_text().replace("example-search", "renamed-search")) + self.git("add", ".") + self.git("commit", "-qm", "Rename task") + result = self.cli("check_tasks.py", "--merge-ready", "--base", base) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_nested_task_directories_fail_validation(self): + nested = self.repo / "tasks/alice/nested-search" + shutil.copytree(self.task, nested) + self.assertIn(nested, discover(self.repo)) + result = self.cli("check_tasks.py") + self.assertNotEqual(result.returncode, 0) + self.assertIn("tasks/", result.stdout) + + def test_incremental_manifest_preserves_old_entries_without_old_data(self): + target = self.repo / "tasks/example-search" + entry, data = self.asset(target) + old = {"path": "tasks/reasoning-query-rewriting/old.bin", "size_bytes": 500, "sha256": "b" * 64, "license": "fixture"} + snapshot = self.root / "official.json" + snapshot.write_text(json.dumps({"schema_version": 1, "files": [old], "extra": "preserve"})) + output = self.root / "upload" + self.assertEqual(prepare(snapshot, target, data, output), 1) + merged = json.loads((output / "manifest.json").read_text()) + self.assertEqual(merged["files"][0], old) + self.assertEqual(merged["extra"], "preserve") + self.assertFalse((output / old["path"]).exists()) + self.assertTrue((output / entry["source"]["filename"]).is_file()) + with self.assertRaises(ValueError): + prepare(snapshot, target, data, output) + # Existing official paths are never replaced, even if bytes match. + snapshot.write_text(json.dumps(merged)) + with self.assertRaises(ValueError): + prepare(snapshot, target, data, self.root / "collision") + snapshot.write_text(json.dumps({"schema_version": 1, "files": [old]})) + (data / entry["source"]["filename"]).write_text("corrupt") + with self.assertRaises(ValueError): + prepare(snapshot, target, data, self.root / "corrupt") + self.assertFalse((self.root / "corrupt").exists()) + + def test_incremental_rejects_traversal_and_symlinks(self): + target = self.repo / "tasks/example-search" + entry, data = self.asset(target) + snapshot = self.root / "official.json" + snapshot.write_text(json.dumps({"schema_version": 1, "files": [ + {"path": "../escape", "size_bytes": 1, "sha256": "a" * 64}]})) + with self.assertRaises(ValueError): + prepare(snapshot, target, data, self.root / "upload") + snapshot.write_text(json.dumps({"schema_version": 1, "files": []})) + (data / "link").symlink_to(self.root) + with self.assertRaises(ValueError): + prepare(snapshot, target, data, self.root / "upload") + + +if __name__ == "__main__": + unittest.main() diff --git a/task-submissions/README.md b/task-submissions/README.md deleted file mode 100644 index a142a96..0000000 --- a/task-submissions/README.md +++ /dev/null @@ -1,34 +0,0 @@ -# Task submissions - -Create new tasks at `task-submissions/`, with `[task].name` set to -`search-swe/`. Use at most five lowercase, hyphen-separated words, -starting with a letter, such as `example-search`. The `task-` prefix and `all` -are reserved. Names must be unique across submissions and -formal tasks; check active PRs for conflicting names before handoff. - -One PR may add multiple tasks. Use actual `task.toml` authors and correct Git -author metadata. - -```bash -python .agents/skills/create-searchswe-task/scripts/scaffold_task.py example-search \ - --author 'Alice Example' --hardware cpu -python scripts/check_submission.py task-submissions/example-search -python scripts/download_assets.py --task-path task-submissions/example-search --dry-run -python scripts/run_task.py --task-path task-submissions/example-search \ - --agent pi --model deepseek/deepseek-flash --dry-run -``` - -A scaffold intentionally fails verification until implemented. Automatic task -discovery only includes formal `tasks/*/task.toml` packages; submissions need -explicit `--task-path`. See the [complete contribution guide](../docs/contributing.md). - -Before promotion, run a configured coding-model trial alongside grader checks. -Use the trajectory and scores to refine the task setting where needed, revalidate -substantive changes, and report the final tested revision and any missing evidence. -The authoring skill explains how to request local configuration of missing API keys. - -Promote every package to `tasks/` in the same PR, preserving its -name. Each pure move is committed separately from finalization; preserve the -contributor's commits with a merge commit. Do not reuse a submission path in -the same PR after removing or promoting it. No submission package enters main; -this README stays as the workflow entry point. From 05a88a10010b492739678dfcb7681546ff17fd6c Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Sat, 10 Oct 2026 21:16:49 +0800 Subject: [PATCH 2/3] Improve readability of task instructions --- tasks/agentic-search/instruction.md | 33 ++++++--- .../code-embedding-finetuning/instruction.md | 6 +- tasks/long-document-reranking/instruction.md | 32 +++++++-- tasks/long-pdf-evidence/instruction.md | 67 +++++++++++++++++-- .../instruction.md | 36 ++++++++-- tasks/query-encoder-alignment/instruction.md | 14 +++- .../reasoning-query-rewriting/instruction.md | 30 +++++++-- tasks/scientific-paper-qa/instruction.md | 29 ++++++-- tasks/sparse-retrieval-pruning/instruction.md | 27 ++++++-- 9 files changed, 223 insertions(+), 51 deletions(-) diff --git a/tasks/agentic-search/instruction.md b/tasks/agentic-search/instruction.md index c0a81cf..48276c4 100644 --- a/tasks/agentic-search/instruction.md +++ b/tasks/agentic-search/instruction.md @@ -1,19 +1,36 @@ Some of the questions I need to research combine several conditions, and I want a small set of documents that I can read together to address the full question. Please improve the supplied ReAct retrieval starter so that each such question produces exactly five distinct documents from the corpus that best cover its combined information need. The goal is to maximize Gold Recall@5 on held-out questions while preserving the interfaces and resource constraints. -There is already a retrieval-and-selection workflow to build on under /app/starter, with executable /app/build.sh and /app/run.sh entry points. Read /app/starter/README.md to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If OPENROUTER_API_KEY is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective. +There is already a retrieval-and-selection workflow to build on under `/app/starter`, with executable `/app/build.sh` and `/app/run.sh` entry points. Read `/app/starter/README.md` to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If `OPENROUTER_API_KEY` is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective. -Please read /task/docs/environment.md for the installed Python environment, runtime, packages and tools, and /task/docs/available_resources.md for optional retrieval and generative APIs, runtime environment-variable and credential handling, exact model allowlists, provider and endpoint restrictions, usage limits and jailbreak penalties. Follow the documented restrictions and do not assume an unlisted provider, model or endpoint is authorized. Harbor injects the documented credentials at runtime; do not put them in the task package or Docker image. These documents and the rest of /task are read-only. Only create or modify submission files under /app, and fit implementation, validation and debugging into the agent's 120-minute allowance. +Please read `/task/docs/environment.md` for the installed Python environment, runtime, packages and tools, and `/task/docs/available_resources.md` for optional retrieval and generative APIs, runtime environment-variable and credential handling, exact model allowlists, provider and endpoint restrictions, usage limits and jailbreak penalties. Follow the documented restrictions and do not assume an unlisted provider, model or endpoint is authorized. Harbor injects the documented credentials at runtime; do not put them in the task package or Docker image. These documents and the rest of `/task` are read-only. Only create or modify submission files under `/app`, and fit implementation, validation and debugging into the agent's 120-minute allowance. -The documents to select from are in /task/data/corpus.jsonl. To check whether the selected set covers a question well, there are 20 public development queries in /task/data/validation/queries.jsonl, using the same record schema as hidden questions, and their gold relevance labels are /task/data/validation/qrels_gold.txt in TREC format, query_id 0 docid 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment. +The documents to select from are in `/task/data/corpus.jsonl`. To check whether the selected set covers a question well, there are 20 public development queries in `/task/data/validation/queries.jsonl`, using the same record schema as hidden questions, and their gold relevance labels are `/task/data/validation/qrels_gold.txt` in TREC format, `query_id` 0 `docid` 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment. -The preparation interface is /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must finish within 600 seconds, having built all required indexes and runtime artifacts and started the search service. Return exit status 0 only when that service is ready for queries, and keep it alive across repeated run.sh calls. Use the supplied --index-dir for persistent indexes and other build artifacts; the verifier must not have to know any additional fixed paths. The final submission must contain executable build.sh and run.sh under /app. You can also keep optional modules in /app/src and optional implementation or self-test notes in /app/README.md. +The preparation interface is: -After a successful build, the query call is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Process every supplied query and write exactly one result object per query. You may choose any internal protocol or service endpoint that works with build.sh and that command-line interface. The verifier builds once and invokes run.sh once per hidden query, with up to five invocations in parallel. The search service and entry point must support those concurrent invocations sharing the same --index-dir, with separate input and output files. They share the task's CPU, memory, index and service resources. +```bash +/app/build.sh \ + --corpus /task/data/corpus.jsonl \ + --index-dir /app/index +``` + +It must finish within 600 seconds, having built all required indexes and runtime artifacts and started the search service. Return exit status 0 only when that service is ready for queries, and keep it alive across repeated `run.sh` calls. Use the supplied `--index-dir` for persistent indexes and other build artifacts; the verifier must not have to know any additional fixed paths. The final submission must contain executable `build.sh` and `run.sh` under `/app`. You can also keep optional modules in `/app/src` and optional implementation or self-test notes in `/app/README.md`. + +After a successful build, the query call is: + +```bash +/app/run.sh \ + --index-dir /app/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl +``` + +Process every supplied query and write exactly one result object per query. You may choose any internal protocol or service endpoint that works with `build.sh` and that command-line interface. The verifier builds once and invokes `run.sh` once per hidden query, with up to five invocations in parallel. The search service and entry point must support those concurrent invocations sharing the same `--index-dir`, with separate input and output files. They share the task's CPU, memory, index and service resources. Each query is allowed at most 20 retrieval rounds. A search action returning a candidate set counts as one round; putting independent searches into a batch does not combine them into a single round. There is no individual query time limit, but the whole query execution shares a 15,300-second budget and the verifier phase has a 15,600-second limit. -The output is JSONL, with a line like {"query_id":"example-1","doc_ids":["100","101","102","103","104"]}. query_id must be a string preserving the input query's identifier. doc_ids must be an array of exactly five non-empty strings, all distinct and matching actual docid values in /task/data/corpus.jsonl. These are the final five documents to read for that question, even if your system went through many internal retrieval rounds to select them. Emit exactly one record per input query, with no missing, duplicate or extra query IDs. +The output is JSONL, with a line like `{"query_id":"example-1","doc_ids":["100","101","102","103","104"]}`. `query_id` must be a string preserving the input query's identifier. `doc_ids` must be an array of exactly five non-empty strings, all distinct and matching actual `docid` values in `/task/data/corpus.jsonl`. These are the final five documents to read for that question, even if your system went through many internal retrieval rounds to select them. Emit exactly one record per input query, with no missing, duplicate or extra query IDs. -Harbor transfers /app to a separate verifier with the same runtime and private split after the agent phase. Your build.sh must restart the service there. The verifier checks genuine corpus retrieval, compliance with the resource/model/provider/usage restrictions and the build, shared-time and 20-round limits, executable entry points, and output validity. Failed execution or invalid output receives score 0 for the entire submission. +Harbor transfers `/app` to a separate verifier with the same runtime and private split after the agent phase. Your `build.sh` must restart the service there. The verifier checks genuine corpus retrieval, compliance with the resource/model/provider/usage restrictions and the build, shared-time and 20-round limits, executable entry points, and output validity. Failed execution or invalid output receives score 0 for the entire submission. -The coverage of those five selected documents is measured only by Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward. +The coverage of those five selected documents is measured only by Gold Recall@5, with every hidden query weighted equally. For a query q, `R_q` is your five returned document IDs and `G_q` is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of `G_q`. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward. diff --git a/tasks/code-embedding-finetuning/instruction.md b/tasks/code-embedding-finetuning/instruction.md index d323c28..4cc1cb6 100644 --- a/tasks/code-embedding-finetuning/instruction.md +++ b/tasks/code-embedding-finetuning/instruction.md @@ -1,9 +1,9 @@ I need to find relevant code in the supplied corpus from a search query, and there is already a code-retrieval embedder to start from. Please use the provided corpus and training data to improve the matches it produces, and deliver the updated model as a checkpoint that can be loaded for retrieval. It will be compared with the original checkpoint using exact Top-1 retrieval on held-out queries. Larger improvement in held-out Accuracy@1 is better, subject to preserving the supplied backbone. -Please read /task/docs/environment.md and /task/docs/available_resources.md first. The supplied backbone is mounted read-only under /task/models, and your starting point must be /task/models/bge-base-en-v1.5. Preserve its architecture, parameter names, parameter shapes and parameter count. All of /task is read-only. +Please read `/task/docs/environment.md` and `/task/docs/available_resources.md` first. The supplied backbone is mounted read-only under `/task/models`, and your starting point must be `/task/models/bge-base-en-v1.5`. Preserve its architecture, parameter names, parameter shapes and parameter count. All of `/task` is read-only. -The corpus and training data for improving the embedder are under /task/data. To check whether your changes are helping retrieval during development, there is also a fixed public validation split of 10 queries with labels under /task/data/validation. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable; both development and the submitted checkpoint must rely on the supplied local resources. +The corpus and training data for improving the embedder are under `/task/data`. To check whether your changes are helping retrieval during development, there is also a fixed public validation split of 10 queries with labels under `/task/data/validation`. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable; both development and the submitted checkpoint must rely on the supplied local resources. -I need to be able to load the improved model for retrieval in a separate environment, so write it to /app/submission/model as a complete, self-contained offline checkpoint. Both AutoTokenizer.from_pretrained and AutoModel.from_pretrained must be able to load it with local_files_only=True. The verifier collects only /app/submission and ignores files outside that directory, so the checkpoint must load without depending on uncollected files. +I need to be able to load the improved model for retrieval in a separate environment, so write it to `/app/submission/model` as a complete, self-contained offline checkpoint. Both `AutoTokenizer.from_pretrained` and `AutoModel.from_pretrained` must be able to load it with `local_files_only=True`. The verifier collects only `/app/submission` and ignores files outside that directory, so the checkpoint must load without depending on uncollected files. You have a 12-hour agent phase. Task-specific submission verification has a 1-hour limit, which excludes the separate integrity judge. A timeout of the agent phase, the task-specific verification, or the integrity judge results in score 0. Once you finish, the checkpoint is copied into a separate verifier environment, where its architecture is checked against the supplied backbone. The verifier evaluates the original and submitted checkpoints with the same fixed embedding implementation and exact-search implementation. Larger improvement in held-out Accuracy@1 is better. diff --git a/tasks/long-document-reranking/instruction.md b/tasks/long-document-reranking/instruction.md index 1f6da0b..1d80c1c 100644 --- a/tasks/long-document-reranking/instruction.md +++ b/tasks/long-document-reranking/instruction.md @@ -1,13 +1,31 @@ -I need to find useful information in long financial documents, and each query already comes with a BM25 Top-100 candidate set. The next step is to decide which of those documents should be read first. Please improve the supplied reranking pipeline so that the five returned candidates are as relevant to the question as possible. Use the supplied local BAAI/bge-reranker-large model to rerank the fixed candidates, maximizing retrieval quality on held-out financial queries within the required interfaces and resource limits. +I need to find useful information in long financial documents, and each query already comes with a BM25 Top-100 candidate set. The next step is to decide which of those documents should be read first. Please improve the supplied reranking pipeline so that the five returned candidates are as relevant to the question as possible. Use the supplied local `BAAI/bge-reranker-large` model to rerank the fixed candidates, maximizing retrieval quality on held-out financial queries within the required interfaces and resource limits. -The part to improve is how the existing candidates are reranked; the candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The supplied model is mounted at /task/models/bge-reranker-large. Read /task/docs/environment.md and /task/docs/available_resources.md for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. +The part to improve is how the existing candidates are reranked; the candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The supplied model is mounted at `/task/models/bge-reranker-large`. Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. -The full documents are available in /task/data/corpus.jsonl. To see how changes affect the ranking, use the public development queries at /task/data/validation/queries.jsonl and their normalized labels at /task/data/validation/ground_truth.jsonl. The public query records use the same schema as hidden queries. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. +The full documents are available in `/task/data/corpus.jsonl`. To see how changes affect the ranking, use the public development queries at `/task/data/validation/queries.jsonl` and their normalized labels at `/task/data/validation/ground_truth.jsonl`. The public query records use the same schema as hidden queries. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. -Create or modify submission files only under /app. Treat /task, /tests and the task-provided documents as read-only. The total agent allowance for implementation, validation and debugging is 120 minutes. The final package needs executable /app/build.sh and /app/run.sh files; optional implementation modules can go in /app/src and optional notes and self-test details in /app/README.md. During verification you run as a non-root user and /app is read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts, including preprocessing and cache state, and the verifier must not need to know any additional fixed paths. +Create or modify submission files only under `/app`. Treat `/task`, `/tests` and the task-provided documents as read-only. The total agent allowance for implementation, validation and debugging is 120 minutes. The final package needs executable `/app/build.sh` and `/app/run.sh` files; optional implementation modules can go in `/app/src` and optional notes and self-test details in `/app/README.md`. During verification you run as a non-root user and `/app` is read-only. Only the supplied `--index-dir` is writable for persistent and runtime-generated artifacts, including preprocessing and cache state, and the verifier must not need to know any additional fixed paths. -The verifier starts with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must prepare all required corpus, model and runtime artifacts, start the reranking service, and return exit status 0 only when the system can process queries. It must keep that service available across repeated run.sh calls. After build.sh succeeds, queries arrive through /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Every query record carries its fixed BM25 Top-100 candidates; rerank only those candidates and write exactly one result object for each input query. The internal protocol or service endpoint is your choice as long as it works with the build process and the specified interface. +The verifier starts with: -Write JSONL output with a line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. That example shows the fields of a result item; --top-k 5 requires exactly five items for each valid query. Keep the corresponding input identifier in query_id so the reranked documents can be matched back to their question. Each doc_id must identify one of that query's supplied candidates and each score must be finite and numeric. Document IDs must not repeat within a query. Sort by descending score and, for equal scores, preserve original candidate order as the tie-break. +```bash +/app/build.sh \ + --corpus /task/data/corpus.jsonl \ + --index-dir /app/index +``` -After the agent phase, Harbor uses the task runtime and private split to verify genuine reranking, the fixed candidate set, and use of the unchanged supplied local model. It checks that build.sh and run.sh are executable, waits for the build to return successfully, invokes the query entry point and validates the JSONL structure, query coverage, count of results, duplicates, candidate document IDs, scores and ranking. Invalid output or failure at an executable gate gives a zero score. To assess whether the first five documents contain useful material for the question, a valid submission is scored only by Accuracy@5: a query earns 1 if at least one relevant corpus document is among its first five results and 0 otherwise, and the final score is 100 times the average of those values. +It must prepare all required corpus, model and runtime artifacts, start the reranking service, and return exit status 0 only when the system can process queries. It must keep that service available across repeated `run.sh` calls. After `build.sh` succeeds, queries arrive through: + +```bash +/app/run.sh \ + --index-dir /app/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl \ + --top-k 5 +``` + +Every query record carries its fixed BM25 Top-100 candidates; rerank only those candidates and write exactly one result object for each input query. The internal protocol or service endpoint is your choice as long as it works with the build process and the specified interface. + +Write JSONL output with a line shaped like `{"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}`. That example shows the fields of a result item; `--top-k 5` requires exactly five items for each valid query. Keep the corresponding input identifier in `query_id` so the reranked documents can be matched back to their question. Each `doc_id` must identify one of that query's supplied candidates and each score must be finite and numeric. Document IDs must not repeat within a query. Sort by descending score and, for equal scores, preserve original candidate order as the tie-break. + +After the agent phase, Harbor uses the task runtime and private split to verify genuine reranking, the fixed candidate set, and use of the unchanged supplied local model. It checks that `build.sh` and `run.sh` are executable, waits for the build to return successfully, invokes the query entry point and validates the JSONL structure, query coverage, count of results, duplicates, candidate document IDs, scores and ranking. Invalid output or failure at an executable gate gives a zero score. To assess whether the first five documents contain useful material for the question, a valid submission is scored only by Accuracy@5: a query earns 1 if at least one relevant corpus document is among its first five results and 0 otherwise, and the final score is 100 times the average of those values. diff --git a/tasks/long-pdf-evidence/instruction.md b/tasks/long-pdf-evidence/instruction.md index 7cd96b3..d150f49 100644 --- a/tasks/long-pdf-evidence/instruction.md +++ b/tasks/long-pdf-evidence/instruction.md @@ -1,18 +1,71 @@ I need to look up passages in a long PDF repeatedly. Often I have a description of the material I want to find and need to locate the original pages so I can check the surrounding text. Please build an executable evidence-localization system that takes a natural-language query and returns the five most relevant physical pages, with an evidence excerpt and a relevance score for each, so I can go back to the document and verify them. Aim to maximize localization quality on held-out queries; you may use any retrieval architecture that satisfies the executable interface and resource constraints. -The file to work with is in /task/data/corpus, which contains exactly one PDF. It is the same long document used for development and hidden evaluation. To help you check the page finder while developing it, public queries are available in /task/data/validation/queries.jsonl, following the same record schema as hidden queries, and /task/data/validation/golden_answers.jsonl contains their reference physical pages and supporting excerpts. For a returned page number to lead back to the right place in the document, use the 1-based physical page index in the PDF. Please build genuine localization over this document, without hidden labels, hard-coded query-to-location mappings, precomputed query-to-answer mappings or answer files, external datasets containing evaluation judgments, or external search or answer services that supply evidence locations or evaluation answers. +The file to work with is in `/task/data/corpus`, which contains exactly one PDF. It is the same long document used for development and hidden evaluation. To help you check the page finder while developing it, public queries are available in `/task/data/validation/queries.jsonl`, following the same record schema as hidden queries, and `/task/data/validation/golden_answers.jsonl` contains their reference physical pages and supporting excerpts. For a returned page number to lead back to the right place in the document, use the 1-based physical page index in the PDF. Please build genuine localization over this document, without hidden labels, hard-coded query-to-location mappings, precomputed query-to-answer mappings or answer files, external datasets containing evaluation judgments, or external search or answer services that supply evidence locations or evaluation answers. -Read /task/docs/environment.md and /task/docs/available_resources.md before implementing. They describe the installed Python environment, runtime, packages and system tools, and optional retrieval and generative resources, including exact model allowlists, provider and endpoint restrictions, runtime environment-variable and credential handling, usage rules and jailbreak penalties. Only listed resources may be used according to those restrictions; an unlisted model, provider or endpoint is not implicitly permitted. Harbor injects the documented credentials at runtime, and you must not put them in the package or Docker image. Both documents are read-only task data. Treat all of /task as read-only and create or modify submission files only under /app. Your implementation, validation and debugging must fit into the agent's 120-minute allowance. +Read `/task/docs/environment.md` and `/task/docs/available_resources.md` before implementing. They describe the installed Python environment, runtime, packages and system tools, and optional retrieval and generative resources, including exact model allowlists, provider and endpoint restrictions, runtime environment-variable and credential handling, usage rules and jailbreak penalties. Only listed resources may be used according to those restrictions; an unlisted model, provider or endpoint is not implicitly permitted. Harbor injects the documented credentials at runtime, and you must not put them in the package or Docker image. Both documents are read-only task data. Treat all of `/task` as read-only and create or modify submission files only under `/app`. Your implementation, validation and debugging must fit into the agent's 120-minute allowance. -I need to be able to prepare the document once and then submit queries to the running service, so provide executable /app/build.sh and /app/run.sh entry points. Optional modules may live in /app/src and optional implementation notes and self-test details in /app/README.md. The verifier prepares the system by invoking /app/build.sh --corpus /task/data/corpus --index-dir /app/index. That command must create all required index and runtime artifacts, start the search service, and return exit status 0 only when the service is ready for queries. The service must continue working across repeated run.sh calls. The supplied --index-dir is where persistent indexes and other build artifacts belong. Honor the passed paths: verification mounts the same PDF at a different location, and its index and output directories may differ from these examples. +I need to be able to prepare the document once and then submit queries to the running service, so provide executable `/app/build.sh` and `/app/run.sh` entry points. Optional modules may live in `/app/src` and optional implementation notes and self-test details in `/app/README.md`. The verifier prepares the system by invoking: -Once build.sh has returned successfully, the query interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Each input line looks like {"query_id":"opaque-query-id","query":"Describe the passage to locate."}. Query IDs are opaque, all queries concern the sole PDF, and no target field is supplied. Process every input query and write exactly one result object per query. The verifier invokes run.sh separately for each query, with up to five concurrent processes sharing the index and any service started by the build. Choose any internal protocol or endpoint that works with this build lifecycle and command-line interface. +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /app/index +``` -Each output line must be a JSON object like {"query_id":"query-id","results":[{"page":12,"evidence":"text from page 12","score":0.9},{"page":13,"evidence":"text from page 13","score":0.8},{"page":25,"evidence":"text from page 25","score":0.7},{"page":26,"evidence":"text from page 26","score":0.6},{"page":30,"evidence":"text from page 30","score":0.5}]}. Preserve the input query_id and return exactly five items when invoked with --top-k 5. Each item needs a valid physical page from this PDF, a non-empty evidence string from that page, and a finite numeric score. Pages must be unique within the query, results must be in descending score order, and equal scores must be resolved by ascending page number. The evidence text must actually correspond to the reported page so that the passage can be checked there. +That command must create all required index and runtime artifacts, start the search service, and return exit status 0 only when the service is ready for queries. The service must continue working across repeated `run.sh` calls. The supplied `--index-dir` is where persistent indexes and other build artifacts belong. Honor the passed paths: verification mounts the same PDF at a different location, and its index and output directories may differ from these examples. -After the agent phase, Harbor transfers /app to a fresh verifier environment with the same runtime. That environment gets the same PDF and private queries, but not the public validation queries or labels. Agent-phase processes are not transferred, so build.sh must restart the service. Submission commands execute as an unprivileged user; anything needed at evaluation time must already be in the base image or installed under /app. +Once `build.sh` has returned successfully, the query interface is: -The verifier gives build.sh 2,400 seconds. Every query has its own run.sh process with a 900-second limit and up to five such processes may run concurrently, but they all share a 1,800-second execution budget. Questions still unfinished or not yet launched when that shared deadline arrives score zero. Time spent queuing does not consume a query's individual allowance, although the shared deadline keeps advancing. The runner gets an additional 60 seconds for cleanup and result collection, and all of this sits within a 10,800-second verifier phase. +```bash +/app/run.sh \ + --index-dir /app/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl \ + --top-k 5 +``` + +Each input line looks like `{"query_id":"opaque-query-id","query":"Describe the passage to locate."}`. Query IDs are opaque, all queries concern the sole PDF, and no `target` field is supplied. Process every input query and write exactly one result object per query. The verifier invokes `run.sh` separately for each query, with up to five concurrent processes sharing the index and any service started by the build. Choose any internal protocol or endpoint that works with this build lifecycle and command-line interface. + +Each output line must be a JSON object like: + +```json +{ + "query_id": "query-id", + "results": [ + { + "page": 12, + "evidence": "text from page 12", + "score": 0.9 + }, + { + "page": 13, + "evidence": "text from page 13", + "score": 0.8 + }, + { + "page": 25, + "evidence": "text from page 25", + "score": 0.7 + }, + { + "page": 26, + "evidence": "text from page 26", + "score": 0.6 + }, + { + "page": 30, + "evidence": "text from page 30", + "score": 0.5 + } + ] +} +``` + +Preserve the input `query_id` and return exactly five items when invoked with `--top-k 5`. Each item needs a valid physical page from this PDF, a non-empty evidence string from that page, and a finite numeric score. Pages must be unique within the query, results must be in descending score order, and equal scores must be resolved by ascending page number. The evidence text must actually correspond to the reported page so that the passage can be checked there. + +After the agent phase, Harbor transfers `/app` to a fresh verifier environment with the same runtime. That environment gets the same PDF and private queries, but not the public validation queries or labels. Agent-phase processes are not transferred, so `build.sh` must restart the service. Submission commands execute as an unprivileged user; anything needed at evaluation time must already be in the base image or installed under `/app`. + +The verifier gives `build.sh` 2,400 seconds. Every query has its own `run.sh` process with a 900-second limit and up to five such processes may run concurrently, but they all share a 1,800-second execution budget. Questions still unfinished or not yet launched when that shared deadline arrives score zero. Time spent queuing does not consume a query's individual allowance, although the shared deadline keeps advancing. The runner gets an additional 60 seconds for cleanup and result collection, and all of this sits within a 10,800-second verifier phase. The verifier checks the executable entry points and service startup, real evidence localization, and each query's JSONL structure, result count, page numbers, evidence, scores, duplicates and ranking. A failed or timed-out query scores zero while the others continue; missing or invalid output also scores zero for that query and stays in the average's denominator. A failure to build the shared index or initialize verification prevents every query from scoring. Page relevance is checked deterministically against gold labels, with no model judging answer correctness. A model audit checks trajectory compliance only. diff --git a/tasks/memory-constrained-dense-retrieval/instruction.md b/tasks/memory-constrained-dense-retrieval/instruction.md index 4b94b6c..372dfa2 100644 --- a/tasks/memory-constrained-dense-retrieval/instruction.md +++ b/tasks/memory-constrained-dense-retrieval/instruction.md @@ -1,15 +1,37 @@ I have a collection whose vectors are already computed, and I need to find relevant items whenever a query vector is supplied. Please turn these files into an executable search system that prepares the collection and then serves repeated queries, returning a ranked set of matching corpus items each time. Both preparation time and the wait for each result have to fit the limits below. Maximize retrieval quality on held-out queries within the evaluator's resource limits; you may choose any indexing, storage and retrieval architecture that satisfies those limits and the required interfaces. -The collection is already available in /task/data/vectors.f32: 1,500,000 normalized float32 corpus vectors, each with dimension 1024. /task/data/metadata.jsonl has 1,500,000 rows in one-to-one correspondence with them, and /task/data/vector_config.json describes that vector collection. For trying queries against that collection, the public development vectors are /task/data/validation/query_vectors.f32, with matching metadata at /task/data/validation/query_metadata.jsonl and configuration at /task/data/validation/vector_config.json. You also have /task/data/validation/queries.jsonl for the public query records and /task/data/validation/ground_truth.jsonl for their normalized labels. The final evaluation uses held-out query vectors and private relevance judgments, disjoint from these public examples and not copied into your environment. +The collection is already available in `/task/data/vectors.f32`: 1,500,000 normalized float32 corpus vectors, each with dimension 1024. `/task/data/metadata.jsonl` has 1,500,000 rows in one-to-one correspondence with them, and `/task/data/vector_config.json` describes that vector collection. For trying queries against that collection, the public development vectors are `/task/data/validation/query_vectors.f32`, with matching metadata at `/task/data/validation/query_metadata.jsonl` and configuration at `/task/data/validation/vector_config.json`. You also have `/task/data/validation/queries.jsonl` for the public query records and `/task/data/validation/ground_truth.jsonl` for their normalized labels. The final evaluation uses held-out query vectors and private relevance judgments, disjoint from these public examples and not copied into your environment. -Please read /task/docs/environment.md before implementing anything so you know the installed Python environment, system runtime, packages and tools. That document is read-only task data, as is /task generally. Create or modify submission files only under /app. Retrieval must use only the supplied precomputed vectors and task data: no external models, retrieval services, external datasets containing evaluation results, or precomputed query-to-result mappings. Implement genuine vector retrieval over the supplied corpus. Do not obtain answers from hidden labels, hard-coded query-to-document mappings, precomputed answer files, or external datasets containing the evaluation judgments. Plan your implementation, validation and debugging within the agent's total allowance of 120 minutes. +Please read `/task/docs/environment.md` before implementing anything so you know the installed Python environment, system runtime, packages and tools. That document is read-only task data, as is `/task` generally. Create or modify submission files only under `/app`. Retrieval must use only the supplied precomputed vectors and task data: no external models, retrieval services, external datasets containing evaluation results, or precomputed query-to-result mappings. Implement genuine vector retrieval over the supplied corpus. Do not obtain answers from hidden labels, hard-coded query-to-document mappings, precomputed answer files, or external datasets containing the evaluation judgments. Plan your implementation, validation and debugging within the agent's total allowance of 120 minutes. -The final submission must include executable /app/build.sh and /app/run.sh files. Optional implementation modules can go in /app/src, and optional notes and self-test details in /app/README.md. Keep in mind that verification runs your submission as a non-root user with /app read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts. The evaluator must not have to know another fixed path for the indexes, runtime files or other build artifacts. +The final submission must include executable `/app/build.sh` and `/app/run.sh` files. Optional implementation modules can go in `/app/src`, and optional notes and self-test details in `/app/README.md`. Keep in mind that verification runs your submission as a non-root user with `/app` read-only. Only the supplied `--index-dir` is writable for persistent and runtime-generated artifacts. The evaluator must not have to know another fixed path for the indexes, runtime files or other build artifacts. -To prepare the system, the verifier will call /app/build.sh --vectors /task/data/vectors.f32 --metadata /task/data/metadata.jsonl --config /task/data/vector_config.json --index-dir /app/index. Have that command construct every required index and runtime artifact, start the search service, and return exit status 0 only when the service is ready to accept queries. The whole build process, including index construction and service startup, must finish within 120 seconds. The service must then remain available for repeated run.sh calls, so the prepared collection stays ready as more queries are submitted. +To prepare the system, the verifier will call: -The query invocation looks like /app/run.sh --index-dir /app/index --query-vectors /task/data/validation/query_vectors.f32 --query-metadata /task/data/validation/query_metadata.jsonl --query-config /task/data/validation/vector_config.json --output /tmp/results.jsonl --top-k 3. Process every query supplied to that invocation and write exactly one result object for each input query. Your internal protocol or service endpoint is your choice provided it works with build.sh and this command-line interface. The verifier may start run.sh separately for every query. Each such invocation has just 0.5 seconds of end-to-end wall-clock time, measured from process start through process exit, so the entire invocation must fit rather than merely the search calculation inside it. +```bash +/app/build.sh \ + --vectors /task/data/vectors.f32 \ + --metadata /task/data/metadata.jsonl \ + --config /task/data/vector_config.json \ + --index-dir /app/index +``` -The output is JSONL. A line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}, with exactly three result items when --top-k 3 is supplied. Preserve the corresponding input query's identifier as query_id so each group of matches can be associated with its query vector. Every item needs a doc_id from the supplied corpus and a finite numeric score; no document ID may repeat within a query. Order the items by descending score and break equal-score ties using corpus order. +Have that command construct every required index and runtime artifact, start the search service, and return exit status 0 only when the service is ready to accept queries. The whole build process, including index construction and service startup, must finish within 120 seconds. The service must then remain available for repeated `run.sh` calls, so the prepared collection stays ready as more queries are submitted. -After the agent phase, Harbor uses the task runtime and private test split to check retrieval integrity, executable entry points, build completion and service readiness, query execution, and the validity of the JSONL structure, query coverage, result counts, duplicate handling, document IDs, scores and ranking. A bad output or failed run.sh invocation gives that query zero credit, while the verifier continues with the other queries. A hidden query earns 1 only if its invocation exits successfully within 0.5 seconds, its output is valid, and at least one relevant document is in the first three results; otherwise it earns 0. The primary metric is the average of these latency-gated Accuracy@3 values. The normalized reward is that average, and the displayed score is 100 times the average. A failed build or failed retrieval-integrity check makes the overall score 0. +The query invocation looks like: + +```bash +/app/run.sh \ + --index-dir /app/index \ + --query-vectors /task/data/validation/query_vectors.f32 \ + --query-metadata /task/data/validation/query_metadata.jsonl \ + --query-config /task/data/validation/vector_config.json \ + --output /tmp/results.jsonl \ + --top-k 3 +``` + +Process every query supplied to that invocation and write exactly one result object for each input query. Your internal protocol or service endpoint is your choice provided it works with `build.sh` and this command-line interface. The verifier may start `run.sh` separately for every query. Each such invocation has just 0.5 seconds of end-to-end wall-clock time, measured from process start through process exit, so the entire invocation must fit rather than merely the search calculation inside it. + +The output is JSONL. A line has the shape `{"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}`, with exactly three result items when `--top-k 3` is supplied. Preserve the corresponding input query's identifier as `query_id` so each group of matches can be associated with its query vector. Every item needs a `doc_id` from the supplied corpus and a finite numeric score; no document ID may repeat within a query. Order the items by descending score and break equal-score ties using corpus order. + +After the agent phase, Harbor uses the task runtime and private test split to check retrieval integrity, executable entry points, build completion and service readiness, query execution, and the validity of the JSONL structure, query coverage, result counts, duplicate handling, document IDs, scores and ranking. A bad output or failed `run.sh` invocation gives that query zero credit, while the verifier continues with the other queries. A hidden query earns 1 only if its invocation exits successfully within 0.5 seconds, its output is valid, and at least one relevant document is in the first three results; otherwise it earns 0. The primary metric is the average of these latency-gated Accuracy@3 values. The normalized reward is that average, and the displayed score is 100 times the average. A failed build or failed retrieval-integrity check makes the overall score 0. diff --git a/tasks/query-encoder-alignment/instruction.md b/tasks/query-encoder-alignment/instruction.md index b74f0d0..6105bf3 100644 --- a/tasks/query-encoder-alignment/instruction.md +++ b/tasks/query-encoder-alignment/instruction.md @@ -1,11 +1,21 @@ The document collection has already been converted into vectors. I now need to take a query and find the corpus document that best addresses it, using those existing representations. Please build the query-side retrieval system around the supplied 0.6B query-encoder backbone, with the document vectors kept fixed throughout the work. The goal is to maximize held-out Accuracy@1 while meeting the fixed runtime gate, so improvements need to make the first result more useful while keeping the complete query run within the allowed runtime. -Before deciding how to improve the query side, read /task/docs/environment.md and /task/docs/available_resources.md. The fixed vectors, corpus mapping and public validation split are under /task/data, and the supplied query backbone and reference starter model are under /task/models. Use /task/models/Qwen3-Embedding-0.6B as the basis of your query-side system. Treat all of /task as read-only and put the complete submission under /app/submission. The vectors supplied through --doc-vectors must remain unchanged: do not modify, replace, re-encode or reorder them. Preserve the row correspondence between /task/data/doc.npy and /task/data/corpus.jsonl so a retrieved vector continues to identify the same document. +Before deciding how to improve the query side, read `/task/docs/environment.md` and `/task/docs/available_resources.md`. The fixed vectors, corpus mapping and public validation split are under `/task/data`, and the supplied query backbone and reference starter model are under `/task/models`. Use `/task/models/Qwen3-Embedding-0.6B` as the basis of your query-side system. Treat all of `/task` as read-only and put the complete submission under `/app/submission`. The vectors supplied through `--doc-vectors` must remain unchanged: do not modify, replace, re-encode or reorder them. Preserve the row correspondence between `/task/data/doc.npy` and `/task/data/corpus.jsonl` so a retrieved vector continues to identify the same document. Agent-stage network access is available, but the system you submit must be self-contained and run without network access. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers, or external retrieval, reranking, generation or answer services. The hidden retrieval queries and their private relevance labels are held out, disjoint from the public validation split and unavailable during the agent phase. You have 12 hours for the agent phase. Task-specific submission verification has a 70-minute limit, excluding the separate integrity judge. If the agent phase, task-specific verification or integrity judge times out, the score is 0. -The verifier's call is /app/submission/run.sh --doc-vectors /task/data/doc.npy --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 1. Please make run.sh executable, process every query exactly once in input order, write deterministic results, exit successfully and leave no background processes. The output is JSONL with each line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. With --top-k 1, return exactly one corpus document and a finite numeric score. Ranking is by descending score, with corpus order breaking ties. Include the executable entry point and every required local file under /app/submission, since that directory alone is copied into the verifier. +The verifier's call is: + +```bash +/app/submission/run.sh \ + --doc-vectors /task/data/doc.npy \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl \ + --top-k 1 +``` + +Please make `run.sh` executable, process every query exactly once in input order, write deterministic results, exit successfully and leave no background processes. The output is JSONL with each line shaped like `{"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}`. With `--top-k 1`, return exactly one corpus document and a finite numeric score. Ranking is by descending score, with corpus order breaking ties. Include the executable entry point and every required local file under `/app/submission`, since that directory alone is copied into the verifier. To check whether the query-side work makes the first match more useful, the verifier runs both the fixed reference starter and your submission on the same private queries and the same document vectors. It checks vector integrity, output format, determinism and background processes. Submissions that meet the fixed runtime gate are evaluated on held-out Accuracy@1. diff --git a/tasks/reasoning-query-rewriting/instruction.md b/tasks/reasoning-query-rewriting/instruction.md index f9c59b7..bdb3f0a 100644 --- a/tasks/reasoning-query-rewriting/instruction.md +++ b/tasks/reasoning-query-rewriting/instruction.md @@ -1,15 +1,33 @@ I need to look up material in this document collection using questions written in ordinary language. Once I have a question, I want a short ranked set of relevant documents to read. Please build an executable retrieval system over the supplied corpus that retrieves and ranks the best matches for each natural-language query. The goal is to maximize retrieval quality on held-out queries while satisfying the executable interface and resource constraints described here. -Before choosing your approach, read /task/docs/environment.md and /task/docs/available_resources.md. They describe the installed Python environment, system runtime, available packages and tools, and the retrieval and generative API resources that are actually permitted, including their model allowlist, providers, endpoints, credential handling, usage restrictions and jailbreak penalties. Those documents are read-only task data. You may use an external model or service only if that resource document explicitly permits it, and only in its documented retrieval or query-transformation role. An unlisted provider, model or endpoint is not an option. Harbor injects the documented credentials at runtime; do not put them into the task package or Docker image. +Before choosing your approach, read `/task/docs/environment.md` and `/task/docs/available_resources.md`. They describe the installed Python environment, system runtime, available packages and tools, and the retrieval and generative API resources that are actually permitted, including their model allowlist, providers, endpoints, credential handling, usage restrictions and jailbreak penalties. Those documents are read-only task data. You may use an external model or service only if that resource document explicitly permits it, and only in its documented retrieval or query-transformation role. An unlisted provider, model or endpoint is not an option. Harbor injects the documented credentials at runtime; do not put them into the task package or Docker image. -The collection to search is /task/data/corpus.jsonl. You can try out the retrieval system with the public questions in /task/data/validation/queries.jsonl, which use the same record schema as hidden queries, and check the matches against their normalized labels in /task/data/validation/ground_truth.jsonl. Use the supplied corpus and task data to build real retrieval. Do not obtain relevance judgments or results from hidden labels, hard-coded query-to-document mappings, precomputed query-to-result mappings or answer files, external datasets containing evaluation results or judgments, an external search engine, or an external retrieval service. The resource permissions above do not authorize those shortcuts. The questions used for the final evaluation and their private relevance judgments are held out, disjoint from the public development examples, and never copied into your environment. +The collection to search is `/task/data/corpus.jsonl`. You can try out the retrieval system with the public questions in `/task/data/validation/queries.jsonl`, which use the same record schema as hidden queries, and check the matches against their normalized labels in `/task/data/validation/ground_truth.jsonl`. Use the supplied corpus and task data to build real retrieval. Do not obtain relevance judgments or results from hidden labels, hard-coded query-to-document mappings, precomputed query-to-result mappings or answer files, external datasets containing evaluation results or judgments, an external search engine, or an external retrieval service. The resource permissions above do not authorize those shortcuts. The questions used for the final evaluation and their private relevance judgments are held out, disjoint from the public development examples, and never copied into your environment. -All submission files you create or change must stay under /app, and /task is read-only. Implementation, validation and debugging must fit within the 120-minute agent phase. The final submission must contain executable /app/build.sh and /app/run.sh files. You may keep implementation modules in /app/src and implementation notes or self-test details in /app/README.md, but those two additions are optional. During verification the submission runs as a non-root user and /app is read-only; the supplied --index-dir is the only writable location for persistent and runtime-generated artifacts. +All submission files you create or change must stay under `/app`, and `/task` is read-only. Implementation, validation and debugging must fit within the 120-minute agent phase. The final submission must contain executable `/app/build.sh` and `/app/run.sh` files. You may keep implementation modules in `/app/src` and implementation notes or self-test details in `/app/README.md`, but those two additions are optional. During verification the submission runs as a non-root user and `/app` is read-only; the supplied `--index-dir` is the only writable location for persistent and runtime-generated artifacts. -To make the collection ready to search, the verifier starts by invoking /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. This must construct every required local index and runtime artifact and return exit status 0 when they are ready. A background service is fine if useful, but the verifier will not keep a foreground process alive for you: build.sh must finish successfully before any query invocation begins. The --index-dir argument identifies the place for persistent indexes and other build artifacts, and the verifier must not need to know any additional fixed paths. +To make the collection ready to search, the verifier starts by invoking: -Once that build succeeds, the query entry point is invoked as /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 3. Process every supplied query and write exactly one result object for each input query to the requested output. Choose whatever internal protocol or service endpoint works with your build process and this command-line interface. Write JSONL output, where a line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}; the single result shown illustrates an item's fields, while an invocation with --top-k 3 must actually return exactly three items for every valid query. +```bash +/app/build.sh \ + --corpus /task/data/corpus.jsonl \ + --index-dir /app/index +``` -To keep each set of matches associated with the question that produced it, preserve the corresponding input identifier in query_id. Each returned doc_id must identify a document in the supplied corpus, each score must be finite and numeric, and a query must never contain duplicate document IDs. Rank results from highest to lowest score, using corpus order to resolve equal scores. +This must construct every required local index and runtime artifact and return exit status 0 when they are ready. A background service is fine if useful, but the verifier will not keep a foreground process alive for you: `build.sh` must finish successfully before any query invocation begins. The `--index-dir` argument identifies the place for persistent indexes and other build artifacts, and the verifier must not need to know any additional fixed paths. + +Once that build succeeds, the query entry point is invoked as: + +```bash +/app/run.sh \ + --index-dir /app/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl \ + --top-k 3 +``` + +Process every supplied query and write exactly one result object for each input query to the requested output. Choose whatever internal protocol or service endpoint works with your build process and this command-line interface. Write JSONL output, where a line has the shape `{"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}`; the single result shown illustrates an item's fields, while an invocation with `--top-k 3` must actually return exactly three items for every valid query. + +To keep each set of matches associated with the question that produced it, preserve the corresponding input identifier in `query_id`. Each returned `doc_id` must identify a document in the supplied corpus, each score must be finite and numeric, and a query must never contain duplicate document IDs. Rank results from highest to lowest score, using corpus order to resolve equal scores. After your work, Harbor runs the submission with the task runtime and private test split. It checks genuine corpus retrieval, all the resource and model/provider/endpoint/credential/usage restrictions, successful execution through the specified interfaces, and the JSONL structure, coverage of queries, result counts, duplicate handling, document IDs, scores and ranking. Invalid output or failure at an executable gate receives score 0. The usefulness of the first three matches is measured, for a valid submission, only by Accuracy@3: a query earns 1 when at least one relevant corpus document appears in its first three returned results and 0 otherwise, and the displayed task score is 100 times the average of those query values. diff --git a/tasks/scientific-paper-qa/instruction.md b/tasks/scientific-paper-qa/instruction.md index e9bc20d..ed9f89c 100644 --- a/tasks/scientific-paper-qa/instruction.md +++ b/tasks/scientific-paper-qa/instruction.md @@ -1,15 +1,32 @@ I need to answer scientific questions using a collection of papers and then check which paper supports each answer. Please build an executable retrieval-augmented question answering system over the supplied PDFs so that I can submit a question, get a concise answer, and identify the supporting paper to consult. The goal is to maximize answer correctness and source attribution quality on held-out questions. You may use any method that satisfies the executable interface and resource constraints. -The material to work from is a collection of 300 scientific-paper PDFs in /task/data/corpus. To check both the answers and the papers they point to during development, use the public questions in /task/data/validation/queries.jsonl and their reference answers and evidence document IDs in /task/data/validation/golden_answers.jsonl. Those public questions use the same record schema as hidden questions. Use these papers to answer, without precomputed query-to-answer mappings, hidden labels, hard-coded evaluation answers, or external datasets or services containing evaluation answers. The held-out queries and private relevance judgments are disjoint from the development examples and unavailable in your environment. +The material to work from is a collection of 300 scientific-paper PDFs in `/task/data/corpus`. To check both the answers and the papers they point to during development, use the public questions in `/task/data/validation/queries.jsonl` and their reference answers and evidence document IDs in `/task/data/validation/golden_answers.jsonl`. Those public questions use the same record schema as hidden questions. Use these papers to answer, without precomputed query-to-answer mappings, hidden labels, hard-coded evaluation answers, or external datasets or services containing evaluation answers. The held-out queries and private relevance judgments are disjoint from the development examples and unavailable in your environment. -Before implementing the system, read /task/docs/environment.md for the installed Python environment, runtime, packages and system tools, and /task/docs/available_resources.md for the optional retrieval and generative APIs, exact model allowlists, provider and endpoint restrictions, usage rules, runtime credential handling and jailbreak penalties. Follow those restrictions rather than assuming an unlisted service, model or endpoint is allowed; unauthorized datasets and services are prohibited. Harbor supplies the documented credentials at runtime, and they must never be put in the task package or Docker image. These documents and all of /task are read-only. Create or change submission files only under /app, and finish implementation, validation and debugging within the 120-minute agent allowance. +Before implementing the system, read `/task/docs/environment.md` for the installed Python environment, runtime, packages and system tools, and `/task/docs/available_resources.md` for the optional retrieval and generative APIs, exact model allowlists, provider and endpoint restrictions, usage rules, runtime credential handling and jailbreak penalties. Follow those restrictions rather than assuming an unlisted service, model or endpoint is allowed; unauthorized datasets and services are prohibited. Harbor supplies the documented credentials at runtime, and they must never be put in the task package or Docker image. These documents and all of `/task` are read-only. Create or change submission files only under `/app`, and finish implementation, validation and debugging within the 120-minute agent allowance. -To prepare the papers for repeated questions, the final submission must include executable /app/build.sh and /app/run.sh files. You can organize optional modules under /app/src and leave optional implementation and self-test notes in /app/README.md. The preparation call is /app/build.sh --corpus /task/data/corpus --index-dir /app/index. Build all required indexes and runtime artifacts, start the search service and return exit status 0 only after it is ready for queries. It must stay available through repeated run.sh invocations. Use the supplied --index-dir for persistent indexes and other build artifacts, and honor supplied paths generally: the verifier's index and output directories may differ from the example locations. The build must finish within 3600 seconds. +To prepare the papers for repeated questions, the final submission must include executable `/app/build.sh` and `/app/run.sh` files. You can organize optional modules under `/app/src` and leave optional implementation and self-test notes in `/app/README.md`. The preparation call is: -After a successful build, the query interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Whatever internal protocol or service endpoint you use must work with the build process and this interface. Process every supplied query and write exactly one result object for each. The verifier builds once and then invokes run.sh once for each hidden query, with up to five invocations in parallel. They share the task's CPU, memory, index and service resources. Each query has 900 seconds, counted from the start of its run.sh process; waiting for an available slot does not use up that query's individual allowance. +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /app/index +``` -For the output, write one JSON object per line in the form {"query_id":"query-id","answer":"The answer for the query","evidence":"document-id"}. Preserve each input query's identifier in query_id, provide a non-empty answer string that directly answers the question, and include evidence identifying the supporting document from the supplied corpus. The evidence value must be the PDF filename with its .pdf extension removed, so document-id.pdf is reported as document-id. Both an answer and a valid supporting corpus document ID are required for every query, so each response can be traced back to a paper in the collection. +Build all required indexes and runtime artifacts, start the search service and return exit status 0 only after it is ready for queries. It must stay available through repeated `run.sh` invocations. Use the supplied `--index-dir` for persistent indexes and other build artifacts, and honor supplied paths generally: the verifier's index and output directories may differ from the example locations. The build must finish within 3600 seconds. -Harbor will transfer /app into a separate verifier with the same runtime and the private split after you finish; your build.sh must restart the service there. The verifier phase has 14,400 seconds overall, including the arrangement of a 3,600-second build and 900 seconds per query with up to five concurrent queries. It checks retrieval and answer integrity, resource compliance, executable interfaces and output validity. A failed execution or invalid output gives score 0 to the entire submission, rather than just dropping the affected question. +After a successful build, the query interface is: + +```bash +/app/run.sh \ + --index-dir /app/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl +``` + +Whatever internal protocol or service endpoint you use must work with the build process and this interface. Process every supplied query and write exactly one result object for each. The verifier builds once and then invokes `run.sh` once for each hidden query, with up to five invocations in parallel. They share the task's CPU, memory, index and service resources. Each query has 900 seconds, counted from the start of its `run.sh` process; waiting for an available slot does not use up that query's individual allowance. + +For the output, write one JSON object per line in the form `{"query_id":"query-id","answer":"The answer for the query","evidence":"document-id"}`. Preserve each input query's identifier in `query_id`, provide a non-empty answer string that directly answers the question, and include evidence identifying the supporting document from the supplied corpus. The evidence value must be the PDF filename with its `.pdf` extension removed, so `document-id.pdf` is reported as `document-id`. Both an answer and a valid supporting corpus document ID are required for every query, so each response can be traced back to a paper in the collection. + +Harbor will transfer `/app` into a separate verifier with the same runtime and the private split after you finish; your `build.sh` must restart the service there. The verifier phase has 14,400 seconds overall, including the arrangement of a 3,600-second build and 900 seconds per query with up to five concurrent queries. It checks retrieval and answer integrity, resource compliance, executable interfaces and output validity. A failed execution or invalid output gives score 0 to the entire submission, rather than just dropping the affected question. The answer and its source are evaluated together using LLMJudgeAccuracy, equally averaged over every hidden query. For a query to earn 1, its evidence document must match the reference and an LLM judge must accept its answer as semantically equivalent to the reference answer; otherwise it earns 0. The answer judge only evaluates an answer after its evidence document matches. The normalized reward is the mean of those query values, and the displayed score is 100 times that reward. An independent trajectory audit also checks compliance with the task and resource restrictions; a failed audit makes the entire submission's reward and score 0. diff --git a/tasks/sparse-retrieval-pruning/instruction.md b/tasks/sparse-retrieval-pruning/instruction.md index bc3a17c..f4394b8 100644 --- a/tasks/sparse-retrieval-pruning/instruction.md +++ b/tasks/sparse-retrieval-pruning/instruction.md @@ -2,17 +2,34 @@ I need to run a collection of searches over a large corpus and reduce the time s The faster result still needs to retrieve the relevant documents. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the unpruned reference starter wall time measured by the verifier in the same environment. -Please read /task/docs/environment.md for the installed runtime and resource limits, /task/docs/available_resources.md for the local dependency policy, and /task/docs/index_format.md before changing the starter index. That format is an implementation detail and may be replaced. All of /task is read-only, and submission files must only be created or modified under /app. The agent phase allows up to 120 minutes. Do not use precomputed hidden query-to-document mappings, hidden relevance judgments, or external datasets or services containing evaluation answers. The formal build and search run offline and must not need credentials, remote models, package downloads or external services. +Please read `/task/docs/environment.md` for the installed runtime and resource limits, `/task/docs/available_resources.md` for the local dependency policy, and `/task/docs/index_format.md` before changing the starter index. That format is an implementation detail and may be replaced. All of `/task` is read-only, and submission files must only be created or modified under `/app`. The agent phase allows up to 120 minutes. Do not use precomputed hidden query-to-document mappings, hidden relevance judgments, or external datasets or services containing evaluation answers. The formal build and search run offline and must not need credentials, remote models, package downloads or external services. -Implement all submission logic in Python source code. build.sh and run.sh may be thin Bash wrappers for argument parsing and launching Python, but indexing, retrieval, scoring, pruning and ranking must all be Python. Do not add or compile native submission source, including C, C++, Rust or Go, and do not have build.sh generate a native executable. Preinstalled Python packages such as NumPy are available through their normal APIs. +Implement all submission logic in Python source code. `build.sh` and `run.sh` may be thin Bash wrappers for argument parsing and launching Python, but indexing, retrieval, scoring, pruning and ranking must all be Python. Do not add or compile native submission source, including C, C++, Rust or Go, and do not have `build.sh` generate a native executable. Preinstalled Python packages such as NumPy are available through their normal APIs. Both the submission build and search path must be single-process and single-threaded. Do not use multiprocessing, worker pools, threading, joblib, Ray or parallel query execution. The verifier environment is limited to one CPU, so extra threads cannot gain more aggregate CPU throughput, but that CPU limit does not itself make parallel execution authorized or valid. -The collection to search is /task/data/corpus.jsonl. To check retrieval quality as you reduce search time, public development material includes /task/data/validation/queries.jsonl for validation queries, /task/data/validation/qrels.tsv for relevance judgments, /task/data/validation/stats.json for term and document-frequency statistics, and /task/data/validation/reference_top100.jsonl for regression rankings. The formal hidden query split and its relevance judgments belong to the verifier. Document and query files are JSONL: a document looks like {"doc_id":"document-id","terms":["term-id", "term-id"]} and a query like {"query_id":"query-id","terms":["term-id", "term-id"]}. The repeated placeholder in those examples denotes the shape of the array; actual terms within a row are unique opaque strings. Their lexical value, apparent numeric value, input order and position do not encode relevance. +The collection to search is `/task/data/corpus.jsonl`. To check retrieval quality as you reduce search time, public development material includes `/task/data/validation/queries.jsonl` for validation queries, `/task/data/validation/qrels.tsv` for relevance judgments, `/task/data/validation/stats.json` for term and document-frequency statistics, and `/task/data/validation/reference_top100.jsonl` for regression rankings. The formal hidden query split and its relevance judgments belong to the verifier. Document and query files are JSONL: a document looks like `{"doc_id":"document-id","terms":["term-id", "term-id"]}` and a query like `{"query_id":"query-id","terms":["term-id", "term-id"]}`. The repeated placeholder in those examples denotes the shape of the array; actual terms within a row are unique opaque strings. Their lexical value, apparent numeric value, input order and position do not encode relevance. -Provide executable /app/build.sh and /app/run.sh files in the final submission. Optional Python modules may live in /app/src and optional notes in /app/README.md. The verifier builds with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /path/to/index. Build retrieval state from the supplied corpus and create every persistent index and runtime artifact beneath that supplied --index-dir. Return status 0 only when the index is ready. The build may use public statistics next to the corpus, but it must not require additional undeclared paths or undeclared persistent files from the verifier. +Provide executable `/app/build.sh` and `/app/run.sh` files in the final submission. Optional Python modules may live in `/app/src` and optional notes in `/app/README.md`. The verifier builds with: -Once the build succeeds, the search interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. The search run must get all of its required runtime state from the supplied index directory and query file. Process each supplied query exactly once, preserve input query order, and write one result object per query. A JSONL output line looks like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. query_id must identify its corresponding input query. Return at most 100 results, with unique doc_id values that occur in the supplied corpus and scores that are finite and numeric. Order results by descending score, resolving equal scores by ascending lexicographic order of the opaque document ID. Fewer than 100 results is valid, though it can reduce Recall@100. +```bash +/app/build.sh \ + --corpus /task/data/corpus.jsonl \ + --index-dir /path/to/index +``` + +Build retrieval state from the supplied corpus and create every persistent index and runtime artifact beneath that supplied `--index-dir`. Return status 0 only when the index is ready. The build may use public statistics next to the corpus, but it must not require additional undeclared paths or undeclared persistent files from the verifier. + +Once the build succeeds, the search interface is: + +```bash +/app/run.sh \ + --index-dir /app/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl +``` + +The search run must get all of its required runtime state from the supplied index directory and query file. Process each supplied query exactly once, preserve input query order, and write one result object per query. A JSONL output line looks like `{"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}`. `query_id` must identify its corresponding input query. Return at most 100 results, with unique `doc_id` values that occur in the supplied corpus and scores that are finite and numeric. Order results by descending score, resolving equal scores by ascending lexicographic order of the opaque document ID. Fewer than 100 results is valid, though it can reduce Recall@100. The speed improvement will be measured against the unpruned reference starter used by the verifier as its latency baseline. It retains every corpus term and posting, applies no DF, prefix or other posting-list pruning by default, uses the set-valued corpus and query inputs without weights, builds a consistent dictionary, metadata and posting layout, and performs unpruned binary/IDF-overlap search. Any optional pruning flag in starter code is only for explicit local experiments and is not enabled for the verifier's baseline. From 4ab2dfa61fc337bb0ec395965f2f71e8403f5ef3 Mon Sep 17 00:00:00 2001 From: hanhainebula <2512674094@qq.com> Date: Sun, 11 Oct 2026 01:06:28 +0800 Subject: [PATCH 3/3] Correct long-document reranking descriptions and model paths --- tasks/long-document-reranking/instruction.md | 4 ++-- tasks/long-document-reranking/raw-instruction.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tasks/long-document-reranking/instruction.md b/tasks/long-document-reranking/instruction.md index 1d80c1c..5bc6cfb 100644 --- a/tasks/long-document-reranking/instruction.md +++ b/tasks/long-document-reranking/instruction.md @@ -1,8 +1,8 @@ -I need to find useful information in long financial documents, and each query already comes with a BM25 Top-100 candidate set. The next step is to decide which of those documents should be read first. Please improve the supplied reranking pipeline so that the five returned candidates are as relevant to the question as possible. Use the supplied local `BAAI/bge-reranker-large` model to rerank the fixed candidates, maximizing retrieval quality on held-out financial queries within the required interfaces and resource limits. +I need to find useful information in long documents, and each query already comes with a BM25 Top-100 candidate set. The next step is to decide which of those documents should be read first. Please improve the supplied reranking pipeline so that the five returned candidates are as relevant to the question as possible. Use the supplied local `BAAI/bge-reranker-large` model to rerank the fixed candidates, maximizing retrieval quality on held-out queries within the required interfaces and resource limits. The part to improve is how the existing candidates are reranked; the candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The supplied model is mounted at `/task/models/bge-reranker-large`. Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. -The full documents are available in `/task/data/corpus.jsonl`. To see how changes affect the ranking, use the public development queries at `/task/data/validation/queries.jsonl` and their normalized labels at `/task/data/validation/ground_truth.jsonl`. The public query records use the same schema as hidden queries. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. +The full documents are available in `/task/data/corpus.jsonl`. To see how changes affect the ranking, use the public development queries at `/task/data/validation/queries.jsonl` and their normalized labels at `/task/data/validation/ground_truth.jsonl`. The public query records use the same schema as hidden queries. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. Create or modify submission files only under `/app`. Treat `/task`, `/tests` and the task-provided documents as read-only. The total agent allowance for implementation, validation and debugging is 120 minutes. The final package needs executable `/app/build.sh` and `/app/run.sh` files; optional implementation modules can go in `/app/src` and optional notes and self-test details in `/app/README.md`. During verification you run as a non-root user and `/app` is read-only. Only the supplied `--index-dir` is writable for persistent and runtime-generated artifacts, including preprocessing and cache state, and the verifier must not need to know any additional fixed paths. diff --git a/tasks/long-document-reranking/raw-instruction.md b/tasks/long-document-reranking/raw-instruction.md index 9a2d1ef..a072de2 100644 --- a/tasks/long-document-reranking/raw-instruction.md +++ b/tasks/long-document-reranking/raw-instruction.md @@ -2,7 +2,7 @@ ## Task Description -Improve the supplied long-document reranking system. Given a query and its fixed BM25 Top-100 candidate documents, the system should rerank the candidates using the supplied local `/tast/model/bge-reranker-large` model. +Improve the supplied long-document reranking system. Given a query and its fixed BM25 Top-100 candidate documents, the system should rerank the candidates using the supplied local `/task/models/bge-reranker-large` model. The candidate set and reranker model are fixed. You may optimize the reranking pipeline, but may not replace or train the model or modify the supplied candidates. @@ -92,7 +92,7 @@ The finalized submission must contain executable `build.sh` and `run.sh` files u After the Agent phase, the Harbor verifier runs the submission in the same task environment and uses the private test split. The verifier checks the following items: 1. **Reranking integrity.** The submission must implement a genuine reranking pipeline over the candidate documents supplied with each query. It must not obtain relevance judgments or results through hidden labels, hard-coded query-to-document mappings, precomputed answer files, external datasets containing the evaluation judgments, or external retrieval/reranking services. -2. **Candidate-set and model compliance.** The submission must rerank only the fixed BM25 Top-100 candidates supplied with each query and must use the supplied local `/tast/model/bge-reranker-large` model without replacing, training, or modifying it. +2. **Candidate-set and model compliance.** The submission must rerank only the fixed BM25 Top-100 candidates supplied with each query and must use the supplied local `/task/models/bge-reranker-large` model without replacing, training, or modifying it. 3. **Executable and output validity.** The verifier checks that build.sh and run.sh exist and are executable, invokes build.sh, waits for it to return successfully, and then invokes run.sh for the hidden queries. It also checks the JSONL result structure, query coverage, result count, duplicate handling, candidate document IDs, scores, and ranking output. Invalid output or a failed executable gate receives a zero score. 4. **Final retrieval score.** For a valid submission, the final task score is calculated only with `Accuracy@5`: @@ -104,7 +104,7 @@ For an individual query, `Accuracy@5` is `1` when at least one relevant corpus d ## Hidden Test Overview -The hidden evaluation contains held-out financial queries with private relevance judgments. The hidden queries are disjoint from the public development examples and are not copied into the Agent-visible environment. +The hidden evaluation contains held-out queries with private relevance judgments. The hidden queries are disjoint from the public development examples and are not copied into the Agent-visible environment. ## Environment and available resources