diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index bf623f5..346579c 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -32,9 +32,11 @@ Markdown links. user-request rewrite → grader checks → configured model evaluation → feedback and task refinement → revalidation of substantive changes. Preserve `raw-instruction.md` and audit all its conditions in the agent-facing - `instruction.md`. Describe the task, available inputs, resource constraints, - acceptance criteria and optimization objective directly, without invented - requester identities or narrative scenes; the skill defines both gates. + `instruction.md`. Start from a concrete work need and explain how the + requested result will be used, then connect the available inputs, constraints, + acceptance criteria and optimization objective in natural prose. Preserve + practical context without self-introductions or invented identities; the + skill defines the rewrite and validation gates. Before scaffolding, inspect one or two closest reviewed packages under `tasks/` as read-only structural precedents, selected by engineering objective, grading shape, resources and hardware. Apply current repository contracts diff --git a/.agents/skills/create-searchswe-task/SKILL.md b/.agents/skills/create-searchswe-task/SKILL.md index cd15979..4b7d143 100644 --- a/.agents/skills/create-searchswe-task/SKILL.md +++ b/.agents/skills/create-searchswe-task/SKILL.md @@ -92,12 +92,14 @@ Follow the detailed rules in `references/task-authoring.md`: 1. Define the submission interface and grader behavior in `raw-instruction.md`. Preserve an existing structured instruction there before rewriting. Follow [references/instruction-rewrite.md](references/instruction-rewrite.md) to write - the agent-facing `instruction.md`: a direct user request in connected prose - describing the task, current data and starting system, available resources, - constraints, acceptance criteria and optimization objective. Omit - self-introductions, invented identities and fictional scenes. Preserve every - condition and exact interface; do not reveal hidden answers. Audit the two - documents for semantic equivalence before running a model. + the agent-facing `instruction.md`: a natural user request beginning with a + concrete work need and how the result will be used. Carry that practical + context through the supplied data, starting system, resources, constraints, + acceptance criteria and optimization objective. Omit self-introductions and + invented identities; retain the sense of a real task in progress. Preserve + every condition and exact interface without adding obligations for the + setting; do not reveal hidden answers. Audit the two documents for semantic + equivalence before running a model. 2. When adding fixed data/models or external services, read [references/assets-and-resources.md](references/assets-and-resources.md). Build exact manifests and read-only phase-specific mounts. Keep hidden inputs, diff --git a/.agents/skills/create-searchswe-task/references/instruction-rewrite.md b/.agents/skills/create-searchswe-task/references/instruction-rewrite.md index 8cee23a..26d5f79 100644 --- a/.agents/skills/create-searchswe-task/references/instruction-rewrite.md +++ b/.agents/skills/create-searchswe-task/references/instruction-rewrite.md @@ -23,20 +23,36 @@ used for the rewrite so changes can be reviewed. ## Write the request -Write the request a user would give their coding agent to get the work done. -Start with what needs to be built or improved, then connect the current data -and supplied starting system, available resources, restrictions, required -deliverables, acceptance criteria and optimization objective in natural prose. -Explain what counts as better using the actual metric, baseline, quality gates -and resource tradeoffs in the source. Include enough detail to preserve the -whole contract; length should come from those requirements, not filler. - -Do not introduce a named requester, a self-introduction such as "I'm Alex", an -invented role, physical workplace, personal history or fictional motivation. -A direct request such as "I need a system that..." is fine. Use context grounded -in the supplied task, such as vectors that are already computed or a starter -whose retrieval quality needs improvement. Do not invent dataset provenance, -usage claims or a story around the work. +Write the request a user would give their coding agent while working on a +concrete problem. Begin with the work they need to do, the material or system +already available, and how they will use the result. The opening should make +the practical need clear, such as finding a passage to check its original +context, choosing which retrieved documents to read, or reducing the wait for +a complete search workload. + +Keep this sense of use throughout the request. Connect the current data and +starting system, available resources, restrictions, deliverables and acceptance +criteria to the work as it progresses. Explain what counts as better using +the source's metric, baseline, quality gates and resource tradeoffs. Technical +interfaces and scoring details must remain exact, and the whole request should +read as one coherent commission. Adding a contextual opening alone is not +enough if the rest reads as a disconnected specification. + +Use ordinary first-person task language when it fits, without a named requester +or a self-introduction such as "I'm Alex". For example, a PDF-localization task +could begin: "I need to look up passages in a long PDF repeatedly. Often I have +a description of the material I want to find and need to locate the original +pages so I can check the surrounding text." Choose context appropriate to each +task; there is no fixed opening template. A plain task description with "I need" +prepended is not sufficient to establish the work situation. + +Ground the context in the actual task and resources. Do not invent identities, +organizations, biographies or decorative scenes. Do not claim an unsupported +dataset provenance, production deployment, measured defect or usage history. +Practical motivation must not add functionality or acceptance criteria: a +desire to check original pages, for example, does not require a PDF viewer or +a graphical interface unless the source specifies one. Include enough detail +to preserve the whole contract without padding it with a backstory. Use no headings, enumerated requirements, bullet lists, tables, directory trees or fenced code blocks in the final instruction. Exact commands, paths, @@ -73,9 +89,11 @@ Make a small author-only correspondence record: source condition or line range, the request paragraph containing it, and any discrepancy. Review both ways: every raw condition must appear in the prose, and every prose obligation must come from the source or an explicitly resolved contract correction. Read the -whole request for natural flow after this check. Confirm it focuses on the -work, inputs, resources, constraints and measurable outcome, without an -invented persona or setting. +whole request for natural flow after this check. Confirm that a concrete work +need and the intended use of the results are apparent, and remain connected +to the data, resources, constraints and measurable outcome through the body. +Check that there is no self-introduction, invented identity or extra obligation +introduced by the context. Removing a persona must not erase the work situation. Literal comparisons of paths, flags, IDs and numbers catch omissions but cannot prove semantic equivalence. Inspect negations, exceptions, time accounting, diff --git a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py index f34f461..e0bc366 100755 --- a/.agents/skills/create-searchswe-task/scripts/scaffold_task.py +++ b/.agents/skills/create-searchswe-task/scripts/scaffold_task.py @@ -126,10 +126,12 @@ def files_for(task_name: str, hardware: str) -> dict[str, str]: "instruction.md": clean( f""" Replace this scaffold for {task_name} after completing raw-instruction.md. - Write a direct natural-language request describing the task, current - data and starting system, resources, constraints, acceptance criteria, - and optimization objective. Omit self-introductions, invented people, - roles, and fictional scenes. Use connected prose paragraphs without + Begin with a concrete work need and how the result will be used. + Carry that practical context through the current data and starting + system, resources, constraints, acceptance criteria, and optimization + objective. Omit self-introductions and invented identities; preserve + the sense of a real task without adding requirements for the setting. + Use connected prose paragraphs without headings, lists, tables, or fenced blocks. Preserve every source condition and exact interface; follow references/instruction-rewrite.md in the authoring skill and remove this author-only prompt before use. diff --git a/tasks/agentic-search/instruction.md b/tasks/agentic-search/instruction.md index 07eb0f2..c0a81cf 100644 --- a/tasks/agentic-search/instruction.md +++ b/tasks/agentic-search/instruction.md @@ -1,10 +1,10 @@ -Optimize the supplied ReAct retrieval starter over the corpus. For each question combining multiple constraints, the system must select exactly five distinct corpus documents that best address the combined information need. The goal is to maximize Gold Recall@5 on held-out questions while preserving the interfaces and resource constraints. +Some of the questions I need to research combine several conditions, and I want a small set of documents that I can read together to address the full question. Please improve the supplied ReAct retrieval starter so that each such question produces exactly five distinct documents from the corpus that best cover its combined information need. The goal is to maximize Gold Recall@5 on held-out questions while preserving the interfaces and resource constraints. -The starter lives under /app/starter, with executable /app/build.sh and /app/run.sh entry points. Read /app/starter/README.md to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If OPENROUTER_API_KEY is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective. +There is already a retrieval-and-selection workflow to build on under /app/starter, with executable /app/build.sh and /app/run.sh entry points. Read /app/starter/README.md to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If OPENROUTER_API_KEY is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective. Please read /task/docs/environment.md for the installed Python environment, runtime, packages and tools, and /task/docs/available_resources.md for optional retrieval and generative APIs, runtime environment-variable and credential handling, exact model allowlists, provider and endpoint restrictions, usage limits and jailbreak penalties. Follow the documented restrictions and do not assume an unlisted provider, model or endpoint is authorized. Harbor injects the documented credentials at runtime; do not put them in the task package or Docker image. These documents and the rest of /task are read-only. Only create or modify submission files under /app, and fit implementation, validation and debugging into the agent's 120-minute allowance. -The complete corpus is /task/data/corpus.jsonl. There are 20 public development queries in /task/data/validation/queries.jsonl, using the same record schema as the hidden questions, and their gold relevance labels are /task/data/validation/qrels_gold.txt in TREC format, query_id 0 docid 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment. +The documents to select from are in /task/data/corpus.jsonl. To check whether the selected set covers a question well, there are 20 public development queries in /task/data/validation/queries.jsonl, using the same record schema as hidden questions, and their gold relevance labels are /task/data/validation/qrels_gold.txt in TREC format, query_id 0 docid 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment. The preparation interface is /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must finish within 600 seconds, having built all required indexes and runtime artifacts and started the search service. Return exit status 0 only when that service is ready for queries, and keep it alive across repeated run.sh calls. Use the supplied --index-dir for persistent indexes and other build artifacts; the verifier must not have to know any additional fixed paths. The final submission must contain executable build.sh and run.sh under /app. You can also keep optional modules in /app/src and optional implementation or self-test notes in /app/README.md. @@ -12,8 +12,8 @@ After a successful build, the query call is /app/run.sh --index-dir /app/index - Each query is allowed at most 20 retrieval rounds. A search action returning a candidate set counts as one round; putting independent searches into a batch does not combine them into a single round. There is no individual query time limit, but the whole query execution shares a 15,300-second budget and the verifier phase has a 15,600-second limit. -The output is JSONL, with a line like {"query_id":"example-1","doc_ids":["100","101","102","103","104"]}. query_id must be a string preserving the input query's identifier. doc_ids must be an array of exactly five non-empty strings, all distinct and matching actual docid values in /task/data/corpus.jsonl. These are the final five selected documents even if your system went through many internal retrieval rounds. Emit exactly one record per input query, with no missing, duplicate or extra query IDs. +The output is JSONL, with a line like {"query_id":"example-1","doc_ids":["100","101","102","103","104"]}. query_id must be a string preserving the input query's identifier. doc_ids must be an array of exactly five non-empty strings, all distinct and matching actual docid values in /task/data/corpus.jsonl. These are the final five documents to read for that question, even if your system went through many internal retrieval rounds to select them. Emit exactly one record per input query, with no missing, duplicate or extra query IDs. Harbor transfers /app to a separate verifier with the same runtime and private split after the agent phase. Your build.sh must restart the service there. The verifier checks genuine corpus retrieval, compliance with the resource/model/provider/usage restrictions and the build, shared-time and 20-round limits, executable entry points, and output validity. Failed execution or invalid output receives score 0 for the entire submission. -The only final metric is Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward. +The coverage of those five selected documents is measured only by Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward. diff --git a/tasks/code-embedding-finetuning/instruction.md b/tasks/code-embedding-finetuning/instruction.md index aba6fae..d323c28 100644 --- a/tasks/code-embedding-finetuning/instruction.md +++ b/tasks/code-embedding-finetuning/instruction.md @@ -1,9 +1,9 @@ -Improve the supplied code-retrieval embedder using the provided corpus and training data, and deliver the result as a model checkpoint. It will be compared with the original checkpoint using exact Top-1 retrieval on held-out queries. Larger improvement in held-out Accuracy@1 is better, subject to preserving the supplied backbone. +I need to find relevant code in the supplied corpus from a search query, and there is already a code-retrieval embedder to start from. Please use the provided corpus and training data to improve the matches it produces, and deliver the updated model as a checkpoint that can be loaded for retrieval. It will be compared with the original checkpoint using exact Top-1 retrieval on held-out queries. Larger improvement in held-out Accuracy@1 is better, subject to preserving the supplied backbone. Please read /task/docs/environment.md and /task/docs/available_resources.md first. The supplied backbone is mounted read-only under /task/models, and your starting point must be /task/models/bge-base-en-v1.5. Preserve its architecture, parameter names, parameter shapes and parameter count. All of /task is read-only. -The corpus and training data are under /task/data. There is also a fixed public validation split of 10 queries with labels under /task/data/validation for development. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable; both development and the submitted checkpoint must rely on the supplied local resources. +The corpus and training data for improving the embedder are under /task/data. To check whether your changes are helping retrieval during development, there is also a fixed public validation split of 10 queries with labels under /task/data/validation. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable; both development and the submitted checkpoint must rely on the supplied local resources. -Write the final model to /app/submission/model as a complete, self-contained offline checkpoint. Both AutoTokenizer.from_pretrained and AutoModel.from_pretrained must be able to load it with local_files_only=True. The verifier collects only /app/submission and ignores files outside that directory, so the checkpoint must load without depending on uncollected files. +I need to be able to load the improved model for retrieval in a separate environment, so write it to /app/submission/model as a complete, self-contained offline checkpoint. Both AutoTokenizer.from_pretrained and AutoModel.from_pretrained must be able to load it with local_files_only=True. The verifier collects only /app/submission and ignores files outside that directory, so the checkpoint must load without depending on uncollected files. You have a 12-hour agent phase. Task-specific submission verification has a 1-hour limit, which excludes the separate integrity judge. A timeout of the agent phase, the task-specific verification, or the integrity judge results in score 0. Once you finish, the checkpoint is copied into a separate verifier environment, where its architecture is checked against the supplied backbone. The verifier evaluates the original and submitted checkpoints with the same fixed embedding implementation and exact-search implementation. Larger improvement in held-out Accuracy@1 is better. diff --git a/tasks/long-document-reranking/instruction.md b/tasks/long-document-reranking/instruction.md index 852b3f0..1f6da0b 100644 --- a/tasks/long-document-reranking/instruction.md +++ b/tasks/long-document-reranking/instruction.md @@ -1,13 +1,13 @@ -Improve the supplied long-document reranking system. Each query already has a fixed BM25 Top-100 set of candidate documents, which must be reranked with the supplied local BAAI/bge-reranker-large model. The goal is to maximize retrieval quality on held-out financial queries by improving the reranking pipeline within the required interfaces and resource limits. +I need to find useful information in long financial documents, and each query already comes with a BM25 Top-100 candidate set. The next step is to decide which of those documents should be read first. Please improve the supplied reranking pipeline so that the five returned candidates are as relevant to the question as possible. Use the supplied local BAAI/bge-reranker-large model to rerank the fixed candidates, maximizing retrieval quality on held-out financial queries within the required interfaces and resource limits. -The candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The supplied model is mounted at /task/models/bge-reranker-large. Read /task/docs/environment.md and /task/docs/available_resources.md for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. +The part to improve is how the existing candidates are reranked; the candidate set and learned scorer are fixed. Use exactly the candidates supplied with each query: do not add, remove, replace or independently retrieve candidates. You may optimize the surrounding reranking pipeline, but only the supplied local model may perform learned relevance scoring. Do not replace, fine-tune, retrain, distill, quantize or otherwise modify the model, tokenizer or configuration. The supplied model is mounted at /task/models/bge-reranker-large. Read /task/docs/environment.md and /task/docs/available_resources.md for the installed Python environment, system runtime, packages and tools, local resources and network restrictions before implementing. -You have the full corpus at /task/data/corpus.jsonl, public development queries at /task/data/validation/queries.jsonl in the same record schema used by hidden queries, and their normalized labels at /task/data/validation/ground_truth.jsonl. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. +The full documents are available in /task/data/corpus.jsonl. To see how changes affect the ranking, use the public development queries at /task/data/validation/queries.jsonl and their normalized labels at /task/data/validation/ground_truth.jsonl. The public query records use the same schema as hidden queries. Do not use precomputed query-to-document mappings, hidden relevance information, hard-coded mappings, precomputed answer files, external datasets containing evaluation labels or judgments, or external retrieval or reranking services to get relevance judgments or results. Implement genuine reranking over the supplied candidates. The held-out financial questions have private relevance judgments, are disjoint from the public examples and are not copied into your environment. Create or modify submission files only under /app. Treat /task, /tests and the task-provided documents as read-only. The total agent allowance for implementation, validation and debugging is 120 minutes. The final package needs executable /app/build.sh and /app/run.sh files; optional implementation modules can go in /app/src and optional notes and self-test details in /app/README.md. During verification you run as a non-root user and /app is read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts, including preprocessing and cache state, and the verifier must not need to know any additional fixed paths. The verifier starts with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must prepare all required corpus, model and runtime artifacts, start the reranking service, and return exit status 0 only when the system can process queries. It must keep that service available across repeated run.sh calls. After build.sh succeeds, queries arrive through /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Every query record carries its fixed BM25 Top-100 candidates; rerank only those candidates and write exactly one result object for each input query. The internal protocol or service endpoint is your choice as long as it works with the build process and the specified interface. -Write JSONL output with a line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. That example shows the fields of a result item; --top-k 5 requires exactly five items for each valid query. Keep the corresponding input identifier in query_id. Each doc_id must identify one of that query's supplied candidates and each score must be finite and numeric. Document IDs must not repeat within a query. Sort by descending score and, for equal scores, preserve original candidate order as the tie-break. +Write JSONL output with a line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. That example shows the fields of a result item; --top-k 5 requires exactly five items for each valid query. Keep the corresponding input identifier in query_id so the reranked documents can be matched back to their question. Each doc_id must identify one of that query's supplied candidates and each score must be finite and numeric. Document IDs must not repeat within a query. Sort by descending score and, for equal scores, preserve original candidate order as the tie-break. -After the agent phase, Harbor uses the task runtime and private split to verify genuine reranking, the fixed candidate set, and use of the unchanged supplied local model. It checks that build.sh and run.sh are executable, waits for the build to return successfully, invokes the query entry point and validates the JSONL structure, query coverage, count of results, duplicates, candidate document IDs, scores and ranking. Invalid output or failure at an executable gate gives a zero score. A valid submission is scored only by Accuracy@5: a query earns 1 if at least one relevant corpus document is among its first five results and 0 otherwise, and the final score is 100 times the average of those values. +After the agent phase, Harbor uses the task runtime and private split to verify genuine reranking, the fixed candidate set, and use of the unchanged supplied local model. It checks that build.sh and run.sh are executable, waits for the build to return successfully, invokes the query entry point and validates the JSONL structure, query coverage, count of results, duplicates, candidate document IDs, scores and ranking. Invalid output or failure at an executable gate gives a zero score. To assess whether the first five documents contain useful material for the question, a valid submission is scored only by Accuracy@5: a query earns 1 if at least one relevant corpus document is among its first five results and 0 otherwise, and the final score is 100 times the average of those values. diff --git a/tasks/long-pdf-evidence/instruction.md b/tasks/long-pdf-evidence/instruction.md index 1e6a0b0..7cd96b3 100644 --- a/tasks/long-pdf-evidence/instruction.md +++ b/tasks/long-pdf-evidence/instruction.md @@ -1,14 +1,14 @@ -Build an executable evidence-localization system for the supplied single long PDF. Given a natural-language query describing the evidence to find, return the physical pages in that PDF that best contain it. Maximize localization quality on held-out queries using any retrieval architecture that satisfies the executable interface and resource constraints. +I need to look up passages in a long PDF repeatedly. Often I have a description of the material I want to find and need to locate the original pages so I can check the surrounding text. Please build an executable evidence-localization system that takes a natural-language query and returns the five most relevant physical pages, with an evidence excerpt and a relevance score for each, so I can go back to the document and verify them. Aim to maximize localization quality on held-out queries; you may use any retrieval architecture that satisfies the executable interface and resource constraints. -The corpus directory /task/data/corpus contains exactly one PDF, the same long document used for development and hidden evaluation. Public queries are in /task/data/validation/queries.jsonl, following the same record schema as hidden queries, and /task/data/validation/golden_answers.jsonl contains their reference physical pages and supporting excerpts. A page means the 1-based physical page index in the PDF. Please build genuine localization over this document, without hidden labels, hard-coded query-to-location mappings, precomputed query-to-answer mappings or answer files, external datasets containing evaluation judgments, or external search or answer services that supply evidence locations or evaluation answers. +The file to work with is in /task/data/corpus, which contains exactly one PDF. It is the same long document used for development and hidden evaluation. To help you check the page finder while developing it, public queries are available in /task/data/validation/queries.jsonl, following the same record schema as hidden queries, and /task/data/validation/golden_answers.jsonl contains their reference physical pages and supporting excerpts. For a returned page number to lead back to the right place in the document, use the 1-based physical page index in the PDF. Please build genuine localization over this document, without hidden labels, hard-coded query-to-location mappings, precomputed query-to-answer mappings or answer files, external datasets containing evaluation judgments, or external search or answer services that supply evidence locations or evaluation answers. Read /task/docs/environment.md and /task/docs/available_resources.md before implementing. They describe the installed Python environment, runtime, packages and system tools, and optional retrieval and generative resources, including exact model allowlists, provider and endpoint restrictions, runtime environment-variable and credential handling, usage rules and jailbreak penalties. Only listed resources may be used according to those restrictions; an unlisted model, provider or endpoint is not implicitly permitted. Harbor injects the documented credentials at runtime, and you must not put them in the package or Docker image. Both documents are read-only task data. Treat all of /task as read-only and create or modify submission files only under /app. Your implementation, validation and debugging must fit into the agent's 120-minute allowance. -Provide executable /app/build.sh and /app/run.sh entry points. Optional modules may live in /app/src and optional implementation notes and self-test details in /app/README.md. The verifier prepares the system by invoking /app/build.sh --corpus /task/data/corpus --index-dir /app/index. That command must create all required index and runtime artifacts, start the search service, and return exit status 0 only when the service is ready for queries. The service must continue working across repeated run.sh calls. The supplied --index-dir is where persistent indexes and other build artifacts belong. Honor the passed paths: verification mounts the same PDF at a different location, and its index and output directories may differ from these examples. +I need to be able to prepare the document once and then submit queries to the running service, so provide executable /app/build.sh and /app/run.sh entry points. Optional modules may live in /app/src and optional implementation notes and self-test details in /app/README.md. The verifier prepares the system by invoking /app/build.sh --corpus /task/data/corpus --index-dir /app/index. That command must create all required index and runtime artifacts, start the search service, and return exit status 0 only when the service is ready for queries. The service must continue working across repeated run.sh calls. The supplied --index-dir is where persistent indexes and other build artifacts belong. Honor the passed paths: verification mounts the same PDF at a different location, and its index and output directories may differ from these examples. Once build.sh has returned successfully, the query interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 5. Each input line looks like {"query_id":"opaque-query-id","query":"Describe the passage to locate."}. Query IDs are opaque, all queries concern the sole PDF, and no target field is supplied. Process every input query and write exactly one result object per query. The verifier invokes run.sh separately for each query, with up to five concurrent processes sharing the index and any service started by the build. Choose any internal protocol or endpoint that works with this build lifecycle and command-line interface. -Each output line must be a JSON object like {"query_id":"query-id","results":[{"page":12,"evidence":"text from page 12","score":0.9},{"page":13,"evidence":"text from page 13","score":0.8},{"page":25,"evidence":"text from page 25","score":0.7},{"page":26,"evidence":"text from page 26","score":0.6},{"page":30,"evidence":"text from page 30","score":0.5}]}. Preserve the input query_id and return exactly five items when invoked with --top-k 5. Each item needs a valid physical page from this PDF, a non-empty evidence string from that page, and a finite numeric score. Pages must be unique within the query, results must be in descending score order, and equal scores must be resolved by ascending page number. The evidence text must actually correspond to the reported page. +Each output line must be a JSON object like {"query_id":"query-id","results":[{"page":12,"evidence":"text from page 12","score":0.9},{"page":13,"evidence":"text from page 13","score":0.8},{"page":25,"evidence":"text from page 25","score":0.7},{"page":26,"evidence":"text from page 26","score":0.6},{"page":30,"evidence":"text from page 30","score":0.5}]}. Preserve the input query_id and return exactly five items when invoked with --top-k 5. Each item needs a valid physical page from this PDF, a non-empty evidence string from that page, and a finite numeric score. Pages must be unique within the query, results must be in descending score order, and equal scores must be resolved by ascending page number. The evidence text must actually correspond to the reported page so that the passage can be checked there. After the agent phase, Harbor transfers /app to a fresh verifier environment with the same runtime. That environment gets the same PDF and private queries, but not the public validation queries or labels. Agent-phase processes are not transferred, so build.sh must restart the service. Submission commands execute as an unprivileged user; anything needed at evaluation time must already be in the base image or installed under /app. @@ -16,4 +16,4 @@ The verifier gives build.sh 2,400 seconds. Every query has its own run.sh proces The verifier checks the executable entry points and service startup, real evidence localization, and each query's JSONL structure, result count, page numbers, evidence, scores, duplicates and ranking. A failed or timed-out query scores zero while the others continue; missing or invalid output also scores zero for that query and stays in the average's denominator. A failure to build the shared index or initialize verification prevents every query from scoring. Page relevance is checked deterministically against gold labels, with no model judging answer correctness. A model audit checks trajectory compliance only. -For a valid submission that passes that audit, the only localization metric is Recall@5. For one query this is the number of relevant evidence pages retrieved in the top five divided by the number of relevant evidence pages for that query. Reward is average Recall@5 with failed or invalid queries contributing zero, so it lies between 0 and 1. The evaluation report displays 100 times average Recall@5 before the trajectory gate; a failed trajectory audit sets the final reward to zero. Hidden queries and their private page-relevance judgments concern this same PDF but differ from the public examples and are not copied into your environment. Their IDs are independent opaque identifiers, unique across splits, and their order says nothing about page position. +To measure how much of the relevant evidence these five pages let me find, the only localization metric for a valid submission that passes that audit is Recall@5. For one query this is the number of relevant evidence pages retrieved in the top five divided by the number of relevant evidence pages for that query. Reward is average Recall@5 with failed or invalid queries contributing zero, so it lies between 0 and 1. The evaluation report displays 100 times average Recall@5 before the trajectory gate; a failed trajectory audit sets the final reward to zero. Hidden queries and their private page-relevance judgments concern this same PDF but differ from the public examples and are not copied into your environment. Their IDs are independent opaque identifiers, unique across splits, and their order says nothing about page position. diff --git a/tasks/memory-constrained-dense-retrieval/instruction.md b/tasks/memory-constrained-dense-retrieval/instruction.md index 7a5dca4..4b94b6c 100644 --- a/tasks/memory-constrained-dense-retrieval/instruction.md +++ b/tasks/memory-constrained-dense-retrieval/instruction.md @@ -1,15 +1,15 @@ -The corpus and development queries are already represented by precomputed vectors. Build an executable search system that takes a query vector and retrieves and ranks the most relevant corpus items. Maximize retrieval quality on held-out queries within the evaluator's resource limits; you may choose any indexing, storage and retrieval architecture that satisfies those limits and the required interfaces. +I have a collection whose vectors are already computed, and I need to find relevant items whenever a query vector is supplied. Please turn these files into an executable search system that prepares the collection and then serves repeated queries, returning a ranked set of matching corpus items each time. Both preparation time and the wait for each result have to fit the limits below. Maximize retrieval quality on held-out queries within the evaluator's resource limits; you may choose any indexing, storage and retrieval architecture that satisfies those limits and the required interfaces. -/task/data/vectors.f32 contains 1,500,000 normalized float32 corpus vectors, each with dimension 1024. /task/data/metadata.jsonl has 1,500,000 rows in one-to-one correspondence with them, and /task/data/vector_config.json describes that vector collection. The public development vectors are /task/data/validation/query_vectors.f32, with matching metadata at /task/data/validation/query_metadata.jsonl and configuration at /task/data/validation/vector_config.json. You also have /task/data/validation/queries.jsonl for the public query records and /task/data/validation/ground_truth.jsonl for their normalized labels. The final evaluation uses held-out query vectors and private relevance judgments, disjoint from these public examples and not copied into your environment. +The collection is already available in /task/data/vectors.f32: 1,500,000 normalized float32 corpus vectors, each with dimension 1024. /task/data/metadata.jsonl has 1,500,000 rows in one-to-one correspondence with them, and /task/data/vector_config.json describes that vector collection. For trying queries against that collection, the public development vectors are /task/data/validation/query_vectors.f32, with matching metadata at /task/data/validation/query_metadata.jsonl and configuration at /task/data/validation/vector_config.json. You also have /task/data/validation/queries.jsonl for the public query records and /task/data/validation/ground_truth.jsonl for their normalized labels. The final evaluation uses held-out query vectors and private relevance judgments, disjoint from these public examples and not copied into your environment. Please read /task/docs/environment.md before implementing anything so you know the installed Python environment, system runtime, packages and tools. That document is read-only task data, as is /task generally. Create or modify submission files only under /app. Retrieval must use only the supplied precomputed vectors and task data: no external models, retrieval services, external datasets containing evaluation results, or precomputed query-to-result mappings. Implement genuine vector retrieval over the supplied corpus. Do not obtain answers from hidden labels, hard-coded query-to-document mappings, precomputed answer files, or external datasets containing the evaluation judgments. Plan your implementation, validation and debugging within the agent's total allowance of 120 minutes. The final submission must include executable /app/build.sh and /app/run.sh files. Optional implementation modules can go in /app/src, and optional notes and self-test details in /app/README.md. Keep in mind that verification runs your submission as a non-root user with /app read-only. Only the supplied --index-dir is writable for persistent and runtime-generated artifacts. The evaluator must not have to know another fixed path for the indexes, runtime files or other build artifacts. -To prepare the system, the verifier will call /app/build.sh --vectors /task/data/vectors.f32 --metadata /task/data/metadata.jsonl --config /task/data/vector_config.json --index-dir /app/index. Have that command construct every required index and runtime artifact, start the search service, and return exit status 0 only when the service is ready to accept queries. The whole build process, including index construction and service startup, must finish within 120 seconds. The service must then remain available for repeated run.sh calls. +To prepare the system, the verifier will call /app/build.sh --vectors /task/data/vectors.f32 --metadata /task/data/metadata.jsonl --config /task/data/vector_config.json --index-dir /app/index. Have that command construct every required index and runtime artifact, start the search service, and return exit status 0 only when the service is ready to accept queries. The whole build process, including index construction and service startup, must finish within 120 seconds. The service must then remain available for repeated run.sh calls, so the prepared collection stays ready as more queries are submitted. The query invocation looks like /app/run.sh --index-dir /app/index --query-vectors /task/data/validation/query_vectors.f32 --query-metadata /task/data/validation/query_metadata.jsonl --query-config /task/data/validation/vector_config.json --output /tmp/results.jsonl --top-k 3. Process every query supplied to that invocation and write exactly one result object for each input query. Your internal protocol or service endpoint is your choice provided it works with build.sh and this command-line interface. The verifier may start run.sh separately for every query. Each such invocation has just 0.5 seconds of end-to-end wall-clock time, measured from process start through process exit, so the entire invocation must fit rather than merely the search calculation inside it. -The output is JSONL. A line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}, with exactly three result items when --top-k 3 is supplied. Preserve the corresponding input query's identifier as query_id. Every item needs a doc_id from the supplied corpus and a finite numeric score; no document ID may repeat within a query. Order the items by descending score and break equal-score ties using corpus order. +The output is JSONL. A line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}, with exactly three result items when --top-k 3 is supplied. Preserve the corresponding input query's identifier as query_id so each group of matches can be associated with its query vector. Every item needs a doc_id from the supplied corpus and a finite numeric score; no document ID may repeat within a query. Order the items by descending score and break equal-score ties using corpus order. After the agent phase, Harbor uses the task runtime and private test split to check retrieval integrity, executable entry points, build completion and service readiness, query execution, and the validity of the JSONL structure, query coverage, result counts, duplicate handling, document IDs, scores and ranking. A bad output or failed run.sh invocation gives that query zero credit, while the verifier continues with the other queries. A hidden query earns 1 only if its invocation exits successfully within 0.5 seconds, its output is valid, and at least one relevant document is in the first three results; otherwise it earns 0. The primary metric is the average of these latency-gated Accuracy@3 values. The normalized reward is that average, and the displayed score is 100 times the average. A failed build or failed retrieval-integrity check makes the overall score 0. diff --git a/tasks/query-encoder-alignment/instruction.md b/tasks/query-encoder-alignment/instruction.md index ef4062d..b74f0d0 100644 --- a/tasks/query-encoder-alignment/instruction.md +++ b/tasks/query-encoder-alignment/instruction.md @@ -1,6 +1,6 @@ -The document vectors are already fixed. Build a query-side retrieval system for searching them, with a query encoder based on the supplied 0.6B backbone. Maximize held-out Accuracy@1 while meeting the fixed runtime gate and preserving the document vectors. +The document collection has already been converted into vectors. I now need to take a query and find the corpus document that best addresses it, using those existing representations. Please build the query-side retrieval system around the supplied 0.6B query-encoder backbone, with the document vectors kept fixed throughout the work. The goal is to maximize held-out Accuracy@1 while meeting the fixed runtime gate, so improvements need to make the first result more useful while keeping the complete query run within the allowed runtime. -Read /task/docs/environment.md and /task/docs/available_resources.md before you begin. The fixed vectors, corpus mapping and public validation split are under /task/data, and the supplied query backbone and reference starter model are under /task/models. Use /task/models/Qwen3-Embedding-0.6B as the basis of your query-side system. Treat all of /task as read-only and put the complete submission under /app/submission. The vectors supplied through --doc-vectors must remain unchanged: do not modify, replace, re-encode or reorder them. Preserve the row correspondence between /task/data/doc.npy and /task/data/corpus.jsonl. +Before deciding how to improve the query side, read /task/docs/environment.md and /task/docs/available_resources.md. The fixed vectors, corpus mapping and public validation split are under /task/data, and the supplied query backbone and reference starter model are under /task/models. Use /task/models/Qwen3-Embedding-0.6B as the basis of your query-side system. Treat all of /task as read-only and put the complete submission under /app/submission. The vectors supplied through --doc-vectors must remain unchanged: do not modify, replace, re-encode or reorder them. Preserve the row correspondence between /task/data/doc.npy and /task/data/corpus.jsonl so a retrieved vector continues to identify the same document. Agent-stage network access is available, but the system you submit must be self-contained and run without network access. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers, or external retrieval, reranking, generation or answer services. The hidden retrieval queries and their private relevance labels are held out, disjoint from the public validation split and unavailable during the agent phase. @@ -8,4 +8,4 @@ You have 12 hours for the agent phase. Task-specific submission verification has The verifier's call is /app/submission/run.sh --doc-vectors /task/data/doc.npy --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 1. Please make run.sh executable, process every query exactly once in input order, write deterministic results, exit successfully and leave no background processes. The output is JSONL with each line shaped like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. With --top-k 1, return exactly one corpus document and a finite numeric score. Ranking is by descending score, with corpus order breaking ties. Include the executable entry point and every required local file under /app/submission, since that directory alone is copied into the verifier. -After you finish, the verifier runs both the fixed reference starter and your submission on the same private queries and the same document vectors. It checks vector integrity, output format, determinism and background processes. Submissions that meet the fixed runtime gate are evaluated on held-out Accuracy@1. +To check whether the query-side work makes the first match more useful, the verifier runs both the fixed reference starter and your submission on the same private queries and the same document vectors. It checks vector integrity, output format, determinism and background processes. Submissions that meet the fixed runtime gate are evaluated on held-out Accuracy@1. diff --git a/tasks/reasoning-query-rewriting/instruction.md b/tasks/reasoning-query-rewriting/instruction.md index 6f12369..f9c59b7 100644 --- a/tasks/reasoning-query-rewriting/instruction.md +++ b/tasks/reasoning-query-rewriting/instruction.md @@ -1,15 +1,15 @@ -Build an executable retrieval system over the supplied document corpus. Given a natural-language query, it should retrieve and rank the most relevant corpus documents. The goal is to maximize retrieval quality on held-out queries while satisfying the executable interface and resource constraints described here. +I need to look up material in this document collection using questions written in ordinary language. Once I have a question, I want a short ranked set of relevant documents to read. Please build an executable retrieval system over the supplied corpus that retrieves and ranks the best matches for each natural-language query. The goal is to maximize retrieval quality on held-out queries while satisfying the executable interface and resource constraints described here. Before choosing your approach, read /task/docs/environment.md and /task/docs/available_resources.md. They describe the installed Python environment, system runtime, available packages and tools, and the retrieval and generative API resources that are actually permitted, including their model allowlist, providers, endpoints, credential handling, usage restrictions and jailbreak penalties. Those documents are read-only task data. You may use an external model or service only if that resource document explicitly permits it, and only in its documented retrieval or query-transformation role. An unlisted provider, model or endpoint is not an option. Harbor injects the documented credentials at runtime; do not put them into the task package or Docker image. -The complete collection is /task/data/corpus.jsonl. For development, /task/data/validation/queries.jsonl contains public queries in the same record schema as the hidden queries, and /task/data/validation/ground_truth.jsonl contains their normalized labels. Use the supplied corpus and task data to build real retrieval. Do not obtain relevance judgments or results from hidden labels, hard-coded query-to-document mappings, precomputed query-to-result mappings or answer files, external datasets containing evaluation results or judgments, an external search engine, or an external retrieval service. The resource permissions above do not authorize those shortcuts. The questions used for the final evaluation and their private relevance judgments are held out, disjoint from the public development examples, and never copied into your environment. +The collection to search is /task/data/corpus.jsonl. You can try out the retrieval system with the public questions in /task/data/validation/queries.jsonl, which use the same record schema as hidden queries, and check the matches against their normalized labels in /task/data/validation/ground_truth.jsonl. Use the supplied corpus and task data to build real retrieval. Do not obtain relevance judgments or results from hidden labels, hard-coded query-to-document mappings, precomputed query-to-result mappings or answer files, external datasets containing evaluation results or judgments, an external search engine, or an external retrieval service. The resource permissions above do not authorize those shortcuts. The questions used for the final evaluation and their private relevance judgments are held out, disjoint from the public development examples, and never copied into your environment. All submission files you create or change must stay under /app, and /task is read-only. Implementation, validation and debugging must fit within the 120-minute agent phase. The final submission must contain executable /app/build.sh and /app/run.sh files. You may keep implementation modules in /app/src and implementation notes or self-test details in /app/README.md, but those two additions are optional. During verification the submission runs as a non-root user and /app is read-only; the supplied --index-dir is the only writable location for persistent and runtime-generated artifacts. -The verifier starts by invoking /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. This must construct every required local index and runtime artifact and return exit status 0 when they are ready. A background service is fine if useful, but the verifier will not keep a foreground process alive for you: build.sh must finish successfully before any query invocation begins. The --index-dir argument identifies the place for persistent indexes and other build artifacts, and the verifier must not need to know any additional fixed paths. +To make the collection ready to search, the verifier starts by invoking /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. This must construct every required local index and runtime artifact and return exit status 0 when they are ready. A background service is fine if useful, but the verifier will not keep a foreground process alive for you: build.sh must finish successfully before any query invocation begins. The --index-dir argument identifies the place for persistent indexes and other build artifacts, and the verifier must not need to know any additional fixed paths. Once that build succeeds, the query entry point is invoked as /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl --top-k 3. Process every supplied query and write exactly one result object for each input query to the requested output. Choose whatever internal protocol or service endpoint works with your build process and this command-line interface. Write JSONL output, where a line has the shape {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}; the single result shown illustrates an item's fields, while an invocation with --top-k 3 must actually return exactly three items for every valid query. -Please preserve the corresponding input identifier in query_id. Each returned doc_id must identify a document in the supplied corpus, each score must be finite and numeric, and a query must never contain duplicate document IDs. Rank results from highest to lowest score, using corpus order to resolve equal scores. +To keep each set of matches associated with the question that produced it, preserve the corresponding input identifier in query_id. Each returned doc_id must identify a document in the supplied corpus, each score must be finite and numeric, and a query must never contain duplicate document IDs. Rank results from highest to lowest score, using corpus order to resolve equal scores. -After your work, Harbor runs the submission with the task runtime and private test split. It checks genuine corpus retrieval, all the resource and model/provider/endpoint/credential/usage restrictions, successful execution through the specified interfaces, and the JSONL structure, coverage of queries, result counts, duplicate handling, document IDs, scores and ranking. Invalid output or failure at an executable gate receives score 0. For a valid submission, the only final retrieval metric is Accuracy@3: a query earns 1 when at least one relevant corpus document appears in its first three returned results and 0 otherwise, and the displayed task score is 100 times the average of those query values. +After your work, Harbor runs the submission with the task runtime and private test split. It checks genuine corpus retrieval, all the resource and model/provider/endpoint/credential/usage restrictions, successful execution through the specified interfaces, and the JSONL structure, coverage of queries, result counts, duplicate handling, document IDs, scores and ranking. Invalid output or failure at an executable gate receives score 0. The usefulness of the first three matches is measured, for a valid submission, only by Accuracy@3: a query earns 1 when at least one relevant corpus document appears in its first three returned results and 0 otherwise, and the displayed task score is 100 times the average of those query values. diff --git a/tasks/scientific-paper-qa/instruction.md b/tasks/scientific-paper-qa/instruction.md index 58c25e5..e9bc20d 100644 --- a/tasks/scientific-paper-qa/instruction.md +++ b/tasks/scientific-paper-qa/instruction.md @@ -1,15 +1,15 @@ -Build an executable retrieval-augmented question answering system over the supplied scientific-paper PDFs. Given a scientific question, it should return a concise answer and identify the paper supporting that answer. The goal is to maximize answer correctness and source attribution quality on held-out questions. You may use any method that satisfies the executable interface and resource constraints. +I need to answer scientific questions using a collection of papers and then check which paper supports each answer. Please build an executable retrieval-augmented question answering system over the supplied PDFs so that I can submit a question, get a concise answer, and identify the supporting paper to consult. The goal is to maximize answer correctness and source attribution quality on held-out questions. You may use any method that satisfies the executable interface and resource constraints. -There are 300 scientific-paper PDFs in /task/data/corpus. Public development questions are in /task/data/validation/queries.jsonl and use the same record schema as hidden questions; /task/data/validation/golden_answers.jsonl supplies their reference answers and evidence document IDs. Use these papers to answer, without precomputed query-to-answer mappings, hidden labels, hard-coded evaluation answers, or external datasets or services containing evaluation answers. The held-out queries and private relevance judgments are disjoint from the development examples and unavailable in your environment. +The material to work from is a collection of 300 scientific-paper PDFs in /task/data/corpus. To check both the answers and the papers they point to during development, use the public questions in /task/data/validation/queries.jsonl and their reference answers and evidence document IDs in /task/data/validation/golden_answers.jsonl. Those public questions use the same record schema as hidden questions. Use these papers to answer, without precomputed query-to-answer mappings, hidden labels, hard-coded evaluation answers, or external datasets or services containing evaluation answers. The held-out queries and private relevance judgments are disjoint from the development examples and unavailable in your environment. Before implementing the system, read /task/docs/environment.md for the installed Python environment, runtime, packages and system tools, and /task/docs/available_resources.md for the optional retrieval and generative APIs, exact model allowlists, provider and endpoint restrictions, usage rules, runtime credential handling and jailbreak penalties. Follow those restrictions rather than assuming an unlisted service, model or endpoint is allowed; unauthorized datasets and services are prohibited. Harbor supplies the documented credentials at runtime, and they must never be put in the task package or Docker image. These documents and all of /task are read-only. Create or change submission files only under /app, and finish implementation, validation and debugging within the 120-minute agent allowance. -The final submission must include executable /app/build.sh and /app/run.sh files. You can organize optional modules under /app/src and leave optional implementation and self-test notes in /app/README.md. The preparation call is /app/build.sh --corpus /task/data/corpus --index-dir /app/index. Build all required indexes and runtime artifacts, start the search service and return exit status 0 only after it is ready for queries. It must stay available through repeated run.sh invocations. Use the supplied --index-dir for persistent indexes and other build artifacts, and honor supplied paths generally: the verifier's index and output directories may differ from the example locations. The build must finish within 3600 seconds. +To prepare the papers for repeated questions, the final submission must include executable /app/build.sh and /app/run.sh files. You can organize optional modules under /app/src and leave optional implementation and self-test notes in /app/README.md. The preparation call is /app/build.sh --corpus /task/data/corpus --index-dir /app/index. Build all required indexes and runtime artifacts, start the search service and return exit status 0 only after it is ready for queries. It must stay available through repeated run.sh invocations. Use the supplied --index-dir for persistent indexes and other build artifacts, and honor supplied paths generally: the verifier's index and output directories may differ from the example locations. The build must finish within 3600 seconds. After a successful build, the query interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Whatever internal protocol or service endpoint you use must work with the build process and this interface. Process every supplied query and write exactly one result object for each. The verifier builds once and then invokes run.sh once for each hidden query, with up to five invocations in parallel. They share the task's CPU, memory, index and service resources. Each query has 900 seconds, counted from the start of its run.sh process; waiting for an available slot does not use up that query's individual allowance. -For the output, write one JSON object per line in the form {"query_id":"query-id","answer":"The answer for the query","evidence":"document-id"}. Preserve each input query's identifier in query_id, provide a non-empty answer string that directly answers the question, and include evidence identifying the supporting document from the supplied corpus. The evidence value must be the PDF filename with its .pdf extension removed, so document-id.pdf is reported as document-id. Both an answer and a valid supporting corpus document ID are required for every query. +For the output, write one JSON object per line in the form {"query_id":"query-id","answer":"The answer for the query","evidence":"document-id"}. Preserve each input query's identifier in query_id, provide a non-empty answer string that directly answers the question, and include evidence identifying the supporting document from the supplied corpus. The evidence value must be the PDF filename with its .pdf extension removed, so document-id.pdf is reported as document-id. Both an answer and a valid supporting corpus document ID are required for every query, so each response can be traced back to a paper in the collection. Harbor will transfer /app into a separate verifier with the same runtime and the private split after you finish; your build.sh must restart the service there. The verifier phase has 14,400 seconds overall, including the arrangement of a 3,600-second build and 900 seconds per query with up to five concurrent queries. It checks retrieval and answer integrity, resource compliance, executable interfaces and output validity. A failed execution or invalid output gives score 0 to the entire submission, rather than just dropping the affected question. -The final metric is LLMJudgeAccuracy, equally averaged over every hidden query. For a query to earn 1, its evidence document must match the reference and an LLM judge must accept its answer as semantically equivalent to the reference answer; otherwise it earns 0. The answer judge only evaluates an answer after its evidence document matches. The normalized reward is the mean of those query values, and the displayed score is 100 times that reward. An independent trajectory audit also checks compliance with the task and resource restrictions; a failed audit makes the entire submission's reward and score 0. +The answer and its source are evaluated together using LLMJudgeAccuracy, equally averaged over every hidden query. For a query to earn 1, its evidence document must match the reference and an LLM judge must accept its answer as semantically equivalent to the reference answer; otherwise it earns 0. The answer judge only evaluates an answer after its evidence document matches. The normalized reward is the mean of those query values, and the displayed score is 100 times that reward. An independent trajectory audit also checks compliance with the task and resource restrictions; a failed audit makes the entire submission's reward and score 0. diff --git a/tasks/sparse-retrieval-pruning/instruction.md b/tasks/sparse-retrieval-pruning/instruction.md index f8e32e2..bc3a17c 100644 --- a/tasks/sparse-retrieval-pruning/instruction.md +++ b/tasks/sparse-retrieval-pruning/instruction.md @@ -1,6 +1,6 @@ -Optimize the supplied sparse retrieval system over its large corpus. Each document and query is represented by a set of opaque term IDs. The system must return relevant corpus documents while reducing total search wall time for the complete query workload. You may replace the starter index or retrieval algorithm as long as the executable interface and output contract are preserved. +I need to run a collection of searches over a large corpus and reduce the time spent waiting for the whole set of results. There is already a sparse retrieval system to start from, and both documents and queries are represented by sets of opaque term IDs. Please optimize that system so it returns relevant corpus documents with lower total search wall time for the complete query workload. You may replace the starter index or retrieval algorithm as long as the executable interface and output contract are preserved. -Quality comes first in this comparison. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the unpruned reference starter wall time measured by the verifier in the same environment. +The faster result still needs to retrieve the relevant documents. The submission must meet the held-out retrieval-quality requirements, after which reward increases as total hidden-query search wall time falls from 35 percent to 25 percent of the unpruned reference starter wall time measured by the verifier in the same environment. Please read /task/docs/environment.md for the installed runtime and resource limits, /task/docs/available_resources.md for the local dependency policy, and /task/docs/index_format.md before changing the starter index. That format is an implementation detail and may be replaced. All of /task is read-only, and submission files must only be created or modified under /app. The agent phase allows up to 120 minutes. Do not use precomputed hidden query-to-document mappings, hidden relevance judgments, or external datasets or services containing evaluation answers. The formal build and search run offline and must not need credentials, remote models, package downloads or external services. @@ -8,13 +8,13 @@ Implement all submission logic in Python source code. build.sh and run.sh may be Both the submission build and search path must be single-process and single-threaded. Do not use multiprocessing, worker pools, threading, joblib, Ray or parallel query execution. The verifier environment is limited to one CPU, so extra threads cannot gain more aggregate CPU throughput, but that CPU limit does not itself make parallel execution authorized or valid. -The full corpus is /task/data/corpus.jsonl. Public development material includes /task/data/validation/queries.jsonl for validation queries, /task/data/validation/qrels.tsv for relevance judgments, /task/data/validation/stats.json for term and document-frequency statistics, and /task/data/validation/reference_top100.jsonl for regression rankings. The formal hidden query split and its relevance judgments belong to the verifier. Document and query files are JSONL: a document looks like {"doc_id":"document-id","terms":["term-id", "term-id"]} and a query like {"query_id":"query-id","terms":["term-id", "term-id"]}. The repeated placeholder in those examples denotes the shape of the array; actual terms within a row are unique opaque strings. Their lexical value, apparent numeric value, input order and position do not encode relevance. +The collection to search is /task/data/corpus.jsonl. To check retrieval quality as you reduce search time, public development material includes /task/data/validation/queries.jsonl for validation queries, /task/data/validation/qrels.tsv for relevance judgments, /task/data/validation/stats.json for term and document-frequency statistics, and /task/data/validation/reference_top100.jsonl for regression rankings. The formal hidden query split and its relevance judgments belong to the verifier. Document and query files are JSONL: a document looks like {"doc_id":"document-id","terms":["term-id", "term-id"]} and a query like {"query_id":"query-id","terms":["term-id", "term-id"]}. The repeated placeholder in those examples denotes the shape of the array; actual terms within a row are unique opaque strings. Their lexical value, apparent numeric value, input order and position do not encode relevance. Provide executable /app/build.sh and /app/run.sh files in the final submission. Optional Python modules may live in /app/src and optional notes in /app/README.md. The verifier builds with /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /path/to/index. Build retrieval state from the supplied corpus and create every persistent index and runtime artifact beneath that supplied --index-dir. Return status 0 only when the index is ready. The build may use public statistics next to the corpus, but it must not require additional undeclared paths or undeclared persistent files from the verifier. -Once the build succeeds, the search interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Required runtime state must come from the supplied index directory and query file. Process each supplied query exactly once, preserve input query order, and write one result object per query. A JSONL output line looks like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. query_id must identify its corresponding input query. Return at most 100 results, with unique doc_id values that occur in the supplied corpus and scores that are finite and numeric. Order results by descending score, resolving equal scores by ascending lexicographic order of the opaque document ID. Fewer than 100 results is valid, though it can reduce Recall@100. +Once the build succeeds, the search interface is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. The search run must get all of its required runtime state from the supplied index directory and query file. Process each supplied query exactly once, preserve input query order, and write one result object per query. A JSONL output line looks like {"query_id":"query-id","results":[{"doc_id":"document-id","score":0.123}]}. query_id must identify its corresponding input query. Return at most 100 results, with unique doc_id values that occur in the supplied corpus and scores that are finite and numeric. Order results by descending score, resolving equal scores by ascending lexicographic order of the opaque document ID. Fewer than 100 results is valid, though it can reduce Recall@100. -The latency baseline used by the verifier is the unpruned reference starter. It retains every corpus term and posting, applies no DF, prefix or other posting-list pruning by default, uses the set-valued corpus and query inputs without weights, builds a consistent dictionary, metadata and posting layout, and performs unpruned binary/IDF-overlap search. Any optional pruning flag in starter code is only for explicit local experiments and is not enabled for the verifier's baseline. +The speed improvement will be measured against the unpruned reference starter used by the verifier as its latency baseline. It retains every corpus term and posting, applies no DF, prefix or other posting-list pruning by default, uses the set-valued corpus and query inputs without weights, builds a consistent dictionary, metadata and posting layout, and performs unpruned binary/IDF-overlap search. Any optional pruning flag in starter code is only for explicit local experiments and is not enabled for the verifier's baseline. For evaluation, the verifier rebuilds your submission from the full corpus and runs it on a private query split. It checks retrieval integrity, executable behavior, valid output, retrieval quality and search wall time. Hidden results must achieve NDCG@10 of at least 0.89 and qrels Recall@100 of at least 0.99. Candidate and reference wall time are measured by external process wrappers under the same workload and resource allocation. The latency ratio is candidate wall time divided by that verifier-measured starter wall time.