sb-cli cloud evaluation marks all SWE-bench Lite test submissions as failed, while local official eval resolves same predictions

[lnotgy123]

Describe the issue

Hi SWE-bench team,

I submitted predictions to SWE-bench Lite test via sb-cli, but every submitted instance is marked as failed run, not unresolved.

The same predictions can be evaluated successfully with the local official SWE-bench harness.

Environment:

• sb-cli: 0.1.5
• swebench: 4.1.0
• subset/split: swe-bench_lite test

Cloud run:

• run_id: qwen25-coder-7b-swesmith-stage2-v2-smoke5-dict-retry-20260630033410
• submitted_instances: 5
• completed_instances / successful runs: 0
• failed_runs: 5
• resolved_instances: 0
• error_instances: 0

Failed IDs:

• django__django-10914
• django__django-11039
• django__django-11099
• django__django-11133
• django__django-11179

Local official eval on the same 5 predictions:

• completed_instances: 5
• resolved_instances: 5
• error_instances: 0

Prediction format follows the documented dict format:

{
"django__django-10914": {
"model_patch": "...",
"model_name_or_path": "qwen2.5-coder-7b-swesmith-stage2-v2"
}
}

Could you help check why the cloud runner reports all submitted instances as failed runs?

Suggest an improvement to documentation

No response

[Sehlani042]

I can help narrow this down, but I do not think the public details here are enough to distinguish a cloud runner problem from a submission-format/reporting problem yet.

One thing worth checking first: sb-cli accepts JSON dict/list input, while the local harness docs use JSONL rows with instance_id, model_name_or_path, and model_patch. Since this run used the dict-style input, the useful minimal repro bundle would be:

1. the exact 5-instance predictions file passed to sb-cli, with full model_patch values
2. the exact submit command/stdout, e.g.
   sb-cli submit swe-bench_lite test --predictions_path ... --run_id qwen25-coder-7b-swesmith-stage2-v2-smoke5-dict-retry-20260630033410
3. the report artifacts from:
   sb-cli get-report swe-bench_lite test qwen25-coder-7b-swesmith-stage2-v2-smoke5-dict-retry-20260630033410
4. the local harness command plus results.json / instance_results.jsonl for the same 5 predictions
5. one failed instance log from cloud if the report includes per-instance stderr/log details

With those, the first comparison can be very small: verify whether the cloud API received the same five model_patch strings that the local harness evaluated, then compare one instance result path such as django__django-10914 before looking at broader runner behavior.