sb-cli false negatives for tests that are skipped for one-core machines

[klieret]

Note

This should only affect evaluation with sb-cli

Affected Instance

pylint-dev__pylint-6528

Description

The sb-cli evaluation marks pylint-dev__pylint-6528 as resolved: false even though all 4 FAIL_TO_PASS tests pass. The failure is caused by 4 PASS_TO_PASS tests being SKIPPED (not failed) in the Docker environment, which the evaluation framework then counts as regressions.

Root Cause

The Docker container used by sb-cli has only 1 CPU core. Four tests in the PASS_TO_PASS list have a skip condition requiring 2+ cores:

SKIPPED [1] tests/lint/unittest_lint.py:774: Need 2 or more cores for test to be meaningful
SKIPPED [2] tests/lint/unittest_lint.py:803: Need 2 or more cores for test to be meaningful
SKIPPED [1] tests/test_self.py:1050: Need 2 or more cores for test to be meaningful

The gold patch also changes _discover_files from @staticmethod to instance method (same as the submitted patch), so these tests would be skipped for the gold patch as well in the same environment. This means the instance is unsolvable in a single-core environment — the gold patch itself would also be marked resolved: false.

Additional Notes

• The overall pytest result line confirms: 2 failed, 171 passed, 4 skipped, 1 xfailed
• The 2 actual failures (test_generate_toml_config, test_generate_toml_config_disable_symbolic_names) are caused by a pre-existing tomlkit version incompatibility in the Docker image and are not in the PASS_TO_PASS list.

[FazeelUsmani]

Hi klieret
I think the root cause is slightly different from the issue description. It's a parser bug, not just a grading-semantics issue.

In swebench/harness/log_parsers/python.py, the pytest parser does line.split()[1] to get the test node ID. That works for PASSED tests/foo.py::test_bar, but pytest's SKIPPED summary format is:

SKIPPED [1] tests/lint/unittest_lint.py:774: Need 2 or more cores for test to be meaningful

So split()[1] returns "[1]". The status map gets polluted with garbage keys and the real node IDs never appear. test_failed() then sees the P2P tests as missing from the map and defaults to failed → false regression.

A few ways to fix it:

1. Parser only - handle the SKIPPED format. Doesn't fully fix this bug on its own: the summary line doesn't contain the node ID, so the test would still be missing from the map.
2. Grading only - treat "missing from map" as success for P2P. Fixes the symptom but hides real regressions (e.g. test collection failures, deleted tests).
3. Both - parse SKIPPED node IDs from pytest's verbose section earlier in the log (where the full ID is present), then update P2P grading to treat SKIPPED as success. The grading change becomes narrow and safe once the parser is correct.

I'd go with (3), but (1) alone is a valid smaller-scope option if you'd rather handle grading semantics separately. Either way I'd add pylint-dev__pylint-6528 as a regression fixture.

Let me know which approach seems reasonable to you, I will open a PR.