Split test examples using pytest-split and improve test clean-up - #1312
Split test examples using pytest-split and improve test clean-up#1312lrandersson wants to merge 10 commits into
Conversation
conda-bot
added
the
cla-signed
| "examples", | ||
| ] | ||
| addopts = [ | ||
| "--store-durations", # pytest-split, update duration cache |
There was a problem hiding this comment.
7194cca to
379a130
Compare
8e3ee06 to
f8d220a
Compare
| exit 1 | ||
| fi | ||
|
|
||
| # mamba occasionally fails to load libmamba.2.dylib on macOS CI right after |
There was a problem hiding this comment.
All the changes in this file is to see if it resolves the intermittent issue:
INFO:constructor.main:Successfully created '/private/var/folders/df/djsxfhc17x95674wsm_g8s980000gn/T/pytest-of-runner/pytest-1/test_example_miniforge_mamba2_0/installer/Miniforge3-mamba2-25.1.1-0-MacOSX-arm64.sh'.
dyld[31994]: Library not loaded: @rpath/libmamba.2.dylib
Referenced from: <A07BA708-81EE-3434-9A5E-50BBA7FCC449> /private/var/folders/df/djsxfhc17x95674wsm_g8s980000gn/T/pytest-of-runner/pytest-1/test_example_miniforge_mamba2_0/install/batch/pkgs/mamba-2.0.8-h105ca85_2/bin/mamba
Reason: tried: '/private/var/folders/df/djsxfhc17x95674wsm_g8s980000gn/T/pytest-of-runner/pytest-1/test_example_miniforge_mamba2_0/install/batch/pkgs/mamba-2.0.8-h105ca85_2/bin/../lib/libmamba.2.dylib' (no such file), '/private/var/folders/df/djsxfhc17x95674wsm_g8s980000gn/T/pytest-of-runner/pytest-1/test_example_miniforge_mamba2_0/install/batch/pkgs/mamba-2.0.8-h105ca85_2/bin/../lib/libmamba.2.dylib' (no such file), '/usr/local/lib/libmamba.2.dylib' (no such file), '/usr/lib/libmamba.2.dylib' (no such file, not in dyld cache)
/private/var/folders/df/djsxfhc17x95674wsm_g8s980000gn/T/pytest-of-runner/pytest-1/test_example_miniforge_mamba2_0/install/batch/pkgs/post_install.sh: line 48: 31994 Abort trap: 6 mamba info
ERROR: executing post_install.sh failed
When debugging it seems like mamba --version passes, but mamba info directly after fails. Therefore, perhaps this will help mitigate the flakiness of this test.
lrandersson
changed the title
| jobs: | ||
| tests: | ||
| name: ${{ matrix.os }}, Python ${{ matrix.python-version }}, ${{ matrix.conda-standalone }} | ||
| name: ${{ matrix.os }}, Python ${{ matrix.python-version }}, ${{ matrix.conda-standalone }}${{ matrix.split-group && format(', group {0}/3', matrix.split-group) || '' }} |
There was a problem hiding this comment.
Since you are only deploying two groups for macOS, hard-coding 3 makes it looks like tests are missing.
There was a problem hiding this comment.
Good catch (see below)
| conda-standalone: conda-standalone-onedir | ||
| check-docs-schema: true | ||
| # MACOS | ||
| # MACOS (split into 2 groups per Python version) |
There was a problem hiding this comment.
Why split macOS in two groups? Because of the queue? This matrix is hard to manage with every scenario hard-coded.
There was a problem hiding this comment.
Originally I wasn't planning on modifying macOS, but then I did anyway for testing purposes, and then I did 2 groups instead of 3 because we always have to sit and wait for the macOS runner queue.
I've now split it into 3 groups: 592f6cf
I'm also fine just leaving macOS as is (no split at all) since the main bottle neck is Windows, but I still think doing something about macOS would be good for us since sometimes the test duration reach almost 90 minutes.
There was a problem hiding this comment.
I agree that we should split macOS. I don't mind it being split in two necessarily as long as we don't introduce coverage gaps. Runner availability is a real concern for sure.
3 participants
Description
This PR tests a different approach to solve #1306, related PR with another approach in #1310.
In this PR we use
pytest-splitand simply separates the Windows test examples into 3 groups and macOS into 2 groups.https://github.com/jerry-git/pytest-split/tree/master
I also expanded the scope to include some test cleanup to minimize the chances of more intermittent issues with memory/disk-space related CI issues. Also removed the artifact archiving since it seems not to be used.
Benchmarks:
#1306 (comment)
Checklist - did you ...
newsdirectory (using the template) for the next release's release notes?