Investigate parallelizing `test_examples.py` across CI runners

[lrandersson]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

test_examples.py currently runs as a single serial pytest step per OS/Python/conda-standalone matrix entry in main.yml. Investigate splitting it across multiple parallel jobs/runners per OS, and decide on an approach.

Options to look into:

• pytest-split (available on conda-forge) — balances test groups by recorded duration; needs a --durations-based baseline, or falls back to even count-splitting.
• Manual -k/marker-based grouping — no new dependency; tests grouped by hand into buckets and run separately per shard via -k.

Note: some tests touch OS-global state (Windows registry, Start Menu, macOS keychain, a hardcoded C:/ProgramData/conda/.condarc, fixed Docker image tags). Running shards as separate CI jobs/runners avoids collisions from this; running them concurrently on one machine (e.g. pytest-xdist -n auto in a single job) would not, without further changes.

Why is this needed?

Reduce test times, Illustrative arithmetic (current full-run time ÷ N, no overhead accounted for):

OS	Full run	÷2	÷3	÷4
Windows	~100 min	~50 min	~33 min	~25 min
macOS	~85 min	~43 min	~28 min	~21 min
Linux	~30 min	~15 min	~10 min	~8 min

What should happen?

Examples split into multiple runners, reducing overall testing time.

Additional Context

Decisions needed

1. Which mechanism to use (pytest-split vs. manual grouping).
2. Number of shards per OS.
3. Whether shard boundaries should be based on measured per-test timing (pytest --durations=0) rather than a manual guess.
4. How to handle steps that assume one test run per matrix entry: coverage upload (Codecov integration flag), the installers-* artifact upload, and the check-docs-schema step.

Notes:

• pytest-xdist-style same-machine parallelization should not be attempted without first auditing tests that mutate shared OS-global state (e.g. test_initialization registry writes, conftest.py keychain mutation, test_uninstallation_standalone's hardcoded .condarc path, test_docker_image_build's fixed image tag).
• This issue is for deciding shard count and split mechanism; implementation should follow in a separate PR.

[lrandersson]

I explored a few approaches for splitting the CI test runners:

What I tried:

1. Split by installer type with -k filtering - Duplicate each Windows/macOS matrix entry with an installer-type (exe/msi/sh/pkg), derive the pytest -k filter from it. Works well, no new deps, keeps construct.yaml as source of truth. Downside: relies on naming convention (don't use msi/exe/pkg/sh in test names). Run test examples in more parallel instances #1310
2. pytest-split plugin - Split into N groups by test duration. Balances runtime but adds a dependency. Split test examples using pytest-split and improve test clean-up #1312 (can also be applied to macOS but ignored for now for testing purposes).

Considered but I didn't try:
3. Separate test files per installer type - Most structural but requires significant refactoring and likely code duplication.
4. Auto-apply pytest markers via conftest hook - Cleaner than -k substring matching but moves config away from construct.yaml.
5. A new config file with test groups - Centralized but adds maintenance burden when adding/removing tests.

[marcoesters]

I don't think an additional test dependency is a big deal here. conda is using pytest-split. What are the benchmarks?

[lrandersson]

I don't think an additional test dependency is a big deal here. conda is using pytest-split. What are the benchmarks?

@marcoesters as this PR is in its current state, I ran the tests four times and logged all the times and compared it to a run from master, these are the results (as of right now):
CI wall-clock time (i.e. how long until the slowest job finishes) across 4 sample PR runs vs. 1 baseline run on main:

Slowest job's wall-clock time, before vs. after splitting tests with pytest-split (4 sample PR runs averaged):

Platform	Before	After	Savings
Windows	105m	~51m	~51% faster
macOS	90m	~51m	~43% faster
Ubuntu (not split)	...	...	no change