Extend nat-lab.sh for natlab-subset sharding - #1937
Conversation
489a30d to
86ab3a6
Compare
b3444ad to
35ec855
Compare
35ec855 to
86ab3a6
Compare
86ab3a6 to
4baa9da
Compare
4baa9da to
77a8067
Compare
86c8464 to
8c78ece
Compare
8c78ece to
21b9913
Compare
|
|
||
| # CI load balancing only - see the shard_* markers in pyproject.toml. These tests need | ||
| # nothing beyond the plain docker services. | ||
| pytestmark = pytest.mark.shard_windows |
There was a problem hiding this comment.
Does this mean that the tests will run only on Windows container? 😕 This doesn't seem right since nordvpnlite was never intended to run on Windows
There was a problem hiding this comment.
Nah, it only will run on Windows shards, it will use the pure docker containers only (i.e. even if this test file contains some tests using additional containers, it is always used with DOCKER_ONLY, so only these using the basic containers will run)
There was a problem hiding this comment.
Could we document somehow more explicitly what Windows shards entail and how Windows shards is different from regular shard?
Reading it for the first time gives the impression that only Windows containers / VMs are started, and/or that only Windows tests are run in this shard. Does the Windows shard start all the regular natlab containers + Windows VMs?
I wonder if there is a more "elegant way" to assign the shards, personally find it puzzling at first glance to see mark windows inside of nordvpnlite.py.
|
|
||
| # CI load balancing only - see the shard_* markers in pyproject.toml. The windows/mac | ||
| # params below keep their own marks, so they still run on their own shards. | ||
| pytestmark = pytest.mark.shard_openwrt |
There was a problem hiding this comment.
I'm confused, does this includes or excludes the tests from OpenWRT?
If the former, why would we want to test UPNP only on OpenWRT?
There was a problem hiding this comment.
windows and mac tests are excluded from this load balancing, because we always use it with DOCKER_ONLY filter in CI config, so only these using basic containers are under this mark and these will be moved to openwrt shard to better balance the tests between them
|
|
||
| # CI load balancing only - see the shard_* markers in pyproject.toml. These tests need | ||
| # nothing beyond the plain docker services. | ||
| pytestmark = pytest.mark.shard_mac |
There was a problem hiding this comment.
These tests need nothing beyond the plain docker services.
Is this true for the macOS VM though?
Also, how does this interact with the mac shard, none of the tests below specify ConnectionTag.VM_MAC?
There was a problem hiding this comment.
The tests don't need anything more than lite-mode, but because we have a bit too much litemode tests, it makes sense to move them to other shards to balance it a bit
There was a problem hiding this comment.
I see so this just manually "assigns" the test to the shard that runs the macOS tests?
What happens if I accidentally mark test_openwrt.py with shard_mac?
d8c6d1e to
86273da
Compare
86273da to
3b0b2b9
Compare
3b0b2b9 to
cce3445
Compare
cce3445 to
0abdfba
Compare
895137b to
739d891
Compare
Using the given JSON description of natlab shards and known test durations determine the test split and use the proper test set for the considered shard in conftest.
27f12b9 to
3082347
Compare
Currently the test success is dependent on when it is run among the other tests - it should be able to be run as the first test, though, so in this commit we add some robustness to it.
3082347 to
f0bf4e2
Compare
3 participants
Problem
Every nat-lab shard is quite heavy running the whole nat-lab
Solution
Let's add additional options for running nat-lab, so we can divide tests into shards which run only particular subset of all nat-lab containers, thus reducing the overall load
☑️ Definition of Done checklist