How to config the network generation so that it approaches the performance of the real Tor network?

[higher-order]

I noticed that for smaller networks, such as 5% ones, its performance in the simulation is way better than the real Tor network. From the plots, their goodput lines are a lot further on the right, while their error rate lines are a lot further on the left (close to 0%), compared to the "Public Tor" line. This is shown in the attached PDF. "Original" is a network that is simulated as is after generation, and "sbws" is a network that additionally has sbws running.

tornet.plot.pages.pdf

Therefore I assume the generated 5% network doesn't reflect real Tor's performance. I thought maybe I should change the parameters to tornettools generate to increase the number of clients. This is because although the number of relays is indeed 5% of what's in the real world, the number of clients seems to be much fewer than 5% of the millions of clients in the real world. But there are quite a few parameters, such as network scale, process scale, server scale, load scale, etc. I reviewed the Once is Never Enough paper to see what are the parameters used there. But in section 4.3, it says a process scale of 0.01 is used, which is the current default. It's just that the network scale is 31%. And the plotted lines are pretty close to the real Tor's lines.

So now I'm wondering if I should change the params for 5% networks at all so that their lines approach real Tor's, or are the params already optimal and changing them will actually deviate the simulation from reality? If I do need to change, which value should I pick for which param?

[robgjansen]

Well, you've already deviated from reality quite a bit by downsampling to 5%. Think about that in a more extreme case: if you chose a 0.000000001% network, there is no magic combination of parameters that will make it match public Tor performance. Similarly, it is unlikely you will have a perfect match on all of the graphs when you're using a 5% network.

Anyway, with that in mind, what you are aiming for is "the best we can do given the constraint of a 5% network". That might get kinda close to public Tor, but probably not in all metrics.

I think the main parameter you want to play around with is --load_scale (for tornettools generate). Increasing it means more simultaneous transfers, which will generally make download times increase (slow down) and increase relay goodput (the perf CDFs move to the right). Decreasing it means fewer simultaneous transfers, which will generally make download times decrease (speed up) and decrease relay goodput (the perf CDFs move to the left).

I'm not sure that you should be unsatisfied with the performance you're getting without tweaking the parameters. You're in a 5% network, meaning you are far from real Tor network characteristics. That's not necessarily a bad thing, it is a different type of network, but you can still compare what performance would be like within that type of network. But if you want to try to get a little closer, then look at the --load_scale.

***NOTE: the parameters in the Never Enough paper were "optimized" not only for a given network size as you mentioned, but also based on what the Tor network looked like at that point in time (we used consensus/relay data from a specific time period). If you are now using consensus/relay data from a more recent network/time period, that means many things have shifted (e.g., relay locations, bandwidth distribution, etc.). So adjustments could be warranted.

[higher-order]

@robgjansen Thank you for your response. I'm confused about two things:

1. It is a little counter-intuitive to me when you said increasing the load scale will slow down the download but increase the goodput. Wouldn't slower download result in lower goodput?
2. My current understanding leads me to think it makes more sense to increase process scale rather than load scale. I checked the generation log of the 5% network, and it says "We will use 376 TGen client processes to emulate 37617.15 Tor exit users". But from the Tor metrics site, it seems like on average there are 4 million users. 5% of that would be 200K, which is about 5 times of 37617.15. Doesn't that mean proportionally, there are fewer clients in the generated network and I should add more by dividing the default process scale by 5?

[higher-order]

@robgjansen Some curious result: I divided process scale by 5 so that 5x clients are simulated. The simulation took much longer, but the goodput was very similar. #0858 is the one with 5x clients.

tornet.plot.pages.pdf

[robgjansen]

1. Increasing load_scale means the clients will attempt more downloads at the same time, meaning we are pushing even more bytes through the network up to the relays' maximum throughputs. The relays goodput will tend toward the maximum amount allowed by the Tor rate limits or the Shadow link rate limits. There are more downloads happening at the same time, and more congestion in Tor, and that means each single download will get slower.

2. Your understanding of process_scale is not correct. For performance and memory reasons, each Tor "user" that we simulate does not get its own Tor client process in the simulation. Most "users" are idle for long times, and running an idle Tor client process takes a lot of memory. Thus, we have a workaround where we use a smaller number of client processes to run all of the downloads from multiple "users". This reduces the overall memory footprint of the simulation, with minimal adverse effects on the simulated performance. When you change the process scale, the total number of downloads does not change - it just means we use a different number of Tor clients to perform those downloads. This exactly explains the results you are seeing, similar goodput but a longer simulation - it takes more time to run 5x as many Tor processes.

So indeed, if you want to push more bytes through the network to make it more congested and thus increase download times, you need to increase the density of the downloads that are happening in parallel. To do that you increase load_scale.