Equivalent of TechEmpower full stack inclusive of database workload

[jzabroski]

TechEmpower had benchmark like Fortunes that also included the database networking stack. For example, this would show the benefits of tokio-postgres or postgres driver in Rust versus the Pgsql nuget package in .NET.

I'm actually genuinely interested to see this, as currently HttpArena makes C# seem God Tier for real world applications. As a C# application developer, I would love for that to be true, but have to be a skeptic first. If there was a port of the TechEmpower Fortunes benchmark, this would help a lot.

I think these will also matter more in the future as people will see the cumulative penalty of slower stacks.

[MDA2AV]

There is a postgres (async-db) and fortunes test in HTTP/1.1

we also had a crud one which was quite more complex but very difficult to balance between technologies. The reason why database test is unscored is because this benchmark aims to compare only framework capabilities, postgres driver is something external

[jzabroski]

We might be talking about slightly different things, so forgive me if this reply flies past your points above:

1. TechEmpower had deeper insight into real world tech stacks, because it included data access layer libraries on top of database drivers. For example, in .NET, Dapper and EntityFrameworkCore were both a part of Fortunes.
2. While you can look at micro-benchmark frameworks in individual languages today, like Frans Bouma's ORM benchmark for .NET, and infer how that would apply to your particular favorite language, TechEmpower further allowed judging across language ecosystems how entire tech stack choices played out.

The reason why database test is unscored is because this benchmark aims to compare only framework capabilities, postgres driver is something external

Is that because this is called http-arena? Was not sure if you were saying my request is out of scope, and that the postgres piece is just meant as a simple control arm. I agree Postgres is generally a good database engine, as a control arm, to target because it supports query pipelines across its various language-specific drivers, like tokio-postgres and Npgsql. Other engines like SQL Server have proprietary communication protocols that are synchronous in nature, which risks measuring blocking I/O overhead rather than CPU instructions throughput (perf_event_open).

[MDA2AV]

I personally find techempower's fortunes test to not be a good example of a benchmark, from the database network stack to the fortune rendering there were no clear rules and the results were meaningless as some frameworks used a pg driver, others used ORMs, some pre rendered, others used a rendering engine.

I think the problem is that as C# developers we have a very opinionated way of doing certain things when it comes to building REST applications(and I am guilty of that when designing some of the benchmarks we have here), object mappers and threadpool continuations async await being two of them. Unfortunately while a EF Core or ORM benchmark can make a lot of sense in a C# only or Java scenario, not every technology or runtime fits in that box, let's look for example at the current fastest C# framework genhttp-ioxide, its runtime is not optimized for Dapper or EF Core but its very own pg driver which will always be way faster than aspnet with an object mapper. Other technologies have their very own opinionated way on doing database networking that has nothing to do with C#'s, so how can we create a benchmark test that doesn't compare apples to oranges here?