use quarkus managed datasources and multi-datasource support at runtime

[barreiro]

Summary:

• Switch from manual datasource management to Quarkus-managed datasources, removing the custom DatasourceConfiguration and DuckDbDatasourceConfiguration classes that
  manually created Agroal connection pools. Datasource configuration is now handled entirely through application.properties, enabling Quarkus's built-in multi-datasource
  support and dev services (e.g. automatic PostgreSQL provisioning in dev/test).
• Drop DuckDB dependency. It was unused and only configured in the removed DuckDbDatasourceConfiguration.
• Migrate all entities from PanacheEntity to PanacheEntityBase with explicit @Id and @GeneratedValue(strategy = IDENTITY), which is required for SQLite compatibility (SQLite
  does not support PostgreSQL-style sequences).
• Add SQLite connection initializer (SqliteConnectionInitializer) using Agroal's AgroalPoolInterceptor to set WAL mode, synchronous=normal, and busy_timeout pragmas on each
  new SQLite connection, replacing the old import.sql approach.
• Enable work queue retries — WorkService.RETRY_LIMIT changed from 0 to 3, with error logging downgraded from error to debug for transient failures.
• Add retry logic for concurrent uploads on SQLite — FolderService.upload() now retries up to 3 times on PessimisticLockException (SQLITE_BUSY). The CLI upload path also
  collects futures and waits for all processing to complete before exiting.
• Drop Hibernate JDBC statement batching (statement-batch-size, order-inserts, order-updates) — the codebase uses already uses persist(), so batching was never actually taking effect. With GenerationType.IDENTITY, it forces Hibernate to flush each INSERT immediately to retrieve the generated ID.
• CI matrix for dual-database testing — both ci.yml and pull-request.yml now run the test suite against PostgreSQL and SQLite in parallel using a build matrix (-Dsqlite
  Maven profile).
• Unify test cleanup — FreshDb.dropRows() uses a single code path (DELETE statements) for both databases, with proper FK-safe ordering and transaction rollback handling. The
  CliProfile now provides a fixed test DB path directly.

[stalep]

One suggestion regarding the GenerationType.IDENTITY change: since we're on Hibernate 7 (7.3.2 via Quarkus 3.35.1), there's a @NativeGenerator annotation that gives us the best of both worlds:

@Id
@GeneratedValue
@org.hibernate.annotations.NativeGenerator
public Long id;

This tells Hibernate to use the dialect-native strategy:

• PostgreSQL: SEQUENCE (native sequences, enables batch inserts with statement-batch-size=100)
• SQLite: IDENTITY (fast native autoincrement, zero contention)

I tested locally — 342 tests pass on both PostgreSQL and SQLite with @NativeGenerator. The batching config (statement-batch-size, order-inserts, order-updates) can be re-added since it only takes effect when SEQUENCE is active.

With IDENTITY (current PR), Hibernate must flush after every INSERT to retrieve the generated ID, which disables batch inserts on PostgreSQL. Hibernate was batch-inserting (the BatchImpl.execute() stack traces confirm it), so this is a meaningful performance regression for PostgreSQL.

@NativeGenerator avoids this by using SEQUENCE on PG (pre-allocates 50 IDs, enables batching) while still using IDENTITY on SQLite.

[barreiro]

thanks @stalep, well spotted !!

I took your suggestion for ValueEntity which is the only that really benefits from batch inserts.

[willr3]

Does this change result in a single cli jar (target/cli/h5m.jar) that can work with either sqlite or postgresql based on changing some command line parameter? I'm concerned about removing the DatasourceConfiguration.getPath as that is what allowed for environment variable control of the local sqlite database location using H5M_PATH. Why did we need to replace the DtasourceConfiguration?

[willr3]

Migrate all entities from PanacheEntity to PanacheEntityBase with explicit @id and @GeneratedValue(strategy = IDENTITY), which is required for SQLite compatibility (SQLite
does not support PostgreSQL-style sequences).

The sqlite integration was working prior to this PR, what change is requiring the @GeneratedValue(strategy = IDENTITY) or was something not working properly prior to adding that annotation?

[barreiro]

The sqlite integration was working prior to this PR, what change is requiring the @GeneratedValue(strategy = IDENTITY) or was something not working properly prior to adding that annotation?

there is a problem with the sqlite integration: it was not relying on JTA transactions. when adding transactions there are deadlocks caused by increasing the sequences (hibernate does that in it's own tx)

to avoid that deadloak, we change the id genation to identity that never spawn a new tx. this is something that we already discussed, because sequences can cause "holes" in the ids when the server is restarted

the only exception is ValueEntity because it can have a big number of inserts and we want to batch those. that entity now uses NativeGenerator which uses IDENTITY for sqlite and SEQUENCE for postgeSQL

[barreiro]

after rebasing to current main branch, some more changes were needed:

• FolderService.upload(): some changes on how the task is handled in terms of transactions and how the Future is completed from it
• ProcessingService.recalculate(): same thing as above
• WorkService.runInNewTransaction(): added this method to encapsulate retry-able work, without replicating that logic across the code base
• TEST_DB_PATH: use random tmp folder
• WorkQueueRaceTest.rapidUploadsShouldProduceAllValues(): skip this test on sqlite as it fails intermittently after exhausting the default 3 retires