PYR1-1773 Delete GWC layers one at a time during workspace removal

[danielhvs]

Purpose

Delete GWC tile layers one at a time during workspace removal. GeoServer's DiskQuota store bills every delete to a single ___GLOBAL_QUOTA___ row inside a SERIALIZABLE transaction, so concurrent deletes conflict and Postgres aborts the losers with could not serialize access due to concurrent update.

1. New delete-cached-layers! issues the deletes sequentially and still treats 404 as success, as before.
2. remove-workspace! calls it in place of make-parallel-rest-requests.
3. The add path (core.clj:738) keeps its parallel requests - registration throughput depends on them.

Why now

PYR1-1745 moved the DiskQuota store from embedded H2/HSQL to PostgreSQL. The contention already existed; H2 takes a table lock and blocks, Postgres aborts immediately. Counts over the same 8-day window:

1. geoserver-utility-dev (PostgreSQL) - 14,248
2. geoserver-west (H2) - 18, as CannotAcquireLockException: Timeout trying to lock table TILESET
3. geoserver-utility prod (HSQL) - 0

JDBCQuotaStore's constructor hardcodes setIsolationLevel(8) = ISOLATION_SERIALIZABLE, verified by disassembling gwc-diskquota-jdbc-1.28.5.jar. There is no configuration knob for it, so the fix has to be on the caller side.

Related Issues

Closes PYR1-1773

Testing

clojure -M:test-runner - 6 tests, 27 assertions, 0 failures.

The new test redefines make-rest-request to track in-flight requests and asserts peak concurrency of 1 across 20 deletes. Confirmed it has teeth: swapping the mapv for pmap fails it with (not (= 1 11)).

clj-kondo --lint src test - 0 errors, 0 warnings.

Not yet exercised against a live GeoServer; the acceptance check is a dev workspace removal producing zero could not serialize access entries.

Trade-off

Removal gets slower in proportion to layer count - parrot04 carries ~18k gwc-layers. Sequential deletes are the cost of not having a retry loop; if removal wall time becomes a problem, bounded concurrency plus retry-on-serialization-failure is the next step.

[jjt-sig]

LGTM

[danielhvs]

Verified on geoserver-utility-dev with jar geosync-2026.09.22-24d694c-standalone.jar. Reviewed and analyzed on my behalf in collaboration with Claude Code.

Removal - zero new serialization errors

GeoSync	Before	After	New
04-parrot-risk	16835	16835	0
05-parrot-weather	114	114	0
06-parrot-planning	803	803	0

parrot05, two workspaces on port 31338:

18:57:59 Processing Request: {:action "remove", ... nfdrs-constant_20260922_12}
18:58:01 15 cached layers were removed from GeoWebCache.
18:58:01 44 GeoFence data rules were deleted.
18:58:01 Processing Request: {:action "remove", ... nfdrs-variable_20260922_12}
18:58:02 15 cached layers were removed from GeoWebCache.
18:58:02 44 GeoFence data rules were deleted.

parrot04, one 328-layer workspace on port 31337:

18:58:44 Processing Request: {:action "remove", ... fire-risk-forecast_anza_20260922_00}
18:59:11 325 cached layers were removed from GeoWebCache.
18:59:11 1 GeoFence data rules were deleted.

None of the three logged Errors encountered during workspace removal. The last such line is 18:49:24, before this jar went live. Workspace dirs are gone; geoserver04's gwc-layers dropped from 18,082 to 13,120.

Cost

325 sequential deletes took 27s, about 83ms each. The 15-layer weather workspaces took 1-2s. Acceptable at current layer counts, and the bounded-concurrency-plus-retry option stays open if it stops being.

Re-registration

herd trigger sync-fire-weather-forecast brought all five parrot05 workspaces back, including fire-weather-forecast_adswrf_20260922_00.

Three Errors encountered during layer **registration** lines showed up during the re-register. That is the add path, which this PR leaves parallel and untouched. Cause is {:status 500, :body "Index 0 out of bounds for length 0"} and it is pre-existing - 19 occurrences today starting 00:44:00, well before this jar. Tracking separately.