Repository navigation
renderd: fix PostGIS memory growth with parameterize_style - #502
Conversation
…ry leak When parameterize_style is configured, render() previously created a full copy of the Mapnik Map object and called set_datasource() on each affected layer for every single tile render. Each set_datasource() call creates a new PostGIS datasource, which registers a new connection pool entry in Mapnik's global ConnectionManager singleton. These pools were never released, causing renderd to exhaust memory rapidly under load. SQLite-backed styles were unaffected because the SQLite datasource has no global connection pool. Fix: add a std::map<string, mapnik::Map> parameterized_map_cache to xmlmapconfig. On the first render for a given options string the parameterized Map is built as before and then moved into the cache. Subsequent renders with the same options reuse the cached Map directly, calling set_datasource() only once per (thread, style, options) combination. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #502 +/- ##
==========================================
- Coverage 81.54% 81.01% -0.53%
==========================================
Files 24 24
Lines 5645 5267 -378
==========================================
- Hits 4603 4267 -336
+ Misses 1042 1000 -42 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for merging this and #509! For what it's worth, OSM UK's tile server has been running the complete series in production, rendering a PostGIS-backed style (24 million HM Land Registry INSPIRE polygons). Under repeated bursts of 1,000 concurrent requests, renderd's memory settles at about 90 MB and stays there, where the old packaged version used to grow until the server ran out of memory. The remaining PRs (#503, #504, #506–#508, #510–#513) all still merge cleanly onto the current |
|
Thank you again very much, I will be looking over the remaining pull requests and commenting where needed. |
|
Update: OSM UK have now moved their live tile server onto the machine running this series of fixes (mod_tile/renderd built from my branch), and everything is working. There was a certificate issue during the switchover, but that was down to the IP address change, not mod_tile or renderd. renderd's memory has been stable at around 86 MB since. |
Fixes renderd running out of memory with PostGIS-backed styles when
parameterize_styleis configured (e.g. multilingual maps). This is the problem we hit at OSM UK on Ubuntu with Postgres; single-threaded SQLite was unaffected.render()built a full copy of the Mapnik map and calledset_datasource()on each affected layer for every tile. Each PostGIS datasource registers a new connection-pool entry in Mapnik's globalConnectionManager, and those were never released, so memory grew until the server ran out.Parameterised maps are now built once and cached per render thread per options string, so
set_datasource()runs once per (thread, style, options) combination. The cache is small in practice (e.g. one entry per requested language).This replaces part of #497, split up as requested there. Independent of the other PRs in this series; it applies to
masteron its own and merges cleanly with each of them in any order.🤖 Generated with Claude Code
https://claude.ai/code/session_01YForD7qSL9LT6FhMuAoKPV