Tiles generator: derive PMTiles bounds from the pyramid, reject unsupported config

src-community/hakunapi-tiles-generator/README.md

@@ -21,6 +21,7 @@ hakunapi-tiles-generator [options] <config.properties> <control.properties> <out
   --max-zoom <n>    finest zoom to write (default: the control file's maximum)
   --bbox <minx,miny,maxx,maxy>
                     limit to this extent, in the tiling scheme's CRS
+                    (default: the whole tiling scheme, which at a fine zoom is a very large number of tiles)
   --threads <n>     parallel render workers (default: available processors)
   --generator <id>  vector tile generator type (default: hakunapi)
 ```
@@ -37,42 +38,93 @@ java -cp target/hakunapi-tiles-generator.jar:<source>.jar:<proj>.jar \
     fi.nls.hakunapi.tiles.generator.TilesGenerator service.properties control.properties tiles.gpkg
 ```
 
+## Example
+
+A GeoPackage `parks.gpkg` with one collection, `parks`, in EPSG:3857, and the
+service file:
+
+```properties
+api.title = Parks
+api.version = 0.1
+api.description = Tile generator example
+api.license.name = N/A
+api.license.url = N/A
+servers = dev
+servers.dev.url = http://localhost:8080/features
+servers.dev.description = Development server
+formats = json
+srid = 3857
+db.classes = fi.nls.hakunapi.source.gpkg.GpkgSimpleSource
+collections = parks
+collections.parks.db = parks.gpkg
+collections.parks.srid = 3857
+collections.parks.parameters = kind
+```
+
+with the control file:
+
+```properties
+tileMatrixSet = WebMercatorQuad
+ranges = overview, detail
+
+overview.zoom = 10-12
+overview.collections = parks
+overview.parks.filter = kind = 1
+
+detail.zoom = 13-14
+detail.collections = parks
+```
+
+```
+$ java -cp <jars> fi.nls.hakunapi.tiles.generator.TilesGenerator \
+      --bbox 1483000,6890000,1497000,6897000 service.properties control.properties tiles.gpkg
+Wrote 23 tiles into tiles.gpkg
+  parks                                23
+
+$ java -cp <jars> fi.nls.hakunapi.tiles.generator.PMTilesConvert tiles.gpkg parks parks.pmtiles
+Wrote parks.pmtiles (2,657 bytes) in 0 s
+```
+
 ## The two configuration files
 
 The **service properties file** is an ordinary hakunapi configuration and says
 what the data *is*: `db.classes`, `collections.*`, and the `tileMatrixSets.*`
 block defining the tiling scheme. `tiles.layers.*` is **not** read — a run's
 content comes from the control file.
 
-The **control file** defines the run:
+The **control file** defines the run (collection ids, property names and
+filters below are illustrative):
 
 ```properties
-tileMatrixSet = ETRS-TM35FIN
+tileMatrixSet = WebMercatorQuad
 ranges = overview, mid, detail
 
 overview.zoom = 0-7
-overview.collections = maastoalue, tieviiva
-overview.maastoalue.filter = kohdeluokka IN (36200, 36313)
-overview.tieviiva.filter = kohdeluokka <= 12132
+overview.collections = landuse, roads
+overview.landuse.filter = class IN (1, 2)
+overview.roads.filter = rank <= 2
 
 mid.zoom = 8-11
-mid.collections = maastoalue, tieviiva, rakennus
-mid.tieviiva.filter = kohdeluokka <= 12141
+mid.collections = landuse, roads, buildings
+mid.roads.filter = rank <= 4
 
 detail.zoom = 12-14
-detail.collections = maastoalue, tieviiva, rakennus, korkeuskayra
+detail.collections = landuse, roads, buildings, contours
 ```
 
 | key | meaning |
 | --- | --- |
-| `tileMatrixSet` | tiling scheme to render into, by id; its definition is read from the service file's `tileMatrixSets.*` block |
+| `tileMatrixSet` | tiling scheme to render into, by id: `WebMercatorQuad` is built in, others are read from the service file's `tileMatrixSets.*` block |
 | `ranges` | the zoom ranges, by name |
 | `<range>.zoom` | inclusive zoom range, `n` or `n-m` |
 | `<range>.collections` | collection ids rendered at these zooms, one MVT layer each |
-| `<range>.<collection>.filter` | CQL2 text, parsed with the API's own parser |
+| `<range>.<collection>.filter` | CQL2 text, parsed with the API's own parser; may only name properties the collection declares in `collections.<id>.parameters`, and may not call functions (the generator registers none) |
 | `sources` | output tables, by source id, comma-separated; default: one table per rendered collection |
 | `<source>.collections` | collection ids in this source, comma-separated; each rendered collection belongs to exactly one source |
 
+A source id must differ from every range name: both are configured by
+`<name>.collections`, so a shared name would silently read one key twice.
+
 Ranges must cover the rendered zooms exactly: a gap or an overlap is a
 configuration error. The whole file — collection ids and filters included — is
 validated before any rendering starts, so a typo cannot fail a long run an hour
@@ -125,6 +177,40 @@ A GeoPackage may hold several pyramids, so the rules are per *table*:
 | present | absent | create the table in it |
 | present | present | error: refuses to overwrite |
 
-A run that is interrupted leaves whatever it had committed. That is a partial
-pyramid with holes in it, not a prefix of the traversal, so it cannot be
-continued — rerun into a fresh output.
+A run that is interrupted or fails leaves whatever it had committed and exits
+non-zero, logging which tables are incomplete. That is a partial pyramid with
+holes in it, not a prefix of the traversal, so it cannot be continued — delete
+it and rerun. The file is written without a rollback journal and without
+`fsync` for speed, so a run that crashed outright may leave a file SQLite
+cannot open at all; discard it the same way.
+
+Each table gets a `gpkg_extensions` row, `hakunapi_mvt` on `tile_data`, because
+the GeoPackage tiles model otherwise reads as raster images; a reader that does
+not know the extension should leave the table alone.
+
+## Converting to PMTiles
+
+```
+PMTilesConvert [options] <pyramid.gpkg> <table> <output.pmtiles>
+
+  --raw             the pyramid's tiles are uncompressed (default: gzip, which the generator stores)
+  --flip            the pyramid counts tile_row from the bottom (TMS)
+  --bbox <minLon,minLat,maxLon,maxLat>
+                    WGS84 extent recorded in the header
+  --center <lon,lat>
+                    point a viewer opens at
+  --center-zoom <n> zoom a viewer opens at (default: the lowest zoom in the pyramid)
+```
+
+Without `--bbox`, the header extent is that of the tiles in the finest zoom of
+the table, converted to WGS84 with the pyramid's own SRID; the centre is the
+middle of that extent. Nothing is assumed about where the data is. Converting
+a CRS other than EPSG:4326 needs a projection factory on the classpath
+(`-Dhakunapi.proj=<class>`, default `fi.nls.hakunapi.proj.gt.GeoToolsProjectionTransformerFactory`,
+as for a service configuration); without one, pass `--bbox`.
+
+The archive keeps the pyramid's own tile addressing. Every zoom must be a
+`2^z` by `2^z` matrix, because a PMTiles tile id is a Hilbert index over a
+square grid, and the archive carries no projection: a viewer that assumes
+`WebMercatorQuad` mis-places tiles generated for any other scheme. The metadata
+says so. `go-pmtiles verify` accepts archives written this way.

src-community/hakunapi-tiles-generator/src/main/java/fi/nls/hakunapi/tiles/generator/GeneratorControl.java

@@ -98,7 +98,8 @@ private static Filter parseFilter(String rangeName, String collectionId, String
             return parser.parse(ft, cql2, srid);
         } catch (IllegalArgumentException e) {
             throw new IllegalArgumentException("control file: range " + rangeName + " filter for collection '"
-                    + collectionId + "' is not valid " + parser.getCode() + ": " + e.getMessage());
+                    + collectionId + "' is not valid " + parser.getCode() + ": " + e.getMessage()
+                    + (String.valueOf(e.getMessage()).contains("Unmapped Function") ? " (the generator registers no functions)" : ""));
         }
     }
 

src-community/hakunapi-tiles-generator/src/main/java/fi/nls/hakunapi/tiles/generator/PMTilesConvert.java

@@ -70,6 +70,14 @@ private static int run(String[] args) throws Exception {
             if (parsed.centerZoom < 0) {
                 parsed.centerZoom = (int) addresses.get(0)[1];
             }
+            if (parsed.bboxE7 == null) {
+                parsed.bboxE7 = toE7(PyramidBounds.wgs84(c, parsed.table, parsed.flip));
+            }
+            if (parsed.centerE7 == null) {
+                parsed.centerE7 = new int[] {
+                        (int) (((long) parsed.bboxE7[0] + parsed.bboxE7[2]) / 2),
+                        (int) (((long) parsed.bboxE7[1] + parsed.bboxE7[3]) / 2) };
+            }
             written = convert(c, parsed, addresses);
         }
         long seconds = (System.currentTimeMillis() - started) / 1000;
@@ -78,6 +86,14 @@ private static int run(String[] args) throws Exception {
         return 0;
     }
 
+    private static int[] toE7(double[] degrees) {
+        int[] e7 = new int[degrees.length];
+        for (int i = 0; i < e7.length; i++) {
+            e7[i] = (int) Math.round(degrees[i] * 1e7);
+        }
+        return e7;
+    }
+
     /** The PMTiles tile id is a Hilbert index over a 2^z square grid. */
     private static void checkSquareMatrices(Connection c, String table) throws SQLException {
         String sql = "SELECT m.zoom_level, m.matrix_width, m.matrix_height FROM gpkg_tile_matrix m"
@@ -238,7 +254,9 @@ private static void usage() {
         System.err.println("                   (default: gzip, which is what the generator stores)");
         System.err.println("  --flip           the pyramid counts tile_row from the bottom (TMS)");
         System.err.println("  --bbox <minLon,minLat,maxLon,maxLat>");
-        System.err.println("                   WGS84 extent recorded in the header (default: Finland, 19,59.4,31.8,70.1)");
+        System.err.println("                   WGS84 extent recorded in the header (default: the extent of the tiles");
+        System.err.println("                   at the finest zoom)");
+        System.err.println("  --center <lon,lat> point a viewer should open at (default: the middle of the extent)");
         System.err.println("  --center-zoom <n>  zoom a viewer should open at (default: the lowest zoom in the pyramid)");
     }
 
@@ -249,16 +267,17 @@ private static class Args {
         Path outputPath;
         byte tileCompression = Compression.GZIP;
         boolean flip;
-        int[] bboxE7 = { 190000000, 594000000, 318000000, 701000000 };
-        int[] centerE7 = { 250000000, 645000000 };
+        int[] bboxE7;
+        int[] centerE7;
         int centerZoom = -1;
 
         static Args parse(String[] args) {
             Args a = new Args();
             List<String> positional = new ArrayList<>(3);
             for (int i = 0; i < args.length; i++) {
                 String arg = args[i];
-                if (("--bbox".equals(arg) || "--center-zoom".equals(arg)) && i + 1 >= args.length) {
+                if (("--bbox".equals(arg) || "--center".equals(arg) || "--center-zoom".equals(arg))
+                        && i + 1 >= args.length) {
                     System.err.println("Option " + arg + " needs a value");
                     return null;
                 }
@@ -270,10 +289,10 @@ static Args parse(String[] args) {
                     a.flip = true;
                     break;
                 case "--bbox":
-                    a.bboxE7 = parseBboxE7(args[++i]);
-                    a.centerE7 = new int[] {
-                            (int) (((long) a.bboxE7[0] + a.bboxE7[2]) / 2),
-                            (int) (((long) a.bboxE7[1] + a.bboxE7[3]) / 2) };
+                    a.bboxE7 = parseE7(args[++i], 4, "--bbox must be minLon,minLat,maxLon,maxLat");
+                    break;
+                case "--center":
+                    a.centerE7 = parseE7(args[++i], 2, "--center must be lon,lat");
                     break;
                 case "--center-zoom":
                     a.centerZoom = Integer.parseInt(args[++i]);
@@ -295,16 +314,16 @@ static Args parse(String[] args) {
             return a;
         }
 
-        private static int[] parseBboxE7(String csv) {
+        private static int[] parseE7(String csv, int n, String message) {
             String[] parts = csv.split(",");
-            if (parts.length != 4) {
-                throw new IllegalArgumentException("--bbox must be minLon,minLat,maxLon,maxLat");
+            if (parts.length != n) {
+                throw new IllegalArgumentException(message);
             }
-            int[] bbox = new int[4];
-            for (int i = 0; i < 4; i++) {
-                bbox[i] = (int) Math.round(Double.parseDouble(parts[i].trim()) * 1e7);
+            int[] e7 = new int[n];
+            for (int i = 0; i < n; i++) {
+                e7[i] = (int) Math.round(Double.parseDouble(parts[i].trim()) * 1e7);
             }
-            return bbox;
+            return e7;
         }
 
     }

src-community/hakunapi-tiles-generator/src/main/java/fi/nls/hakunapi/tiles/generator/PyramidBounds.java

@@ -0,0 +1,121 @@
+package fi.nls.hakunapi.tiles.generator;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+
+import fi.nls.hakunapi.core.projection.ProjectionTransformer;
+import fi.nls.hakunapi.core.projection.ProjectionTransformerFactory;
+import fi.nls.hakunapi.gpkg.GPKG;
+
+/** The WGS84 extent of the tiles a pyramid table holds, not of the tiling scheme it was generated for. */
+public class PyramidBounds {
+
+    public static final String PROJ_PROPERTY = "hakunapi.proj";
+    public static final String DEFAULT_PROJ = "fi.nls.hakunapi.proj.gt.GeoToolsProjectionTransformerFactory";
+
+    private static final int WGS84 = 4326;
+
+    private PyramidBounds() {
+        // Block init
+    }
+
+    /** Tiles of the finest zoom only: a tile at a coarse zoom spans the whole scheme and says nothing about the data. */
+    public static double[] wgs84(Connection c, String table, boolean flip) throws Exception {
+        int zoom;
+        try (Statement st = c.createStatement();
+                ResultSet rs = st.executeQuery("SELECT MAX(zoom_level) FROM " + GPKG.quote(table))) {
+            rs.next();
+            zoom = rs.getInt(1);
+        }
+
+        long minCol, maxCol, minRow, maxRow;
+        String sql = "SELECT MIN(tile_column), MAX(tile_column), MIN(tile_row), MAX(tile_row) FROM "
+                + GPKG.quote(table) + " WHERE zoom_level = ?";
+        try (PreparedStatement ps = c.prepareStatement(sql)) {
+            ps.setInt(1, zoom);
+            try (ResultSet rs = ps.executeQuery()) {
+                rs.next();
+                minCol = rs.getLong(1);
+                maxCol = rs.getLong(2);
+                minRow = rs.getLong(3);
+                maxRow = rs.getLong(4);
+            }
+        }
+
+        int srid;
+        double originX, originY, spanX, spanY;
+        long matrixHeight;
+        sql = "SELECT s.srs_id, s.min_x, s.max_y, m.matrix_height, m.tile_width * m.pixel_x_size,"
+                + " m.tile_height * m.pixel_y_size FROM gpkg_tile_matrix_set s"
+                + " JOIN gpkg_tile_matrix m ON m.table_name = s.table_name"
+                + " WHERE s.table_name = ? AND m.zoom_level = ?";
+        try (PreparedStatement ps = c.prepareStatement(sql)) {
+            ps.setString(1, table);
+            ps.setInt(2, zoom);
+            try (ResultSet rs = ps.executeQuery()) {
+                if (!rs.next()) {
+                    throw new IllegalArgumentException("Table " + table
+                            + " has no tile matrix for zoom " + zoom + "; pass --bbox");
+                }
+                srid = rs.getInt(1);
+                originX = rs.getDouble(2);
+                originY = rs.getDouble(3);
+                matrixHeight = rs.getLong(4);
+                spanX = rs.getDouble(5);
+                spanY = rs.getDouble(6);
+            }
+        }
+
+        if (flip) {
+            long top = matrixHeight - 1 - maxRow;
+            maxRow = matrixHeight - 1 - minRow;
+            minRow = top;
+        }
+
+        double minX = originX + minCol * spanX;
+        double maxX = originX + (maxCol + 1) * spanX;
+        double maxY = originY - minRow * spanY;
+        double minY = originY - (maxRow + 1) * spanY;
+
+        // Corners and edge midpoints: a projected rectangle is not a rectangle in lon/lat
+        double midX = (minX + maxX) / 2;
+        double midY = (minY + maxY) / 2;
+        double[] xy = { minX, minY, midX, minY, maxX, minY, maxX, midY, maxX, maxY, midX, maxY, minX, maxY, minX, midY };
+        if (srid != WGS84) {
+            toCRS84(srid).transformInPlace(xy, 0, xy.length / 2);
+        }
+
+        double[] bbox = { Double.POSITIVE_INFINITY, Double.POSITIVE_INFINITY,
+                Double.NEGATIVE_INFINITY, Double.NEGATIVE_INFINITY };
+        for (int i = 0; i < xy.length; i++) {
+            if (!Double.isFinite(xy[i])) {
+                throw new IllegalArgumentException("The pyramid's extent lies outside the projection's valid area"
+                        + " in EPSG:" + srid + "; pass --bbox");
+            }
+        }
+        for (int i = 0; i < xy.length; i += 2) {
+            bbox[0] = Math.min(bbox[0], xy[i]);
+            bbox[1] = Math.min(bbox[1], xy[i + 1]);
+            bbox[2] = Math.max(bbox[2], xy[i]);
+            bbox[3] = Math.max(bbox[3], xy[i + 1]);
+        }
+        return bbox;
+    }
+
+    private static ProjectionTransformer toCRS84(int srid) {
+        String className = System.getProperty(PROJ_PROPERTY, DEFAULT_PROJ);
+        try {
+            ProjectionTransformerFactory factory = (ProjectionTransformerFactory) Class.forName(className)
+                    .getDeclaredConstructor().newInstance();
+            return factory.toCRS84(srid);
+        } catch (Exception | LinkageError e) {
+            throw new IllegalArgumentException("Cannot convert the pyramid's EPSG:" + srid
+                    + " extent to WGS84 with " + className + " (" + e + "); put a projection factory on the"
+                    + " classpath (-D" + PROJ_PROPERTY + "=<class>) or pass --bbox");
+        }
+    }
+
+}