feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata by HuggeK · Pull Request #735 · srcfl/ftw

HuggeK · 2026-07-31T08:59:39Z

Derives PV array tilt, azimuth and kWp from Lantmäteriet open geodata, so a
Swedish site can stop typing panel angles in by hand.

Stacked on #734 (which adds the coverage registry this registers into), which
is itself stacked on #718.

Shape

roofmodel/ is a separate Python package alongside optimizer/, reached at
arm's length: core spawns it, passes coordinates and the operator's own
Geotorget credentials, and reads back one versioned roof_model.json.

That boundary is the point. LiDAR segmentation drags in a compiled point-cloud
stack, runs for minutes, and is onboarding-time work rather than runtime work.
In a time-boxed subprocess it cannot stall the control tick, cannot leak into the
daemon, and can be absent entirely — which is the normal case, since the data
exists only for Sweden.

Method

Follows the SPAN paper
(Yavuzdoğan, Renewable Energy 2023): iterative RANSAC pulls one roof surface at
a time out of the cloud, then DBSCAN splits faces that satisfy the same plane
equation but sit in different places — two wings of a building fit one plane and
are not one roof.

The method only. No code is taken from SPAN's GPL QGIS plugin; dependencies are
numpy, scikit-learn and requests, all BSD, so the module carries no copyleft
obligation.

Two implementation choices worth flagging

Plane fitting is total-least-squares, not z = ax + by + c. An ordinary
regression of height on position minimises vertical error, which
systematically biases steep roofs. The smallest singular vector of the
mean-centred points minimises perpendicular distance instead, and stays
well-conditioned as the surface approaches vertical.

SWEREF 99 TM is implemented directly rather than via pyproj. Lantmäteriet
publishes in EPSG:3006 and FTW stores WGS84, so every request crosses that
boundary — but it is exactly one projection with fixed parameters, and shipping
a full PROJ build for that is disproportionate on a Pi. The Krüger series is
~120 lines and verified two ways: against the projection's exact analytic
properties (easting on the central meridian is exactly 500000 at every
latitude, and the origin maps to (0, 500000)), and by sub-millimetre round trips
from Smygehuk to Treriksröset.

What it produces

input: 1392 LiDAR returns   (synthetic gable + detached garage)

  tilt  azimuth   area m2  points   truth
  35.0    180.0      87.9     576   35 deg south
  35.0      0.0      87.9     576   35 deg north
  18.0    270.4      23.8     181   18 deg west

-- as candidate PV arrays (north face dropped, too little yield) --
   Roof south       12.31 kWp   tilt 35.0  az 180.0  (88 m2)
   Roof west         3.33 kWp   tilt 18.0  az 270.4  (24 m2)

GET /api/roofmodel reports availability and coverage; POST /api/roofmodel/derive runs a derive and returns the proposal. Lantmäteriet also
joins /api/data-sources as a Sweden-only, credential-gated source.

Deliberately not wired: auto-applying to config

The RFC sketched automatic auto-fill of weather.pv_arrays. This PR returns the
proposal instead and leaves applying it as a separate explicit act. Derivation is
a best guess from a point cloud that may be years old, packing factor is an
assumption, and silently rewriting an operator's panel config is a change they
should make knowingly. Happy to wire the auto-apply if you'd rather — it is a
small change on top, but it is a behaviour decision, not an implementation
detail.

Testing

57 Python tests. Every geometry case is a synthetic roof whose tilt, azimuth
and area are known exactly by construction, and the generator is the inverse of
the code under test rather than a copy of it — so the assertions check recovered
geometry against ground truth, not against a previous run. Covered: six
orientations, gable splitting, two buildings sharing a plane, sloped-vs-footprint
area, realistic LiDAR noise, wall rejection, determinism.

11 Go tests. The subprocess contract is exercised end-to-end against a stub
module, so argument passing, stdout parsing, stderr error surfacing, schema
rejection and the timeout are all real rather than mocked.

A limitation the tests pin rather than hide

Sweden is a long diagonal, so no lat/lon rectangle traces its border — any box
containing Sweden also contains parts of Norway. Oslo is the clearest case: west
of Sweden, inside the box.

Rather than tighten the box until it starts excluding real Swedish addresses near
the border, the box stays generous and the STAC search is the authority: no tiles
come back, and the module reports "Sweden only". There are tests for both halves —
that Oslo is admitted by design, and that Strömstad, Haparanda, Karesuando and
Smygehuk are never excluded.

Not covered: the live Lantmäteriet fetch. It needs Geotorget credentials for
Byggnad-vektor and Laserdata-skog, which are the operator's own; the HTTP layer
is tested against a fake session instead, including the documented quirk that
properties.datetime is still being backfilled through 2026 and the laser strip
datum is the fallback. To try it live:

pip install -e roofmodel[geo]
python -m ftw_roofmodel --lat 59.33 --lon 18.07 --username <user> --token <token>

Part of the roadmap in discussion #717.

Seen running

Find buildings here → STAC search → GeoPackage decoded with sqlite3 + raw WKB → two candidates, distances correct after the axis-order fix (details in this comment):

Read roof from LiDAR on the picked footprint: two planes found, the north face dropped per the rule, the form pre-filled — "Filled in 1 array(s) from 2 roof plane(s). Laser data from 2019-06-30. Review them and press Save." The capture date came through the datum fallback, since the STAC datetime is still being backfilled:

The section's help bubble, which used to truncate at the first quoted product name and clip at the modal edge, now renders whole and flips upward when it sits low (29fb3e23):

Runbook: how this was demoed (reproducible)

go build -o ftw-demo.exe ./cmd/ftw from the branch, plus go run ./cmd/sim-ferroamp.
A demo config: Stockholm (59.3293, 18.0686), provider: open_meteo, two pv_arrays (7 kWp S @ 35°, 3 kWp W @ 35°), roofmodel.enabled with command: python, module_dir: roofmodel.
./ftw-demo.exe -config config.demo.yaml -backfill 30 — 518 400 synthetic history rows from the repo's own generator.
Launch, and let the pvperf boot timer fire: it fetched 745 hourly rows from SMHI's production STRÅNG API, live and scored all 30 days.
Lantmäteriet stood in locally — the credentials are the operator's own, so a local server spoke real STAC and served a real GeoPackage (byte-assembled per OGC 12-128r19, EPSG:3006) and a synthetic LiDAR tile through the module's documented decode seam (laspy absent → the whole-tile fallback path, which is itself a shipped path).
Driven with headless Edge over the DevTools protocol — nothing installed; screenshots are unretouched viewport captures.

Everything else — every line of core, the module, the UI — was the shipped code from this branch.

…coverage STRANG becomes a real irradiance source rather than an unreferenced client, every external data source now says where in the world it works, and the location picker moves off Leaflet. STRANG's parameter set was mapped against the live API because SMHI's apidocs pages 404: exactly 116-122 exist. Identification was confirmed by physics rather than by guessing - at solar noon 121 + 122 = 723.0 + 87.5 = 810.5, exactly parameter 117, and 119 caps at 60, i.e. minutes within the hour. STRANG publishes no cloud cover; it is a radiation model. Cloudiness is instead derived from sunshine duration as 1 - minutes/60, which is observed rather than inferred but coarser: blind to thin cirrus, undefined at night. CloudCover() therefore returns an explicit unknown instead of defaulting to clear, because those two lead to opposite decisions. The new coverage registry makes an existing silence explicit. STRANG is Nordic-only and every price provider is European, so sites elsewhere were getting empty results with no explanation (srcfl#726). GET /api/data-sources now reports area, countries, licence and whether each source reaches this site, and the Weather tab renders it under the map. Bounds are advisory: STRANG's grid is rotated, so a lat/lon box can only ever be a superset - all four in-box corners were probed and returned no data. False is definitive, true means worth trying. Scoring now declines to start outside the domain instead of retrying nightly forever. Stacked on srcfl#718, which carries the plane-of-array wiring this builds on. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

A Swedish site can stop typing panel angles in by hand. The new roofmodel/ module reads Lantmateriet LiDAR around the site and recovers each roof face's tilt, azimuth and usable area, which pre-fill weather.pv_arrays. Structured as a separate Python package alongside optimizer/, reached at arm's length: core spawns it, passes coordinates and the operator's own Geotorget credentials, and reads one versioned roof_model.json from stdout. Segmentation drags in a compiled point-cloud stack and runs for minutes, so a time-boxed subprocess keeps it off the control tick and lets it be absent entirely - the normal case, since the data exists only for Sweden. The pipeline follows the SPAN method: iterative RANSAC pulls one surface at a time out of the cloud, then DBSCAN splits faces sharing a plane equation but not a location, because two wings of a building fit the same plane and are not the same roof. Method only - no code is taken from SPAN's GPL QGIS plugin, and the dependencies (numpy, scikit-learn, requests) are all BSD. SWEREF 99 TM is implemented directly rather than via pyproj: it is one projection with fixed parameters, and a full PROJ build is disproportionate for that. Verified against the projection's exact analytic properties - easting on the central meridian is exactly 500000 at every latitude - and by sub-millimetre round trips from Smygehuk to Treriksroeset. Applying derived arrays to config is deliberately a separate act. Derivation is a best guess from a point cloud that may be years old, and silently rewriting an operator's panel config is a change they should make knowingly. Also generalises the __pycache__ ignore rule, which was optimizer-specific and would otherwise need repeating per module. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

The roofmodel tests passed on Windows and failed on Linux, which is where CI runs them. Two separate bugs, both caused by stubbing the module with a shell script: - The stub handed back "sh" as the command, and the service invokes it as `<cmd> -m ftw_roofmodel ...`. dash reads that as "run the script named ftw_roofmodel", cannot open it, and exits 2. The stub never ran at all, so five tests failed against an error that had nothing to do with what they were asserting. - The module-error stub echoed an unquoted JSON document. dash strips the double quotes, so `{"error":"..."}` reached the parser as `{error:...}` and could not be unmarshalled. cmd.exe keeps them, which is the only reason that test ever passed. The stub is now this test binary re-executed with an environment variable telling it which behaviour to play. Nothing goes through a shell, so there is no quoting to get wrong and no per-platform behaviour to diverge. The timeout test consequently runs everywhere instead of being skipped on Windows. Two assertions the shell stub could not make: that the site, credentials and radius actually survive the process boundary, and that PYTHONPATH is set -- without it the module is only importable if installed system-wide, which on a Pi it is not. Also that --vostok stays absent unless configured, so a GPL tool is never invoked by default. The Python module's tests did not run in CI at all: the workflow runs `pytest -q optimizer/tests` and nothing else, so 75 tests covering the plane fitting every derived tilt and azimuth depends on were never executed upstream. Added a roofmodel job mirroring the optimizer's, wired into the required-check gate so a failure blocks rather than being reported and ignored. It installs without the `geo` extra on purpose: LAZ decoding pulls a compiled backend, and everything except the point-cloud read is exercised without it. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

Settings -> Weather gains a Geotorget credential form and a building picker. Press "Find buildings here" and the footprints near the marker are drawn on the map and listed beside it; click yours, press "Read roof from LiDAR", and the PV arrays fill in with one entry per usable roof face. The form is filled but nothing is saved. The operator sees the numbers, corrects what is wrong and presses Save. FTW does not rewrite a panel configuration on its own: the derivation is a guess from a scan that may be years old, and only the operator knows whether that face has panels on it. Picking a building is not cosmetic. Without a footprint the module segments whatever stands inside its search radius, and the plane fitting is global -- a fitted plane is infinite, so a roof at azimuth 180 is z = f(y) with no x term and extends across the whole tile. A second building sharing that ridge orientation lands inside its inlier band however far away it is, and the two lose returns to each other. Measured on a synthetic pair: a detached garage recovered 93% of its true area and split into two fragments while coplanar with the house, against 100% and one clean face once clipped to its own footprint. The clip buffers the outline by a metre first, because roofs overhang their walls and the eaves carry the lowest returns. Frames are detected rather than assumed. GeoJSON mandates WGS84 but Lantmateriet publishes this catalogue in SWEREF 99 TM and its STAC search takes a SWEREF bbox, so ring coordinates are classified by magnitude -- six- and seven-figure numbers are projected metres, degrees never are. New GET /api/roofmodel/buildings lists footprints as GeoJSON, honouring an explicit lat/lon so the picker can search where the marker is rather than where the last save put it. POST /api/roofmodel/derive accepts a building_id. GET /api/roofmodel reports has_credentials so the UI can stop asking. The Geotorget token now masks and restores like every other secret. It never appears in an API response, and saving an unrelated setting no longer wipes it -- the settings form returns the blank it was given, which without PreserveMaskedSecrets would have deleted the stored credential. docs/roof-geometry.md covers ordering the two Geotorget products, what the derived kWp does and does not mean (an upper bound on what fits, not what is installed), and what each failure message is telling you. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

Both roof-geometry products are STAC APIs behind one Geotorget account, and they differ only in what their items point at: Byggnad Nedladdning, vektor delivers GeoPackage, Laserdata Nedladdning, Skog delivers LAZ organised as COPC. Assets are now selected by declared media type rather than by guessing at asset key names, so a catalogue that calls its asset "punktmoln" instead of "data" keeps working, and a thumbnail is never handed back as a point cloud. Building footprints are read straight out of the GeoPackage with the standard library. A GeoPackage is a SQLite database holding geometry as WKB, both published formats with fixed layouts, so this costs about a hundred lines against a GDAL dependency that will not install on a Pi without a compiler. Previously only inline STAC geometry was handled and the asset-backed case -- the normal one -- returned nothing at all. Because COPC indexes points into an octree, picking a building now also makes the download small: only the octree nodes covering that footprint are range-requested, instead of a 2.5 km tile running to hundreds of megabytes. Plain .laz assets, hosts that ignore Range, and builds of laspy without COPC support all fall back to reading the tile whole -- slower, same answer -- and the model records which path ran as source.fetch, since that also decides what returns_in_radius is counted over. Also corrects the coplanar-absorption measurement quoted in buildings.py and docs/roof-geometry.md. A fresh run does not reproduce the 93%/two-fragment figures: one pass over a house and a garage 40 m apart consumes all 576 of the house's south-face returns and all 256 of the garage's as one surface, and gives identical output at every separation from 3 m to 40 m -- which is the signature of a global infinite-plane fit rather than a proximity effect. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

… from Rings read from a GeoPackage arrive x=easting first, the way GIS files store them. Rings converted from inline WGS84 geometry were stored the way wgs84_to_sweref99tm returns them: northing first. Everything downstream assumed the second convention, so every GeoPackage-sourced building -- the normal Lantmateriet case -- reported its centroid near (4 N, 63 E) in the Indian Ocean, an 8 000 km "distance", and a picker map with nothing visible on it. The suite missed it because every assertion was axis-blind: areas and SWEREF centroids survive a consistent swap. It surfaced the first time the module ran against a server and a human looked at where the buildings landed. ring_sweref now always holds (easting, northing): rings from degrees are swapped into it, and projected rings are normalised per point -- eastings stay under a million metres and northings start above six, so each point states its own order and an EPSG-registry-ordered export (north first) is folded in rather than mis-read. The WGS84 accessors and the site distance unpack accordingly. A regression test feeds the same square in both orders and requires real Stockholm coordinates and a sub-50 m distance back from each; the fixture files mis-unpacked the projection the same way the code did, and now match reality (LAZ x is an easting). Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

HuggeK · 2026-08-05T12:06:26Z

Found and fixed while demoing: GeoPackage buildings rendered in the Indian Ocean

Driving the picker end-to-end against a locally served GeoPackage put both candidate buildings at lat ≈ 4°, lon ≈ 63°, "8 352 577 m away" — for the normal Byggnad-vektor case.

The cause is an axis-order split. wgs84_to_sweref99tm returns (northing, easting), so rings converted from inline WGS84 geometry were stored north-first — while rings decoded from a GeoPackage passed through as published, east-first. Everything downstream (centroid_wgs84, ring_wgs84, the site distance) assumed the first convention.

Why 133 tests missed it

Every existing assertion was axis-blind: polygon areas and SWEREF centroids survive a consistent swap, and the clip compares the ring against points synthesized by the same mis-unpacked helper — the test fixtures encoded the bug in the same way the code did (e, n = wgs84_to_sweref99tm(...) unpacks a northing into e). Nothing ever asked "is this WGS84 output actually in Sweden?" until a human looked at a map.

Fixed in a07fc0c0: ring_sweref now always holds (easting, northing) — rings from degrees are swapped on conversion, and projected rings are normalised per point (eastings stay under a million metres, northings start above six, so each point states its own order; an EPSG-registry-ordered export is folded in rather than mis-read). The regression test feeds the same square in both orders and requires real Stockholm coordinates and a sub-50 m distance back from each. The fixture files now build points the way real LAZ delivers them (x = easting). 134 tests pass; #736 is restacked on top.

After the fix, the same run shows the picker list as "108 m² · 0 m away" and "30 m² · 18 m away", footprints on the map where the buildings stand, and a derive that fills the form from the picked footprint with "Laser data from 2019-06-30" out of the datum fallback.

The roofmodel section's help text ends with two quoted product names, and the bubble cut off mid-sentence right before them. escHtml used the textContent/innerHTML trick, which never escapes quotes -- so every caller that builds an attribute (data-help, title, value) had its text terminated at the first embedded quote, with the remainder parsed as junk attribute names. Plain string replaces now cover both quote kinds. With the full text rendering, the second failure appeared: the bubble opens downward and the modal body is the scroll container that clips it, so a badge low in the last section has less room below than a long help text needs. settings.js now toggles .help-up on hover when the badge sits in the lower part of the visible modal, and the bubble grows upward into room that exists. Measured on hover because scroll position, not the badge, decides which way is open. Verified in headless Edge: the full text renders at both scroll positions, upward when low, downward when high. Co-authored-by: HuggeK <48095810+HuggeK@users.noreply.github.com> Signed-off-by: Hugo Karlsson <48095810+HuggeK@users.noreply.github.com>

frahlg · 2026-08-07T09:14:58Z

This PR currently has right of way on .github/workflows/test.yml. After #865 merged, every push to master fails in the later app-contract job: the workflow calls gh pr view "${{ github.event.pull_request.number }}" on a push event, where the number is empty.

Please rebase onto current master and make that pair-selection step event-aware: read the live PR body only for pull_request; use ftw-webapp@main for push and other events. Add a push-path check if practical. This also needs to be resolved before the smaller setup-python bump in #785 can move without rewriting this file underneath your branch.

HuggeK mentioned this pull request Jul 31, 2026

feat(roofmodel): optional shadow-aware irradiance via vostok #736

Draft

HuggeK force-pushed the roofmodel branch from bc52303 to 934eda4 Compare July 31, 2026 13:44

This was referenced Aug 4, 2026

fix(control): bound every driver command with a deadline #791

Merged

fix(models): version learned PV and load state against its feature vector #793

Merged

fix(config): enforce owner-only ACL on Windows saves #797

Draft

claude and others added 5 commits August 5, 2026 12:37

HuggeK force-pushed the roofmodel branch from ee93827 to f42c34b Compare August 5, 2026 10:44

This was referenced Aug 7, 2026

chore(drivers): move bundled pin to f4ab266e #863

Merged

chore(ci): bump actions/setup-python from 6 to 7 #785

Open

Navigation Menu

Uh oh!

feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata - #735

feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata#735
HuggeK wants to merge 7 commits into
srcfl:masterfrom
HuggeK:roofmodel

HuggeK commented Jul 31, 2026 •

edited

Loading

Uh oh!

HuggeK commented Aug 5, 2026

Uh oh!

frahlg commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

HuggeK commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Shape

Method

What it produces

Deliberately not wired: auto-applying to config

Testing

Seen running

Uh oh!

HuggeK commented Aug 5, 2026

Found and fixed while demoing: GeoPackage buildings rendered in the Indian Ocean

Uh oh!

frahlg commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HuggeK commented Jul 31, 2026 •

edited

Loading