Repository navigation
Dashboard - #241
Open
SimonSadler wants to merge 26 commits into
Open
Dashboard#241SimonSadler wants to merge 26 commits into
SimonSadler wants to merge 26 commits into
Conversation
SimonSadler
force-pushed
the
dashboard
branch
from
September 23, 2026 13:47
7cc84e9 to
27211ca
Compare
The beginnings of a web-based dashboard for viewing and comparing cyclone trajectories. The datasette-geojson-map plugin has specific requirements for the GeoJSON geometry column; the "geometry" field of a GeoJSON structure must be the root. A tweak has been made to the construction of the track GeoJSON in order to support this. The change is temporary; the plugin will be replaced or improved in later versions. Tests have not been updated so some will fail.
To replace the existing Leaflet map. MapLibre is more modern, fully featured and faster than Leaflet. Extensive tests show it can easily handle 1,00,000 points and tracks. The layer system also allows simpler overlays. MapLibre also correctly renders 0–360 longitudes so there is no longer any need to wrap longitude values to the -180–180 range. This solves the problem where lines cross the antimeridian.
Add a [dashboard] extra to the main pyproject file for installing Datasette and the custom plugin.
Loading GeoJSON into the map was slow on very large datasets (around 2s). Reducing data size and side-stepping some data conversion on the server-side has reduced this to 0.5s. There are other options for optimisation that can be taken based on eventual usage. Two of these are: storing GeoJSON in one blob rather than row by row, and using a more optimal format such as MapLibre Vector Tiles.
Tracks and points are added as map layers via a magic `layer_` column in the database view. Each layer is coloured accessibly for visual distinction.
Also removes the (undesired) background layer without further configuration.
Previously GeoJSON was built and stored in the database at the time of creation. This big shift generates the required GeoJSON at map rendering time. Performance is very similar to the previous approach as that also required some data wrangling to merge it into one dataset. Dynamic generation gives the following benefits: - Flexibility of structuring GeoJSON data in any way required. - Property names come from the query, not baked in or abbreviated. - Layering does not require the slight hack of hooking up the column name to a fixed property name. - Filtering is more flexible and powerful because views are now based on observation points rather than tracks. - The database is much smaller without the GeoJSON data (which was the same data as stored in the tables, just in text format). - No need to retrieve or hide GeoJSON columns from the table view. This initial change is for observation points only.
Track lines are formed from observations by grouping on the trajectory_id. This is done via a generalised group_by setting and system that allows for future expansion.
Pre-built GeoJSON is no longer required in the database now that it is generated just-in-time on the client-side. The benefits of dynamic creation (repeated from a previous commit): - Flexibility of structuring GeoJSON data in any way required. - Property names come from the query, not baked in or abbreviated. - Layering does not require the slight hack of hooking up the column name to a fixed property name. - Filtering is more flexible and powerful because views are now based on observation points rather than tracks. - The database is much smaller without the GeoJSON data (which was the same data as stored in the tables, just in text format). - No need to retrieve or hide GeoJSON columns from the table view.
- Reduce map height so query controls are more present. - Disable map scroll-zoom so page is easier to scroll. - Disable drag-rotate to avoid confusion. - Add compass control. - Only add a line layer if lines were created. - Change pointer when hovering over points and lines.
Colours can now be specified for single and multiple layers. Style improvements for layer control and facets. The best map for dark mode so far is the MapTiler `dataviz-v4-dark`. An API key is required so this option is not included in the config.
SimonSadler
force-pushed
the
dashboard
branch
from
September 23, 2026 20:52
27211ca to
168ef0d
Compare
Datasette facets added for swifter searching.
Point radius can now be linked to property value.
Opacity can now be linked to a property value. This allows for point and line visual variation without adding multiple layers. Layers can now be used solely for comparing distinct outputs/collections. The palette configuration has been adapted for distinction in readiness for this. The layer control opacity slider doesn't support opacity expressions so it has been disabled to accommodate this.
Datasette v1.0 is still in alpha, but it's very mature at this point and contains fixes that are beneficial (fixed position of facets being one). I've seen no issues and the changes were minimal.
SimonSadler
marked this pull request as ready for review
October 2, 2026 07:54
The default influence radius has also been tweaked for best results with our datasets.
sjavis
approved these changes
Oct 6, 2026
sjavis
left a comment
sjavis
left a comment
Collaborator
There was a problem hiding this comment.
This is looking good, I've been able to run the dashboard and no issues are leaping out at me. I have a few minor suggestions for the README including how it could be added to the documentation. But there is not anything significant blocking it from being merged.
|
|
||
| The following settings are available: | ||
|
|
||
| - `basemap` URL of the MapLibre-compatible basemap to project onto the map. See the [basemap gallery](https://madewithmaplibre.com/basemaps/gallery) for alternatives. The MapTiler dataviz map ([light](https://www.maptiler.com/maps/#style=dataviz-v4)/[dark](https://www.maptiler.com/maps/#style=dataviz-v4-dark)) has proven effective (API key required). |
Collaborator
There was a problem hiding this comment.
Suggested change
| - `basemap` URL of the MapLibre-compatible basemap to project onto the map. See the [basemap gallery](https://madewithmaplibre.com/basemaps/gallery) for alternatives. The MapTiler dataviz map ([light](https://www.maptiler.com/maps/#style=dataviz-v4)/[dark](https://www.maptiler.com/maps/#style=dataviz-v4-dark)) has proven effective (API key required). | |
| - `basemap` URL of the MapLibre-compatible basemap to project onto the map. See the [basemap gallery](https://madewithmaplibre.com/basemaps/gallery) for alternatives. The MapTiler dataviz map ([light](https://www.maptiler.com/maps/#style=dataviz-v4)/[dark](https://www.maptiler.com/maps/#style=dataviz-v4-dark)) has proven effective ([API key required](https://docs.maptiler.com/guides/credentials/api-key/)). |
| datasette serve <database.db> --metadata dashboard/metadata.yaml --config dashboard/datasette.yaml --static static:dashboard/static | ||
|
|
||
| - `metadata.yaml` applies the database and table metadata. | ||
| - `datasette.yaml` applies the dashboard theme, map configuration and server settings. |
Collaborator
There was a problem hiding this comment.
I would explicitly say that this file can be edited to configure the dashboard. It isn't currently clear that it is intended to be edited.
|
|
||
| ### Configuration | ||
|
|
||
| The map can be configured in the `datasette-maplibre` section of the `metadata.yaml` file. *The `datasette` server process must be restarted for changes to take effect*. |
Collaborator
There was a problem hiding this comment.
Suggested change
| The map can be configured in the `datasette-maplibre` section of the `metadata.yaml` file. *The `datasette` server process must be restarted for changes to take effect*. | |
| The map can be configured in the `datasette-maplibre` section of the `datasette.yaml` file. *The `datasette` server process must be restarted for changes to take effect*. |
Collaborator
There was a problem hiding this comment.
It would be nice to include the README directly in the documentation. It should be possible using the myst markdown parser:
- Make a
docs/dashboard/readme_link.mdwith:
```{include} ../../dashboard/README.md
```
- Add it to the
toctreeindocs/index.rst:
Data Formats and Manipulation <data/index>
Dashboard <dashboard/readme_link>
Developer Information <developer/index>
- Add
"myst-parser"to thedocdependencies inpyproject.toml. - Add
"myst_parser"to theextensionsindocs/conf.py(I don't know why it can't be consistent with the hyphen/underscore)
Small changes to enhance visual appearance: - Opacity setting added for points and line layers. This gives better results than using a dynamic opacity based on property. It's the density of points or lines and the subtlety of the opacity that make it look more professional. - Maximum layers setting removed. This had the potential to cause confusion over what was seen in layers versus the data actually loaded (in the heatmap for example). - A return to cycling the layer palette colours instead of using the last colour for all subsequent layers. It's more helpful that way. - The table name is now used as the layer name for single layers. - Layer palettes now begin with the same colour used by the single layer for consistency. - Stylesheet tweaks around the layer control panel. - Opacity control re-added to the layer control panel.
This allows a coastline to be drawn above all other layers, ensuring that land edges remain visible even when covered with other data.
Switch to the static API method that adds a hash to resources so they can be reloaded when changed. This assists development workflow when using the Datasette `--reload` switch. The static method is new for Datasette v1.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A web-based dashboard for viewing and comparing cyclone trajectories. Tracks and observations are viewed on a map.
See dashboard/README.md for full instructions.
In short, once you have your
venvsetup:I've sent a prebuilt database file via Slack that can get you started quickly.
Copy the file to the root of your checkout and then run this command (this will be simplified to one statement in later versions):
This takes you to the Datasette homepage, where you can select your database and explore tables and views. The views have the most useful data structures and also trigger the map view.
These direct links will take you to the data views:
http://127.0.0.1:8001/6hr_track_mm_oceans_1980on/points
http://127.0.0.1:8001/6hr_track_mm_oceans_1980on/tracks_layer_year
Note that the ocean and landfall detection are in a separate branch, so the filters are not as useful as the version I've been showing recently. The database I've attached does have the ocean and landfall data within so if you link the metadata to your database as described in the README, you can then add the
ocean_idandlandfallcolumns to thefacetskey, under thepointsview, for example.The line count is mostly due to the two style sheets. Changes to existing code are fairly minimal.