[TeaDrinkingProgrammer]

Description

At the LF Energy summit, @nicburgt ran into the FlexMeasures UI breaking during demos on the spotty wifi: missing styles, tables and charts not rendering (flexmeasures-openadr3#4). The cause: the UI loads almost all of its frontend libraries from CDNs (jsdelivr, unpkg, cdnjs, cdn.datatables.net, ajax.aspnetcdn.com, d3js.org), plus its font from Google Fonts and its logo from artwork.lfenergy.org. The same goes for anyone running FlexMeasures without (reliable) internet access.

We moved to CDNs in #21 (issue #18) as "just a small speed tweak", explicitly open to revisiting "if someone reports a problem with CDNs failing". This is that report. The speed argument has also weakened since: browsers partition their HTTP cache per site, so a library cached by another site no longer helps.

I propose serving everything from FlexMeasures itself:

• Vendored files: all 19 libraries live under flexmeasures/ui/static/vendor/<lib>/<version>/, committed, so the wheel and Docker image ship them without a build step (about 5 MB).
• One place for versions: flexmeasures/ui/utils/vendor_versions.py, available in every template as js_versions (now a Jinja global, so login and error pages get it too).
• One place for tags: templates/includes/vendor.html holds Jinja macros, the only place a third-party <script>/<link> is written. Templates call those.
• Caching: versioned vendor files are served with Cache-Control: max-age=31536000, immutable, since a new version gets a new path.
• Asset type icons: the iconify SVGs in SVG_ICON_MAPPING are served locally too (separate commit, easy to drop if we'd rather not).

While at it I cleaned up the versioning, which had drifted:

• DataTables loaded 2.0.8 and then 1.10.22 on top of it. Every table is set up after both have loaded, so 1.10.22 is what ran on every page: I kept that and dropped 2.0.8 (and the unused Buttons CSS).
• Leaflet loaded at 1.7.1 on asset pages and 1.9.4 on the dashboard: unified on 1.9.4 (MarkerCluster likewise on 1.5.3).
• Unpinned references are now pinned: D3 v6 to 6.7.0, Litepicker to 2.0.12 (today's latest), echarts@5 to 5.6.0.
• No other upgrades: this PR is meant to change where files come from, not what they do. Bumps can follow separately.

Breaking for plugins (the part I'm least happy with): FLEXMEASURES_JS_VERSIONS is deprecated and ignored, with a warning if a host still sets it, and it no longer has a default. A plugin that reads config["FLEXMEASURES_JS_VERSIONS"] directly will get a KeyError. Plugin templates reading js_versions.vega etc. keep working, with the same keys.

• Added changelog item in documentation/changelog.rst

Look & Feel

Nothing should look different. The only visible changes are minor version alignments: the DataTables sort icons come from 1.10.22 instead of 1.10.0, and asset page maps run Leaflet 1.9.4 like the dashboard already did.

How to test

• pytest flexmeasures/ui/tests/test_vendor_assets.py: checks that every vendored file is served, CSS font/image references resolve, no template or stylesheet points at an external host, the login page loads local assets, the deprecation warning fires, and versioned files are cached long.
• Offline: open the UI, set devtools Network to "Offline" (or block outbound traffic), and click through dashboard, assets, an asset's context/graphs/properties/status/automations, a sensor page, users and accounts. I did this with a headless Chromium blocking every non-localhost request: all pages render with no console errors, and the only external requests left are Mapbox map tiles.

Further Improvements

• Map tiles still come from Mapbox: offline, maps show markers on an empty background. Fixing that needs a tile server, which I think is out of scope here.
• Licences: we now redistribute these files, and not all notices come along (e.g. JSON-Editor's LICENSE.txt, ECharts' NOTICE, CC BY attribution for one Font Awesome icon). I'd like input on how we want to handle this: a licence table in static/vendor/README.md, or a LICENSES directory.
• The icon paths are root-relative (/ui/static/...), like some existing paths, so they break under a URL-prefix deployment.
• The login page now also loads the chart libraries. Harmless once cached, and the dashboard needs them right after login anyway.

Related Items

Refs #18, #21. Came up in flexmeasures-openadr3#4.

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #35236841 | 📁 Comparing c645670 against latest (e676b09)

  🔍 Preview build  

27 files changed · + 2 added · ± 25 modified

+ Added

• _autosummary/flexmeasures.ui.utils.fetch_vendor_assets.html
• _autosummary/flexmeasures.ui.utils.vendor_assets.html

± Modified

• changelog.html
• configuration.html
• genindex.html
• py-modindex.html
• _autosummary/flexmeasures.api.common.rate_limiting.html
• _autosummary/flexmeasures.api.common.utils.api_utils.html
• _autosummary/flexmeasures.api.common.utils.args_parsing.html
• _autosummary/flexmeasures.api.v3_0.sensors.html
• _autosummary/flexmeasures.data.models.reporting.aggregator.html
• _autosummary/flexmeasures.data.schemas.reporting.aggregation.html
• and 15 more...

[Flix6x]

Thanks for this, @TeaDrinkingProgrammer! Before we go further, I opened #2718 to discuss how we want to ship these libraries in the long run. The macros, vendor_versions.py and the version clean-up look worth keeping either way.

[Flix6x]

Hi @TeaDrinkingProgrammer, following #2718, where @nhoening and I agreed on fetching the libraries from a manifest with a CDN fallback (option B), I pushed a rework of this PR on top of your commits. A lot of your work carried over unchanged: the vendor.html macros as the one place for third-party tags, the version clean-up, the cache headers, the deprecation of FLEXMEASURES_JS_VERSIONS, the local logos and icons. If you disagree with the direction, please say so on #2718, so we keep that discussion in one place.

What changed

• A manifest instead of committed files. flexmeasures/ui/utils/vendor_assets.py replaces vendor_versions.py. It records each library's version, its files with their SHA-256 hashes, and its licence. JS_VERSIONS is derived from it and keeps the legacy keys, so plugin templates reading js_versions.vega keep working. The library files are no longer committed, and static/vendor/ is git-ignored.
• A fetch command. python -m flexmeasures.ui.utils.fetch_vendor_assets downloads the files and each library's licence into static/vendor/<key>/<version>/, and fails on any file whose hash doesn't match. --print-hashes <key> prints the hashes for a new version, so a bump means editing a version, printing its hashes and pasting them in.
• CDN fallback. The macros link a file from FlexMeasures if it was fetched, and from its CDN otherwise. The CDN tags then carry the recorded hash as their integrity attribute, so browsers check each file too. Local paths follow each npm package's own layout, so the relative font and image references inside the stylesheets resolve the same way locally and on the CDN.
• Docker image. The image fetches the libraries at build time, and the Docker CI job checks that it has all of them. The wheel doesn't include them, so headless users don't download them: it stays at 2.8 MB, even when built from a checkout that fetched them.
• Fonts and sort icons. The Poppins @font-face rules and the DataTables sort-icon rules moved from flexmeasures.css into the head() macro, since a static stylesheet can't tell whether those files were fetched. That also removes a stray my at the top of flexmeasures.css, which made browsers drop the first rule in it.
• Icons. The asset type icons stay committed, in static/icons/, because Iconify renders them on request, so they have no fixed release to fetch from. A README there lists each icon set's licence, and the licence texts are included.
• Startup log. On startup, FlexMeasures logs at INFO level when the libraries were not fetched, with the command that fetches them.
• Docs. A new section in the installation docs covers serving the UI's libraries without internet access, and the changelog entry and configuration docs are updated.

How I checked it

• Every recorded hash matches the live upstream file. Your committed copies matched upstream byte for byte too, so nothing changes in what browsers run. I kept DataTables on cdn.datatables.net, because its npm build differs from the files the UI loaded so far.

• I clicked through the UI (login, dashboard, assets, an asset's context, graphs, properties and status pages, a sensor page, users, accounts) in a headless browser, comparing main with this branch in two modes:

  • Not fetched: every library loads from its CDN with no integrity or CORS errors.
  • Fetched, with every non-local host blocked: no request leaves localhost apart from Mapbox tiles.

  Charts render the same as on main once settled, the date picker opens, and Poppins and the sort icons apply. The login page also loses the vega is not defined console error that main has, since the chart libraries now load there.

• The new tests cover both modes, the fetch command's hash check and the startup log, and each fails when the behaviour it covers is broken.

Follow-ups, stacked on this branch:

• Check the UI's third-party libraries for advisories every week #2745: a weekly check of these libraries against the OSV advisory database, for Track updates and advisories for the UI's frontend libraries #2725.
• Upgrade Vega to 5.33.1 #2743: Vega 5.22.1 → 5.33.1.
• Upgrade DataTables to 1.13.11 #2744: DataTables 1.10.22 → 1.13.11.

[Flix6x]

@TeaDrinkingProgrammer, one more change: we'd like hosts to keep the option of loading other versions, so FLEXMEASURES_JS_VERSIONS is no longer deprecated. I pushed that as c645670.

How the setting works now

• It only lists the libraries a host wants at another version, on top of the versions in vendor_assets.py. Its default is {}.
• Each entry is a version, or a version together with SHA-256 hashes:

  FLEXMEASURES_JS_VERSIONS = {
      "vega": "5.33.1",                     # loaded from its CDN, unchecked
      "vegaembed": ("6.26.0", "<sha256>"),  # a library with one file
      "leaflet": ("1.9.5", {"dist/leaflet.css": "<sha256>", ...}),  # a hash per file
  }

• With hashes, a library is handled like the recorded ones: CDN tags carry the integrity attribute, and the fetch command stores and checks its files. The command reads the setting from the config file FlexMeasures reads, or from the one passed with --config.
• Without hashes, browsers load the library from its CDN unchecked, and it isn't fetched. The startup log warns about that, with the command that prints the entry to paste: python -m flexmeasures.ui.utils.fetch_vendor_assets --print-hashes vega --version 5.33.1.
• Files: an overridden version is expected to have the same files as the recorded one, since the templates name them. A licence of another version is fetched without a hash check, because it isn't executed. If it's missing, as for Vega 5.33.1 on npm, only a warning is logged.
• Copies of the old defaults: a version equal to the recorded one is ignored. A host who copied the setting's old defaults into their config therefore keeps the hash checks.
• Mistakes: hashes that can't be matched to a library's files stop the app from starting, rather than letting it load files unchecked. That covers one hash for a library with several files, a dict that misses a file, or a malformed hash. An unknown library key is logged and ignored.
• Templates: js_versions follows the setting, so plugin templates see the versions actually loaded.

The configuration docs describe the forms, and I updated the changelog entry. I checked it end to end against the CDNs, overriding Vega and Leaflet with hashes and D3 without, and every new test fails when the behaviour it covers is broken.

One more follow-up, stacked on this branch: #2746 has FlexMeasures fetch missing library files itself, in the background after the first request a web process serves. Hosts then no longer need to run the command after each upgrade.

As before, if you'd rather go another way, please say so on #2718.

[Flix6x]

Thanks for this. It was about time we revisited this.