Serve UI assets from FlexMeasures instead of CDNs

.github/workflows/docker-build.yml

@@ -53,6 +53,9 @@ jobs:
           >> .env
       - name: Keep running flexmeasures container in background
         run: docker run -t -d --env-file .env --network=host --name fm-container flexmeasures:latest
+      - name: Check that the image serves the UI's libraries itself
+        run: docker exec fm-container python -c
+          "from flexmeasures.ui.utils.vendor_assets import missing_files; assert not missing_files(), missing_files()"
       - name: Execute database upgrade
         run: docker exec --env-file .env fm-container flexmeasures
           db upgrade

.gitignore

@@ -28,6 +28,8 @@ __pycache__
 notebooks/.ipynb_checkpoints/
 
 flexmeasures/ui/static/documentation
+# The UI's third-party libraries, when fetched with: python -m flexmeasures.ui.utils.fetch_vendor_assets
+flexmeasures/ui/static/vendor/
 documentation/img/screenshot_*
 documentation/_autosummary/
 generic_asset_fm_user_ownership.sql

Dockerfile

@@ -50,6 +50,11 @@ RUN --mount=type=cache,target=/root/.cache/uv \
     SETUPTOOLS_SCM_PRETEND_VERSION="${FLEXMEASURES_VERSION}" \
     uv sync --frozen --reinstall-package flexmeasures --no-dev --no-editable
 
+# Fetch the UI's third-party libraries into the installed package, so the UI also works without internet access.
+# Each file is checked against the hash recorded in flexmeasures/ui/utils/vendor_assets.py, and the build fails on a mismatch.
+# It runs outside /app, because python -m would otherwise import the source copy in /app/flexmeasures, which the runtime image leaves behind.
+RUN cd / && python -m flexmeasures.ui.utils.fetch_vendor_assets
+
 # Install gunicorn separately since it's not a dependency of the project
 RUN --mount=type=cache,target=/root/.cache/uv \
     uv pip install gunicorn==25.0.3

documentation/changelog.rst

@@ -40,6 +40,7 @@ New features
 
 Infrastructure / Support
 -------------------------
+* The UI can work without internet access: the Docker image serves the UI's JavaScript and CSS libraries itself, and a pip-installed FlexMeasures can fetch them once (see :ref:`install-ui-libraries`), while otherwise browsers keep loading them from CDNs; the ``FLEXMEASURES_JS_VERSIONS`` setting now only lists the versions a host wants to differ, optionally with the hashes of their files, so plugins reading it should read the ``js_versions`` template variable instead [see `PR #2711 <https://www.github.com/FlexMeasures/flexmeasures/pull/2711>`_]
 * ``add_beliefs_to_db_and_enqueue_forecasting_jobs`` now raises ``BeliefReplacementNotAllowed`` rather than an ``IntegrityError`` when saving would change existing values, so plugins catching ``IntegrityError`` around it should catch the new exception instead [see `PR #2722 <https://www.github.com/FlexMeasures/flexmeasures/pull/2722>`_]
 * ``[GET] /sensors/(id)`` now answers a sensor which does not exist with a 404, as the asset endpoints do, rather than with a 422, so that a client can tell a sensor which is gone apart from a request it got wrong [see `PR #2650 <https://www.github.com/FlexMeasures/flexmeasures/pull/2650>`_]
 * Speed up preparing a storage schedule by validating each device's constraints with vectorised comparisons instead of one ``DataFrame.eval`` per check (from ~25 ms to ~3 ms per device); the helpers ``validate_constraint`` and ``sanitize_expression`` in ``flexmeasures.data.models.planning.storage``, which nothing in FlexMeasures called any more, are removed, with ``validate_constraints`` replacing them [see `PR #2641 <https://www.github.com/FlexMeasures/flexmeasures/pull/2641>`_]

documentation/configuration.rst

@@ -267,7 +267,30 @@ Default: ``{"renewables": ["solar", "wind"], "EVSE": ["one-way_evse", "two-way_e
 FLEXMEASURES_JS_VERSIONS
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
-Default: ``{"vega": "5.22.1", "vegaembed": "6.20.8", "vegalite": "5.2.0"}``
+Versions of the UI's JavaScript and CSS libraries to load, in place of the versions listed in ``flexmeasures/ui/utils/vendor_assets.py``.
+Only list the libraries you want to differ, as a listed version stays in place when an upgrade of FlexMeasures bumps that library.
+A library's key maps to its version, optionally together with the SHA-256 hashes of its files:
+
+.. code-block:: python
+
+    FLEXMEASURES_JS_VERSIONS = {
+        # A version only: browsers load the library from its CDN without checking it.
+        "vega": "5.33.1",
+        # A version and the hash of the library's one file.
+        "vegaembed": ("6.26.0", "<sha256>"),
+        # A version and the hash of each of the library's files.
+        "leaflet": ("1.9.5", {"dist/leaflet.css": "<sha256>", "dist/leaflet-src.min.js": "<sha256>", ...}),
+    }
+
+With hashes, browsers check each file they load from a CDN, and the library can be served by FlexMeasures itself (see :ref:`install-ui-libraries`).
+FlexMeasures expects another version to have the same files as the listed one.
+This prints the setting for another version, with the hashes of its files:
+
+.. code-block:: bash
+
+   $ python -m flexmeasures.ui.utils.fetch_vendor_assets --print-hashes vega --version 5.33.1
+
+Default: ``{}``
 
 
 Timing

documentation/host/installation.rst

@@ -451,6 +451,26 @@ This suffices for a quick start. For an actually secure secret, here is a Python
 Finally, you should set ``SECURITY_TWO_FACTOR_RESCUE_MAIL`` to an email address under which you can be reached by users who cannot use their 2nd means of authentication. 
 
 
+.. _install-ui-libraries:
+
+Serve the UI's libraries without internet access
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The UI uses third-party JavaScript and CSS libraries, such as Bootstrap and Vega.
+The Docker image comes with them, so FlexMeasures serves them itself.
+A FlexMeasures installed via `pip` does not, so browsers load them from public CDNs, checking each file against its expected hash.
+
+To serve them from FlexMeasures instead, for example on a network without internet access, fetch them once:
+
+.. code-block:: bash
+
+   $ python -m flexmeasures.ui.utils.fetch_vendor_assets
+
+This downloads each library's files and licence into FlexMeasures' installation, and fails on any file that does not match its expected hash.
+It fetches the versions set by :ref:`FLEXMEASURES_JS_VERSIONS <configuration>` too, read from the config file FlexMeasures reads (or from the one passed with ``--config``), except for versions set without hashes.
+Run it again after upgrading FlexMeasures, as an upgrade can bump a library's version.
+
+
 Where to go from here?
 ------------------------
 

documentation/tut/building_uis.rst

@@ -212,8 +212,8 @@ First, we define a div tag for the chart and a basic layout (full width). We als
 
     <script src="https://d3js.org/d3.v6.min.js"></script>
     <script src="https://cdn.jsdelivr.net/npm/vega@5.22.1"></script>
-    <script src="https://cdn.jsdelivr.net/npm/vega-lite@5.2.0"></script>
-    <script src="https://cdn.jsdelivr.net/npm/vega-embed@6.20.8"></script>
+    <script src="https://cdn.jsdelivr.net/npm/vega-lite@5.5.0"></script>
+    <script src="https://cdn.jsdelivr.net/npm/vega-embed@6.21.0"></script>
     <script>
         vega.expressionFunction('quantityWithUnitFormat', function(datum, params) {
             return d3.format(params[0])(datum) + " " + params[1];
@@ -264,7 +264,11 @@ Here are some common parameter choices for our JavaScript function:
 
 As FlexMeasures uses `the Vega-Lite Grammar of Interactive Graphics <https://vega.github.io/vega-lite/>`_ internally, we also need to import this library to render the chart (see the ``script`` tags above). It's crucial to note that FlexMeasures is not transferring images across HTTP here, just information needed to render them.
 
-.. note:: It's best to match the visualization library versions you use in your frontend to those used by FlexMeasures. These are set by the FLEXMEASURES_JS_VERSIONS config (see :ref:`configuration`) with defaults kept in ``flexmeasures/utils/config_defaults``.
+.. note:: It's best to match the visualization library versions you use in your frontend to those used by FlexMeasures, which are listed in ``flexmeasures/ui/utils/vendor_assets.py``.
+          A plugin page which extends FlexMeasures' ``base.html`` already loads them, from FlexMeasures itself if they were fetched (see :ref:`install-ui-libraries`), and from CDNs otherwise.
+          A plugin page served by FlexMeasures which does not extend ``base.html`` can load the very same files:
+          ``{% import "includes/vendor.html" as vendor %}{{ vendor.scripts() }}`` loads Vega, Vega-Lite, Vega-Embed and the other chart libraries,
+          and templates can read the versions as ``js_versions``, e.g. ``{{ js_versions.vegalite }}``.
 
 Now let's call this function when the HTML page is opened, to embed our chart:
 

flexmeasures/ui/__init__.py

@@ -32,9 +32,38 @@
     parse_config_entry_by_account_roles,
     find_first_applicable_config_entry,
 )
+from flexmeasures.ui.utils.vendor_assets import (
+    APP_EXTENSION,
+    active_libraries,
+    libraries_with_overrides,
+    missing_files,
+    unhashed_libraries,
+    vendor_attrs,
+    vendor_url,
+)
+
+
+class UIBlueprint(Blueprint):
+    """A blueprint whose vendored files, under static/vendor/<lib>/<version>/, browsers may keep for a year."""
+
+    def send_static_file(self, filename: str):
+        response = super().send_static_file(filename)
+        parts = filename.split("/")
+        libraries = active_libraries()
+        # A new version of a library gets a new path, so a cached copy is never stale.
+        if (
+            len(parts) > 3
+            and parts[0] == "vendor"
+            and parts[1] in libraries
+            and libraries[parts[1]].version == parts[2]
+        ):
+            response.cache_control.max_age = 31536000
+            response.cache_control.immutable = True
+        return response
+
 
 # The ui blueprint. It is registered with the Flask app (see app.py)
-flexmeasures_ui = Blueprint(
+flexmeasures_ui = UIBlueprint(
     "flexmeasures_ui",
     __name__,
     static_folder="static",
@@ -81,8 +110,34 @@ def favicon():
     from flexmeasures.ui.error_handlers import add_html_error_views
 
     add_html_error_views(app)
+    load_vendor_libraries(app)
     add_jinja_filters(app)
     add_jinja_variables(app)
+    log_where_vendor_files_load_from(app)
+
+
+def load_vendor_libraries(app: Flask):
+    """Settle which versions of the UI's libraries the app loads, applying the FLEXMEASURES_JS_VERSIONS setting."""
+    app.extensions[APP_EXTENSION] = libraries_with_overrides(
+        app.config.get("FLEXMEASURES_JS_VERSIONS")
+    )
+
+
+def log_where_vendor_files_load_from(app: Flask):
+    """Tell hosts whether browsers load the UI's libraries from CDNs, and how to serve them from FlexMeasures instead."""
+    libraries = app.extensions[APP_EXTENSION]
+    missing = missing_files(libraries)
+    if missing:
+        app.logger.info(
+            f"{len(missing)} file(s) of the UI's libraries were not fetched, so browsers load them from CDNs. "
+            "To serve them from FlexMeasures, for example without internet access, run: python -m flexmeasures.ui.utils.fetch_vendor_assets"
+        )
+    for key in unhashed_libraries(libraries):
+        version = libraries[key].version
+        app.logger.warning(
+            f"FLEXMEASURES_JS_VERSIONS sets {key} to {version} without hashes, so browsers load it from its CDN without checking it, and it is not fetched. "
+            f"To have it checked, set the hashes printed by: python -m flexmeasures.ui.utils.fetch_vendor_assets --print-hashes {key} --version {version}"
+        )
 
 
 def _tolerate_non_json_job_data():
@@ -221,6 +276,12 @@ def add_jinja_variables(app):
         ("FLEXMEASURES_PUBLIC_DEMO_CREDENTIALS", ""),
     ):
         app.jinja_env.globals[v] = app.config.get(v, d)
+    # A global rather than a per-render variable, so pages rendered without render_flexmeasures_template (e.g. login, errors) also get it.
+    app.jinja_env.globals["js_versions"] = {
+        key: library.version for key, library in app.extensions[APP_EXTENSION].items()
+    }
+    app.jinja_env.globals["vendor_url"] = vendor_url
+    app.jinja_env.globals["vendor_attrs"] = vendor_attrs
     app.jinja_env.globals["sphinx_docs_exist"] = (
         True
         if os.path.exists(

flexmeasures/ui/static/css/flexmeasures.css

@@ -1,32 +1,6 @@
-my/*
+/*
   Styling for the FlexMeasures Platform
 */
-/* 
-
-/* devanagari */
-@font-face {
-  font-family: 'Poppins';
-  font-style: normal;
-  font-weight: 400;
-  src: url(https://fonts.gstatic.com/s/poppins/v20/pxiEyp8kv8JHgFVrJJbecmNE.woff2) format('woff2');
-  unicode-range: U+0900-097F, U+1CD0-1CF6, U+1CF8-1CF9, U+200C-200D, U+20A8, U+20B9, U+25CC, U+A830-A839, U+A8E0-A8FB;
-}
-/* latin-ext */
-@font-face {
-  font-family: 'Poppins';
-  font-style: normal;
-  font-weight: 400;
-  src: url(https://fonts.gstatic.com/s/poppins/v20/pxiEyp8kv8JHgFVrJJnecmNE.woff2) format('woff2');
-  unicode-range: U+0100-024F, U+0259, U+1E00-1EFF, U+2020, U+20A0-20AB, U+20AD-20CF, U+2113, U+2C60-2C7F, U+A720-A7FF;
-}
-/* latin */
-@font-face {
-  font-family: 'Poppins';
-  font-style: normal;
-  font-weight: 400;
-  src: url(https://fonts.gstatic.com/s/poppins/v20/pxiEyp8kv8JHgFVrJJfecg.woff2) format('woff2');
-  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;
-}
 
 body {
     padding-top: 100px;  /* This default is overridden by flexmeasures.js to support a fluid navbar */
@@ -854,17 +828,6 @@ div.dataTables_paginate {
      vertical-align: middle;
 }
 
-/* sorting icons on left when header is not right aligned */
-
-table.dataTable thead tr th:not(.text-right).sorting_asc {
-  background: url("https://cdn.datatables.net/1.10.0/images/sort_asc.png") no-repeat center left;
-}
-table.dataTable thead tr th:not(.text-right).sorting_desc {
-  background: url("https://cdn.datatables.net/1.10.0/images/sort_desc.png") no-repeat center left;
-}
-table.dataTable thead tr th:not(.text-right).sorting {
-  background: url("https://cdn.datatables.net/1.10.0/images/sort_both.png") no-repeat center left;
-}
 table.dataTable thead tr th.no-sort {
     padding: 9px;
 }