diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 583c4d79b1..4d6f33c605 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -47,6 +47,7 @@ Infrastructure / Support * Upgrade DataTables, which draws the UI's lists, from 1.10.22 to 1.13.11, its latest 1.x release [see `PR #2744 `_] * Speed up preparing a storage schedule by validating each device's constraints with vectorised comparisons instead of one ``DataFrame.eval`` per check (from ~25 ms to ~3 ms per device); the helpers ``validate_constraint`` and ``sanitize_expression`` in ``flexmeasures.data.models.planning.storage``, which nothing in FlexMeasures called any more, are removed, with ``validate_constraints`` replacing them [see `PR #2641 `_] * Rate-limit simplification: Triggers for schedules and forecasts are now always counted per organisation, against one budget that all of its users and assets share, so plans no longer choose what triggers are counted against, and the ``FLEXMEASURES_API_RATE_LIMIT_KEY`` setting is deprecated and ignored. Hosts who set it (or a plan's rate-limit key) to ``account+asset`` or ``user`` may want to raise the trigger rate limit of the affected plans, or ``FLEXMEASURES_API_TRIGGER_RATE_LIMIT``, before upgrading [see `PR #2679 `_] +* A pip-installed FlexMeasures now fetches the UI's JavaScript and CSS libraries itself, in the background once it serves its first request, so it serves them itself without a host running a command, also after an upgrade bumps one of them; set ``FLEXMEASURES_FETCH_UI_LIBRARIES`` to ``False`` to switch this off [see `PR #2746 `_] * Rate-limit simplification: The API's default rate limit is now counted per organisation rather than per user, so that all users and API clients of an organisation share the one budget its plan sets; it stays at 500 requests per minute, which hosts whose organisations run several integrations may want to raise on those organisations' plans (``default_rate_limit``) or server-wide (``FLEXMEASURES_API_DEFAULT_RATE_LIMIT``) before upgrading [see `PR #2677 `_] * Rate-limit clarification: When a request is refused for hitting a rate limit, the server log now says who made it (account and user), on which endpoint, against which limit and, for the trigger limit, about which asset, so hosts can tell which integration used up an account's budget; with debug logging on, each trigger that spends budget is logged, too. The organisation audit log also records one breach per API or trigger limit window, with the plan, effective limit and how far into the window the breach occurred. The plan tooltip notes when play mode disables both limits [see `PR #2678 `_] * Rate-limit simplification: Hosts can reset an organisation's rate limits with ``flexmeasures edit reset-rate-limit``, so that an organisation whose integration has been fixed after running into a rate limit need not wait out the rest of the window. A rate limit (on a plan, or in ``FLEXMEASURES_API_DEFAULT_RATE_LIMIT`` or ``FLEXMEASURES_API_TRIGGER_RATE_LIMIT``) now holds one limit: several limits separated by semicolons used to be accepted without being documented, but are now refused, and a server with such a setting no longer starts [see `PR #2680 `_] diff --git a/documentation/configuration.rst b/documentation/configuration.rst index 0df349c0bb..29be6eba51 100644 --- a/documentation/configuration.rst +++ b/documentation/configuration.rst @@ -292,6 +292,14 @@ This prints the setting for another version, with the hashes of its files: Default: ``{}`` +FLEXMEASURES_FETCH_UI_LIBRARIES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Whether FlexMeasures fetches the files of the UI's JavaScript and CSS libraries which it is missing, so that it serves them itself rather than browsers loading them from CDNs. +The first request that a web server process serves starts the download, in the background (see :ref:`install-ui-libraries`). + +Default: ``True`` + Timing ------ diff --git a/documentation/host/installation.rst b/documentation/host/installation.rst index 8a4d5d0afd..569b4ca016 100644 --- a/documentation/host/installation.rst +++ b/documentation/host/installation.rst @@ -460,15 +460,23 @@ The UI uses third-party JavaScript and CSS libraries, such as Bootstrap and Vega The Docker image comes with them, so FlexMeasures serves them itself. A FlexMeasures installed via `pip` does not, so browsers load them from public CDNs, checking each file against its expected hash. -To serve them from FlexMeasures instead, for example on a network without internet access, fetch them once: +To serve them from FlexMeasures instead, for example for users on a network without internet access, FlexMeasures fetches them itself: +once it serves its first request, it downloads the files it is missing in the background, while browsers keep loading them from CDNs until it is done. +That also happens after an upgrade of FlexMeasures bumps a library's version. +The server needs internet access for this, and write access to FlexMeasures' installation. +If several web server processes run on one host, only one of them fetches. + +You can also fetch them yourself, for example on a server without internet access, before taking its installation offline: .. code-block:: bash $ python -m flexmeasures.ui.utils.fetch_vendor_assets -This downloads each library's files and licence into FlexMeasures' installation, and fails on any file that does not match its expected hash. -It fetches the versions set by :ref:`FLEXMEASURES_JS_VERSIONS ` too, read from the config file FlexMeasures reads (or from the one passed with ``--config``), except for versions set without hashes. -Run it again after upgrading FlexMeasures, as an upgrade can bump a library's version. +Both download each library's files and licence into FlexMeasures' installation, and fail on any file that does not match its expected hash. +They fetch the versions set by :ref:`FLEXMEASURES_JS_VERSIONS ` too, except for versions set without hashes. +The command reads that setting from the config file FlexMeasures reads, or from the one passed with ``--config``. +To stop FlexMeasures from fetching the libraries itself, set :ref:`FLEXMEASURES_FETCH_UI_LIBRARIES ` to ``False``. +Then run the command again after each upgrade of FlexMeasures, as an upgrade can bump a library's version. Where to go from here? diff --git a/flexmeasures/ui/__init__.py b/flexmeasures/ui/__init__.py index 96fe14d1e6..a8d31003ea 100644 --- a/flexmeasures/ui/__init__.py +++ b/flexmeasures/ui/__init__.py @@ -4,6 +4,7 @@ import json import os +import threading from datetime import date, datetime from flask import current_app, Flask, Blueprint, send_from_directory, request @@ -35,6 +36,8 @@ from flexmeasures.ui.utils.vendor_assets import ( APP_EXTENSION, active_libraries, + fetch_unless_fetching, + fetched_before, libraries_with_overrides, missing_files, unhashed_libraries, @@ -114,6 +117,7 @@ def favicon(): add_jinja_filters(app) add_jinja_variables(app) log_where_vendor_files_load_from(app) + fetch_vendor_files_on_first_request(app) def load_vendor_libraries(app: Flask): @@ -127,8 +131,15 @@ def log_where_vendor_files_load_from(app: Flask): """Tell hosts whether browsers load the UI's libraries from CDNs, and how to serve them from FlexMeasures instead.""" libraries = app.extensions[APP_EXTENSION] missing = missing_files(libraries) - if missing: + if missing and app.config.get("FLEXMEASURES_FETCH_UI_LIBRARIES"): app.logger.info( + f"{len(missing)} file(s) of the UI's libraries were not fetched yet, so browsers load them from CDNs. " + "FlexMeasures fetches them in the background once it serves its first request." + ) + elif missing: + # Files fetched before, but missing now, most likely belong to versions which an upgrade bumped. + log = app.logger.warning if fetched_before() else app.logger.info + log( f"{len(missing)} file(s) of the UI's libraries were not fetched, so browsers load them from CDNs. " "To serve them from FlexMeasures, for example without internet access, run: python -m flexmeasures.ui.utils.fetch_vendor_assets" ) @@ -140,6 +151,48 @@ def log_where_vendor_files_load_from(app: Flask): ) +def fetch_vendor_files_on_first_request(app: Flask): + """Have the first request which a process serves start fetching the UI's missing library files, in the background. + + Only web processes serve requests, so CLI commands and job workers never fetch. + The request does not wait for the fetch, and browsers load the files from CDNs until they are fetched. + Of the web processes on a host, only the first one to get the lock fetches. + """ + if not app.config.get("FLEXMEASURES_FETCH_UI_LIBRARIES"): + return + libraries = app.extensions[APP_EXTENSION] + # Each web process has its own copy of this, also when gunicorn forks them from a process which loaded the app. + started = threading.Lock() + + @app.before_request + def start_fetching_vendor_files(): + if not started.acquire(blocking=False): + return + thread = threading.Thread( + target=_fetch_vendor_files, + args=(app, libraries), + name="fetch-ui-libraries", + daemon=True, + ) + app.extensions["flexmeasures_vendor_fetch"] = thread + thread.start() + + +def _fetch_vendor_files(app: Flask, libraries): + try: + written = fetch_unless_fetching(libraries) + except (OSError, ValueError) as exc: + app.logger.warning( + f"Could not fetch the UI's libraries, so browsers keep loading them from CDNs: {exc}. " + "To fetch them by hand, run: python -m flexmeasures.ui.utils.fetch_vendor_assets" + ) + return + if written: + app.logger.info( + f"Fetched {written} file(s) of the UI's libraries, which FlexMeasures serves itself from now on." + ) + + def _tolerate_non_json_job_data(): """Let rq-dashboard render jobs whose meta data is not JSON-serializable. diff --git a/flexmeasures/ui/tests/test_vendor_assets.py b/flexmeasures/ui/tests/test_vendor_assets.py index cf5462e592..a1702d5f77 100644 --- a/flexmeasures/ui/tests/test_vendor_assets.py +++ b/flexmeasures/ui/tests/test_vendor_assets.py @@ -483,3 +483,127 @@ def test_unversioned_files_keep_default_caching(client): assert response.status_code == 200 assert response.cache_control.max_age is None response.close() + + +@pytest.fixture +def web_app(not_fetched): + """A bare app on which FlexMeasures' fetching on the first request is set up, so tests can count its fetches.""" + from flask import Flask + + from flexmeasures.ui import fetch_vendor_files_on_first_request + + def make(fetch_setting: bool = True): + web_app = Flask("web_app_under_test") + web_app.config["FLEXMEASURES_FETCH_UI_LIBRARIES"] = fetch_setting + web_app.extensions[APP_EXTENSION] = VENDOR_LIBRARIES + web_app.add_url_rule("/", "index", lambda: "page") + fetch_vendor_files_on_first_request(web_app) + return web_app + + return make + + +@pytest.fixture +def blocking_fetch(monkeypatch): + """Replace fetching with one that waits until the test releases it, and counts how often it was started.""" + import threading + + started = threading.Semaphore(0) + release = threading.Event() + calls = [] + + def fake_fetch(libraries): + calls.append(libraries) + started.release() + release.wait(timeout=5) + return 3 + + monkeypatch.setattr("flexmeasures.ui.fetch_unless_fetching", fake_fetch) + yield started, release, calls + release.set() + + +def test_first_request_fetches_in_the_background(web_app, blocking_fetch): + """The first request starts fetching, without waiting for it, and later requests do not fetch again.""" + started, release, calls = blocking_fetch + app_ = web_app() + client_ = app_.test_client() + + # The fetch is still blocked, so getting a response shows that the request did not wait for it. + assert client_.get("/").data == b"page" + assert app_.extensions["flexmeasures_vendor_fetch"].is_alive() + assert started.acquire(timeout=10) + assert client_.get("/").data == b"page" + release.set() + app_.extensions["flexmeasures_vendor_fetch"].join(timeout=10) + assert len(calls) == 1 + + +def test_no_fetching_when_switched_off(web_app, blocking_fetch): + _, _, calls = blocking_fetch + app_ = web_app(fetch_setting=False) + assert app_.test_client().get("/").data == b"page" + assert "flexmeasures_vendor_fetch" not in app_.extensions + assert calls == [] + + +def test_failed_background_fetch_is_logged(web_app, monkeypatch, caplog): + """A server without internet access logs that it could not fetch, and keeps serving pages.""" + + def failing_fetch(libraries): + raise urllib.error.URLError("no internet") + + monkeypatch.setattr("flexmeasures.ui.fetch_unless_fetching", failing_fetch) + app_ = web_app() + with caplog.at_level(logging.WARNING): + assert app_.test_client().get("/").data == b"page" + app_.extensions["flexmeasures_vendor_fetch"].join(timeout=10) + assert "Could not fetch the UI's libraries" in caplog.text + + +@pytest.mark.skipif(vendor_assets.fcntl is None, reason="file locks need fcntl") +def test_only_one_process_fetches(not_fetched, fake_library): + """While one process holds the lock, another process does not fetch, and fetches once the lock is free.""" + import fcntl + + _, downloads, download = fake_library + libraries = vendor_assets.VENDOR_LIBRARIES + vendor_assets.VENDOR_DIR.mkdir(parents=True) + # Another open file description of the lock file conflicts with the one held here, as one in another process would. + with open(vendor_assets.VENDOR_DIR / vendor_assets.LOCK_FILE, "w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + assert vendor_assets.fetch_unless_fetching(libraries, download) is None + assert downloads == [] + assert vendor_assets.fetch_unless_fetching(libraries, download) == 3 + # Once nothing is missing, the lock is not even taken. + with open(vendor_assets.VENDOR_DIR / vendor_assets.LOCK_FILE, "w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + assert vendor_assets.fetch_unless_fetching(libraries, download) == 0 + + +@pytest.mark.parametrize("fetched_earlier", [False, True]) +def test_startup_log_warns_when_fetched_files_went_missing( + app, caplog, monkeypatch, not_fetched, fetched_earlier +): + """When FlexMeasures does not fetch, a host who fetched before but misses files now, most likely after an upgrade, is warned.""" + monkeypatch.setitem(app.config, "FLEXMEASURES_FETCH_UI_LIBRARIES", False) + if fetched_earlier: + old_file = vendor_assets.VENDOR_DIR / "vega" / "5.0.0" / "build" / "vega.min.js" + old_file.parent.mkdir(parents=True) + old_file.write_text("vega 5.0.0") + + with caplog.at_level(logging.INFO): + log_where_vendor_files_load_from(app) + + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert "run: python -m flexmeasures.ui.utils.fetch_vendor_assets" in caplog.text + assert bool(warnings) is fetched_earlier + + +def test_startup_log_says_fetching_follows(app, caplog, monkeypatch, not_fetched): + monkeypatch.setitem(app.config, "FLEXMEASURES_FETCH_UI_LIBRARIES", True) + with caplog.at_level(logging.INFO): + log_where_vendor_files_load_from(app) + assert ( + "fetches them in the background once it serves its first request" in caplog.text + ) diff --git a/flexmeasures/ui/utils/vendor_assets.py b/flexmeasures/ui/utils/vendor_assets.py index 94c4ae15b0..a655f05ca0 100644 --- a/flexmeasures/ui/utils/vendor_assets.py +++ b/flexmeasures/ui/utils/vendor_assets.py @@ -3,7 +3,8 @@ Each library is served by FlexMeasures from static/vendor/// when its files were fetched there, and loaded from its CDN otherwise, with the hash recorded below as the tag's integrity attribute. The Docker image comes with the files fetched. -To serve them from a pip-installed FlexMeasures, for example without internet access, fetch them once: +A pip-installed FlexMeasures fetches them in the background once it serves its first request, +unless the FLEXMEASURES_FETCH_UI_LIBRARIES setting is switched off; then fetch them with: python -m flexmeasures.ui.utils.fetch_vendor_assets @@ -24,6 +25,7 @@ import base64 import hashlib import logging +import os import re import urllib.error import urllib.request @@ -34,6 +36,11 @@ from flask import current_app, has_app_context, url_for from markupsafe import Markup, escape +try: + import fcntl +except ImportError: # Windows + fcntl = None # type: ignore[assignment] + VENDOR_DIR = Path(__file__).resolve().parents[1] / "static" / "vendor" NPM_CDN = "https://cdn.jsdelivr.net/npm/{name}@{version}/" @@ -296,6 +303,9 @@ def license_path(self) -> str: SHA256 = re.compile(r"[0-9a-f]{64}") +# Held by the process that fetches the libraries, so the other processes on the host leave it to that one. +LOCK_FILE = ".fetch.lock" + def _logger() -> logging.Logger: """The app's logger, or this module's outside an app, such as in the fetch command. @@ -508,12 +518,41 @@ def _store( f"{url} does not match the hash recorded for {key} {libraries[key].version}." ) target.parent.mkdir(parents=True, exist_ok=True) - partial = target.with_name(target.name + ".part") + # Named after the process, so two processes fetching at once do not write into the same file. + partial = target.with_name(f"{target.name}.{os.getpid()}.part") partial.write_bytes(content) partial.replace(target) return 1 +def fetch_unless_fetching( + libraries: dict[str, VendorLibrary], + download: Callable[[str], bytes] = download, +) -> int | None: + """Fetch the libraries' missing files, unless another process on this host is fetching them. + + Returns the number of files written, or None if another process holds the lock. + """ + if not missing_files(libraries): + return 0 + VENDOR_DIR.mkdir(parents=True, exist_ok=True) + with open(VENDOR_DIR / LOCK_FILE, "w") as lock: + if fcntl is not None: + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + return None + # The lock is released when the file closes, also when the process dies. + return fetch(download=download, libraries=libraries) + + +def fetched_before() -> bool: + """Whether any library files were fetched into static/vendor/, for example before an upgrade bumped some versions.""" + return VENDOR_DIR.is_dir() and any( + path.is_file() and path.name != LOCK_FILE for path in VENDOR_DIR.rglob("*") + ) + + def print_hashes( key: str, download: Callable[[str], bytes] = download, diff --git a/flexmeasures/utils/config_defaults.py b/flexmeasures/utils/config_defaults.py index 7183ac7e6b..bc80904ee1 100644 --- a/flexmeasures/utils/config_defaults.py +++ b/flexmeasures/utils/config_defaults.py @@ -196,6 +196,8 @@ class Config(object): FLEXMEASURES_API_RATE_LIMIT_KEY: str = "account" # Versions of the UI's libraries to load instead of those in flexmeasures/ui/utils/vendor_assets.py FLEXMEASURES_JS_VERSIONS: dict = {} + # Fetch the UI's libraries in the background, so FlexMeasures serves them itself rather than browsers loading them from CDNs + FLEXMEASURES_FETCH_UI_LIBRARIES: bool = True FLEXMEASURES_JSON_COMPACT = False OPENAPI_VERSION = "3.1.2" JSON_SORT_KEYS = False @@ -286,6 +288,7 @@ class TestingConfig(Config): DEBUG: bool = True # this seems to be important for logging in, not sure why LOGGING_LEVEL: int = logging.INFO FLEXMEASURES_CREATE_TEMPLATE_ASSETS_ON_STARTUP: bool = False + FLEXMEASURES_FETCH_UI_LIBRARIES: bool = False WTF_CSRF_ENABLED: bool = False # also necessary for logging in during tests SECRET_KEY: str = "dummy-key-for-testing"