diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index c17ccc3ff2..6329fd9a4a 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -5,60 +5,54 @@ API change log .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace. -v3.0-40 | October XX, 2026 +v3.0-34 | November XX, 2026 """""""""""""""""""""""""""" -- The default rate limit, which applies to every endpoint, is now counted per account rather than per user, so that all users and integrations (anything calling the API with a token) of an account share one budget. Unauthenticated requests are still counted per IP address. A consultant counts against their own account, also when acting on a client account's assets. The trigger limit is unaffected. -- Requests which the UI makes with a user's session, rather than with an API token, count against a budget of their own, per user (``FLEXMEASURES_UI_RATE_LIMIT``), rather than against the account's default rate limit. + +**Automations** + +- Added ``GET /api/v3_0/assets//automations``, ``POST /api/v3_0/assets//automations``, ``GET /api/v3_0/assets//automations/``, ``PATCH /api/v3_0/assets//automations/`` and ``DELETE /api/v3_0/assets//automations/`` for listing, inspecting and managing an asset's automations, including the sensors an automation reads from and writes to. Managing them requires the same permission as writing data under the asset, and an automation may only involve sensors its creator can access: read access to the sensors it reads data from, and permission to record data on the sensors it writes to (a ``403`` otherwise). Creation and update accept a ``timezone``, in which the automation's cron expression is interpreted; it defaults to the asset's own timezone, taken from the asset's timezone attribute or one of its sensors, and to the server's ``FLEXMEASURES_TIMEZONE`` if the asset has neither. +- Automation responses report the IANA ``timezone``, a ``cursor`` (the most recent run the automation committed to, which advances just before queueing, so it does not indicate that queueing or the run itself succeeded), a ``next_run`` (the next scheduled run, excluding any catch-up run still pending, and null while the automation is inactive), both as clock times in the automation's own timezone, and a ``schedule-revision``, which counts the execution-affecting edits made to the automation's schedule. Their ``job-stats`` and ``redis-connection-err`` fields are spelled in kebab-case, like the rest. +- Automation detail responses add a ``run-stats`` object, summarising the automation's durable runs and describing the most recent ones: their scheduled time, dispatch state (``pending``, ``claimed``, ``partially_queued``, ``queued`` or ``failed``), execution state (``pending``, ``running``, ``succeeded``, ``failed`` or ``canceled``), attempt count, intended and queued job counts, timestamps, last error, latest attempt, and the individual jobs they created. They also report the data generator's stored configuration as ``config`` on the automation's ``source``, such as the model a forecaster trains and the period it trains over. +- ``GET /api/v3_0/assets//automations`` also lists the automations of the assets below the asset, at any depth, as far as the caller may read them, and each entry names the asset it is defined on in ``asset`` and ``asset-name``. Pass ``include-child-assets=false`` to list only the asset's own automations. +- Added ``POST /api/v3_0/assets//automations//trigger``, to run one automation now, once, on top of its recurring runs, which leaves its recurrence alone and works for inactive automations, too. The response is the standard job response, extended with ``n_jobs``: how many jobs the run queued. Triggering requires the same permission as writing data under the asset, and falls under the trigger rate limit. +- ``POST /api/v3_0/assets//automations`` accepts ``source``, the ID of an existing data source to reuse, instead of a ``data-generator`` and its ``config``; the automation then runs the data generator the source stores, and records its results under that same source. Naming a source is refused with a ``403`` unless the caller may read it, and with a ``422`` where it is combined with ``data-generator`` or ``config``, where it stores no data generator of the automation's type, or where the automation schedules (a schedule automation's source follows from the asset and the flex config). This holds for automation types registered by a plugin, too. +- ``POST /api/v3_0/assets//copy`` now copies the automations of each copied asset, too, inactive and with a fresh cursor, so a copy inherits neither the original's run history nor its queued jobs. Sensor references in a copied automation's parameters and generator configuration are pointed at the copied sensors; a reference to a sensor outside the copied assets is kept only where both the destination organisation and the user making the copy may read it. A copy that lands in another organisation records under a data source of that organisation. Schedule automations are skipped for now, as is any automation that cannot be copied safely, and the response lists each skipped one under a new ``skipped-automations`` field, as ``id``, ``name``, ``asset`` and ``reason``. + +**Data sources and sensor data** + +- Added ``GET /api/v3_0/sources/`` to show the full record of one data source, including the attributes in which data generators store their configuration. +- ``GET /api/v3_0/sources`` now accepts ``filter``, matching search terms against a source's name, its model and its id prefix, ``type``, narrowing the list to one source type, and ``limit``, returning only that many sources, the most recently created ones first. The ``types`` in the response still cover every source the caller may read. +- ``GET /api/v3_0/sources`` and ``GET /api/v3_0/sources/`` now decide access per data source: a source is readable when it belongs to an organisation the caller may read, when an automation they may read computes under it, or when it has recorded data on a sensor they may read. The listing holds only the first two kinds, which are the sources the caller may reuse for an automation of their own, and ``attributes`` is returned only for those, because a data generator's configuration names the sensors it runs on. A data source which names no organisation is no longer readable by every authenticated user for that reason alone. +- Data sources returned by ``GET /api/v3_0/sources`` and ``GET /api/v3_0/sources/`` now carry the ``account_id`` of the organisation they record for, where only sources of type ``user`` used to. Source filters that select by organisation, such as ``source-account`` on ``GET /api/v3_0/sensors//data`` and inside sensor references in a flex-model or flex-context, therefore also match data computed by forecasters, schedulers and reporters. A source belonging to no organisation is one the host runs for everyone. - ``POST /api/v3_0/sensors//data/upload`` now accepts ``prior``, as ``POST /api/v3_0/sensors//data`` does: all uploaded data has been recorded prior to this belief time. It cannot be combined with ``belief-time-measured-instantly``. -- ``GET /api/v3_0/assets//jobs`` now accepts ``page`` and ``per-page`` to return jobs newest first, with ``num-records`` and ``filtered-records`` counts. Paginated results can be sorted by enqueue time or queue with ``sort-by`` and ``sort-dir``. Requests without ``page`` retain the full list and previous response shape. -- Added ``DELETE /api/v3_0/sensors//annotations``, which deletes a sensor's annotations, optionally only those of one ``source`` or lying entirely between ``start`` and ``until``. An annotation that is also registered on another sensor, an asset or an account is only removed from this sensor. It requires permission to delete the sensor, like ``DELETE /api/v3_0/sensors//data``, and reports how many annotations it removed from the sensor as ``deleted``. -- A sensor reference inside a flex-model or flex-context takes the keys listed under :ref:`api_notation` and no others. Unknown fields now get a ``422`` naming the nearest key the reference does take, rather than being read past. Mind the source filters: a data request takes ``"source"`` and ``"source-type"``, each naming one value, while a sensor reference takes ``"sources"`` and ``"source-types"``, each with a list, so a filter written in the singular there used to do nothing at all. +- A sensor reference inside a flex-model or flex-context takes the keys listed under :ref:`api_notation` and no others: an unknown key now gets a ``422`` naming the nearest key the reference does take, rather than being read past. Mind the source filters: a data request takes ``"source"`` and ``"source-type"``, each naming one value, while a sensor reference takes ``"sources"`` and ``"source-types"``, each with a list. - Deleting a sensor leaves ``"sensor": null`` in the stored flex-model and flex-context references to it that carry a default, bounds or source filters, which ``GET /api/v3_0/assets/`` returns as such (see :ref:`deleted_referenced_sensor`). A sensor reference with ``"sensor": null`` sent to the API gets a ``422``, so a flex-context or flex-model holding one can only be saved again once a sensor is picked for it. +- ``GET /api/v3_0/sensors//stats`` now reports an ``All sources`` entry summarising every data source, whenever more than one recorded. Its mean divides by the values that were summed, not by ``Number of values``, which also counts rows holding NaN. +- Added ``DELETE /api/v3_0/sensors//annotations``, which deletes a sensor's annotations, optionally only those of one ``source`` or lying entirely between ``start`` and ``until``, and reports how many it removed as ``deleted``. An annotation that is also registered on another sensor, an asset or an account is only removed from this sensor. It requires permission to delete the sensor, like ``DELETE /api/v3_0/sensors//data``. - ``POST /api/v3_0/accounts//annotations``, ``POST /api/v3_0/assets//annotations`` and ``POST /api/v3_0/sensors//annotations`` now accept an ``end`` equal to the ``start``, which makes an instantaneous annotation, where they used to reply with a ``422``. An ``end`` before the ``start`` is still refused. -- ``POST /api/v3_0/assets//automations`` now accepts ``source``, the ID of an existing data source to reuse, instead of a ``data-generator`` and its ``config``. The source hands over the data generator it stores, and the automation records its results under that same source, so naming one is refused with a ``403`` unless the caller may read it, and with a ``422`` where it is combined with ``data-generator`` or ``config``, where it stores no data generator of the automation's type, or where the automation schedules (a schedule automation's source follows from the asset and the flex config). -- ``GET /api/v3_0/sources`` now accepts ``filter``, matching search terms against a source's name, its model and its id prefix, ``type``, narrowing the list to one source type, and ``limit``, returning only that many sources, the most recently created ones first. The ``types`` in the response still cover every source the caller may read, so that one call can both search the sources and offer the types to search by. -- ``GET /api/v3_0/sources`` and ``GET /api/v3_0/sources/`` now decide access per data source, rather than only by the organisation a source names. A source is readable when it belongs to an organisation the caller may read, when an automation they may read computes under it, or when it has recorded data on a sensor they may read. The listing holds only the first two, which are the sources the caller may reuse for an automation of their own, and ``attributes`` is returned only for those two as well, because a data generator's configuration names the sensors it runs on. A data source which names no organisation is no longer readable for that reason alone, where it used to be readable by every authenticated user. -- Data sources returned by ``GET /api/v3_0/sources`` and ``GET /api/v3_0/sources/`` now carry the ``account_id`` of the organisation they record for, where only sources of type ``user`` used to. Source filters that select by organisation, such as ``source-account`` on ``GET /api/v3_0/sensors//data`` and inside sensor references in a flex-model or flex-context, therefore match data computed by forecasters, schedulers and reporters, where they used to match user-recorded data only. A source belonging to no organisation is one the host runs for everyone. -- ``POST /api/v3_0/assets//automations`` now also refuses a ``source`` which is not the caller's to work with for automation types registered by a plugin, where the refusal used to cover the built-in types only. -- ``GET /api/v3_0/sensors//status`` now returns ``missing-members`` with the status of reporter data whose most recent aggregate of a device group, old enough to count as stale, had to leave out members, because they did not report: the ``id``, ``name`` and ``asset-name`` of each of those sensors. The field is left out where no member is missing, as for most sensors. -- ``GET /api/v3_0/assets//chart`` now answers an unknown ``chart-type`` with a ``422``. -v3.0-39 | September 22, 2026 -"""""""""""""""""""""""""""" -- ``POST /api/v3_0/assets//copy`` now copies the automations of each copied asset, too. A copied automation keeps its name, type, cron expression and timezone, but starts out inactive and with a fresh cursor, so it inherits neither the original's run history nor its queued jobs. Sensor references in its parameters and in its generator configuration are pointed at the copied sensors; a reference to a sensor outside the copied assets is kept only where both the destination organisation and the user making the copy may read it. Forecast and report automations are copied; a schedule automation is skipped for now, as its parameters point at sensors in ways a copy cannot follow. A copy that lands in another organisation records under a data source of that organisation. An automation that cannot be copied safely is skipped rather than failing the copy, and the response lists each one under a new ``skipped-automations`` field, as ``id``, ``name``, ``asset`` and ``reason``. +**Plans and rate limits** -v3.0-38 | September 16, 2026 -"""""""""""""""""""""""""""" -- ``GET /api/v3_0/assets//automations`` now lists the automations of the assets below the asset as well, at any depth, so that a site asset reports everything that runs below it, and each entry names the asset it is defined on in ``asset`` and ``asset-name``. ``GET /api/v3_0/assets//jobs`` lists the jobs of the assets below the asset in the same way. Pass ``include-child-assets=false`` to either one to list only what belongs to the asset itself. Only the assets below it which the caller may read are included, as a child asset can belong to another organisation than its parent. -- ``GET /api/v3_0/assets//automations/`` now reports the data generator's stored configuration as ``config`` on the automation's ``source``, such as the model a forecaster trains and the period it trains over. The configuration is what separates one data source from another of the same model. +- The default rate limit, which applies to every endpoint, is now counted per account rather than per user, so that all users and integrations (anything calling the API with a token) of an account share one budget. Unauthenticated requests are still counted per IP address, and a consultant counts against their own account, also when acting on a client account's assets. The trigger limit is unaffected. +- Requests which the UI makes with a user's session, rather than with an API token, count against a budget of their own, per user (``FLEXMEASURES_UI_RATE_LIMIT``), rather than against the account's default rate limit. -v3.0-37 | September 15, 2026 -"""""""""""""""""""""""""""" -- Added ``POST /api/v3_0/assets//automations``, ``PATCH /api/v3_0/assets//automations/`` and ``DELETE /api/v3_0/assets//automations/`` for managing an asset's automations. They require the same permission as writing data under the asset, and an automation may only involve sensors that its creator can access: read access to the sensors it reads data from, and permission to record data on the sensors it writes to (a ``403`` otherwise). Both the creation and the update accept a ``timezone``, in which the automation's cron expression is interpreted; it defaults to the asset's own timezone, taken from the asset's timezone attribute or one of its sensors, and to the server's ``FLEXMEASURES_TIMEZONE`` if the asset has neither. The automation endpoints also report a ``next_run``: the next scheduled run, null while the automation is inactive, and excluding any catch-up run still pending. Both ``next_run`` and ``cursor`` are now reported as clock times in the automation's own timezone, where ``cursor`` was previously reported in UTC, since a recurrence is read in that timezone. The automation endpoints' last two snake_case fields, ``job_stats`` and ``redis_connection_err``, are now ``job-stats`` and ``redis-connection-err``, like the rest. ``GET /api/v3_0/assets//jobs`` renames its ``redis_connection_err`` to ``redis-connection-err`` as well, so that one field is not spelled two ways; that field shipped in v1.0.0, and is read by FlexMeasures' own UI rather than by the FlexMeasures client. +**Reporting** -v3.0-36 | September 14, 2026 -"""""""""""""""""""""""""""" -- ``GET /api/v3_0/assets`` now scopes a listing the way its parameters say. ``num-records`` respects the ``root`` and ``depth`` constraints, so a listing scoped to an asset subtree reports how many assets that subtree holds, where it used to count every asset in the account and asset-type scope, making a paginated client report the rest as having been filtered out by the search term. And an explicit ``include_public=false`` is now honoured: leaving the parameter out keeps the behaviour it had, where ``all_accessible`` and ``root`` include public assets and a listing of a single account does not, but passing it no longer loses out to those two, so a client can offer the choice. -- ``GET /api/v3_0/sensors//status`` now accepts an optional ``asset_id`` query parameter, naming the asset whose status page the sensor is reported on. - It only affects the reported ``relation``, which previously always claimed that the sensor belongs to the asset shown, also for a sensor of another asset that is only listed because the asset's flex-context or graphs page refers to it. - Without it, the sensor is still reported relative to the asset it belongs to. - Reading the status of a sensor in the context of an asset requires read permission on that asset, too. +- Added ``POST /api/v3_0/assets//reports/trigger`` to queue a one-off report as a background job. It returns ``202 Accepted`` with the canonical ``job`` and ``job-url`` fields, and shares the trigger rate limit with forecast and schedule endpoints. -v3.0-35 | September 9, 2026 -""""""""""""""""""""""""""" -- The ``resolution`` field is now rejected with a ``422 (Unprocessable Entity)`` response unless it spans a positive amount of time. This applies wherever the API accepts one: as a query parameter on ``GET /api/v3_0/sensors//data`` and on the ``chart_data`` endpoints under ``api/dev``, and in the request body of the ``POST`` schedule trigger endpoints. Previously, a zero resolution (such as ``PT0S``) either crashed the request with a ``500`` or was silently ignored, and a negative resolution returned an empty set of values. +**Assets, jobs and charts** -v3.0-34 | September 2, 2026 -""""""""""""""""""""""""""" -- Added ``POST /api/v3_0/assets//automations//trigger``, to run one automation now, once, on top of its recurring runs. The response is the standard job response, extended with ``n_jobs``: how many jobs the run queued. An on-demand run does not affect the automation's recurrence, and inactive automations can be triggered, too. Triggering requires the same permission as writing data under the asset, and falls under the stricter rate limit that the other triggering endpoints share. +- ``GET /api/v3_0/assets//jobs`` now accepts ``page`` and ``per-page`` to return jobs newest first, with ``num-records`` and ``filtered-records`` counts. Paginated results can be sorted by enqueue time or queue with ``sort-by`` and ``sort-dir``. Requests without ``page`` retain the full list and previous response shape. +- ``GET /api/v3_0/assets//jobs`` also lists the jobs of the assets below the asset, at any depth, as far as the caller may read them; pass ``include-child-assets=false`` to list only the asset's own jobs. Its entries include ``created_via`` provenance, and the automation that created a job only where the caller may read that automation. +- ``GET /api/v3_0/assets//jobs`` renames its ``redis_connection_err`` field, which shipped in v1.0.0, to ``redis-connection-err``, as the automation endpoints spell it. It is read by FlexMeasures' own UI rather than by the FlexMeasures client. +- ``GET /api/v3_0/assets`` now scopes a listing the way its parameters say. ``num-records`` respects the ``root`` and ``depth`` constraints, where it used to count every asset in the account and asset-type scope. And an explicit ``include_public=false`` is now honoured, also next to ``all_accessible`` or ``root``; leaving the parameter out keeps the behaviour it had, where ``all_accessible`` and ``root`` include public assets and a listing of a single account does not. +- ``GET /api/v3_0/sensors//status`` now accepts an optional ``asset_id`` query parameter, naming the asset whose status page the sensor is reported on. It only affects the reported ``relation``, which used to claim that the sensor belongs to the asset shown, also for a sensor that is only listed because the asset's flex-context or graphs page refers to it. Reading the status of a sensor in the context of an asset requires read permission on that asset, too. +- ``GET /api/v3_0/sensors//status`` now returns ``missing-members`` for reporter data whose most recent aggregate of a device group, old enough to count as stale, had to leave out members that did not report: the ``id``, ``name`` and ``asset-name`` of each of those sensors. The field is left out where no member is missing. +- ``GET /api/v3_0/assets//chart`` now answers an unknown ``chart-type`` with a ``422``. -v3.0-33 | September 1, 2026 +v3.0-33 | September 9, 2026 """"""""""""""""""""""""""" -- Added ``POST /api/v3_0/assets//reports/trigger`` to queue a one-off report as a background job. It returns ``202 Accepted`` with the canonical ``job`` and ``job-url`` fields, and shares the trigger rate limit with forecast and schedule endpoints. -- Added ``GET /api/v3_0/assets//automations`` and ``GET /api/v3_0/assets//automations/`` for listing and inspecting forecast automations, including the sensors an automation reads from and writes to. Each automation shows the IANA ``timezone`` in which its cron expression is interpreted, and a ``cursor``: the offset-aware UTC time of the most recent run it committed to. The cursor advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded. Asset job entries now include ``created_via`` provenance; automation identity is included only when the caller may read that automation. -- Added ``GET /api/v3_0/sources/`` to show the full record of one data source, including the attributes in which data generators store their configuration. -- Automation responses now also include ``schedule-revision``, which counts the execution-affecting edits made to the automation's schedule, and automation detail responses gained a ``run-stats`` object. It summarizes the durable runs of that automation and describes the most recent ones: their scheduled time, dispatch state (``pending``, ``claimed``, ``partially_queued``, ``queued`` or ``failed``), execution state (``pending``, ``running``, ``succeeded``, ``failed`` or ``canceled``), attempt count, intended and queued job counts, timestamps, last error, latest attempt, and the individual jobs they created. Execution state, and the jobs a run created, are recorded for every type of automation. Both additions are backward compatible: no existing field changed. -- ``GET /api/v3_0/sensors//stats`` now reports an ``All sources`` entry summarising every data source, whenever more than one recorded. Its mean divides by the values that were summed, not by ``Number of values``, which also counts rows holding NaN. +- The ``resolution`` field is now rejected with a ``422 (Unprocessable Entity)`` response unless it spans a positive amount of time. This applies wherever the API accepts one: as a query parameter on ``GET /api/v3_0/sensors//data`` and on the ``chart_data`` endpoints under ``api/dev``, and in the request body of the ``POST`` schedule trigger endpoints. Previously, a zero resolution (such as ``PT0S``) either crashed the request with a ``500`` or was silently ignored, and a negative resolution returned an empty set of values. v3.0-32 | August 11, 2026 """"""""""""""""""""""""" diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 88b857096e..d7aa3b9e51 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -5,7 +5,7 @@ FlexMeasures Changelog -v1.1.0 | September XX, 2026 +v1.1.0 | November XX, 2026 ============================ .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``). @@ -27,142 +27,223 @@ v1.1.0 | September XX, 2026 .. note:: A scheduler's data source now also records the flex config it computed under, so a sensor can carry schedules from several configs, the way it already could for forecasts. Select a data source on the sensor page to see the schedule computed under one configuration. +.. _v1.1.0-add-report-warning: + .. warning:: If you run ``flexmeasures add report`` (for instance, in a cron job), it now saves the reporter's config on the report's data source by default, so its first run after upgrading records under a new data source. Point anything that refers to the old data source by ID at the new one, or add ``--no-save-config`` to keep recording under the old one [see `PR #2764 `_] +.. _v1.1.0-roles-warning: + +.. warning:: The upgrade gives every existing user the new ``account-member`` role. + To give a user the narrower reader or data-integrator access instead, assign them the ``account-reader`` or ``account-data-integrator`` role, and then remove their ``account-member`` role. + +.. warning:: The API's rate limits are now counted per organisation rather than per user, so all users and integrations of an organisation share one budget. + Before upgrading, hosts should check the limits of organisations with several users or integrations; see :ref:`Plans and rate limits `. + New features ------------- + +**Automations** + +* Automations: recurring tasks defined per asset, which compute forecasts, schedules or reports on a cron recurrence read in the automation's own timezone, defined from the CLI, the API or the UI, and dispatched once a minute by ``flexmeasures jobs run-automations``; see :ref:`automations` for what they do, and `Automations, in detail`_ for what each pull request contributed + +**Data sources and sensor data** + * When uploading sensor data, you can now set the time prior to which the data was recorded, such as when moving historical data from another system together with its original recording times [see `PR #2722 `_ and `PR #2734 `_] -* Browse an asset's jobs page by page, so status pages with long job histories load faster; the jobs API supports optional pagination while existing requests still return the full list [see `PR #2619 `_] -* The asset's status page now splits its sensor data and its jobs over two tabs, of which only the opened one loads its data, and it opens the tab you last looked at [see `PR #2470 `_] -* Both tabs of an asset's status page now name the asset each row belongs to, and the jobs tab also lists the jobs of the asset's sub-assets, so a site asset shows what happened anywhere below it, which you can switch off per session [see `PR #2500 `_] -* The ``AggregatorReporter`` can report the measured aggregate of a device group, taking the devices, their sign convention and the sensor to record on from the flex-config that scheduling already uses, so a site's aggregate needs no second description that could drift from the first; a group of groups is resolved to the devices behind it, members can be narrowed to a category such as every PV installation, each sensor's values are converted to the output sensor's unit at its own resolution before being resampled to the output's, and an event that some members did not report is aggregated from what did, under a data source of its own naming the missing members, so that a partial aggregate never passes for a complete one, while one faulty meter no longer blanks the group's aggregate; the asset's status page shows which members an aggregate old enough to count as stale still leaves out [see `PR #2648 `_ and `PR #2741 `_] -* Automations: recurring tasks defined per asset, which compute forecasts, schedules or reports on a cron recurrence read in the automation's own timezone, defined from the CLI, the API or the UI, and dispatched once a minute by ``flexmeasures jobs run-automations``. See :ref:`automations` for what they do, and `Automations, in detail`_ for what each pull request contributed. -* In the UI, users can now add annotation labels to sensor data from the sensor page, over a time range selected on the chart and rounded to whole events, or at an instant clicked on the chart, recording the signed-in user as the source; the sensor page's **Delete data** panel can delete annotations, too, backed by a new API endpoint (``[DELETE] /sensors/(id)/annotations``) [see `PR #2570 `_] -* Clean graphs display to make annotations more useful: Annotation labels on sensor and asset charts no longer flicker while moving between beliefs, can be pinned by clicking their range, use separate rows when pinned and hovered labels would overlap, and show their source and timing context when hovered. Fast chart tooltips now show only value and time by default, plus the sensor on asset charts, with a *Full belief info* switch for the remaining fields [see `PR #2575 `_] -* In the UI, the full record of the data source selected on a sensor page can be inspected, backed by a new API endpoint (``[GET] /sources/(id)``) [see `PR #2290 `_] -* A forecaster can now be told which data sources hold the truth about the sensor it forecasts, the way its regressors already could, so that a sensor that several sources report on is trained on the ones you trust [see `PR #2542 `_ and `PR #2627 `_] -* Forecasters, schedulers and reporters can now clean the data they read, by giving a ``lower``, ``upper`` or ``snap`` bound, so that a sensor with implausible readings can be used without first correcting it at the source [see `PR #2555 `_, `PR #2561 `_, `PR #2627 `_ and `PR #2655 `_] +* Inspect the full record of the data source selected on a sensor page, backed by a new API endpoint (``[GET] /sources/(id)``) [see `PR #2290 `_] + +**Plans and rate limits** + * Hover over an organisation's plan on its page to see its effective API rate limits [see `PR #2524 `_] -* Try out a forecast without recording it, using ``flexmeasures add forecasts --dry-run``, which computes the forecast in full and reports the sensor, data source, number of beliefs and event range it would have saved [see `PR #2483 `_] -* Run one-off reports as background jobs from the CLI or the asset API, with sensor-level authorization and a dedicated reporting worker queue [see `PR #2298 `_] -* Changing the selected time range on an asset or sensor chart now only loads the data that is actually new, instead of reloading the whole range, which makes stepping through or extending a long period much faster; reloading the page, or leaving it open for five minutes, still fetches everything afresh [see `PR #2433 `_] -* The statistics table on a sensor page now shows all data sources together by default, as the graph does [see `PR #2462 `_] -* The asset lists on an organisation's page and on the asset overview now show only top-level assets per default, so the sites you are looking for are no longer buried among their sub-assets, and both offer the same two checkboxes to widen the list again: untick *Top-level only* for the whole tree, or tick *Include public assets*, which is off per default [see `PR #2523 `_] -* The audit log of a device group's asset now notes which devices join or leave the group, and when a member moves to another site, or the group moves away from its members, so the history of a group's membership can be read in one place; a device whose group is deleted gets a note, too [see `PR #2723 `_ and `PR #2769 `_] -* When a report automation aggregates only one asset type of a device group, the group's audit log notes which members it starts with, and which members enter or leave that selection as their asset type changes [see `PR #2724 `_] -* Graph and flex-config sensor searches now hide public assets by default unless "Include public assets" is checked and wait for a keyword, unit, or an explicit request before loading results [see `PR #2775 `_] + +**Reporting and forecasting** + +* The ``AggregatorReporter`` can report the measured aggregate of a device group, taking the group's devices, their sign convention and the output sensor from the flex-config that scheduling already uses, and can narrow the members to a category (such as every PV installation) [see `PR #2648 `_] +* Where some members of a device group did not report an event, its aggregate is computed from those that did, under a data source of its own that names the missing members; the asset's status page shows such an aggregate once it goes stale [see `PR #2648 `_ and `PR #2741 `_] +* Run one-off reports as background jobs from the CLI or the asset API, with sensor-level authorization and a dedicated reporting worker queue [see `PR #2298 `_] +* Try out a forecast without recording it, using ``flexmeasures add forecasts --dry-run``, which reports the sensor, data source, number of beliefs and event range it would have saved [see `PR #2483 `_] +* Tell a forecaster which data sources hold the truth about the sensor it forecasts, as its regressors already could, so that a sensor that several sources report on is trained on the ones you trust [see `PR #2542 `_ and `PR #2627 `_] +* Forecasters, schedulers and reporters can now clean the data they read with a ``lower``, ``upper`` or ``snap`` bound, so that a sensor with implausible readings can be used without first correcting it at the source [see `PR #2555 `_, `PR #2561 `_, `PR #2627 `_ and `PR #2655 `_] + +**UI** + +* The asset's status page splits its sensor data and its jobs over two tabs, of which only the opened one loads its data, and it opens the tab you last looked at [see `PR #2470 `_] +* Browse an asset's jobs page by page, so that status pages with long job histories load faster; the jobs API supports optional pagination, while existing requests still return the full list [see `PR #2619 `_] +* Both tabs of an asset's status page name the asset each row belongs to, and the jobs tab also lists the jobs of its sub-assets, which you can switch off per session [see `PR #2500 `_] +* Add annotation labels to sensor data from the sensor page, over a time range selected on the chart or at an instant clicked on it, and delete annotations from its **Delete data** panel, backed by a new API endpoint (``[DELETE] /sensors/(id)/annotations``) [see `PR #2570 `_] +* Annotation labels on sensor and asset charts no longer flicker, can be pinned by clicking their range, no longer overlap when pinned and hovered, and show their source and timing on hover; fast chart tooltips show only value and time by default (plus the sensor on asset charts), with a *Full belief info* switch for the rest [see `PR #2575 `_] +* Changing the selected time range on an asset or sensor chart only loads the data that is new, which makes stepping through or extending a long period much faster [see `PR #2433 `_] +* The statistics table on a sensor page shows all data sources together by default, as the graph does [see `PR #2462 `_] +* The asset lists on an organisation's page and on the asset overview show only top-level assets by default, so that sites are no longer buried among their sub-assets; untick *Top-level only* for the whole tree, or tick *Include public assets*, which is off by default [see `PR #2523 `_] +* Graph and flex-config sensor searches hide public assets unless *Include public assets* is checked, and wait for a keyword, unit or explicit request before loading results [see `PR #2775 `_] +* The audit log of a device group (the asset that devices name as their ``group`` in their flex-model) notes which devices join or leave it, also when a member or the group moves to another site, and a device whose group is deleted gets a note, too [see `PR #2723 `_ and `PR #2769 `_] +* When a report automation aggregates only one asset type of a device group, the group's audit log notes which members it starts with, and which enter or leave that selection as their asset type changes [see `PR #2724 `_] Infrastructure / Support ------------------------- -* Job workers now also log RQ's own messages, such as when each job starts and how it ended, naming each job by its ID [see `PR #2811 `_] -* ``flexmeasures add report`` saves the reporter's config on its data source by default, as the API and automations do, so that reports computed under different configs no longer share one data source; its first run after upgrading saves all of its beliefs once more, under the new data source [see `PR #2764 `_] -* Upgrade certifi and Werkzeug for security reasons; hosts running the Docker image should pull the new image to get certifi's corrected certificate bundle, and hosts listing an IPv6 address in ``TRUSTED_HOSTS`` should know that this version of Werkzeug rejects requests to it (see `Werkzeug issue #3312 `_) [see `PR #2750 `_] -* A forecast hands its trained model to the prediction in memory, instead of writing it to a file and reading it back. A model is kept as a file only when ``model-save-dir`` is given, where each cycle used to leave one behind, often hundreds of MB, under ``flexmeasures/data/models/forecasting/artifacts/models`` relative to the worker's working directory. Hosts can delete that folder after upgrading [see `PR #2701 `_] -* The UI can work without internet access: the Docker image serves the UI's JavaScript and CSS libraries itself, and a pip-installed FlexMeasures can fetch them once (see :ref:`install-ui-libraries`), while otherwise browsers keep loading them from CDNs; the ``FLEXMEASURES_JS_VERSIONS`` setting now only lists the versions a host wants to differ, optionally with the hashes of their files, so plugins reading it should read the ``js_versions`` template variable instead [see `PR #2711 `_ and `PR #2747 `_] -* Upgrade Vega, which draws the UI's charts, from 5.22.1 to 5.33.1, its latest 5.x release [see `PR #2743 `_] -* ``[GET] /sensors/(id)`` now answers a sensor which does not exist with a 404, as the asset endpoints do, rather than with a 422, so that a client can tell a sensor which is gone apart from a request it got wrong [see `PR #2650 `_] -* Upgrade DataTables, which draws the UI's lists, from 1.10.22 to 1.13.11, its latest 1.x release [see `PR #2744 `_] -* Speed up preparing a storage schedule by validating each device's constraints with vectorised comparisons instead of one ``DataFrame.eval`` per check (from ~25 ms to ~3 ms per device); the helpers ``validate_constraint`` and ``sanitize_expression`` in ``flexmeasures.data.models.planning.storage``, which nothing in FlexMeasures called any more, are removed, with ``validate_constraints`` replacing them [see `PR #2641 `_] -* A pip-installed FlexMeasures now fetches the UI's JavaScript and CSS libraries itself, in the background once it serves its first request, so it serves them itself without a host running a command, also after an upgrade bumps one of them; set ``FLEXMEASURES_FETCH_UI_LIBRARIES`` to ``False`` to switch this off [see `PR #2746 `_] -* Rate-limit simplification: The API's default rate limit is now counted per organisation rather than per user, so that all users and integrations (anything calling the API with a token) of an organisation share the one budget its plan sets; it stays at 500 requests per minute, which hosts whose organisations run several integrations may want to raise on those organisations' plans (``default_rate_limit``) or server-wide (``FLEXMEASURES_API_DEFAULT_RATE_LIMIT``) before upgrading [see `PR #2677 `_] -* Rate-limit simplification: Triggers for schedules and forecasts are now always counted per organisation, against one budget that all of its users and assets share, so plans no longer choose what triggers are counted against, and the ``FLEXMEASURES_API_RATE_LIMIT_KEY`` setting is deprecated and ignored. Hosts who set it (or a plan's rate-limit key) to ``account+asset`` or ``user`` may want to raise the trigger rate limit of the affected plans, or ``FLEXMEASURES_API_TRIGGER_RATE_LIMIT``, before upgrading [see `PR #2679 `_] -* Rate-limit clarification: The UI's own calls to the API, which it makes with the user's session, now count against a budget per user (``FLEXMEASURES_UI_RATE_LIMIT``, 100 requests per minute by default) rather than against their organisation's, so that an integration which uses up its organisation's budget cannot lock that organisation's users out of the UI, and browsing does not spend the budget its integrations need; plans do not change this budget [see `PR #2700 `_] -* Rate-limit clarification: When a request is refused for hitting a rate limit, the server log now says who made it (account and user), on which endpoint, against which limit and, for the trigger limit, about which asset, so hosts can tell which integration used up an account's budget; with debug logging on, each trigger that spends budget is logged, too. The organisation audit log also records one breach per API or trigger limit window, with the plan, effective limit and how far into the window the breach occurred. The plan tooltip notes when play mode disables both limits [see `PR #2678 `_] -* Rate-limit feature: Hosts can reset an organisation's rate limits with ``flexmeasures edit reset-rate-limit``, so that an organisation whose integration has been fixed after running into a rate limit need not wait out the rest of the window. A rate limit (on a plan, or in ``FLEXMEASURES_API_DEFAULT_RATE_LIMIT`` or ``FLEXMEASURES_API_TRIGGER_RATE_LIMIT``) now holds one limit: several limits separated by semicolons used to be accepted without being documented, but are now refused, and a server with such a setting no longer starts [see `PR #2680 `_] -* Authorization now uses named permissions granted by code-defined user roles and checked against each resource's ACL. New ``account-reader`` and ``account-data-integrator`` roles support scoped access to home-account resources; the ``consultant`` role grants access only through consultancy ACLs on client accounts. Custom roles explicitly named in an ACL retain access to the four preexisting permissions on matching resources. A data migration assigns ``account-member`` to existing users and provisions the narrower roles. To give an existing user reader or data-integrator access, remove their ``account-member`` role after assigning the new role. Account readers cannot initiate password recovery; an account administrator or consultant can initiate a reset for them. Users from other organisations can no longer open a sensor page they cannot read. The user page shows each role's description in a tooltip [see `PR #2607 `_ and `PR #2720 `_] -* A forecast automation's cycle jobs are held to the sensors the automation was checked against, as the jobs of every other kind of data generator are [see `PR #2699 `_] -* Remove the last state-of-charge bookkeeping that served the sunset API versions 1 and 2 [see `PR #2665 `_] -* A data source now belongs to the organisation it records for, where only sources of type ``user`` used to. The organisation is part of what identifies a source, so two organisations running the same data generator under the same configuration each record under their own source, instead of sharing one because their configurations happen to hash alike; a ``source-account`` filter consequently finds generated data, where it used to match user-recorded data only. The sources that already exist are coupled on upgrade, to the organisation of the assets their data sits on [see `PR #2631 `_] -* ``flexmeasures.data.utils.get_data_source``, which plugins and scripts use to make sure they have a data source, now takes the organisation that source belongs to, as ``get_or_create_source`` does. Naming none asks for a source belonging to none, which is what a script the host runs for everyone records under. Without it, a script could be handed another organisation's source, and two organisations running the same script would leave the lookup finding two sources where it expected one [see `PR #2613 `_] -* Separate built-in forecast computation from service-managed saving, preserving per-cycle commits and existing plugin entrypoints while avoiding new data sources during synchronous dry runs [see `PR #2645 `_] -* Who may read a data source is now decided per source, rather than only by the organisation the source names. A source is readable when it belongs to an organisation you may read, when an automation you may read computes under it, or when it has recorded data on a sensor you may read. A source which names no organisation, which is what most data generators' sources look like today, is no longer readable by everyone for that reason alone. The configuration a data generator stores on its source is handed over only for the first two cases, because such a configuration names the sensors it runs on [see `PR #2584 `_] -* A schedule whose commitments constrain the solver identically, in that they bind the same devices against the same baseline over the same time steps, now solves them as one commitment priced on their summed prices, rather than as one apiece. A commitment that is not convex on its own can then be carried by one whose prices more than compensate, so such a schedule stays a linear program instead of needing the solver's sign variables, and it is a smaller one. Each commitment is still reported with its own costs, worked out from its own prices [see `PR #2635 `_] -* ``asset.sensors``, and the other lists of related objects that had no order of their own (such as ``account.users`` or ``asset.child_assets``), now list them in order of creation (by id), where they used to come in whatever order the database returned, so that code which takes one by position gets the same one on every database [see `PR #2637 `_] -* Speed up the test suite (two to three times faster), by hashing test passwords with the cheapest argon2 settings instead of Flask-Security's defaults, emptying the test database between tests instead of recreating it, and letting more tests share one database per module; plugins that reuse the ``setup_accounts``, ``setup_roles_users`` and similar fixtures should know that these now commit their data, and that the schema is created once per test session [see `PR #2638 `_] -* The endpoints supporting the UI moved from ``/api/dev`` to ``/api/ui``, where the old prefix keeps working until FlexMeasures v2 [see `PR #2578 `_] -* Find the built-in schedulers, reporters and forecasters from an explicit list, instead of importing every module under ``flexmeasures.data.models`` at start-up to look for them. Shortens boot time for every process and keeps an unrelated broken module from stopping the app; plugins keep being discovered as before [see `PR #2566 `_] + +**Data sources and the upgrade** + +* A data source now belongs to the organisation it records for, where only ``user`` sources used to, so that two organisations running the same data generator under the same configuration each record under their own source, and a ``source-account`` filter also finds generated data; the upgrade couples existing sources to the organisation of the assets their data sits on [see `PR #2634 `_] +* Who may read a data source is now decided per source: a source is readable when it belongs to an organisation you may read, when an automation you may read computes under it, or when it recorded data on a sensor you may read, so that a source belonging to no organisation is no longer listed for everyone (reading its data still depends on the sensor, as before); the configuration a data generator stores on its source is only handed over in the first two cases, as it names the sensors the generator reads from and records on [see `PR #2584 `_] +* ``flexmeasures.data.utils.get_data_source``, which plugins and scripts use to make sure they have a data source, now takes the organisation that source belongs to, as ``get_or_create_source`` does; naming none asks for a source belonging to no organisation, which is what a script the host runs for everyone records under [see `PR #2613 `_] +* Drop the nine obsolete tables that predate the ``GenericAsset``/``Sensor`` data model, asking you to confirm first if any of them still hold data, as v0.18.0 missed two of them, and a database downgraded past that release and upgraded again kept all nine [see `PR #2475 `_] +* Speed up sensor data queries and free up disk space by reordering the ``timed_belief`` primary key to lead with ``sensor_id`` and dropping the indexes it makes redundant, in a migration that runs online and so needs no maintenance window (though it can take a while on a large database) [see `PR #2378 `_] + +.. _v1.1.0-rate-limits: + +**Plans and rate limits** + +* The API's default rate limit is now counted per organisation rather than per user, so that all users and integrations (anything calling the API with a token) of an organisation share the one budget its plan sets; it stays at 500 requests per minute, which hosts whose organisations run several integrations may want to raise on those organisations' plans (``default_rate_limit``) or server-wide (``FLEXMEASURES_API_DEFAULT_RATE_LIMIT``) before upgrading [see `PR #2677 `_] +* Triggers for schedules and forecasts are now always counted per organisation, so that the ``FLEXMEASURES_API_RATE_LIMIT_KEY`` setting is deprecated and ignored; hosts who set it (or a plan's rate-limit key) to ``account+asset`` or ``user`` may want to raise the trigger rate limit of the affected plans, or ``FLEXMEASURES_API_TRIGGER_RATE_LIMIT``, before upgrading [see `PR #2679 `_] +* The UI's own calls to the API count against a budget per user (``FLEXMEASURES_UI_RATE_LIMIT``, 100 requests per minute by default) rather than against their organisation's, so that an integration which uses up its organisation's budget cannot lock that organisation's users out of the UI; plans do not change this budget [see `PR #2700 `_] +* Hosts can reset an organisation's rate limits with ``flexmeasures edit reset-rate-limit``, so that an organisation whose integration has been fixed need not wait out the rest of the window [see `PR #2680 `_] +* A rate limit (on a plan, or in ``FLEXMEASURES_API_DEFAULT_RATE_LIMIT`` or ``FLEXMEASURES_API_TRIGGER_RATE_LIMIT``) now holds one limit: several limits separated by semicolons used to be accepted without being documented, but are now refused, and a server with such a setting no longer starts [see `PR #2680 `_] +* A request refused for hitting a rate limit is logged with who made it (organisation and user), on which endpoint, against which limit and, for the trigger limit, about which asset, and the organisation's audit log records one breach per limit window, with the plan, the effective limit and how far into the window it occurred; the plan tooltip notes when play mode disables both limits [see `PR #2678 `_] + +**Reporting and forecasting** + +* ``flexmeasures add report`` saves the reporter's config on its data source by default, as the API and automations do, so that reports computed under different configs no longer share one data source; its first run after upgrading saves all of its beliefs once more, under the new data source (see :ref:`this warning `) [see `PR #2764 `_] +* Every kind of data generator (e.g. a ``Forecaster``, ``Scheduler`` or ``Reporter``) now has its results checked and saved in the same way: all of a run's outputs are saved or none of them; a plugin's data generator that records on a sensor it was not checked against now fails with ``GeneratorWritesUncheckedSensor`` [see `PR #2689 `_ and `PR #2571 `_] +* Separate the built-in forecast computation from the saving of its results, keeping per-cycle commits and existing plugin entry points, so that a synchronous dry run creates no data sources [see `PR #2645 `_] +* A forecast that repeats the belief right before it is no longer stored again, like any other data; plugins that save forecasts through ``save_forecast`` can pass ``save_changed_beliefs_only=False`` to record every belief [see `PR #2688 `_] +* A forecast hands its trained model to the prediction in memory, and keeps it as a file only when ``model-save-dir`` is given; hosts can delete the ``flexmeasures/data/models/forecasting/artifacts/models`` folder (relative to the worker's working directory) after upgrading, where each cycle used to leave a model behind, often hundreds of MB [see `PR #2701 `_] +* Preparing a forecast's inputs is faster [see `PR #2640 `_] +* Report parameters name their belief time ``prior``, as schedule and forecast parameters do; the name ``belief_time`` is still accepted, while the ``belief_time`` of an ``input`` entry, which selects the beliefs to report on rather than stamping the report, is not renamed by this change [see `PR #2551 `_] + +**Scheduling** + +* A schedule whose commitments bind the same devices against the same baseline over the same time steps now solves them as one commitment priced on their summed prices, so that more such schedules stay a linear program, and a smaller one, while each commitment is still reported with its own costs [see `PR #2635 `_] +* Speed up preparing a storage schedule by validating device constraints with vectorised comparisons (from ~25 ms to ~3 ms per device); the unused helpers ``validate_constraint`` and ``sanitize_expression`` in ``flexmeasures.data.models.planning.storage`` are removed, with ``validate_constraints`` replacing them [see `PR #2641 `_] * ``MetaStorageScheduler``, an internal base class that was never meant to be selected, is no longer registered and can no longer be named as a custom scheduler; name ``StorageScheduler`` instead [see `PR #2566 `_] -* Import pyomo, LightGBM and the Sentry helpers only when they are used, cutting about 0.35 s off the boot time of every process [see `PR #2565 `_] * ``flexmeasures.data.models.planning.linear_optimization`` no longer exposes pyomo's ``UnknownSolver``, which nothing in FlexMeasures used; import it from ``pyomo.environ`` instead [see `PR #2565 `_] -* Report parameters name their belief time ``prior``, as schedule and forecast parameters do; the name ``belief_time`` is still accepted, while the ``belief_time`` of an ``input`` entry, which selects the beliefs to report on rather than stamping the report, keeps its name [see `PR #2551 `_] -* A test now holds new API and CLI field names to kebab-case, listing the names that predate the convention so that the list can only shrink [see `PR #2547 `_] -* Drop the nine obsolete tables that predate the ``GenericAsset``/``Sensor`` data model, asking you to confirm first if any of them still hold data, which cleans up after v0.18.0, where seven of them were dropped but ``asset_type`` and ``weather_sensor_type`` were missed, and where a database that was downgraded past that release and upgraded again kept all nine [see `PR #2475 `_] -* Speed up sensor data queries and free up disk space by reordering the ``timed_belief`` primary key to lead with ``sensor_id`` and dropping the indexes it makes redundant, in a migration that runs online and so needs no maintenance window (though it can take a while on a large database) [see `PR #2378 `_] + +**Access and security** + +* Authorization now uses named permissions, granted by user roles defined in code and checked against each resource's ACL: new ``account-reader`` and ``account-data-integrator`` roles give scoped access within the user's own organisation, the ``consultant`` role grants access only through consultancy ACLs on client organisations, and custom roles named in an ACL keep the four preexisting permissions on matching resources; the upgrade gives existing users the ``account-member`` role and provisions the narrower roles (see :ref:`this warning `); users with the ``account-reader`` role cannot start password recovery themselves (an organisation admin or consultant can start a reset for them), users from other organisations can no longer open a sensor page they cannot read, and the user page shows each role's description in a tooltip [see `PR #2607 `_ and `PR #2720 `_] +* Logging out now takes a POST request, so that another site cannot log users out by linking to ``/logout``; the UI's *Log out* button submits a form, and plugins that link to ``/logout`` should do the same [see `PR #2604 `_] +* Upgrade certifi and Werkzeug for security reasons; hosts running the Docker image should pull the new image to get certifi's corrected certificate bundle, and hosts listing an IPv6 address in ``TRUSTED_HOSTS`` should know that this version of Werkzeug rejects requests to it (see `Werkzeug issue #3312 `_) [see `PR #2750 `_] +* Stop installing the libraries for passkey, authenticator-app and SMS sign-in, which FlexMeasures does not support, as it ships email-based two-factor authentication only; to enable the ``authenticator`` method anyway, install ``qrcode`` [see `PR #2564 `_] +* Durations for Flask-Security, such as ``SECURITY_TWO_FACTOR_LOGIN_VALIDITY``, can now also be given as an ISO 8601 duration (e.g. ``"P1W"``), and in weeks (e.g. ``"1 week"``) on any version of Flask-Security [see `PR #2596 `_] + +**API and plugins** + +* The endpoints supporting the UI moved from ``/api/dev`` to ``/api/ui``, where the old prefix keeps working until FlexMeasures v2 [see `PR #2578 `_] +* ``[GET] /sensors/(id)`` now answers a sensor which does not exist with a 404 rather than a 422, as the asset endpoints do, so that a client can tell a sensor which is gone apart from a request it got wrong [see `PR #2650 `_] +* Add ``FLEXMEASURES_DEPRECATION_AND_SUNSET`` so hosts can configure deprecation and sunset dates and information links per deprecated API version [see `PR #2362 `_] +* Remove the last state-of-charge bookkeeping that served the sunset API versions 1 and 2 [see `PR #2665 `_] +* Settings that a plugin declares in its ``__settings__`` can now be set as environment variables, next to the config file (which still wins), and can declare a ``default`` to fall back to; a missing setting is reported with whether such a default applies [see `PR #2501 `_] +* ``asset.sensors``, and the other lists of related objects that had no order of their own (such as ``account.users`` or ``asset.child_assets``), now list them in order of creation (by id), so that code which takes one by position gets the same one on every database [see `PR #2637 `_] + +**Performance** + +* Speed up saving beliefs that overlap stored data, such as a re-computed schedule or re-posted measurements, which cuts saving a week of 15-minute beliefs from ~5 s to ~10 ms [see `PR #2639 `_] * Look up which data sources recorded for which sensors from a small summary table instead of scanning the beliefs table [see `PR #2382 `_] * The asset page no longer slows down with the amount of data a sensor holds while working out how up to date each of its sensors is [see `PR #2463 `_] -* Shrink the Docker image by excluding dev-only dependencies, pruning stray ``docs``/``examples`` payloads bundled by ``sktime``/``scikit-base`` (issue: https://github.com/sktime/sktime/issues/10891), stripping the symbol tables that the compiled extensions ship with, and dropping the ``sktime``-backed belief-formation extra of ``timely-beliefs``, which FlexMeasures does not use [see `PR #2438 `_, `PR #2439 `_ and `PR #2440 `_] -* Require an exact ``uv`` version (``0.12.7``) via ``[tool.uv].required-version``, which keeps ``uv.lock`` changes reproducible across local development, CI, Docker and Read the Docs, but does mean that anyone running ``uv`` in a FlexMeasures checkout — plugin developers and self-hosters included — needs that same version [see `PR #2451 `_] -* Speed up ``GET /api/v3_0/assets`` on large catalogs by eager-loading the ``owner``, ``generic_asset_type`` and ``child_assets`` relations alongside the already eager-loaded ``sensors``, instead of lazy-loading each of them once per asset, which made the SQL statement count grow linearly with the number of assets returned [see `PR #2515 `_] -* Speed up app boot by ~1s by dropping the ``inflect`` dependency in favor of the already-used, much lighter ``inflection`` package; API error messages that list several values, such as the permissions a request requires or the units a quantity accepts, no longer place a comma before the final "and" or "or" [see `PR #2514 `_] -* The developer and hosting docs now say plainly that the Cbc solver is a system dependency, needed for the tests and for Cbc-based scheduling, and give the command to install it on Debian/Ubuntu, Fedora and macOS [see `PR #2597 `_] -* Speed up container startup and app boot by installing FlexMeasures itself as a regular (non-editable) package in the Docker image, so its own source files are bytecode-compiled at build time like every other dependency, by letting ``gunicorn`` load the app once before forking its workers, in the Docker image and in the ``docker-compose.yml`` dev stack, by reusing the Alembic head-revision check the app already runs at boot instead of repeating its database round trip, and by deferring the ``altair`` import until a chart is actually rendered [see `PR #2512 `_, `PR #2757 `_ and `PR #2811 `_] +* Speed up ``GET /api/v3_0/assets`` on large catalogs, where the number of database queries grew with the number of assets returned [see `PR #2515 `_] +* On an asset with charge points, the Default chart no longer loads the charging-session sensors it doesn't draw, and the Charge Points chart loads only the sensors it draws [see `PR #2790 `_] +* Speed up loading chart data for an asset with many sensors, by no longer repeating two queries for each sensor [see `PR #2759 `_] +* Speed up container startup and app boot, by installing FlexMeasures as a regular (non-editable) package in the Docker image, by letting ``gunicorn`` load the app once before forking its workers (in the Docker image and in the ``docker-compose.yml`` dev stack), by skipping a repeated database round trip at boot, and by deferring the ``altair`` import until a chart is drawn [see `PR #2512 `_, `PR #2757 `_ and `PR #2811 `_] +* Find the built-in schedulers, reporters and forecasters from an explicit list, instead of importing every module under ``flexmeasures.data.models`` at start-up, which shortens the boot time of every process and keeps an unrelated broken module from stopping the app; plugins keep being discovered as before [see `PR #2566 `_] +* Import pyomo, LightGBM and the Sentry helpers only when they are used, cutting about 0.35 s off the boot time of every process [see `PR #2565 `_] +* Speed up app boot by ~1 s by replacing the ``inflect`` dependency with the lighter ``inflection`` package; API error messages that list several values no longer place a comma before the final "and" or "or" [see `PR #2514 `_] + +**Hosting and dependencies** + +* Job workers now also log RQ's own messages, such as when each job starts and how it ended, naming each job by its ID [see `PR #2811 `_] +* The UI can work without internet access: the Docker image serves the UI's JavaScript and CSS libraries itself, and a pip-installed FlexMeasures fetches them in the background once it serves its first request, also after an upgrade bumps one of them, while files not fetched (yet) are loaded from CDNs, checked against their hashes (set ``FLEXMEASURES_FETCH_UI_LIBRARIES`` to ``False`` to switch fetching off, and see :ref:`install-ui-libraries` to fetch them by hand); the ``FLEXMEASURES_JS_VERSIONS`` setting now only lists the versions a host wants to differ, optionally with the hashes of their files, so plugins reading it should read the ``js_versions`` template variable instead [see `PR #2711 `_, `PR #2746 `_ and `PR #2747 `_] +* Support SQLAlchemy 2.1 and upgrade dependencies, including timely-beliefs 4.4 and Flask-Security-Too 5.9; a ``SQLALCHEMY_DATABASE_URI`` that names no driver (``postgresql://``) now connects through psycopg 3, so to keep using psycopg2, name it (``postgresql+psycopg2://``), while ``flexmeasures db-ops dump`` and ``restore`` now also work with a URI that names a driver [see `PR #2596 `_, `PR #2601 `_ and `PR #2808 `_] +* Upgrade Vega, which draws the UI's charts, from 5.22.1 to 5.33.1 [see `PR #2743 `_] +* Upgrade DataTables, which draws the UI's lists, from 1.10.22 to 1.13.11 [see `PR #2744 `_] +* Shrink the Docker image by excluding dev-only dependencies, pruning the ``docs`` and ``examples`` bundled by ``sktime``/``scikit-base``, stripping the symbol tables of compiled extensions, and dropping the ``forecast`` dependency extra of ``timely-beliefs``, whose ``sktime``-based belief formation FlexMeasures never used [see `PR #2438 `_, `PR #2439 `_ and `PR #2440 `_] +* Require an exact ``uv`` version (``0.12.7``) via ``[tool.uv].required-version``, which keeps ``uv.lock`` changes reproducible, but means that anyone running ``uv`` in a FlexMeasures checkout, plugin developers and self-hosters included, needs that same version [see `PR #2451 `_] +* A CLI command called with an invalid option value now logs one error line, so that a cron job which captures only the log file still records why the command failed [see `PR #2544 `_] + +**Development and documentation** + +* Speed up the test suite two to three times; plugins that reuse the ``setup_accounts``, ``setup_roles_users`` and similar fixtures should know that these now commit their data, and that the schema is created once per test session [see `PR #2638 `_] * The UI's JavaScript modules can now be tested, by running them in a headless browser from pytest, without adding a Node.js toolchain [see `PR #2435 `_] -* Test much more of the UI's JavaScript, and move the asset tree's layout and the flex-context editor's logic out of their templates into modules, so that they can be tested too [see `PR #2574 `_] -* Add ``FLEXMEASURES_DEPRECATION_AND_SUNSET`` so hosts can configure deprecation and sunset dates and information links per deprecated API version [see `PR #2362 `_]. -* A CLI command that is called with an invalid option value now logs one error line, so that a cron job which captures only the log file still records why the command failed, where previously Click reported it on stderr alone and nothing was written [see `PR #2544 `_] -* Settings that a plugin declares in its ``__settings__`` can now be set as environment variables, next to being set in the config file (which still wins), can declare a ``default`` to fall back to, and are reported as missing with a message that says whether such a default applies or the setting stays unset [see `PR #2501 `_] -* Support SQLAlchemy 2.1 and upgrade dependencies, including timely-beliefs 4.4 and Flask-Security-Too 5.9, and connect to the database through psycopg 3 when ``SQLALCHEMY_DATABASE_URI`` names no driver (``postgresql://``), where to keep using psycopg2 you can name it (``postgresql+psycopg2://``), while ``flexmeasures db-ops dump`` and ``restore`` now also work with a URI that names a driver [see `PR #2596 `_, `PR #2601 `_ and `PR #2808 `_] -* Durations for Flask-Security, such as ``SECURITY_TWO_FACTOR_LOGIN_VALIDITY``, can now also be given as an ISO 8601 duration (e.g. ``"P1W"``), and in weeks (e.g. ``"1 week"``) on any version of Flask-Security [see `PR #2596 `_] -* Logging out now takes a POST request, so that another site cannot log users out by linking to ``/logout``; the UI's *Log out* button submits a form, and plugins that link to ``/logout`` should do the same [see `PR #2604 `_] -* ``uv run poe clean-db`` now hands the database it creates to the ``--db-user``, so that setting up a development database also works on PostgreSQL 15 and newer, where only a database's owner may create tables in its ``public`` schema [see `PR #2590 `_] -* Speed up saving beliefs that overlap stored data, such as a re-computed schedule or re-posted measurements, by comparing all new beliefs with the stored ones in one pass instead of one belief at a time, which cuts a week of 15-minute beliefs from ~5 s to ~10 ms [see `PR #2639 `_] -* A forecast that repeats the belief right before it is no longer stored again, like any other data, so a report input that asks only for forecasts made at most some time ahead (``horizons_at_most``) can find fewer of them; plugins that save forecasts through ``save_forecast`` can pass ``save_changed_beliefs_only=False`` to record every belief [see `PR #2688 `_] -* Every kind of data generator (e.g. a ``Forecaster``, ``Scheduler`` or ``Reporter``) now has its results checked and saved in the same way: all of a run's outputs are saved or none of them, and recording on a sensor the generator was not checked against raises ``GeneratorWritesUncheckedSensor`` [see `PR #2689 `_ and `PR #2571 `_] -* Stop installing the libraries for passkey, authenticator-app and SMS sign-in, which FlexMeasures does not support, as it ships email-based two-factor authentication only; to enable the ``authenticator`` method anyway, install ``qrcode`` [see `PR #2564 `_] -* Speed up loading chart data for an asset with many sensors, by skipping two queries that were repeated for each sensor: the look-up of the materialized view of most recent beliefs, which is now done once for all sensors (and the view is still used if a host created it), and the query for the parent asset of each sensor [see `PR #2759 `_] -* Add a concepts page on data pipelining, which describes in one place how time-series data is referenced (sensor references, source filters, fallback values), cleaned on the way in and bounded on the way out, and link to it from the feature pages and tutorials [see `PR #2721 `_] -* On an asset with charge points, the Default chart no longer loads the charging-session sensors it doesn't draw, and the Charge Points chart loads only the sensors it draws; switching between them loads that chart's data [see `PR #2790 `_] +* Test much more of the UI's JavaScript, moving the asset tree's layout and the flex-context editor's logic out of their templates into modules [see `PR #2574 `_] +* A test now holds new API and CLI field names to kebab-case, listing the names that predate the convention so that the list can only shrink [see `PR #2547 `_] +* ``uv run poe clean-db`` now hands the database it creates to the ``--db-user``, so that setting up a development database also works on PostgreSQL 15 and newer [see `PR #2590 `_] +* The developer and hosting docs now say plainly that the Cbc solver is a system dependency, needed for the tests and for Cbc-based scheduling, and give the command to install it on Debian/Ubuntu, Fedora and macOS [see `PR #2597 `_] +* Add a concepts page on data pipelining, which describes in one place how time-series data is referenced (sensor references, source filters, fallback values), cleaned on the way in and bounded on the way out [see `PR #2721 `_] +* The README and docs index describe FlexMeasures as a scheduling and optimization engine, rather than as a whole EMS [see `PR #2751 `_] Bugfixes ----------- -* In the Docker image, ``pip install``, which our docs and the compose stack use to add plugins and their requirements, now installs into the environment FlexMeasures runs from, rather than where FlexMeasures could not import it [see `PR #2811 `_] -* Scheduling workers no longer print the HiGHS solver's banner and log to the console, unless ``LOGGING_LEVEL`` is ``DEBUG``; setting ``output_flag`` in ``FLEXMEASURES_LP_SOLVER_OPTIONS`` now also suppresses the banner. And with ``appsi_highs`` (or another Pyomo appsi solver), the solver's output no longer passes through FlexMeasures' log handlers, which repeated each line with ever more prefixes and could hang a worker; that output now goes to stdout only, so it no longer reaches the log file [see `PR #2794 `_ and `PR #2807 `_] -* Allow consultants to create assets in client organisations, and reject attempts to change an asset’s organisation through the API [see `PR #2786 `_] -* The buttons for deleting an asset and resetting a password now act through the API, and the audit log pages and the page listing an organisation's assets check the same permissions as the API [see `PR #2791 `_] -* Remember the selected graph date range when navigating between sensors and assets, including when using fast charts or cached data [see `PR #2774 `_] -* An ``AggregatorReporter`` input that holds more than one belief about an event, such as one read with ``most-recent-beliefs-only: false``, is now refused with a message naming the sensor and the event, where it used to fail with a ``KeyError`` [see `PR #2741 `_] -* Fix ``flexmeasures show beliefs`` using the timezone of the first sensor instead of the one given with ``--timezone``; by default, the CSV file now has its timestamps in the timezone of the first sensor with data, with a UTC offset, instead of in UTC without an offset [see `PR #2695 `_ and `PR #2793 `_] -* ``[GET] /sensors`` lists sensors in order of their ID, where the order used to be up to the database [see `PR #2756 `_] + +**Automations** + +* Offset chains, as in an automation's window or ``flexmeasures add report --start-offset``, now count days as calendar days across a DST transition, so that a window no longer collapses or skips a day when run in the hour around midnight on such a weekend, and ``HB`` and ``DB`` no longer raise an error in the repeated hour or in a DST gap [see `PR #2771 `_] + +**Data sources and sensor data** + +* Two identical scheduling requests arriving at the same moment no longer create two data sources for one scheduler configuration, which made every later schedule under it fail; such requests now share one job, and a deployment that already has such duplicates schedules again [see `PR #2612 `_] +* Where several data sources report the same event, which one a search keeps is now decided the same way every time: a source version only counts against other versions of that source, and a caller that lists its sources gets the order it asked for [see `PR #2494 `_] +* Saving several beliefs about the same event at once no longer drops a belief whose value changes back to an earlier one; ``flexmeasures edit resample-data`` lost such beliefs for good [see `PR #2687 `_] * Posting or uploading sensor data with a ``prior`` beyond the year 2262 is refused with an explanation, where it used to fail with a server error [see `PR #2735 `_] -* The sensor upload form now reports whether a queued upload succeeded, and a refused replacement of existing values is explained rather than shown as a database error [see `PR #2722 `_] -* Keep the current-time bar visible during sensor chart playback when no data is visible yet [see `PR #2722 `_] -* The flex-config and graph dialogues now say so when a configuration refers to a sensor which no longer exists, as a sensor deleted directly in the database leaves such references behind, where they used to show it as a sensor with an undefined name and unit: each dialogue names the dangling reference, lets you remove it, and warns above the configuration how many there are, the graph dialogue no longer reports such a sensor as having a different unit, and an asset's status page no longer fails to load over a flex-context field whose sensor is gone; a sensor that merely failed to load, for instance on a server error, is said to be just that, rather than deleted [see `PR #2650 `_] -* Database migration command errors now appear in CLI output, including the error when multiple Alembic heads exist. [see `PR #2607 `_] -* A failed job now says why in the same way on every queue, in the API, the CLI and the asset page's jobs table: a failed forecasting job no longer reads ``dict: {...}`` and now shows its timeout hint, and a failed report or plugin automation no longer reads ``str: ...`` [see `PR #2698 `_] -* The result of a reporting or scheduling job now counts the beliefs it saved rather than those it computed, so a report's ``n_rows`` and a schedule's ``num-beliefs`` read 0 when a rerun only repeats what is on record, and ``flexmeasures add forecasts`` no longer drops what a forecaster registered by a plugin returns [see `PR #2689 `_ and `PR #2571 `_] +* ``GenericAsset.set_attribute`` and ``Sensor.set_attribute``, which plugins and scripts use, now create an attribute that does not exist yet, instead of silently ignoring it [see `PR #2649 `_] +* A sensor reference in a flex-model or flex-context now refuses a key it does not take, naming the nearest key it does take, where it used to read past it, so that a source filter written in the singular (``source: 42`` rather than ``sources: [42]``) silently did not filter; a stored flex config carrying such a key starts failing, which is how it stops being ignored, and `Notation `_ sets out which spelling belongs where [see `PR #2656 `_] +* Deleting a sensor now unsets it in the flex-model and flex-context references to it that carry a default, bounds or source filters, which used to keep pointing at the deleted sensor; until a new sensor is picked, schedules use the reference's default as a fixed value, or, without a default, ignore the field, so that an ancestor asset's value for it applies, and the UI marks such fields and lists them when you delete a sensor [see `PR #2726 `_, `PR #2767 `_ and `PR #2783 `_] + +**Reporting and forecasting** + +* An ``AggregatorReporter`` input that holds more than one belief about an event, such as one read with ``most-recent-beliefs-only: false``, is now refused with a message naming the sensor and the event, where it used to fail with a ``KeyError`` [see `PR #2741 `_] +* A ``PandasReporter`` with ``droplevels`` no longer fails with ``BeliefDataframe has more than one row per event`` when several data sources reported the same input event, as happens for a while when a job reporting on a rolling window gets a new data source; it keeps one belief per event, unless an input sets ``one_deterministic_belief_per_event`` to ``false`` [see `PR #2595 `_] +* The result of a reporting or scheduling job now counts the beliefs it saved rather than those it computed, so that a report's ``n_rows`` and a schedule's ``num-beliefs`` read 0 when a rerun only repeats what is on record [see `PR #2689 `_ and `PR #2571 `_] +* A forecaster registered by a plugin no longer needs to save its forecasts itself, but can return them instead: ``flexmeasures add forecasts`` now saves them, where it used to drop them while reporting them as created [see `PR #2689 `_ and `PR #2571 `_] * Running the same forecast twice from the same start no longer fails on the beliefs it already recorded [see `PR #2688 `_] -* Saving several beliefs about the same event at once no longer drops a belief whose value changes back to an earlier one, which made the most recent belief look like a value that had already been revised; ``flexmeasures edit resample-data`` lost such beliefs for good [see `PR #2687 `_] -* A schedule with several commitments, one of which pays more for deviating downwards than it costs to deviate upwards, no longer comes back as infeasible or unbounded: whether a commitment needs the solver's sign variables is now decided for each commitment on its own, rather than from every commitment's prices added together, where another commitment's prices could hide it [see `PR #2631 `_] -* Setting an attribute that does not exist yet on an asset or sensor now takes effect, instead of being silently ignored [see `PR #2649 `_] -* A sensor reference in a flex-model or flex-context used to accept a key it does not take and read past it, so a source filter written in the singular — ``source: 42``, as a data request takes it, rather than ``sources: [42]`` — dropped the filtering it was meant to apply. Such a key is now refused, naming the nearest key the reference does take. A stored flex config carrying one starts failing, which is how it stops being ignored, and `Notation `_ sets out which spelling belongs where [see `PR #2656 `_] -* Charts now highlight a hovered or pinned annotation in the secondary colour, in the fast chart as well as in Vega-Lite, where the sensor page's Vega-Lite chart was the only one to do so; annotation text is shown in near-black again, as the light grey was hard to read on white; shift-click pins several annotations at once in every chart, as it already did in the sensor page's Vega-Lite chart; and instantaneous annotations can now be hovered and pinned in Vega-Lite charts, too [see `PR #2570 `_] -* Whether a forecaster takes a regressor's belief as realized now follows the knowledge horizon of the regressor's sensor, as in timely-beliefs; it used to take a belief as realized once it was recorded after its event started, which kept all past readings of an instantaneous regressor (such as a thermometer read on the hour) and all past day-ahead prices (published the day before) from the model, and let a belief recorded halfway through an hourly event count as realized [see `PR #2692 `_] -* A forecaster's ``missing-threshold`` let through more missing data than it allowed, because a gap counted one row fewer than it spans, so that data missing every other row counted as complete; forecasting with a regressor recorded instantaneously failed on a division by zero; and preparing a forecast's inputs is now faster, because the gaps in each sensor's data are filled with pandas, instead of converting to a Darts ``TimeSeries`` for every simulated forecast step [see `PR #2640 `_] -* Two identical scheduling requests arriving at the same moment no longer create two data sources for one scheduler configuration, which made every later schedule under that configuration fail; such requests now share one job, and a deployment that already has such duplicates schedules again [see `PR #2612 `_] -* With the ``highspy`` solver, a site where any device has an operation mode could no longer run a converter coupling several commodities (such as a CHP) above one unit of its normalised flow, which made a must-run converter infeasible and silently capped a flexible one [see `PR #2606 `_] -* Creating an organisation now records the action in its audit log, whether it is created in the UI, through the API or from the CLI [see `PR #2580 `_] +* Whether a forecaster takes a regressor's belief as realized now follows the knowledge horizon of the regressor's sensor, as in timely-beliefs, so that the past readings of an instantaneous regressor (such as a thermometer read on the hour) and day-ahead prices (beliefs about future periods, which count as realized once the auction has cleared) are no longer kept from the model [see `PR #2692 `_] +* A forecaster's ``missing-threshold`` no longer lets through more missing data than it allows, where data missing every other row counted as complete, and forecasting with a regressor recorded instantaneously no longer fails on a division by zero [see `PR #2640 `_] +* A forecaster given two or more past regressors, or two or more future regressors, no longer trains on each of them several times over [see `PR #2560 `_] + +**Scheduling** + +* Scheduling workers no longer print the HiGHS solver's banner and log to the console, unless ``LOGGING_LEVEL`` is ``DEBUG``, and setting ``output_flag`` in ``FLEXMEASURES_LP_SOLVER_OPTIONS`` now also suppresses the banner; with ``appsi_highs`` (or another Pyomo appsi solver), the solver's output no longer passes through FlexMeasures' log handlers, where it could hang a worker, but goes to stdout only, so it no longer reaches the log file [see `PR #2794 `_ and `PR #2807 `_] +* A schedule with several commitments, one of which pays more for deviating downwards than it costs to deviate upwards, no longer comes back as infeasible or unbounded, as each commitment's prices are now checked on their own rather than added together [see `PR #2631 `_] +* With the ``highspy`` solver, a converter coupling several commodities (such as a CHP) on a site where any device has an operation mode is no longer capped at one unit of its normalised flow, which made a must-run converter infeasible and silently capped a flexible one [see `PR #2606 `_] +* A new commitment's prices now default to the currency of the price sensors in the flex-context, where they fell back to EUR unless a fixed price was set [see `PR #2574 `_] +* ``flexmeasures add schedule --dry-run`` combined with ``--as-job`` is now rejected, where the flag used to be dropped and the queued job saved its schedule anyway, and a dry run says how many beliefs it would have saved and which events they cover [see `PR #2483 `_] + +**UI** + +* Remember the selected graph date range when navigating between sensors and assets, including when using fast charts or cached data [see `PR #2774 `_] +* Keep the current-time bar visible during sensor chart playback when no data is visible yet [see `PR #2722 `_] +* The sensor upload form now reports whether a queued upload succeeded, and a refused replacement of existing values is explained rather than shown as a database error [see `PR #2722 `_] +* In every chart, a hovered or pinned annotation is highlighted in the secondary colour and shift-click pins several annotations at once, annotation text is shown in near-black again for readability, and instantaneous annotations can be hovered and pinned in Vega-Lite charts, too [see `PR #2570 `_] +* A chart of a sensor with a single data point in view, such as a daily sensor of which only one day was recorded, drew that point as a bar spanning most of the chart; a bar now covers exactly the event it reports [see `PR #2511 `_] +* Saving an asset chart as PNG or SVG no longer draws the legend over the graph when it lists many sensors, and the exported image spells out the sensor names that the on-screen legend abbreviates [see `PR #2517 `_] +* Chart requests for a sensor or an asset now accept only the names of chart types, and building a chart no longer changes the chart settings that later charts start from, such as a heatmap moving the legend of later charts to the right [see `PR #2801 `_] +* The flex-config and graph dialogues now say so when a configuration refers to a sensor which no longer exists, where they used to show it as a sensor with an undefined name and unit: each dialogue names the dangling reference and lets you remove it, and an asset's status page no longer fails to load over a flex-context field whose sensor is gone [see `PR #2650 `_] +* An asset's status page showed only some of the sensors it reported on, in an order that changed between reloads, and it reported on fixed quantities from the flex-context, which have no data to be up to date with [see `PR #2489 `_] +* The asset status page no longer reports "no data recorded" for a sensor recorded by a source type it did not know about (such as the ``CLI script`` source of ``flexmeasures add beliefs``), shows a red light for data that is not knowable yet only for sources expected to deliver future data (such as forecasters and schedulers), no longer claims that a sensor listed because the flex-context or graphs refer to it belongs to the asset, and explains its traffic lights on hover again [see `PR #2474 `_] +* A KPI on the asset page no longer counts an event once per data source that reported it, which could make a total come out higher than any source reported [see `PR #2472 `_] +* The time range sent when loading an asset's KPIs was off by the viewer's UTC offset, so KPIs could cover the wrong days [see `PR #2435 `_] +* Asset type groups on the asset pages now read ``PVs``, ``EVs``, ``CHPs`` and ``Factories``, rather than ``PVS``, ``EVS``, ``CHPS`` and ``Factorys`` [see `PR #2514 `_] * Show dates in the organisation audit log in the same local format as the asset and user audit logs, with the full date and time on hover [see `PR #2664 `_] -* A ``PandasReporter`` with ``droplevels`` failed with ``BeliefDataframe has more than one row per event`` wherever several data sources reported the same input event, as happens for a while when a job reporting on a rolling window gets a new data source; it now keeps one belief per event, chosen as in any other search asking for that, while an input can still ask for all beliefs by setting ``one_deterministic_belief_per_event`` to ``false`` [see `PR #2595 `_] -* ``GET /api/v3_0/assets`` without a sort parameter no longer skips assets when paging, because the query now always orders by the asset id, which is unique, so every page break falls on a stable boundary [see `PR #2585 `_] -* A forecaster given more than one regressor of the same kind trained on each of them several times over, because every regressor's data was collected once per regressor [see `PR #2560 `_] * Organisation audit logs now show only changed fields with their previous and new values, and user role and active-status changes identify the affected user [see `PR #2522 `_] -* ``GET /api/v3_0/assets/(id)/jobs`` now reports its Redis connection error as ``redis-connection-err`` rather than ``redis_connection_err``, so the field is spelled the same way there as on the automation endpoints that also report it [see `PR #2545 `_] -* Where several data sources report the same event, which one a search keeps is now decided the same way every time: a source version only counts against other versions of that source, and a caller that lists its sources gets the order it asked for [see `PR #2494 `_] -* A KPI on the asset page counted an event once per data source that reported it, so a total could come out higher than any source reported; it now reduces one value per event, from the latest source version and the most recent belief within it [see `PR #2472 `_] -* ``flexmeasures add schedule --dry-run`` no longer saves a schedule when it is combined with ``--as-job``, where the flag used to be dropped without a word and the queued job stored its schedule anyway; that combination is now rejected, and a dry run says how many beliefs it would have saved and which events they cover [see `PR #2483 `_] -* The time range sent when loading an asset's KPIs was off by the viewer's UTC offset, so KPIs could cover the wrong days [see `PR #2435 `_] -* An asset's status page showed only some of the sensors it reported on, in an order that changed between reloads, and it reported on fixed quantities from the flex-context, which have no data to be up to date with [see `PR #2489 `_] -* Saving an asset chart as PNG or SVG drew the legend over the graph whenever it listed more sensors than fit beside a subplot; the exported image now makes room for every entry beside its own plot, without shrinking the plot, and spells out the sensor names that the on-screen legend abbreviates [see `PR #2517 `_] -* A chart of a sensor with a single data point in view, such as a daily sensor of which only one day was recorded, drew that point as a bar spanning most of the chart and stretched the shown time range around it; a bar now covers exactly the event it reports, whatever the chart holds [see `PR #2511 `_] -* Asset type groups were named by a pluralization that mangled acronyms and nouns ending in -y, so the asset pages listed ``PVS``, ``EVS``, ``CHPS`` and ``Factorys``; they now read ``PVs``, ``EVs``, ``CHPs`` and ``Factories`` [see `PR #2514 `_] -* A new commitment's prices now default to the currency of the price sensors in the flex-context, where they fell back to EUR unless a fixed price was set [see `PR #2574 `_] +* Creating an organisation now records the action in its audit log, whether it is created in the UI, through the API or from the CLI [see `PR #2580 `_] + +**Access and security** + +* Allow consultants to create assets in client organisations, and reject attempts to change an asset's organisation through the API [see `PR #2786 `_] +* The buttons for deleting an asset and resetting a password now act through the API, and the audit log pages and the page listing an organisation's assets check the same permissions as the API [see `PR #2791 `_] * A page failing on an unexpected error now shows the error page, where browsers behind a proxy reported a broken or insecure connection instead, and such an error no longer reveals the database query that failed [see `PR #2596 `_] -* Fixed several ways in which the asset status page misreported its sensors: it reported "no data recorded" for a sensor recorded by a source type it did not know about (such as the ``CLI script`` source under which ``flexmeasures add beliefs`` records), it showed a red light for data that is not knowable yet, which is only a reason for concern for sources that are expected to deliver future data such as forecasters and schedulers, it claimed that every sensor it lists belongs to the asset shown, also for sensors of another asset that are only listed because the asset's flex-context or graphs page refers to them, and it no longer explained a traffic light when hovering it [see `PR #2474 `_] + +**API, CLI and jobs** + +* ``[GET] /sensors`` lists sensors in order of their ID, where the order used to be up to the database [see `PR #2756 `_] +* ``GET /api/v3_0/assets`` without a sort parameter no longer skips assets when paging [see `PR #2583 `_] +* ``GET /api/v3_0/assets/(id)/jobs`` now reports its Redis connection error as ``redis-connection-err`` rather than ``redis_connection_err``, as the automation endpoints do [see `PR #2545 `_] +* A failed job now says why in the same way on every queue, in the API, the CLI and the asset page's jobs table: a failed forecasting job no longer reads ``dict: {...}`` and now shows its timeout hint, and a failed report or plugin automation no longer reads ``str: ...`` [see `PR #2698 `_] +* Fix ``flexmeasures show beliefs`` using the timezone of the first sensor instead of the one given with ``--timezone``; by default, the CSV file now has its timestamps in the timezone of the first sensor with data, with a UTC offset, instead of in UTC without an offset [see `PR #2695 `_ and `PR #2793 `_] +* Database migration command errors now appear in CLI output, including the error when multiple Alembic heads exist [see `PR #2607 `_] * Several FlexMeasures processes starting at once right after a database migration, such as the web server's workers and a CLI command, no longer fail to start because each tried to create the same starter templates [see `PR #2600 `_] -* Deleting a sensor now unsets it in the flex-model and flex-context references to it that carry a default, bounds or source filters, which used to stay behind pointing at the deleted sensor; the rest of such a reference is kept for when a new sensor is picked, and until then, schedules use its default as a fixed value, or, without a default, ignore the field, so that an ancestor asset's value for it applies; the flex-config editors mark such fields, the asset's pages show a toast linking to them, and deleting a sensor lists the fields that will need a new one [see `PR #2726 `_, `PR #2767 `_ and `PR #2783 `_] -* Offset chains, as in an automation's window or ``flexmeasures add report --start-offset``, now count days as calendar days across a DST transition, so a window no longer collapses or skips a day when run in the hour around midnight on such a weekend, and ``HB`` and ``DB`` no longer raise an error in the repeated hour or in a DST gap [see `PR #2771 `_] -* Chart requests for a sensor or an asset now accept only the names of chart types, and building a chart no longer changes the chart settings that later charts start from, such as a heatmap moving the legend of later charts to the right [see `PR #2801 `_] + +**Hosting** + +* In the Docker image, ``pip install`` (which the docs and the compose stack use to add plugins) now installs where FlexMeasures can import it [see `PR #2811 `_] Automations, in detail @@ -172,18 +253,17 @@ Automations arrived over several pull requests. This is what each of them contri * The first roundtrip, for forecasts: an automation defined per asset, with new CLI commands (``flexmeasures add|edit|delete automation``), a dispatcher (``flexmeasures jobs run-automations``), a UI page and read endpoints (``[GET] /assets/(id)/automations``). Each automation interprets its recurrence in its own timezone, and runs missed while the runner was down are caught up once, coalesced into one current forecast. An automation's details link to the sensors it reads from and writes to, a sensor's page lists the automations feeding it, and deleting a sensor warns about the automations that use it. Jobs now also record whether they were created via the CLI, the API or an automation [see `PR #2290 `_, `PR #2396 `_, `PR #2582 `_, `PR #2593 `_ and `PR #2628 `_] * Schedules as well as forecasts: a schedule automation stores what the schedule trigger endpoint accepts, and schedules from each run's own time [see `PR #2293 `_] * A scheduler's data source now also records the flex config the scheduler computed under, so a schedule can be traced back to the configuration that produced it, and a schedule automation points at such a data source, the way a forecast automation points at its forecaster's [see `PR #2464 `_] -* A single automation can now be run on demand, from the CLI (``flexmeasures jobs run-automation``), the API (``POST /assets//automations//trigger``) and the asset's *Automations* page (a *Run now* button), which is useful to try out a new automation, to re-run one after fixing what made it fail, or to refresh its results after late input data arrived [see `PR #2460 `_] -* Automations can be created, edited and deleted in the UI and through new API endpoints (``[POST|PATCH|DELETE] /assets/(id)/automations``), by whoever may add data under the asset, with their recurrence expressed in a selectable IANA timezone, and only involving sensors they can access themselves (read access to the sensors an automation reads, and permission to record data on the sensors it writes to). The *Automations* page shows when each active automation is due to run next, as a clock time in the automation's own timezone, so a recurrence no longer has to be read back from its cron string; the same time is available as ``next_run`` on the automations API endpoints, and is null while an automation is inactive. It is the next scheduled clock time, so it excludes catch-up work still pending, and it follows the dispatcher's daylight-saving rules, taking the first fold of a repeated local time and the first valid minute after a skipped one. Each row's *Run now*, *Edit*, *Copy*, *Activate*/*Deactivate* and *Delete* controls are collected into a single *Actions* menu, leaving *Details* beside it, so a long listing carries two controls per row instead of five. *Copy* opens the creation form filled in from that automation, and leaves the copy inactive [see `PR #2294 `_, `PR #2563 `_, `PR #2591 `_ and `PR #2647 `_] +* A single automation can now be run on demand, from the CLI (``flexmeasures jobs run-automation``), the API (``POST /assets//automations//trigger``) and the asset's *Automations* page (a *Run now* button), for instance to try out a new automation, to re-run one after fixing what made it fail, or to refresh its results after late input data arrived [see `PR #2460 `_] +* Automations can be created, edited and deleted in the UI and through new API endpoints (``[POST|PATCH|DELETE] /assets/(id)/automations``), by whoever may add data under the asset, with their recurrence expressed in a selectable IANA timezone, and only involving sensors they can access themselves (read access to the sensors an automation reads, and permission to record data on the sensors it writes to). The *Automations* page shows when each active automation is due to run next, as a clock time in the automation's own timezone, which is also available as ``next_run`` on the automations API endpoints. Each row's *Run now*, *Edit*, *Copy*, *Activate*/*Deactivate* and *Delete* controls are collected into a single *Actions* menu beside *Details*, and *Copy* opens the creation form filled in from that automation, leaving the copy inactive [see `PR #2294 `_, `PR #2563 `_, `PR #2591 `_ and `PR #2647 `_] * An automation's output sensors are checked against its creator's permissions when the automation is created, and the schedules it computes are held to exactly those sensors: a scheduler that returns results for any other sensor is refused, rather than recording on a sensor that was only ever checked for read access [see `PR #2536 `_] +* A forecast automation's cycle jobs can likewise only record on the sensors the automation was checked against, as the jobs of schedule, report and plugin automations already could [see `PR #2699 `_] * Reports as well as forecasts and schedules: a report automation stores report parameters, with its reporter and the reporter's configuration on a data source, and reports on a period resolved afresh on each run, either from ``start-offset`` and ``end-offset`` applied to the run time in the automation's timezone, or since the last successful report ended, while a fixed ``start`` or ``end`` is refused; a report job records only on the sensors the automation was checked against [see `PR #2297 `_] * Every automation times its runs the same way: a fixed ``start``, ``end`` or ``prior`` in its parameters is refused, as every run would share that moment, and two of ``start-offset``, ``end-offset`` and ``duration`` describe the period each run covers instead, with the offsets applied to the time the run was due on the automation's own clock, so that, for instance, a schedule automation can plan the whole of the next day [see `PR #2551 `_] * Look up automations from the command line with ``flexmeasures show automations``, which lists them all (inactive ones included) with the IDs that the edit, delete and run commands expect, and, with ``--id``, shows a single automation's recurrence, cursor, parameters and the sensors it reads from and writes to [see `PR #2533 `_] -* The *Automations* page lists the automations of the assets below an asset too, says when each automation runs next, keeps itself up to date, and shows the configuration of an automation's data source, which the *New automation* form now also lets you set [see `PR #2554 `_ and `PR #2299 `_] +* The *Automations* page lists the automations of the assets below an asset too, says when each automation runs next, keeps itself up to date, and shows the configuration of an automation's data source, which the *New automation* form now also lets you set [see `PR #2554 `_, `PR #2299 `_ and `PR #2584 `_] * Automations now keep a durable record of every scheduled run, so a forecast or schedule run which failed before queueing any work is simply picked up again, while one which failed halfway only queues the jobs it still owes; an automation's details show, per run, what it queued, how many attempts that took, and how its jobs ended [see `PR #2457 `_, `PR #2657 `_ and `PR #2670 `_] * Copying an asset now copies the automations on it and on its descendant assets as well, pointing them at the copied sensors and leaving them switched off, so they can be checked before they start running; an automation that cannot be copied safely is skipped and reported, rather than holding up the copy [see `PR #2531 `_] -* Plugins can register automation types with their own validated configuration, parameters and worker queue, with data provenance and sensor authorization. A plugin job's result is kept for as long as results on its own queue are kept, and reports the number of beliefs it saved [see `PR #2553 `_ and `PR #2644 `_] -* The *Automations* page lists the automations of the assets below an asset too, says when each automation runs next, keeps itself up to date, and shows the configuration of an automation's data source, which the *New automation* form now also lets you set [see `PR #2554 `_ and `PR #2584 `_] -* Update Readme and docs index w.r.t. FlexMeasures being described more as scheduling and optimization engine, not per se a whole EMS (see [see `PR #2751 `_] +* Plugins can register automation types with their own validated configuration, parameters and worker queue, with data provenance and sensor authorization. A plugin job's result is kept for as long as results on its own queue are kept, and reports the number of beliefs it saved [see `PR #2553 `_ and `PR #2644 `_] v1.0.1 | September 9, 2026 diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst index 7a12c1bff9..80b38ecab0 100644 --- a/documentation/cli/change_log.rst +++ b/documentation/cli/change_log.rst @@ -4,30 +4,48 @@ FlexMeasures CLI Changelog ************************** -since v1.1.0 | September XX, 2026 +since v1.1.0 | November XX, 2026 ================================= +**Automations** + * Add ``flexmeasures add automation``, ``flexmeasures edit automation`` and ``flexmeasures delete automation`` to manage automations (recurring tasks on an asset, with ``--type forecasting``, ``--type scheduling`` or ``--type reporting`` saying which task to automate). Each automation carries its own IANA timezone (``--timezone``), in which its cron expression is interpreted. * Add ``flexmeasures show automations`` to list all automations, including inactive ones, with the IDs which the edit, delete and run commands expect, and ``flexmeasures show automations --id `` to show one automation in detail: its recurrence in words, timezone, cursor, data generator, parameters and the sensors it reads from and writes to. -* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions. Run this command once per minute. Each scheduled run is claimed durably, so its jobs are queued exactly once even when several runners overlap. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once. +* Add ``flexmeasures jobs run-automations`` to queue jobs for all automations that are due to run this minute from standard five-field cron expressions; run this command once per minute. Each scheduled run is claimed and recorded durably, so its jobs are queued exactly once even when several runners overlap, and a forecast or schedule run whose queueing did not finish is retried, resuming from its stored plan with the job IDs it already queued. Another runner only takes over a run once the claim lease of the runner holding it has expired. Runs missed while the runner was down are caught up once, with several missed forecast runs coalesced into the latest useful forecast, and a run at a skipped or repeated daylight-saving-time hour happens exactly once. The command reports the run and attempt it is working on, and records each attempt's owner, outcome and error. * Add ``flexmeasures jobs run-automation --automation `` to queue the jobs for a single run of one automation, now, on top of its recurring runs. This leaves the automation's cursor alone, so its next recurring run still happens as scheduled, and inactive automations can be run this way, too. -* ``flexmeasures jobs run-automations`` now records a durable run for each scheduled run it claims, and retries the forecast and schedule runs whose queueing did not finish. A run which failed before queueing anything is dispatched again in full, and one which queued only part of its jobs resumes from its stored plan, reusing the job IDs it already queued. Each attempt is recorded with its owner, outcome and error, and the command reports the run and attempt it is working on. A run is only picked up by another runner once the claim lease of the runner holding it has expired, which is how a runner that died mid-queueing hands its work over. -* ``flexmeasures edit automation`` now counts up the automation's schedule revision whenever it rebases the cursor (on a changed cron string or timezone, or on reactivation), which keeps the durable runs of the old and the new schedule apart, even at the same scheduled UTC time. -* ``flexmeasures add automation --type scheduling`` refuses a flex config field which fixes a moment in time, such as ``soc-at-start`` or a ``soc-targets`` entry with a ``datetime``, naming the field: a recurring schedule automation computes a fresh schedule on every run, so such a value would be stale on the next one. * ``flexmeasures add automation`` refuses a fixed ``start``, ``end`` or ``prior`` in the parameters of any automation, as every run would share that moment. The parameters take ``start-offset`` and ``end-offset`` instead, or either one with a ``duration``, applied to the time each run was due, on the automation's own clock, and so do the new ``--start-offset`` and ``--end-offset`` options, next to ``--duration``. -* ``flexmeasures add report`` saves the reporter's config on its data source by default. As a report then records under a new source, anything referring to its old source by ID should be pointed at the new one after the first run. Pass ``--no-save-config`` to leave the config out and keep recording under the old source; ``--save-config`` is still accepted. -* ``flexmeasures add report`` saves all of a report's outputs or none of them, where each output used to be committed on its own, and says for each output how many beliefs it saved. ``flexmeasures add forecasts`` saves the forecasts of a forecaster that a plugin registers, which used to be dropped while the command reported them as created. -* Add ``flexmeasures add report --as-job`` and the ``reporting`` worker queue for asynchronous one-off reports. +* ``flexmeasures add automation --type scheduling`` refuses a flex config field which fixes a moment in time, such as ``soc-at-start`` or a ``soc-targets`` entry with a ``datetime``, naming the field, as such a value would be stale on the next run. +* ``flexmeasures edit automation`` counts up the automation's schedule revision whenever it rebases the cursor (on a changed cron string or timezone, or on reactivation), which keeps the durable runs of the old and the new schedule apart. +* ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run. + +**Data sources and sensor data** + * Add ``flexmeasures jobs check-source-references`` to list waiting jobs, including reporting jobs, that refer to a data source which no longer exists, or which the database upgrade to v1.1 will merge into an identical one. Run it before upgrading, so that you can wait for the jobs it lists or cancel them yourself, and again after. It is expected to be dropped again in v1.2, as from v1.1 the database refuses duplicate data sources. -* ``flexmeasures add forecasts`` no longer keeps every trained model as a file; pass ``--model-save-dir`` to keep them. -* Add ``flexmeasures add forecasts --dry-run`` to compute a forecast and see what it would record, without saving anything to the database. It cannot be combined with ``--as-job``. -* Add ``flexmeasures edit reset-rate-limit --account `` to reset an account's rate limits, so that its requests are accepted again straight away. Pass ``--limit trigger`` or ``--limit default`` to reset only one of them. +* Fix ``flexmeasures show beliefs --timezone``, which used the timezone of the first sensor instead of the given one. By default, the CSV file now has its timestamps in the timezone of the first sensor with data, with a UTC offset, where it used to have them in UTC without an offset. An unknown timezone name is now refused, and the option warns when it is given without ``--to-file``, as the terminal plot shows UTC. + +**Plans and rate limits** + +* Add ``flexmeasures edit reset-rate-limit --account `` to reset an organisation's rate limits, so that its requests are accepted again straight away. Pass ``--limit trigger`` or ``--limit default`` to reset only one of them. +* ``flexmeasures show account`` now displays the organisation's plan and its effective default API and trigger rate limits, including server defaults. * ``flexmeasures add plan`` and ``flexmeasures edit plan`` refuse a rate limit which holds several limits (separated by semicolons), as each rate limit is now one limit. -* Remove the ``--rate-limit-key`` option from ``flexmeasures add plan`` and ``flexmeasures edit plan`` (including ``--clear rate-limit-key``), and its column from ``flexmeasures show plans``, as triggers are now always counted per account. ``flexmeasures show account`` now displays the account's plan and effective default API and trigger rate limits, including server defaults. +* Remove the ``--rate-limit-key`` option from ``flexmeasures add plan`` and ``flexmeasures edit plan`` (including ``--clear rate-limit-key``), and its column from ``flexmeasures show plans``, as triggers are now always counted per organisation. + +**Reporting and forecasting** + +* Add ``flexmeasures add report --as-job`` and the ``reporting`` worker queue for asynchronous one-off reports. +* Add ``flexmeasures add forecasts --dry-run`` to compute a forecast and see what it would record, without saving anything to the database. It cannot be combined with ``--as-job``. +* ``flexmeasures add report`` saves the reporter's config on its data source by default. As a report then records under a new source, anything referring to its old source by ID should be pointed at the new one after the first run. Pass ``--no-save-config`` to leave the config out and keep recording under the old source; ``--save-config`` is still accepted. +* ``flexmeasures add report`` saves all of a report's outputs or none of them, where each output used to be committed on its own, and says for each output how many beliefs it saved. +* ``flexmeasures add forecasts`` saves the forecasts of a forecaster that a plugin registers, which used to be dropped while the command reported them as created. * ``flexmeasures add forecasts`` no longer fails when it runs again from the same ``--from-date``, and says how many of the forecast beliefs it computed were new and saved. +* ``flexmeasures add forecasts`` no longer keeps every trained model as a file; pass ``--model-save-dir`` to keep them. + +**Scheduling** + * Fix ``flexmeasures add schedule --dry-run`` saving a schedule when combined with ``--as-job``: the combination is now rejected, and a dry run reports the beliefs and events it would have saved. -* Fix ``flexmeasures show beliefs --timezone``, which used the timezone of the first sensor instead of the given one. By default, the CSV file now has its timestamps in the timezone of the first sensor with data, with a UTC offset, where it used to have them in UTC without an offset. An unknown timezone name is now refused, and the option warns when it is given without ``--to-file``, as the terminal plot shows UTC. -* ``flexmeasures delete sensor`` now warns which automations read from or write to a sensor before it is deleted, as an automation refers to its sensors by ID and would fail on its next run. + +**Assets** + * ``flexmeasures delete asset`` now records the deletion in the asset's audit log, notes the membership changes it makes in the audit logs of device groups, and removes the asset from the graphs of other assets, as deleting an asset through the API or the UI does. since v1.0.1 | September 9, 2026