feat(FRBC): correct a usage forecast no plan can absorb, instead of clipping it

CHANGELOG-S2.rst

@@ -19,12 +19,13 @@ Added
 * ``FRBCSimple`` can record what a resource manager declares about its storage, so it can be seen and planned with. Give it any of the new ``soc_minima_sensor_id``, ``soc_maxima_sensor_id``, ``usage_forecast_sensor_id`` and ``leakage_behaviour_sensor_id``; a declaration without a sensor is not recorded.
 
   - A fill-level target profile is recorded on the minima and maxima sensors, if both are given. It is recorded on the handler's ``resolution`` steps, starting from the step its start falls in. It is kept inside the declared fill-level range: a target past either end, such as a night setback below the range, is cut back to the range, and a target band wholly outside the range is recorded at its nearest end, with a warning. If one of the two bounds cannot be recorded, this is logged and the other is still recorded.
-  - A usage forecast is recorded in ``<energy_unit>/s``. A negative expected usage is recorded as zero, with a warning.
-  - A leakage behaviour is recorded as a storage efficiency, each time a schedule is requested (see Changed).
+  - A usage forecast is recorded in ``fill_rate_unit`` (see below). A negative expected usage, which S2 allows, is recorded as stated. Without a usage correction (see below), a schedule request carries the latest forecast without its negative values, since FlexMeasures would plan them as a gain the storage cannot refuse. A FlexMeasures server before v0.23.0 only plans with what is recorded, so there a negative expected usage is recorded as zero, with a warning [`#286 <https://github.com/FlexMeasures/flexmeasures-client/pull/286>`__].
+  - A leakage behaviour is recorded as the leakage rate it states, in ``fill_rate_unit``, each time a schedule is requested (see Changed) [`#289 <https://github.com/FlexMeasures/flexmeasures-client/pull/289>`__].
 
   [`#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__, `#259 <https://github.com/FlexMeasures/flexmeasures-client/pull/259>`__, `#271 <https://github.com/FlexMeasures/flexmeasures-client/pull/271>`__, `#274 <https://github.com/FlexMeasures/flexmeasures-client/pull/274>`__]
-* ``FRBCSimple`` takes an ``energy_unit`` (default ``kWh``). The state of charge, the fill-level band, the fill rate and the usage forecast are recorded in this unit (the last two per second). Allowed units are ``J``, ``kJ``, ``MJ``, ``Wh``, ``kWh`` and ``MWh``; any other unit is refused when the handler is created. In a flex model, the state of charge is stated in ``kWh`` or ``MWh``, converted from the other units, because FlexMeasures accepts no others [`#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__, `#260 <https://github.com/FlexMeasures/flexmeasures-client/pull/260>`__]
-* ``FRBCSimple`` takes a ``fill_level_scale`` (default 1). Every fill level and fill rate sent to FlexMeasures is multiplied by it: the state of charge, the flex model, the fill-level band, the fill rate, the usage forecast and the storage efficiency. Translating a schedule into instructions keeps the resource manager's own units, because it matches fill levels against the operation modes, which are stated in those units. A scale of zero or below is refused. ``FRBCSimple`` also takes a ``power_unit`` (default ``W``), which an actuator's power is converted into and posted in [`#236 <https://github.com/FlexMeasures/flexmeasures-client/pull/236>`__, `#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__, `#245 <https://github.com/FlexMeasures/flexmeasures-client/pull/245>`__, `#252 <https://github.com/FlexMeasures/flexmeasures-client/pull/252>`__]
+* ``FRBCSimple`` takes an ``energy_unit`` (default ``kWh``). The state of charge and the fill-level band are recorded in this unit. Allowed units are ``J``, ``kJ``, ``MJ``, ``Wh``, ``kWh`` and ``MWh``; any other unit is refused when the handler is created. In a flex model, the state of charge is stated in ``kWh`` or ``MWh``, converted from the other units, because FlexMeasures accepts no others [`#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__, `#260 <https://github.com/FlexMeasures/flexmeasures-client/pull/260>`__]
+* ``FRBCSimple`` takes a ``fill_rate_unit`` (default ``W``; also ``kW`` or ``MW``). Rates of fill level, which S2 states per second, are recorded in it: the fill rate, the usage forecast and the leakage. In ``kWh/s``, a heat buffer's leakage of some tens of watts would read about 0.00001. Any other unit is refused when the handler is created [`#289 <https://github.com/FlexMeasures/flexmeasures-client/pull/289>`__]
+* ``FRBCSimple`` takes a ``fill_level_scale`` (default 1). Every fill level and fill rate sent to FlexMeasures is multiplied by it: the state of charge, the flex model, the fill-level band, the fill rate, the usage forecast and the leakage. Translating a schedule into instructions keeps the resource manager's own units, because it matches fill levels against the operation modes, which are stated in those units. A scale of zero or below is refused. ``FRBCSimple`` also takes a ``power_unit`` (default ``W``), which an actuator's power is converted into and posted in [`#236 <https://github.com/FlexMeasures/flexmeasures-client/pull/236>`__, `#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__, `#245 <https://github.com/FlexMeasures/flexmeasures-client/pull/245>`__, `#252 <https://github.com/FlexMeasures/flexmeasures-client/pull/252>`__]
 * ``FRBCSimple`` takes a ``hard_bound_factor`` (default 1.5). It sets how far above the declared fill-level range a schedule's hard upper bound lies. If a schedule starts from a state of charge above that bound, the bound is raised to include it. A factor below 1 is refused: FlexMeasures clips the band to the hard bounds, so a hard upper bound inside the declared range would narrow the band to something the resource manager never asked for [`#245 <https://github.com/FlexMeasures/flexmeasures-client/pull/245>`__]
 * ``FRBCSimple`` can take the actuator's charging efficiency into account. Give it the new ``charging_efficiency_sensor_id``: it records there what the operation modes say about converting drawn power into stored energy, and requests a schedule with it as ``charging-efficiency``. For a heat pump, this ratio is its coefficient of performance. Without it, the scheduler assumes power converts into stored energy one to one, and plans to reach fill-level targets at a power the actuator cannot deliver.
 
@@ -48,13 +49,22 @@ Added
 * An FRBC handler tracks the instruction batch the resource manager is working from, and can revoke instructions it no longer wants carried out. Set its ``send_revocations`` attribute to ``True``. When a new batch replaces that one, the handler revokes the old batch's pending instructions: those without a reported status, or reported as ``NEW`` or ``ACCEPTED``. This is off by default, because a resource manager does not have to honour revocations [`#237 <https://github.com/FlexMeasures/flexmeasures-client/pull/237>`__]
 * New ``FRBC`` methods for handlers. ``filter_instruction_transitions()`` drops instructions that repeat the previous one. ``send_instruction_batch()`` sends a batch that replaces the previous one, unless a newer request has arrived in the meantime. ``system_description_fingerprint()`` tells a changed system description from a re-sent one [`#237 <https://github.com/FlexMeasures/flexmeasures-client/pull/237>`__]
 * ``ControlTypeHandler.send_message()``. Handlers now use it to send messages, instead of putting them on the CEM's sending queue themselves [`#237 <https://github.com/FlexMeasures/flexmeasures-client/pull/237>`__]
-* ``FRBCSimple`` takes an optional ``fill_rate_sensor_id``, and records the actuator's fill rate there. The fill rate is recorded in ``<energy_unit>/s``, not as a power: S2 states a fill rate as fill level per second, which for a heat buffer is not electricity at all [`#236 <https://github.com/FlexMeasures/flexmeasures-client/pull/236>`__, `#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__]
+* ``FRBCSimple`` takes an optional ``fill_rate_sensor_id``, and records the actuator's fill rate there. The fill rate is recorded in ``fill_rate_unit``, not in ``power_unit``: a fill rate is what the actuator delivers into its buffer, which for a heat buffer is not electricity at all [`#236 <https://github.com/FlexMeasures/flexmeasures-client/pull/236>`__, `#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__]
 * ``flexmeasures_client.s2.clock``, with a ``WallClock`` and a ``SimulatedClock``, and ``CEM(clock=...)``. The clock sets the time for the CEM and for every handler it registers. That includes their timers, and the time a recorded state of charge is known from [`#264 <https://github.com/FlexMeasures/flexmeasures-client/pull/264>`__, `#268 <https://github.com/FlexMeasures/flexmeasures-client/pull/268>`__]
 * ``CEM(accept_simulation_metadata=True)`` accepts messages from a co-simulation that are wrapped with the simulated time (``S2Wrapper``), and sets the CEM's clock from them. It is off by default, and then a wrapped message is refused as invalid. The example server turns it on with ``S2_ACCEPT_SIMULATION_METADATA`` [`#264 <https://github.com/FlexMeasures/flexmeasures-client/pull/264>`__]
-* ``flexmeasures_client.s2.site``, with ``FRBC_SITE_SENSORS``, ``ensure_site`` and ``FRBCSiteConfigurator``. It gives each resource manager its own FlexMeasures site, named after it, with an FRBC handler connected to that site's sensors [`#265 <https://github.com/FlexMeasures/flexmeasures-client/pull/265>`__]
+* ``flexmeasures_client.s2.site``, with ``FRBC_SITE_SENSORS``, ``ensure_site`` and ``FRBCSiteConfigurator``. It gives each resource manager its own FlexMeasures site, named after it, with an FRBC handler connected to that site's sensors. The sensors' units suit a residential site and match the handler's defaults: powers and rates of fill level in ``W``, energy and fill levels in ``kWh``, and prices per ``kWh``. For sites at a larger scale, such as charge points, pass ``sensors`` with other units, and the handler's ``power_unit`` and ``fill_rate_unit`` in ``handler_options``. [`#265 <https://github.com/FlexMeasures/flexmeasures-client/pull/265>`__, `#289 <https://github.com/FlexMeasures/flexmeasures-client/pull/289>`__]
 * ``CEM(site_configurator=...)`` sets up a resource manager's site once its details arrive. Until then, the CEM answers its messages with a temporary error. The example server enables this with ``S2_SITE_PER_RESOURCE_MANAGER`` [`#265 <https://github.com/FlexMeasures/flexmeasures-client/pull/265>`__]
 * ``examples/S2-co-simulation``: a FlexMeasures stack with one CEM per simulated home, for co-simulations whose resource managers connect over websockets [`#265 <https://github.com/FlexMeasures/flexmeasures-client/pull/265>`__, `#267 <https://github.com/FlexMeasures/flexmeasures-client/pull/267>`__]
 * ``flexmeasures_client.s2.measurements.MeasurementRecorder`` and ``CEM(measurement_recorder=...)`` record a resource manager's measured power. Use one recorder per server, so that a resource manager that reconnects is not recorded twice [`#266 <https://github.com/FlexMeasures/flexmeasures-client/pull/266>`__]
+* ``FRBCSimple`` can correct a usage forecast that no plan can absorb, instead of leaving the storage without a plan. Give it the new ``usage_correction_sensor_id``; it needs FlexMeasures v1.1.0 or newer. A forced gain at a full storage, or a draw at an empty one that the actuator cannot keep up with, used to make every plan infeasible.
+
+  - The plan may move each step's expected usage towards zero, never past it, and only where nothing physical keeps the storage within its hard bounds. It is recorded on the new sensor.
+  - A negative expected usage is then planned as stated, instead of as zero.
+  - The request sets the breach price of the fill-level band relative to the highest market price over the window (``soc_breach_price_factor``, default 10), instead of leaving it to FlexMeasures. The correction is priced above anything else a unit of stock can be worth, including the site's own stored peak and commitment prices, except a breach of the site's capacity. The site's breach prices are kept from the asset, or else set at FlexMeasures' default, which FlexMeasures itself only fills in for a flex context without commodities, so a site keeps to the capacity it is steered with.
+  - Where a correction cannot be planned, for example without readable prices, the forecast is planned without its negative values, and the log says why.
+  - ``FRBCSiteConfigurator`` takes ``correct_usage`` to give each site the sensor, and the example server does so with ``S2_CORRECT_USAGE``. Both are off by default.
+
+  [`#286 <https://github.com/FlexMeasures/flexmeasures-client/pull/286>`__]
 
 Changed
 -------
@@ -84,7 +94,7 @@ Changed
   - The hard bounds ``soc-min`` and ``soc-max`` are now zero and ``hard_bound_factor`` times the top of the range, raised to include a state of charge above it. In v0.9.0, the declared range itself was the hard bound. So a storage that really was outside it, for example floating above the top or left below the bottom, made the problem infeasible: there was no schedule at all, or one from a fallback scheduler. A storage reported below zero is planned from zero, with a warning, because FlexMeasures refuses a negative ``soc-min``.
   - FlexMeasures older than v0.26.0 cannot relax the band. Then the band is left out, and this is logged, so the storage is planned anywhere between the wide hard bounds.
   - The usage forecast is sent as ``soc-usage`` if its sensor is given and the resource manager provides a forecast (or has sent one anyway).
-  - The leakage behaviour is recorded as a ``storage-efficiency`` with each schedule request, over the window of the schedule. It uses the leakage at the starting fill level, as a share of that fill level (or of the middle of the range, if the storage starts below that). It is only named in the flex model if FlexMeasures confirmed storing it. FlexMeasures takes a storage efficiency as a share of the stock, whereas S2 states a rate, so the loss is exact only at that one fill level.
+  - The leakage behaviour is planned as part of the usage: the leakage rate of the fill-level range the storage starts in is added to ``soc-usage`` over the whole window, and sent with the request. S2 states leakage as a rate, so it is not planned as a ``storage-efficiency``, which FlexMeasures takes as a share of the stock, and which would make the storage leak in proportion to how full it is. A flat rate keeps draining an empty storage, as any forced usage does; with a usage correction, the plan can take that away. It needs FlexMeasures v0.23.0 or newer, which takes a usage with the request; an older server is planned without leakage, and this is logged [`#289 <https://github.com/FlexMeasures/flexmeasures-client/pull/289>`__].
 
   [`#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__, `#245 <https://github.com/FlexMeasures/flexmeasures-client/pull/245>`__, `#259 <https://github.com/FlexMeasures/flexmeasures-client/pull/259>`__, `#271 <https://github.com/FlexMeasures/flexmeasures-client/pull/271>`__, `#273 <https://github.com/FlexMeasures/flexmeasures-client/pull/273>`__]
 * ``FRBCSimple`` posts the state of charge in its ``energy_unit`` (``kWh`` by default), and sets the flex model's ``soc-unit`` to match (to ``kWh`` if ``energy_unit`` is neither ``kWh`` nor ``MWh``). In v0.9.0, every state of charge and every schedule's ``soc-unit`` was labelled ``MWh``, whatever unit the resource manager reported in. On the example server's ``kWh`` sensor, a fill level of 10 was recorded as 10,000. A resource manager that reports in megawatt hours now needs ``energy_unit="MWh"`` [`#244 <https://github.com/FlexMeasures/flexmeasures-client/pull/244>`__]

examples/S2-co-simulation/README.md

@@ -18,6 +18,16 @@ with two switches on:
   resource manager retries. Its schedules start from the simulated now and cover
   24 hours. The site's prices are for whoever runs the simulation to post.
 
+A third switch is off by default:
+
+- `S2_CORRECT_USAGE`: a usage forecast that no plan can absorb within the storage's
+  hard bounds is corrected towards zero, instead of leaving the home without a plan.
+  Each site gets a `usage correction` sensor, which records where the plan corrected
+  the forecast. A negative expected usage is then planned as the gain it states,
+  rather than as none. The request also sets the breach prices of the fill-level band
+  and the site's capacity relative to the market, rather than leaving them to
+  FlexMeasures' defaults. It needs FlexMeasures v1.1.0 or newer.
+
 FlexMeasures reads three settings a co-simulation needs from its instance config
 (`flexmeasures-instance/flexmeasures.cfg` in the flexmeasures checkout), not from the environment:
 

examples/S2-co-simulation/docker-compose.cem.yml

@@ -37,6 +37,8 @@ x-cem: &cem
     S2_ACCEPT_SIMULATION_METADATA: "1"
     # And each home is a site of its own, rather than all of them sharing one.
     S2_SITE_PER_RESOURCE_MANAGER: "1"
+    # Uncomment to correct a usage forecast that no plan can absorb, instead of leaving the home without a plan (see README.md).
+    # S2_CORRECT_USAGE: "1"
   volumes:
     - ../flexmeasures-client:/app/flexmeasures-client:rw
   entrypoint: ["/bin/sh", "-c"]