diff --git a/CHANGELOG-S2.rst b/CHANGELOG-S2.rst index cce3da96..59a3e66e 100644 --- a/CHANGELOG-S2.rst +++ b/CHANGELOG-S2.rst @@ -16,99 +16,145 @@ Other changes: see `Unreleased in the changelog `__] -* ``FRBCSimple`` takes an ``energy_unit``, which a state of charge and the fill-level minima and maxima around it are all recorded in. It is one of ``J``, ``kJ``, ``MJ``, ``Wh``, ``kWh`` and ``MWh``, and is refused when the handler is made otherwise. A state of charge goes into a flex model in ``kWh`` or ``MWh``, converted from any other of these, since FlexMeasures takes no other [`#244 `__, `#260 `__] -* ``FRBCSimple`` takes a ``hard_bound_factor``, defaulting to 1.5, which sets how far above the declared fill-level range the hard upper bound of a schedule sits. The bound is widened further where the state of charge a schedule starts from is above it, and a factor below 1 is refused, since FlexMeasures clips the band into the hard bounds and a bound inside the declared range would narrow the band to something the resource manager never asked for [`#245 `__] -* ``FRBCSimple`` takes an optional ``charging_efficiency_sensor_id``, records on it what the actuator's operation modes say about converting drawn power into stored energy, and asks for a schedule against it as ``charging-efficiency``. The ratio between a mode's fill rate and the power it draws is a heat pump's coefficient of performance, and without it the scheduler plans the storage as if an actuator converted power into stored energy one for one, so a plan reaches a fill-level target at a power the actuator cannot deliver it at. FlexMeasures takes one for the whole device, so where a mode converts differently at different fill levels, the element that applies where the schedule starts answers, and where the modes differ, the lowest goes out, and the choice is logged. A mode that draws power while the storage does not fill, such as a standby or circulation mode, states none, rather than an efficiency of zero that would tell the scheduler no power could fill the storage at all [`#253 `__], [`#256 `__] -* ``fill_rate_in_watts()`` and ``operation_mode_conversion_efficiency()``, which state a fill rate in the watts S2 states a power range in, and take the ratio between the two. A fill rate is per second of whatever unit a fill level is recorded in, so a kilowatt hour per second is 3.6 million watts, and an energy unit neither knows is refused rather than converted by a factor that is wrong by thousands [`#253 `__] -* ``FRBCSimple`` takes ``schedule_attempts``, defaulting to five, and ``schedule_retry_interval``, defaulting to two seconds, which bound how often a schedule is asked for again and how long between two attempts. One attempt is what the handler did before, where any failure cost the storage its instruction for that window [`#254 `__] -* ``FRBCSimple`` takes an optional ``production_price_sensor_id``, for a site where feeding back is priced differently from drawing, as under a feed-in tariff. Without it, both directions are priced by ``price_sensor_id``, as before [`#261 `__] -* ``FRBCSimple`` takes ``relax_site_capacity``: ``True`` lets a schedule breach the site's connection capacity at a price, for a site whose capacity is a target set by others, and ``False`` keeps it a hard limit. Unset, it is left to FlexMeasures, which since v1.0 relaxes it by default, so a home whose connection is a physical limit has to pass ``False`` [`#261 `__] -* An FRBC handler logs the instructions of a batch once, at debug level, by operation-mode name [`#261 `__] -* ``flexmeasures_client.s2.clock``, with a ``WallClock`` and a ``SimulatedClock``, and ``CEM(clock=...)``: the time a CEM and every handler it registers go by, timers and the moment a state of charge is known as of included [`#264 `__, `#268 `__] -* ``CEM(accept_simulation_metadata=True)``, which takes a co-simulation's messages wrapped with the simulated time (``S2Wrapper``) and sets its clock from them. Off by default, a wrapped message is refused as invalid. The example server turns it on with ``S2_ACCEPT_SIMULATION_METADATA`` [`#264 `__] -* ``flexmeasures_client.s2.site``: ``FRBC_SITE_SENSORS``, ``ensure_site`` and ``FRBCSiteConfigurator``, which gives each resource manager a FlexMeasures site of its own, named after it, and an FRBC handler wired to it [`#265 `__] -* ``CEM(site_configurator=...)``, which sets a resource manager's site up once its details arrive, answering its messages with a temporary error until then. The example server does so with ``S2_SITE_PER_RESOURCE_MANAGER`` [`#265 `__] -* ``examples/S2-co-simulation``, a FlexMeasures stack with one CEM per simulated home, for a co-simulation whose resource managers connect over websockets [`#265 `__, `#267 `__] -* ``flexmeasures_client.s2.measurements.MeasurementRecorder`` and ``CEM(measurement_recorder=...)``: where a resource manager's measured power is recorded, one per server so that a reconnecting resource manager is not recorded twice [`#266 `__] +* ``FRBCSimple`` can record what a resource manager declares about its storage, so it can be seen and planned with. Give it any of the new ``soc_minima_sensor_id``, ``soc_maxima_sensor_id``, ``usage_forecast_sensor_id`` and ``leakage_behaviour_sensor_id``; a declaration without a sensor is not recorded. + + - A fill-level target profile is recorded on the minima and maxima sensors, if both are given. It is recorded on the handler's ``resolution`` steps, starting from the step its start falls in. It is kept inside the declared fill-level range: a target past either end, such as a night setback below the range, is cut back to the range, and a target band wholly outside the range is recorded at its nearest end, with a warning. If one of the two bounds cannot be recorded, this is logged and the other is still recorded. + - A usage forecast is recorded in ``fill_rate_unit`` (see below). A negative expected usage is recorded as zero, with a warning. + - A leakage behaviour is recorded as a storage efficiency, each time a schedule is requested (see Changed). + + [`#244 `__, `#259 `__, `#271 `__, `#274 `__] +* ``FRBCSimple`` takes an ``energy_unit`` (default ``kWh``). The state of charge and the fill-level band are recorded in this unit. Allowed units are ``J``, ``kJ``, ``MJ``, ``Wh``, ``kWh`` and ``MWh``; any other unit is refused when the handler is created. In a flex model, the state of charge is stated in ``kWh`` or ``MWh``, converted from the other units, because FlexMeasures accepts no others [`#244 `__, `#260 `__] +* ``FRBCSimple`` takes a ``fill_rate_unit`` (default ``W``; also ``kW`` or ``MW``). Rates of fill level, which S2 states per second, are recorded in it: the fill rate and the usage forecast. In ``kWh/s``, a heat buffer's rates of some tens of watts would read about 0.00001. Any other unit is refused when the handler is created [`#290 `__] +* ``FRBCSimple`` takes a ``fill_level_scale`` (default 1). Every fill level and fill rate sent to FlexMeasures is multiplied by it: the state of charge, the flex model, the fill-level band, the fill rate, the usage forecast and the storage efficiency. Translating a schedule into instructions keeps the resource manager's own units, because it matches fill levels against the operation modes, which are stated in those units. A scale of zero or below is refused. ``FRBCSimple`` also takes a ``power_unit`` (default ``W``), which an actuator's power is converted into and posted in [`#236 `__, `#244 `__, `#245 `__, `#252 `__] +* ``FRBCSimple`` takes a ``hard_bound_factor`` (default 1.5). It sets how far above the declared fill-level range a schedule's hard upper bound lies. If a schedule starts from a state of charge above that bound, the bound is raised to include it. A factor below 1 is refused: FlexMeasures clips the band to the hard bounds, so a hard upper bound inside the declared range would narrow the band to something the resource manager never asked for [`#245 `__] +* ``FRBCSimple`` can take the actuator's charging efficiency into account: how much stored energy the drawn power becomes. For a heat pump, this is its coefficient of performance. Give it the new ``charging_efficiency_sensor_id``. Before each schedule request, it records the efficiency there, over the window of the schedule, and requests the schedule with it as ``charging-efficiency``. Without it, the scheduler assumes power converts into stored energy one to one, and plans to reach fill-level targets at a power the actuator cannot deliver. + + - FlexMeasures takes one efficiency per device, so all operation modes share one sensor and one value. If modes differ, the lowest is used, and the choice is logged. If a mode's efficiency depends on the fill level, the value where the schedule starts is used. A sensor per mode would only help once FlexMeasures can plan with an efficiency per mode (`FlexMeasures #2832 `__). + - A mode that draws power without filling the storage, such as standby or circulation, does not count. Counting it as zero would tell the scheduler that the storage cannot be filled at all. Modes reserved for abnormal conditions do not count either. + - The efficiency is recorded once per window. It is recorded again for the next schedule if FlexMeasures did not confirm storing it. If it cannot be recorded, the schedule is requested without it, and this is logged. + + [`#253 `__, `#256 `__, `#272 `__, `#273 `__] +* ``fill_rate_in_watts()`` converts a fill rate into watts, the unit of S2 power ranges. ``operation_mode_conversion_efficiency()`` gives the ratio between the two at a given fill level. A fill rate is per second, in the unit fill levels are recorded in, so one kilowatt hour per second is 3.6 million watts. An unknown energy unit is refused, rather than converted with a factor that is off by thousands [`#253 `__, `#256 `__] +* ``FRBCSimple`` retries a failed schedule request. Two new arguments set the limits: ``schedule_attempts`` (default 5, at least 1) and ``schedule_retry_interval`` (default 2 seconds). In v0.9.0, a single failure left the storage without an instruction for that window, while the resource manager kept polling for one. + + - Retried: a scheduling job that failed on the server, an unreachable server, a 5xx response, and a bare HTTP 400 from a FlexMeasures older than v0.33.0 (its only way to report a failed job). + - Not retried: failures that would repeat on every attempt, such as an infeasible problem or a scheduler colliding with its own belief. + - A job that no worker picked up in time is checked once more. + - Giving up is logged as an error, saying that the storage is left without an instruction. + + [`#254 `__, `#273 `__] +* ``FRBCSimple`` takes an optional ``production_price_sensor_id``, for sites where feeding back is priced differently from drawing, such as under a feed-in tariff. Without it, ``price_sensor_id`` prices both directions, as before [`#261 `__] +* ``FRBCSimple`` takes ``relax_site_capacity``. With ``True``, a schedule may exceed the site's connection capacity at a price; use this where the capacity is a target set by others. With ``False``, the capacity is a hard limit. If unset, FlexMeasures decides, and since v1.0 it relaxes the capacity by default. So for a home whose connection is a physical limit, pass ``False`` [`#261 `__] +* An FRBC handler logs each instruction batch once, at debug level, listing the operation modes by name [`#261 `__] +* An FRBC handler tracks the instruction batch the resource manager is working from, and can revoke instructions it no longer wants carried out. Set its ``send_revocations`` attribute to ``True``. When a new batch replaces that one, the handler revokes the old batch's pending instructions: those without a reported status, or reported as ``NEW`` or ``ACCEPTED``. This is off by default, because a resource manager does not have to honour revocations [`#237 `__] +* New ``FRBC`` methods for handlers. ``filter_instruction_transitions()`` drops instructions that repeat the previous one. ``send_instruction_batch()`` sends a batch that replaces the previous one, unless a newer request has arrived in the meantime. ``system_description_fingerprint()`` tells a changed system description from a re-sent one [`#237 `__] +* ``ControlTypeHandler.send_message()``. Handlers now use it to send messages, instead of putting them on the CEM's sending queue themselves [`#237 `__] +* ``FRBCSimple`` takes an optional ``fill_rate_sensor_id``, and records the actuator's fill rate there. The fill rate is recorded in ``fill_rate_unit``, not in ``power_unit``: a fill rate is what the actuator delivers into its buffer, which for a heat buffer is not electricity at all [`#236 `__, `#244 `__] +* ``flexmeasures_client.s2.clock``, with a ``WallClock`` and a ``SimulatedClock``, and ``CEM(clock=...)``. The clock sets the time for the CEM and for every handler it registers. That includes their timers, and the time a recorded state of charge is known from [`#264 `__, `#268 `__] +* ``CEM(accept_simulation_metadata=True)`` accepts messages from a co-simulation that are wrapped with the simulated time (``S2Wrapper``), and sets the CEM's clock from them. It is off by default, and then a wrapped message is refused as invalid. The example server turns it on with ``S2_ACCEPT_SIMULATION_METADATA`` [`#264 `__] +* ``flexmeasures_client.s2.site``, with ``FRBC_SITE_SENSORS``, ``ensure_site`` and ``FRBCSiteConfigurator``. It gives each resource manager its own FlexMeasures site, named after it, with an FRBC handler connected to that site's sensors. The sensors' units suit a residential site and match the handler's defaults: powers and rates of fill level in ``W``, energy and fill levels in ``kWh``, and prices per ``kWh``. For sites at a larger scale, such as charge points, pass ``sensors`` with other units, and the handler's ``power_unit`` and ``fill_rate_unit`` in ``handler_options`` [`#265 `__, `#290 `__] +* ``CEM(site_configurator=...)`` sets up a resource manager's site once its details arrive. Until then, the CEM answers its messages with a temporary error. The example server enables this with ``S2_SITE_PER_RESOURCE_MANAGER`` [`#265 `__] +* ``examples/S2-co-simulation``: a FlexMeasures stack with one CEM per simulated home, for co-simulations whose resource managers connect over websockets [`#265 `__, `#267 `__] +* ``flexmeasures_client.s2.measurements.MeasurementRecorder`` and ``CEM(measurement_recorder=...)`` record a resource manager's measured power. Use one recorder per server, so that a resource manager that reconnects is not recorded twice [`#266 `__] Changed ------- -* ``FRBCSimple`` plans one schedule at a time, for the newest request: a storage status arriving while a schedule is planned waits for it, and is dropped where a newer one arrives meanwhile, and a request out of date makes no further attempt after a failed one. Each status used to start a solve of its own beside the one in flight, so the newest is now planned after the attempt in flight rather than beside it [`#269 `__] -* An FRBC handler no longer turns a schedule into instructions where a newer storage status arrived while it was being planned, since sending would drop them anyway [`#261 `__] -* ``FRBC.trigger_schedule()`` takes the number of the request it answers as a keyword argument, so a subclass that overrides it has to accept that argument [`#237 `__] -* ``FRBC.send_storage_status()`` likewise takes the number of the planning request it answers, since a storage status is now what asks for a schedule [`#240 `__] -* ``FRBCSimple`` takes a ``resolution``, defaulting to fifteen minutes, which a schedule's start is lined up with [`#240 `__] -* ``FRBCSimple`` asks for a schedule within the power ranges the actuator declares: a ``consumption-capacity`` and a ``production-capacity`` from the widest range it states, and each distinct range as one of the ``operation-modes``, except where a single range runs through zero and so says no more than the two capacities do. The scheduler then plans only power values the actuator can run at, instead of a fractional power that a resource manager has to round into a full-on block, overshooting the capacity its site is held to. An actuator that cannot produce at all is given a production capacity of zero, where before its storage could be planned as if it were a battery feeding the grid [`#248 `__] -* The ``operation-modes`` field is asked for only of a FlexMeasures at v1.0.0 or newer, which is the release that added it. An older server is asked for a schedule without the power bands, and told so in the log, since a flex model naming a field the server does not know is refused outright and leaves the resource manager with no schedule at all [`#248 `__] -* ``FRBCSimple`` asks for a schedule against what the resource manager declared about its storage: the fill levels it wants to be kept between as ``soc-minima`` and ``soc-maxima``, the usage it expects as ``soc-usage``, and what its storage leaks as ``storage-efficiency``, each only where a sensor holds it. Recording those declarations changed nothing about the schedule that came back until the flex model named them [`#245 `__] -* ``fm_schedule_to_instructions()`` takes the fill levels its own running fill level is held between, so a caller that asked for a schedule against other bounds than the declared fill-level range is answered with the operation modes for the storage states the scheduler actually planned. Without them the declared range is used, as before [`#245 `__] -* ``FRBCSimple`` asks for the declared fill-level range as a band whose breach is priced, with ``relax-soc-constraints`` in the flex context, rather than as the hard ``soc-min`` and ``soc-max``, which become wide bounds at zero and half again above the range, each widened to take in a state of charge beyond them. A hard bound at the declared range made states a storage really reaches infeasible, such as free-floating above the top of its range or being left below the bottom of it by a previous schedule, and an infeasible job is answered with no schedule at all, or, against a FlexMeasures older than v1.0.0, by a fallback scheduler that ignores the band [`#245 `__] -* ``FRBCSimple`` posts a state of charge in ``kWh`` rather than ``MWh``, since S2 is a residential protocol and the example server's own state-of-charge sensor is in kilowatt hours. A state-of-charge sensor in another energy unit still holds the right values, since FlexMeasures converts what is posted into the sensor's own unit [`#244 `__] -* ``FRBCSimple`` asks for a schedule again where an attempt failed in a way another attempt could answer differently. A scheduling job can fail on a transient of its own, from a worker crashing out of its solver to a pandas error over the windows two schedules overlap in, and FlexMeasures refuses a trigger outright where the state of charge it would plan from has not landed yet. Unanswered, such a failure leaves the resource manager polling for an instruction that never comes, since asking for a schedule runs as a task whose exception is logged and dropped. A cause that fails identically on every attempt is never asked about again: a scheduler colliding with its own belief, or an infeasible problem [`#254 `__] -* ``FRBCSimple`` says so, as an error, where a schedule was planned by a fallback scheduler rather than the one it asked for. FlexMeasures answered an infeasible problem that way until v1.0.0 retired it, and an older server either serves that fallback job in place of the failed one or redirects to it, which this client follows, so either way the substitute arrives looking like the schedule that was asked for. It is a coarse charging policy rather than a plan, and it records no state of charge for a later window to be planned from [`#254 `__] -* The example resource manager is a heat pump keeping a hot-water buffer warm: it modulates from 400 to 1500 W at a COP of 4 while the buffer is cold and 3 once it is warm, declares its leakage, a day of hot-water use and a comfort profile, and simulates its buffer from S2's own definitions, following each instruction when it is due and reporting its fill level every ``S2_EXAMPLE_STATUS_INTERVAL`` seconds. It follows only the latest plan, honouring the revocations the example server now sends, and tells the CEM where each instruction stands, so that only what it has not yet carried out is revoked, and sends its leakage, usage forecast and comfort profile again every ``S2_EXAMPLE_FORECAST_INTERVAL`` seconds. The example server records each declaration on a sensor of its own, plans from now, seeds its demo prices again every 12 hours, and sets the site's graphs to show the buffer's fill level within its comfort band, and what fills and draws it [`#259 `__, `#262 `__] +* Breaking for subclasses: ``FRBC.trigger_schedule()`` and ``FRBC.send_storage_status()`` each take a ``request_number`` keyword argument: the number of the planning request they answer. A subclass that overrides either method has to accept it [`#237 `__, `#240 `__] +* Breaking: two functions in ``flexmeasures_client.s2.control_types.FRBC.utils`` changed signature. + + - ``compute_next_fill_level()`` is now ``compute_next_fill_level(fill_level, fill_rate, step, leakage_behaviour=None, usage_rate=0.0)``. The step is a ``timedelta``, and every rate is per second. Before, it was ``(fill_level, fill_rate, leakage_behaviour, deltaT, usage)``, with ``deltaT`` in hours. + - ``power_to_fill_rate_with_metrics()`` returns six values instead of five; the operation-mode factor is new. The tuples that ``explain_choice()`` takes changed the same way. + + [`#256 `__, `#258 `__] +* ``fm_schedule_to_instructions()`` takes three new keyword arguments. ``fill_level_bounds`` are the bounds of the fill level it tracks while translating. Pass them if the schedule was requested with bounds other than the declared fill-level range, so that each step gets the operation mode for the state the scheduler actually planned. Without them, the declared range is used, as before. ``leakage_behaviour`` and ``usage_forecast`` lower the tracked fill level as the storage leaks and is used [`#245 `__, `#258 `__] +* An FRBC handler plans one schedule at a time, and only for the newest request. In v0.9.0, every system description started its own solve, and every solve sent its own instruction batch, even when newer ones had arrived. Now a request that arrives during planning waits. It is dropped if an even newer request arrives. A request that is out of date is not retried after a failure, not turned into instructions, and sends no batch [`#237 `__, `#261 `__, `#269 `__] +* An FRBC handler requests a schedule on every storage status, because a schedule starts from the reported state of charge. It also requests one on a system description that arrives after the latest status and differs from the previous one, including in its validity. So a resource manager gets a plan for whichever of the two it sends first, whether or not its fill level changes. In v0.9.0, only system descriptions triggered planning, every one of them, so re-sending an unchanged description triggered a new plan each time. Now such a description is acknowledged and otherwise ignored. If the state of charge cannot be recorded, this is logged, and the schedule is still planned from it [`#240 `__, `#259 `__, `#273 `__] +* ``FRBCSimple`` takes a ``resolution`` (default 15 minutes). It is the step a schedule is planned on and aligned to. Fill-level target profiles, usage forecasts and storage efficiencies are recorded on this step, and each recorded actuator power and fill rate covers one step. A ``resolution`` or ``schedule_duration`` that is not positive is refused. So is a ``schedule_duration`` that is not a whole number of steps, because FlexMeasures only answers with whole steps. In v0.9.0, a schedule was always read as quarter hours, whatever the power sensor's resolution, and an actuator status was recorded as covering a quarter hour [`#240 `__, `#258 `__] +* A schedule starts on a resolution boundary: the last one at or before the system description's ``valid_from`` plus ``valid_from_shift``, or, if that has passed, the next one after now. ``valid_from_shift`` still defaults to one day. In v0.9.0, a schedule started exactly at ``valid_from`` plus the shift. That moment could lie between steps, which left FlexMeasures without prices for the window. Boundaries are computed in the UTC offset of the moment itself, as FlexMeasures does for a series posted with that offset. In the repeated hour at the end of daylight saving time, the later boundary that is not after the moment is used [`#240 `__, `#259 `__] +* ``FRBCSimple`` keeps a schedule within the power ranges the actuator declares, so the scheduler only plans powers the actuator can run at. Before, it could plan a fractional power, which the resource manager had to round up to a full-on block, exceeding the site's capacity. + + - The widest declared range sets ``consumption-capacity`` and ``production-capacity``. An actuator that cannot produce gets a production capacity of zero; before, its storage could be planned as if it were a battery feeding the grid. + - Each distinct range is also sent as one of the ``operation-modes``, if the server runs FlexMeasures v1.0.0 or newer (the release that added this field). This is skipped if the ranges together cover exactly what the capacities allow, without gaps: then they add nothing, and each one would cost the solver a binary variable per step. An older server is asked without the operation modes, and this is logged, because it would refuse a flex model with a field it does not know. + - Modes that S2 reserves for abnormal conditions are left out of both the capacities and the operation modes, and are never used in instructions: a resource manager refuses an instruction into such a mode if the instruction is not marked as abnormal. An actuator that only has such modes is not planned for, with a warning. + + [`#248 `__, `#272 `__] +* ``FRBCSimple`` treats the declared fill-level range as a soft band whose breach is priced, instead of as hard bounds. It also plans with what the resource manager declared about its storage. + + - The band is sent with every schedule request as ``soc-minima`` and ``soc-maxima`` over the whole window, with ``relax-soc-constraints`` in the flex context. It is the declared range, narrowed by the latest fill-level target profile wherever that profile covers a step, and clipped at zero, because FlexMeasures accepts no negative minima. A newer target profile replaces the previous one, as S2 intends. The band is sent whether or not the band sensors are given; they only record it. + - The hard bounds ``soc-min`` and ``soc-max`` are now zero and ``hard_bound_factor`` times the top of the range, raised to include a state of charge above it. In v0.9.0, the declared range itself was the hard bound. So a storage that really was outside it, for example floating above the top or left below the bottom, made the problem infeasible: there was no schedule at all, or one from a fallback scheduler. A storage reported below zero is planned from zero, with a warning, because FlexMeasures refuses a negative ``soc-min``. + - FlexMeasures older than v0.26.0 cannot relax the band. Then the band is left out, and this is logged, so the storage is planned anywhere between the wide hard bounds. + - The usage forecast is sent as ``soc-usage`` if its sensor is given and the resource manager provides a forecast (or has sent one anyway). + - The leakage behaviour is recorded as a ``storage-efficiency`` with each schedule request, over the window of the schedule. It uses the leakage at the starting fill level, as a share of that fill level (or of the middle of the range, if the storage starts below that). It is only named in the flex model if FlexMeasures confirmed storing it. FlexMeasures takes a storage efficiency as a share of the stock, whereas S2 states a rate, so the loss is exact only at that one fill level. + + [`#244 `__, `#245 `__, `#259 `__, `#271 `__, `#273 `__] +* ``FRBCSimple`` posts the state of charge in its ``energy_unit`` (``kWh`` by default), and sets the flex model's ``soc-unit`` to match (to ``kWh`` if ``energy_unit`` is neither ``kWh`` nor ``MWh``). In v0.9.0, every state of charge and every schedule's ``soc-unit`` was labelled ``MWh``, whatever unit the resource manager reported in. On the example server's ``kWh`` sensor, a fill level of 10 was recorded as 10,000. A resource manager that reports in megawatt hours now needs ``energy_unit="MWh"`` [`#244 `__] +* ``FRBCSimple`` records the actuator's power on the power sensor. It no longer writes to ``rm_discharge_sensor_id``, which is still a required argument. In v0.9.0, it posted the negative fill rate of the first listed operation mode, labelled ``MWh``, to that discharge sensor. + + - The power is now that of the operation mode the actuator reports being in: taken from the power range that applies at the latest fill level, interpolated by the reported factor, and posted as covering one ``resolution`` step. + - The sign follows the power sensor's ``consumption_is_positive`` attribute. If the sensor cannot be read, S2's sign is used, and the reason is logged. The power is converted into ``power_unit``. + - A second status for the same step replaces the first, instead of colliding with it. If the power cannot be recorded, the log gives its value, unit, sensor and time. + + [`#236 `__, `#252 `__, `#256 `__, `#258 `__, `#259 `__] +* ``FRBCSimple`` logs an error if a schedule was made by a fallback scheduler instead of the requested one. Until v1.0.0, FlexMeasures answered an infeasible problem that way. An older server either serves the fallback job in place of the failed one, or redirects to it, which this client follows. Either way, the substitute looks like the requested schedule, but it is only a coarse charging policy, not a plan [`#254 `__] +* An FRBC handler refuses a system description with more than one actuator, answering ``INVALID_CONTENT`` with a diagnostic label, because every plan is made for one actuator. In v0.9.0, it accepted such a description and planned for the first actuator only. + + - An actuator that declares timers, or transitions that rule out switching between some of its operation modes, is accepted with a warning that these are not honoured. Transitions that allow every switch rule nothing out, so they cause no warning; an on/off device that declares both its switches is an example. + - The first ``FRBC.TimerStatus`` for each timer also causes the warning. Every timer status is now stored. + + [`#273 `__] +* The example resource manager is now a heat pump that keeps a hot-water buffer warm. + + - It modulates from 400 to 1500 W, with a COP of 4 while the buffer is cold and 3 once it is warm. It declares its leakage, a day of hot-water use and a comfort profile, and simulates its buffer using S2's own definitions. + - It acknowledges each instruction, and reports the transition it asks for when the instruction's ``execution_time`` arrives, instead of printing it and moving on. It reports its fill level every ``S2_EXAMPLE_STATUS_INTERVAL`` seconds. + - It follows only the latest plan, and honours the revocations the example server now sends. It tells the CEM the status of each instruction, so that only instructions not yet carried out are revoked. + - It re-sends its leakage, usage forecast and comfort profile every ``S2_EXAMPLE_FORECAST_INTERVAL`` seconds. + + The example server records each declaration on its own sensor. It creates its power sensor with ``consumption_is_positive``, as S2 states power, so that measured and scheduled power have the same sign. It seeds its demo prices from the current day, and again every 12 hours, so that the window it plans for keeps its prices. It sets up the site's graphs to show the buffer's fill level within its comfort band, and what fills and drains it [`#241 `__, `#252 `__, `#259 `__, `#262 `__] Removed ------- -* ``flexmeasures_client.s2.cem.get_commodity_unit``, which stated power in kW where S2 states it in W [`#266 `__] -* ``op_mode_compute_factor``, which worked a factor back out of a fill rate and returned zero wherever an element's fill rate is one value; the factor now comes with the operation mode chosen, from its power range [`#258 `__] +* ``flexmeasures_client.s2.cem.get_commodity_unit``. It stated power in kW, while S2 states it in W [`#266 `__] +* ``op_mode_compute_factor``. It derived a factor from a fill rate, and returned zero wherever an element has a single fill rate. The factor now comes with the chosen operation mode, from its power range [`#258 `__] +* ``flexmeasures_client.s2.utils.get_validation_error_summary``. It iterated over a pydantic ``ValidationError``, which is not iterable, so it raised on every error [`#257 `__] Fixed ----- -* A CEM records measured power by the moment it was measured, as one average per step of its sensor's resolution (15 minutes on the example's sites, where it was five), once the step is over by its clock, with the first measurement after that, and in watts, as S2 states it. It binned measurements by its own now, which dropped every measurement a resource manager sent after the fact, as a co-simulation's does, labelled watts as kilowatts, and kept its samples in a buffer shared by every CEM in the process. Recording happens in the background, retried a few times where it fails [`#266 `__] -* A CEM without a sensor for a resource manager's measured power no longer records it on sensor 357, which was whichever sensor of whichever site on the server had that id, and answered it as recorded [`#265 `__] -* A message of the control type a CEM is selecting, arriving before the resource manager confirmed the selection, is taken as that confirmation, since a resource manager sends a control type's messages only once it works in it. A CEM used to stay without a control type, answering each such message with a temporary error that the resource manager retried, as a co-simulation's reconnecting resource managers showed. A message of any other control type activates nothing [`#263 `__] -* Closing a control-type handler cancels the work it still has in flight, recording included, and the example server's session ends with its connection. A resource manager that reconnected used to be served by two sessions planning for it at once [`#263 `__] -* A leakage behaviour is recorded as a storage efficiency each time a schedule is asked for, over the window the schedule covers and as a share of the fill level it starts from, or of the middle of the storage's range where it starts below that, and named in the flex model only where it was recorded. It was recorded once, on arrival, over the hours after the message and as a share of the top of the range, so planning went on without it once those hours had passed, and understated the loss at lower fill levels. FlexMeasures takes a storage efficiency as a share of the stock where S2 states a rate, so the loss is exact only at that one fill level [`#259 `__] -* A resolution boundary is found on the clock of the moment being floored, in the offset it carries, as FlexMeasures floors a series posted with it, and in the repeated hour of an autumn DST change it is the later boundary not after the moment, where it used to be taken for the first pass, an hour or more earlier, so a schedule could start in the past [`#259 `__] -* On a state-of-charge sensor that records instants, a state of charge is recorded as a reading at an instant, where it was recorded over the minute after it, which FlexMeasures took for a forecast. On a sensor with a resolution, which refuses an instant, it is still recorded over a minute [`#259 `__] -* ``FillRateBasedControlTUNES`` posts its usage forecast per hour, which is what a value labelled MW means to FlexMeasures, where it posted a quarter-hour amount that FlexMeasures planned as a quarter of the real draw. It reads it back in the resource manager's own units, per second, dividing out the fill-level scale it was posted with, and plans its instructions with its latest leakage behaviour [`#258 `__] -* ``FRBCSimple`` refuses a ``resolution`` or ``schedule_duration`` that is not positive, and a ``schedule_duration`` that is not a whole number of ``resolution`` steps, since FlexMeasures answers with whole steps only, and such a schedule would be read at the wrong step [`#258 `__] -* Translating a schedule into instructions takes every rate per second, as S2 states it. It advanced its running fill level by a fill rate per second times a step in hours, so the level moved 3600 times too little, and every step of a schedule was answered with the operation mode for the fill level the schedule started from. Leakage now lowers that level, from the element of the latest ``FRBC.LeakageBehaviour`` that applies, where it used to be read from two fields S2 does not have and so never counted, and so does usage, from the latest ``FRBC.UsageForecast``. Each step is followed exactly, as a resource manager follows an instruction: a fill level crossing into another element, or a change in usage within a step, is taken at the moment it happens, and a level that reaches a boundary its far side would push it back from stays at that boundary [`#258 `__] -* A schedule is read at the resolution FlexMeasures answers in, its duration over its number of values, rather than as quarter hours whatever it was, which made a schedule for an hourly or a five-minute power sensor raise, so that no instruction went out [`#258 `__] -* A new instruction goes out wherever the operation mode or its factor changes, including at a constant power where the fill level passes into another element of the mode, rather than only where the power changes. A difference in the factor of a millionth or less, which is the noise of converting a schedule from MW to W, is not a new instruction [`#258 `__] -* An instruction asks for the factor that draws the power the schedule asks for, taken from the operation mode's power range. It was worked back out of the fill rate, which asked for the bottom of the power range wherever the fill rate is one value and the power a range [`#258 `__] -* A leakage behaviour is recorded as a storage efficiency compounded per second, as S2 states the rate, where it was compounded per hour, which declared a buffer losing about 7% of its heat over twelve hours as holding it to within 0.002% [`#258 `__] -* An actuator's power and fill rate are recorded as covering one step of the handler's ``resolution``, rather than a quarter hour whatever it plans on [`#258 `__] -* ``FillRateBasedControlTUNES`` plans, and records its conversion efficiencies, when an operation mode has no diagnostic label, which S2 makes optional. It looked for ``THP`` or ``NES`` inside a label that may be absent, so one unlabelled mode stopped planning, and the recording of the efficiencies of every mode listed after it [`#257 `__] -* ``get_validation_error_summary`` is removed: it iterated a pydantic ``ValidationError``, which is not iterable, so it raised on any error it was given, and nothing called it [`#257 `__] -* A leakage behaviour is recorded as a storage efficiency at the rate of the element that applies at the latest fill level, or of its leakiest element where no fill level is known yet, where it took the last one listed, which S2 does not require to be the top one. Translating a schedule into instructions likewise no longer depends on the order a mode lists its elements in [`#256 `__] -* ``FillRateBasedControlTUNES`` records an ``FRBC.LeakageBehaviour`` instead of answering it with ``PERMANENT_ERROR``. It read its fill-level scale from a misspelt attribute, and the error that raised was taken for a failed post, so no leakage behaviour was ever recorded [`#255 `__] -* ``FRBCSimple`` records a second actuator status for the same moment as superseding the first, rather than colliding with it. FlexMeasures keys a belief on its sensor, its source, the event and the moment it was formed, and two statuses inside one resolution step milliseconds apart were refused with a duplicate key whose belief time had lost its sub-second part, losing the later one. A second apart is unambiguous, and the later status supersedes the earlier, which is what a resource manager reporting twice in a second means. A resource manager sends a status for each instruction it applies, and this handler sends several at once, so the power that survived was the one the actuator had already left behind. Statuses about different steps may interleave, so the spacing is kept per step [`#252 `__, `#259 `__] -* A measurement that cannot be recorded is logged with the value, the unit, the sensor and the moment it was for, rather than disappearing, whether FlexMeasures refused it, the server could not be reached, or the unit it would have been posted in is one nothing converts into. The post runs in a task nobody awaits, so until now a refused measurement left the power sensor holding a schedule and nothing else, with the reason visible only as a dead task in the log [`#252 `__] -* ``FRBCSimple`` converts an actuator's power into the unit it posts in, rather than labelling a value S2 states in watts with whatever ``power_unit`` says. A deployment recording its power sensor in ``kW`` had 883.7 W posted as 883.7 kW, which FlexMeasures stored as the thousandfold it claimed to be; the default of ``W`` needed no conversion and was unaffected [`#252 `__] -* The example server creates its power sensor with ``consumption_is_positive``, as S2 states power, so that an actuator's measured power and the schedule planned for it point the same way on the sensor that holds both [`#252 `__] -* The example resource manager answers an instruction: it acknowledges it, and reports the transition it asks for when the instruction's ``execution_time`` arrives, rather than printing it and carrying on. Answering on arrival, which is what the resource managers written against this example do, reports a mode the device is not in yet, so a customer energy manager records a measurement of a power nothing is drawing: two instructions sent together, hours apart in execution time, were answered within milliseconds of each other and contradicted one another [`#252 `__] -* Importing either S2 example script no longer runs it, so they can be read, imported and tested like any other module [`#251 `__] -* ``FRBCSimple`` posts an actuator's power in the sign the power sensor records consumption in, read from the sensor's ``consumption_is_positive`` attribute, rather than in S2's own convention of positive for consumption. FlexMeasures defaults a plain power sensor to the other convention and flips a schedule's sign on the way in, so a measurement of the actuator consuming 883.7 W sat opposite the plan to consume it, which reads as a device doing the reverse of what it was asked. A sensor that cannot be read is posted to as S2 states it, and the reason logged [`#252 `__] -* ``FRBCSimple`` asks for a schedule in watts, which is what S2 states an operation mode's power ranges in, rather than in whatever unit the power sensor happens to be recorded in. S2 states an operation mode's power ranges in watts, and the example server creates its power sensor in ``kW``, so every scheduled power was read as a thousandth of itself: it then sat nearest the operation mode that draws nothing, and the resource manager was instructed to idle through a schedule that asked it to run. The handler already stated the unit when posting an actuator's power; it just never said which one it wanted back [`#247 `__] -* A state of charge is posted in the unit the handler records fill levels in, instead of a hardcoded ``MWh``, which stopped FlexMeasures converting it: the example server creates its state-of-charge sensor in ``kWh``, so a fill level of 10 was being recorded as 10,000. A schedule is asked for in that same unit, so the bounds the scheduler plans against and the measurements it plans from agree [`#244 `__] -* A state of charge, and the whole flex-model a schedule is asked for with, are scaled by ``fill_level_scale`` like every other fill level the handler records. The scale is applied where the data reaches FlexMeasures and nowhere else: translating a schedule back into instructions keeps the resource manager's own units, since it matches fill levels against the operation modes stated in them [`#244 `__] -* A fill rate is recorded as a rate of fill level, in ``/s``, instead of in the power unit. S2 states both an actuator's fill rate and a usage forecast as fill level per second, which for a heat buffer is not electricity at all, and which even for a battery differs from kilowatts by the seconds in an hour [`#244 `__] -* A fill-level target profile is posted from the resolution boundary its translation lined it up with, rather than from the moment the resource manager sent it, which shifted every value by the difference [`#244 `__] -* An ``FRBCSimple`` handler accepts an ``FRBC.FillLevelTargetProfile``, an ``FRBC.UsageForecast`` and an ``FRBC.LeakageBehaviour`` instead of raising ``NotImplementedError`` for each. The CEM dispatches these as tasks nobody awaits, so every connection by a resource manager that sends all three logged three "Task exception was never retrieved" tracebacks, and recorded none of what they declared [`#244 `__] -* A fill-level target profile is held inside the fill-level range the latest system description declared, so a target reaching past either end of it, such as a night setback below the range, is recorded as the range allows rather than as asked [`#244 `__] -* The example CEM server answers an S2 handshake at once, instead of after configuring its site with FlexMeasures, which took longer than the timeout a resource manager waits for that answer [`#241 `__] -* The example CEM server seeds its demo prices from the current day, instead of from a date hardcoded when it was written, and the example resource manager dates its system description now, so that the window it schedules for has prices. Before both, the two had agreed only by both being hardcoded to the same day [`#241 `__, `#259 `__] -* An FRBC handler asks for a schedule when a storage status reports a new state of charge, and when a system description arrives after the latest status and differs from the one before it, its validity included, so a resource manager is planned for whichever of the two it sends first, and whether or not its fill level moves [`#240 `__, `#259 `__] -* A schedule is asked for from a resolution boundary, instead of from whichever moment a resource manager happened to send its system description, which left FlexMeasures without prices for the window, and from the next resolution boundary rather than the start of the step under way, which a description days old made it slide into the past, and which, planned from the fill level reported mid-step, counted on the whole step where only part was left, so each replan within it asked for less power than the step needed [`#240 `__, `#259 `__] -* An FRBC handler ignores a re-sent system description whose content is unchanged, instead of storing it again [`#240 `__] -* An FRBC handler answers only the newest schedule request, instead of letting every request in a backlog complete its own round trip and send its own instruction batch [`#237 `__] -* An FRBC handler sends an instruction only where it changes what the actuator does, which holds at the sending boundary for every producer of instructions [`#237 `__] -* The instruction-status history is keyed by a string, as its type declares, instead of by the ``UUID`` that a no-op ``cast`` let through, which made every lookup miss [`#237 `__] -* ``FRBCSimple`` reports the power of the operation mode the actuator is actually in, taken from the power range of the element that applies at the latest fill level, and posts it to the power sensor [`#236 `__, `#256 `__] -* A control-type handler can log before it is registered, and one without its own ``close()`` no longer breaks CEM teardown [`#236 `__] - -Added ------ - -* An FRBC handler tracks the instruction batch the resource manager is working from, and ``send_revocations`` makes it revoke the instructions of a batch it replaces that are still pending [`#237 `__] -* ``ControlTypeHandler.send_message()``, which handlers use instead of putting messages on the CEM's sending queue themselves [`#237 `__] -* ``FRBCSimple`` takes an optional ``fill_rate_sensor_id``, and records the actuator's fill rate there instead of discarding it [`#236 `__] +* A full message history now keeps the newest messages. It used to make room by evicting the newest entry, so it kept a session's first messages and lost all later ones except the last. A message stored again under the same id now counts as the latest [`#274 `__] +* ``Handler.is_revoked()`` now finds objects the resource manager revoked. The CEM and its handlers stored a revoked object's id as a ``UUID``, so a lookup by its string id always missed [`#257 `__] +* An FRBC actuator status that arrives before any system description, which can happen after a reconnection, is kept, and recorded once a system description arrives. Only the latest per actuator is kept, since a status describes the actuator's present state. It used to raise an error in a background task [`#274 `__, `#287 `__] +* A CEM records measured power by the time it was measured. It records one average per step of the sensor's resolution (15 minutes on the example's sites, previously five), in watts, as S2 states it. A step is recorded with the first measurement that arrives after the step has ended by the CEM's clock. Before, measurements were grouped by the CEM's current time, which dropped every measurement sent after the fact, as co-simulations do. They were also labelled as kilowatts instead of watts, and kept in a buffer shared by every CEM in the process. Recording happens in the background, with a few retries if it fails [`#266 `__] +* A CEM without a sensor for a resource manager's measured power no longer records L1 measurements on sensor 357. That was whatever sensor had that id on the server, on any site, and the CEM reported the measurement as recorded [`#265 `__] +* A CEM that is selecting a control type now takes a message of that control type as confirmation of the selection, because a resource manager only sends such messages once it uses that control type. Before, the CEM stayed without a control type, and answered each such message with a temporary error, which the resource manager retried; co-simulations whose resource managers reconnect showed this. A message of any other control type activates nothing [`#263 `__] +* Closing a control-type handler cancels its remaining work, including recording. The example server's session now ends when its connection closes; it used to keep working after the resource manager disconnected [`#263 `__] +* A control-type handler can log before it is registered, and a handler without its own ``close()`` no longer breaks the CEM's teardown [`#236 `__] +* An ``FRBCSimple`` handler now accepts ``FRBC.FillLevelTargetProfile``, ``FRBC.UsageForecast`` and ``FRBC.LeakageBehaviour`` messages, instead of raising ``NotImplementedError`` for each. The CEM runs these as tasks that nothing awaits, so a resource manager sending all three caused three "Task exception was never retrieved" tracebacks on every connection [`#244 `__] +* ``FRBCSimple`` requests schedules in watts, the unit of S2 operation-mode power ranges, instead of in the power sensor's unit. The example server's power sensor is in ``kW``, so every scheduled power was read as a thousandth of its value. It then matched the operation mode that draws nothing, and the resource manager was told to idle while the schedule asked it to run [`#247 `__] +* On a state-of-charge sensor that records instants, a state of charge is now recorded as a reading at an instant. Before, it was recorded as covering the minute after it, which FlexMeasures took for a forecast. On a sensor with a resolution, which refuses instants, it is still recorded over one minute [`#259 `__] +* Translating a schedule into instructions now treats every rate as per second, as S2 states it. + + - It used to advance the tracked fill level by a fill rate per second times a step in hours, so the level moved 3600 times too little. As a result, every step got the operation mode for the fill level the schedule started from. + - Leakage now lowers the tracked fill level, using the element of the latest ``FRBC.LeakageBehaviour`` that applies. Before, leakage was read from two fields that S2 does not have, so it was never counted. Usage from the latest ``FRBC.UsageForecast`` now lowers it too. + - Each step is followed exactly, as a resource manager follows an instruction. If the fill level crosses into another element, or usage changes within a step, that is taken into account at the moment it happens. A level that reaches a boundary, beyond which it would be pushed back, stays at that boundary. + + [`#258 `__] +* Translating a schedule into instructions no longer depends on the order in which a mode lists its elements. Where two elements share a boundary, the one that applies is used, not the first one listed [`#256 `__] +* A schedule is read at the resolution FlexMeasures answers in: its duration divided by its number of values. It used to be read as quarter hours, so a schedule for an hourly or a five-minute power sensor raised an error, and no instruction was sent [`#258 `__] +* A new instruction is sent whenever the operation mode or its factor changes, and only then, instead of whenever the power changes. This includes a constant power at which the fill level moves into another element of the mode. A change in the factor of a millionth or less does not count: that is rounding noise from converting a schedule from MW to W [`#237 `__, `#258 `__] +* An instruction now asks for the factor that draws the scheduled power, taken from the operation mode's power range. It used to derive the factor from the fill rate, which asked for the bottom of the power range wherever the fill rate is a single value and the power a range [`#258 `__] +* ``leakage_behaviour_to_storage_efficiency()`` compounds a leakage rate per second, as S2 states it. It compounded it per hour, so a buffer that loses about 7% of its heat over twelve hours was described as losing 0.002%. + + - A new ``fill_level`` keyword argument selects the element that applies at that fill level. Without it, the leakiest element is used. Before, it used the last element listed, which S2 does not require to be the top one. + - A new ``share_of`` keyword argument sets the fill level that the loss is a share of. Without it, that is the top of the highest element's range. + + [`#256 `__, `#258 `__, `#259 `__] +* ``FillRateBasedControlTUNES`` posts its usage forecast per hour, which is what a value in MW means to FlexMeasures. It used to post a quarter-hour amount, which FlexMeasures planned as a quarter of the real usage. It reads the forecast back in the resource manager's own units, per second, dividing out the fill-level scale it was posted with, and plans its instructions with its latest leakage behaviour [`#258 `__] +* ``FillRateBasedControlTUNES`` now plans, and records its conversion efficiencies, when an operation mode has no diagnostic label, which S2 makes optional. It looked for ``THP`` or ``NES`` in a label that may be missing, so a single unlabelled mode stopped planning, and stopped the efficiencies of all modes listed after it from being recorded [`#257 `__] +* ``FillRateBasedControlTUNES`` records an ``FRBC.LeakageBehaviour`` instead of answering it with ``PERMANENT_ERROR``. It read its fill-level scale from a misspelt attribute, and the resulting error was mistaken for a failed post, so no leakage behaviour was ever recorded [`#255 `__] +* Importing either S2 example script no longer runs it, so they can be read, imported and tested like any other module [`#252 `__] [`issue #251 `__] +* The example CEM server answers an S2 handshake immediately. It used to configure its site in FlexMeasures first, which took longer than a resource manager waits for the answer [`#241 `__] v0.9.0 | 2026-04-21 =================== diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 4d0d1f34..f93b4948 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -21,19 +21,20 @@ Added * ``server_version_at_least``, which answers whether the server is of a given version or newer, looking its version up where that is not known yet, for a caller that sends a field or calls an endpoint only where the server has it. Release generations are compared, so a development build such as ``0.33.0.dev26`` counts as ``0.33.0`` [`#248 `__] * ``ingestion_polling_interval``, ``ingestion_polling_max_interval`` and ``ingestion_polling_timeout``, which govern how the ingestion of posted sensor data is waited for, on a tighter profile than the ``job_polling_*`` settings give other background jobs [`#242 `__] +* ``post_sensor_data`` takes ``confirm_ingestion``. With ``True``, it raises an error if it cannot confirm that FlexMeasures ingested the posted data, instead of logging this and carrying on. Use it if your code remembers what it has recorded, and would otherwise never record the data again [`#273 `__] * Optional ``rate_limit_notifier`` and ``job_status_notifier`` callbacks, to report rate limits and the progress of jobs being waited for, instead of logging them [`#225 `__] * ``get_automations``, ``get_automation``, ``add_automation``, ``update_automation``, ``delete_automation``, ``trigger_automation`` and ``trigger_and_await_automation``, to manage and run automations through the API on FlexMeasures v1.1.0 and up [`#230 `__] Changed ------- -* ``add_asset`` no longer requires a ``latitude`` and ``longitude``, which FlexMeasures does not require either, for an asset with no location, such as one standing for a community. ``generic_asset_type_id`` stays required, and half a location is refused [`#261 `__] +* ``add_asset`` no longer requires a ``latitude`` and ``longitude``, just like FlexMeasures. This allows assets without a location, such as one that represents a community. ``generic_asset_type_id`` is still required, and a latitude without a longitude (or the other way around) is refused [`#261 `__] * ``post_sensor_data``, and the deprecated ``post_measurements`` it was renamed from, wait for a server of FlexMeasures 0.33.0 or above that ingests the posted data asynchronously, answering ``202 Accepted`` with a job id, to finish doing so, so that a caller can read the data back or schedule over it; a failed job raises ``JobFailedError``, while a job still pending at the timeout, or one that cannot be looked up, is logged as an error and leaves the post successful. Pass ``await_ingestion=False`` to return as soon as the data is accepted [`#242 `__] Removed ------- -* ``response_handling.check_content_type``, which refused an answer in a JSON variant that aiohttp parses, such as ``application/problem+json``, with an error that raised an ``AttributeError`` of its own when printed. Such an answer is now read like any other. A content type that is not JSON at all is still reported as a ``ConnectionError``, by aiohttp's own check [`#257 `__] +* ``response_handling.check_content_type``. It refused responses in JSON variants that aiohttp can parse, such as ``application/problem+json``, and the error it raised caused an ``AttributeError`` of its own when printed. Such responses are now read like any other. A response that is not JSON at all is still reported as a ``ConnectionError``, by aiohttp's own check [`#257 `__] Fixed ----- diff --git a/src/flexmeasures_client/client.py b/src/flexmeasures_client/client.py index 9252c477..819228e7 100644 --- a/src/flexmeasures_client/client.py +++ b/src/flexmeasures_client/client.py @@ -459,11 +459,11 @@ def report_rate_limit(message: str) -> None: "Request retry timeout while connecting to the API." ) from exception - # request_once has already parsed this answer with aiohttp's json(), - # which refuses a content type that is not JSON, and the loop above reports that as a ConnectionError. - # A check that used to follow here refused the JSON variants aiohttp accepts too, such as application/problem+json, - # with an error that could not even be printed, so it is gone. - # A body that is not JSON under a JSON content type is refused by json() itself, as a JSONDecodeError. + # request_once has already parsed this response with aiohttp's json(). + # json() refuses a content type that is not JSON, and the loop above reports that as a ConnectionError. + # A check used to follow here, but it also refused JSON variants that aiohttp accepts, such as application/problem+json. + # Its error could not even be printed, so the check was removed. + # A body that is not JSON under a JSON content type makes json() itself raise a JSONDecodeError. return await response.json(), response.status async def request_once( @@ -664,6 +664,7 @@ async def post_sensor_data( file_path: str | None = None, belief_time_measured_instantly: bool = False, await_ingestion: bool = True, + confirm_ingestion: bool = False, ): """ Post sensor data for the given time range. @@ -676,19 +677,24 @@ async def post_sensor_data( The method automatically chooses the appropriate API endpoint based on the provided parameters. - Since FlexMeasures 0.33.0, a server with an ingestion worker available answers ``202 Accepted`` with the id of a background job, - rather than ``200 OK``, and ingests the data after the request has returned. - By default this method then waits for that job to finish, - so that a caller which reads the data back, or asks for a schedule over it, sees what it just posted. + Since FlexMeasures 0.33.0, a server with an available ingestion worker responds ``202 Accepted`` instead of ``200 OK``, + with the id of a background job. + The server then ingests the data after the request has returned. + By default, this method waits for that job to finish. + This way, a caller that reads the data back, or asks for a schedule over it, sees the data it just posted. :param await_ingestion: whether to wait for asynchronous ingestion to finish, - when the server answered ``202 Accepted`` with a job id. - Waiting is polled on the ``ingestion_polling_*`` profile, which is tighter than the one other jobs are polled on. + when the server responded ``202 Accepted`` with a job id. + The job is polled on the ``ingestion_polling_*`` profile, which is tighter than the profile for other jobs. A job that fails raises :class:`JobFailedError`. - A job still pending when ``self.ingestion_polling_timeout`` is reached is logged as an error and otherwise ignored, - since the data may well land moments later, and a caller that has waited that long is better served by carrying on than by failing. + A job that is still pending when ``self.ingestion_polling_timeout`` is reached is logged as an error and otherwise ignored. + The data may well land moments later, and a caller that has waited that long is better off carrying on than failing. Pass ``False`` to return as soon as the server has accepted the data. - A synchronous ``200 OK`` response is unaffected either way. + Either way, a synchronous ``200 OK`` response is unaffected. + :param confirm_ingestion: whether to raise when ingestion could not be confirmed while waiting for it, instead of logging and ignoring that. + If the job was still pending at the timeout, this raises :class:`JobTimeoutError`. + If looking the job up failed, this raises the error from the lookup. + Use this if the caller remembers having recorded the data, and would otherwise never record it again. This function raises a ValueError when an unhandled status code is returned. """ @@ -740,6 +746,7 @@ async def post_sensor_data( unit=unit, prior=prior, await_ingestion=await_ingestion, + confirm_ingestion=confirm_ingestion, ) else: # Type assertion to help the type checker understand that file_path is not None @@ -752,6 +759,7 @@ async def post_sensor_data( belief_time_measured_instantly=belief_time_measured_instantly, unit=unit, await_ingestion=await_ingestion, + confirm_ingestion=confirm_ingestion, ) @staticmethod @@ -767,20 +775,26 @@ def _ingestion_job_id(response) -> str | None: job_id = response.get("job") or response.get("job_id") return job_id if isinstance(job_id, str) else None - async def _await_sensor_data_ingestion(self, job_id: str) -> None: + async def _await_sensor_data_ingestion( + self, job_id: str, confirm_ingestion: bool = False + ) -> None: """Wait for an ingestion job to finish, so that the data it carries can be read back. - Waiting is left to :func:`wait_for_job`, so ingestion is backed off like every other job this client waits for, - and reported through the same ``job_status_notifier``, on the ``ingestion_polling_*`` intervals rather than the ``job_polling_*`` ones. - - A failed job propagates as :class:`JobFailedError`, for the caller to handle like any other failed post. - Nothing else does. - A timeout is logged as an error and otherwise ignored, for the reason given in :func:`post_sensor_data`, - and so is a failure to look the job up at all, which says nothing about the data: - the server accepted that, and a job record can be gone, or answered for by something in front of the server, by the time it is asked about. - Raising there would report a post that succeeded as a post that failed. - A lookup that 404s is reported as a ``ConnectionError`` by :func:`request`, which wraps every client error that way, - so that is among the ones caught here, alongside the ``ValueError`` an unexpected status code raises. + Waiting is delegated to :func:`wait_for_job`. + So ingestion is backed off like every other job this client waits for, + and reported through the same ``job_status_notifier``. + The only difference is that it uses the ``ingestion_polling_*`` intervals, instead of the ``job_polling_*`` ones. + + A failed job raises :class:`JobFailedError`, which the caller handles like any other failed post. + By default, no other error is raised. + A timeout is logged as an error and otherwise ignored, for the reason given in :func:`post_sensor_data`. + The same goes for a failure to look the job up at all, which says nothing about the data. + After all, the server did accept the data. + By the time the job is looked up, its record may be gone, or something in front of the server may respond instead. + Raising in these cases would report a successful post as a failed post. + If the caller asked for ingestion to be confirmed, both cases are logged and then raised. + A lookup that returns 404 is reported as a ``ConnectionError`` by :func:`request`, which wraps every client error that way. + So ``ConnectionError`` is among the errors caught here, alongside the ``ValueError`` that an unexpected status code raises. """ try: await self.wait_for_job( @@ -793,6 +807,8 @@ async def _await_sensor_data_ingestion(self, job_id: str) -> None: self.logger.error( f"Ingestion of the data posted as job {job_id} is unconfirmed, so a read of it may come back without it. {e}" ) + if confirm_ingestion: + raise except ( ConnectionError, TimeoutError, @@ -803,6 +819,8 @@ async def _await_sensor_data_ingestion(self, job_id: str) -> None: self.logger.error( f"Ingestion of the data posted as job {job_id} is unconfirmed, because looking that job up failed, so a read of it may come back without it. {e}" ) + if confirm_ingestion: + raise async def _post_sensor_data_json( self, @@ -813,6 +831,7 @@ async def _post_sensor_data_json( unit: str, prior: str | datetime | None = None, await_ingestion: bool = True, + confirm_ingestion: bool = False, ): """ Post sensor data using JSON payload. @@ -849,7 +868,7 @@ async def _post_sensor_data_json( self.logger.info("Sensor data sent successfully via JSON.") if await_ingestion and status == 202 and job_id: - await self._await_sensor_data_ingestion(job_id) + await self._await_sensor_data_ingestion(job_id, confirm_ingestion) return response, status @@ -860,6 +879,7 @@ async def _post_sensor_data_file( belief_time_measured_instantly: bool = False, unit: str | None = None, await_ingestion: bool = True, + confirm_ingestion: bool = False, ): """ Post sensor data using file upload. @@ -981,7 +1001,7 @@ async def _post_sensor_data_file( # Wait for ingestion outside the request timeout above, and outside the error handler that names this an upload failure: # a job takes as long as self.ingestion_polling_timeout allows, which is no concern of the request that queued it. if await_ingestion and status == 202 and upload_job_id: - await self._await_sensor_data_ingestion(upload_job_id) + await self._await_sensor_data_ingestion(upload_job_id, confirm_ingestion) return response_data, status @@ -995,6 +1015,7 @@ async def post_measurements( unit: str, prior: str | datetime | None = None, await_ingestion: bool = True, + confirm_ingestion: bool = False, ): """ Post sensor data for the given time range. @@ -1019,6 +1040,7 @@ async def post_measurements( unit=unit, prior=prior, await_ingestion=await_ingestion, + confirm_ingestion=confirm_ingestion, ) async def get_schedule( @@ -1538,10 +1560,10 @@ async def add_asset( } This function raises a ValueError when an unhandled status code is returned, - when no generic_asset_type_id is given, and when only one of latitude and longitude is. + when no generic_asset_type_id is given, and when only one of latitude and longitude is given. """ if generic_asset_type_id is None: - # Optional in the signature only so that latitude and longitude, before it, can be left out, + # The signature makes this optional only so that latitude and longitude, which come before it, can be left out, # while every argument keeps its position. raise ValueError("add_asset needs a generic_asset_type_id.") if (latitude is None) != (longitude is None): @@ -1553,7 +1575,7 @@ async def add_asset( account_id=account_id, generic_asset_type_id=generic_asset_type_id, ) - # A location is optional to FlexMeasures, which many assets have no use for, such as one standing for a community. + # FlexMeasures does not require a location, and many assets have no use for one, such as an asset that represents a community. if latitude is not None: asset["latitude"] = latitude if longitude is not None: diff --git a/src/flexmeasures_client/s2/__init__.py b/src/flexmeasures_client/s2/__init__.py index 04a7cbcb..9f2ed60e 100644 --- a/src/flexmeasures_client/s2/__init__.py +++ b/src/flexmeasures_client/s2/__init__.py @@ -127,7 +127,7 @@ def __init__(self, max_size: int = 100, timezone: str = "UTC") -> None: self.outgoing_messages_status = SizeLimitOrderedDict(max_size=max_size) self._timezone = ZoneInfo(timezone) - #: The time this handler goes by; a CEM gives each handler it registers its own clock. + #: The clock this handler reads the time from. A CEM sets this to its own clock for each handler it registers. self._clock: Clock = WallClock() self.discover() @@ -253,5 +253,5 @@ def handle_revoke_object(self, message: RevokeObject) -> ReceptionStatus: ) def now(self) -> datetime: - """The current moment by this handler's clock, in its time zone.""" + """Return the current moment by this handler's clock, in the handler's timezone.""" return self._clock.now().astimezone(self._timezone) diff --git a/src/flexmeasures_client/s2/cem.py b/src/flexmeasures_client/s2/cem.py index 6d1f70e4..34122e77 100644 --- a/src/flexmeasures_client/s2/cem.py +++ b/src/flexmeasures_client/s2/cem.py @@ -48,7 +48,7 @@ class ConfiguredHandler(Protocol): - """What a site configurator gives back: the handler for a resource manager, and its measured-power sensors.""" + """What a site configurator returns: the handler for a resource manager, and its measured-power sensors.""" @property def handler(self) -> ControlTypeHandler: ... @@ -60,10 +60,10 @@ def power_sensor_ids(self) -> dict[str, int]: ... def asset_id(self) -> int: ... -#: Gives a resource manager a site and a handler of its own, from its details (see flexmeasures_client.s2.site). +#: A callable that takes a resource manager's details and gives it its own site and handler (see flexmeasures_client.s2.site). SiteConfigurator = Callable[[ResourceManagerDetails], Awaitable[ConfiguredHandler]] -#: The control type each family of control-type messages belongs to, by the prefix of its message type. +#: Maps the prefix of a message type to the control type that its messages belong to. CONTROL_TYPE_BY_MESSAGE_PREFIX: dict[str, ControlType] = { "FRBC.": ControlType.FILL_RATE_BASED_CONTROL, "PEBC.": ControlType.POWER_ENVELOPE_BASED_CONTROL, @@ -111,22 +111,25 @@ def __init__( """ Customer Energy Manager (CEM) - :param site_configurator: where a resource manager gets a site and a handler of its own on connecting; - called with its details, in the background (see FRBCSiteConfigurator). - Until the handler is registered, its messages are answered with TEMPORARY_ERROR. - Without one, the handlers registered up front serve every resource manager. - :param measurement_recorder: where the resource manager's measured power is recorded; one of its own by default. - A server that makes a CEM per connection hands each the same one, - so a resource manager that reconnects is not recorded twice. - :param clock: the time this CEM and every handler it registers go by; - the wall clock by default. - :param accept_simulation_metadata: whether to take messages wrapped with a simulated time, as a co-simulation sends them, + :param site_configurator: gives a resource manager its own site and handler when it connects. + It is called with the resource manager's details, in the background (see FRBCSiteConfigurator). + Until the handler is registered, the CEM responds to the resource manager's messages with TEMPORARY_ERROR. + Without a site configurator, the handlers registered up front serve every resource manager. + By default, there is none. + :param measurement_recorder: records the resource manager's measured power. + By default, the CEM creates its own recorder. + A server that creates a CEM per connection should hand each CEM the same recorder. + Then a resource manager that reconnects does not get its measurements recorded twice. + :param clock: the clock that this CEM and every handler it registers read the time from. + By default, this is the wall clock. + :param accept_simulation_metadata: whether to accept messages wrapped with a simulated time, as a co-simulation sends them, and set the clock to that time (see flexmeasures_client.s2.wrapper.S2Wrapper). - Off by default, since the wrapper is not part of S2, + This is off by default, because the wrapper is not part of S2, and a live CEM should not let a resource manager set its time. - A resource manager that wraps its messages, from its handshake on, - is then refused with INVALID_MESSAGE and cannot connect. - On, the clock has to be a SimulatedClock, and one is made where none is given. + When it is off, a resource manager that wraps its messages from its handshake on is refused with INVALID_MESSAGE, + and cannot connect. + When it is on, the clock must be a SimulatedClock. + If no clock is given, a SimulatedClock is created. """ super(CEM, self).__init__() if accept_simulation_metadata: @@ -149,7 +152,7 @@ def __init__( fm_client, logger=logger ) self._configuring: asyncio.Task | None = None - #: The control type this CEM asked the resource manager to select, and the id of that message, until confirmed. + #: The control type this CEM asked the resource manager to select, and the id of that request. Reset once confirmed. self._selecting: tuple[ControlType, str] | None = None self._default_control_type = default_control_type @@ -163,11 +166,11 @@ def __init__( self._datastore = datastore if datastore is not None else {} def _is_timer_due(self, name: str) -> bool: - # By this handler's clock, so that a co-simulation's timers run on simulated time. + # Use this handler's clock, so that a co-simulation's timers run on simulated time. now = self.now() due_time = self._timers.get(name, now - self._minimum_measurement_period) if due_time.tzinfo is None: - # A timer handed in without a zone was compared with the host's local time, so it is read as that, as before. + # A timer without a timezone used to be compared with the host's local time, so keep reading it as local time. due_time = due_time.astimezone(self._timezone) if due_time <= now: # Get total seconds of the period @@ -185,8 +188,8 @@ def _is_timer_due(self, name: str) -> bool: next_due = now.replace(minute=0, second=0, microsecond=0) + timedelta( seconds=next_tick_seconds ) - # On a period boundary the ceiling is the moment itself, - # and a clock that stands still, as a simulated one does between messages, then found the timer due on every call. + # On a period boundary, the ceiling equals the current moment. + # A clock that stands still (as a simulated clock does between messages) would then find the timer due on every call. if next_due <= now: next_due += self._minimum_measurement_period self._timers[name] = next_due @@ -231,7 +234,7 @@ def register_control_type(self, control_type_handler: ControlTypeHandler): # add fm_client to control_type handler control_type_handler._fm_client = self._fm_client - # The handler goes by the CEM's time, which a co-simulation sets. + # The handler uses the CEM's clock, which a co-simulation sets. control_type_handler._clock = self._clock # add sending queue @@ -261,7 +264,7 @@ async def handle_message(self, message: Dict | pydantic.BaseModel | str): message = json.loads(message) assert isinstance( message, dict - ) # what a message is from here on, whatever form it came in + ) # from here on, the message is a dict, whatever form it came in if "metadata" in message and "message" in message: unwrapped = self._unwrap(message) @@ -273,8 +276,8 @@ async def handle_message(self, message: Dict | pydantic.BaseModel | str): self._activate_implicitly(message) # try to handle the message with the control_type handle - # A control type without a handler yet, as while a resource manager's site is being configured, - # leaves its messages unhandled, so they are answered with a TEMPORARY_ERROR for the resource manager to retry. + # A control type may have no handler yet, for example while the resource manager's site is being configured. + # Its messages are then left unhandled, so the CEM responds with a TEMPORARY_ERROR, and the resource manager retries. handler = self._control_types_handlers.get(self._control_type) if handler is None and self._control_type is not None: self._ensure_configuring() @@ -308,10 +311,11 @@ async def handle_message(self, message: Dict | pydantic.BaseModel | str): await self._sending_queue.put(response) def _unwrap(self, wrapped: dict) -> dict | None: - """The S2 message inside a co-simulation's wrapper, with the clock set to the simulated time it carries. + """Return the S2 message inside a co-simulation's wrapper, and set the clock to the wrapper's simulated time. - Where this CEM does not accept the wrapper, the time cannot be read, or there is no message inside, - the message is refused, the clock is left as it was, and None is returned so that nothing is handled. + The message is refused in three cases: + this CEM does not accept the wrapper, the time cannot be read, or there is no message inside. + Then the clock is left unchanged, and None is returned, so that nothing is handled. """ inner = wrapped["message"] reason = None @@ -345,15 +349,16 @@ def _unwrap(self, wrapped: dict) -> dict | None: return None def _activate_implicitly(self, message: dict) -> None: - """Take a message of the control type this CEM is selecting as the resource manager's acceptance of it. - - A resource manager sends a control type's messages only once that control type is active on its side. - So where one arrives for the control type this CEM asked for, while the CEM still waits for the confirmation, - the resource manager is working in it already. #188 saw the CEM stay in NO_SELECTION while a resource manager, - reconnecting during setup in a co-simulation, kept sending FRBC messages, each answered with a TEMPORARY_ERROR - and retried; whether the confirmation was lost or never sent was not established. - Only the control type being selected is activated, never another one a message happens to belong to, - and its pending confirmation is then no longer waited for. + """Treat a message of the control type being selected as the resource manager's acceptance of that control type. + + A resource manager only sends a control type's messages once that control type is active on its side. + Suppose such a message arrives for the control type this CEM asked for, while the CEM still waits for the confirmation. + Then the resource manager is already working in that control type. + In #188, a resource manager reconnected during setup in a co-simulation, and kept sending FRBC messages. + The CEM stayed in NO_SELECTION, responded to each message with a TEMPORARY_ERROR, and the resource manager retried. + It was not established whether the confirmation was lost or never sent. + This method only activates the control type being selected, never another control type that a message happens to belong to. + After that, the CEM no longer waits for the pending confirmation. """ if self._control_type != ControlType.NO_SELECTION or self._selecting is None: return @@ -481,12 +486,12 @@ async def handle_resource_manager_details(self, message: ResourceManagerDetails) return get_reception_status(message) def _ensure_configuring(self) -> None: - """Set the resource manager's site up in the background, unless that is done or under way. + """Set up the resource manager's site in the background, unless that is already done or under way. - In the background, since a site takes a few requests to FlexMeasures to set up, - and the resource manager waits for a reception status meanwhile. - A setup that failed is tried again on the next message that needs it, - while the resource manager's messages are answered with TEMPORARY_ERROR, which it retries. + Setting up a site takes a few requests to FlexMeasures, and meanwhile the resource manager waits for a reception status. + That is why this runs in the background. + If the setup fails, it is tried again on the next message that needs it. + Meanwhile, the CEM responds to the resource manager's messages with TEMPORARY_ERROR, and the resource manager retries them. """ if ( self._site_configurator is None @@ -523,7 +528,7 @@ async def _configure_site(self, details: ResourceManagerDetails) -> None: async def handle_power_measurement(self, message: PowerMeasurement): if self._site_configurator is not None and not self._control_types_handlers: - # Its site is still being set up, so there is nowhere to record it yet: the resource manager is asked to resend it. + # The site is still being set up, so there is no sensor to record on yet. Ask the resource manager to resend the message. self._ensure_configuring() return get_reception_status(message, ReceptionStatusValues.TEMPORARY_ERROR) @@ -531,8 +536,8 @@ async def handle_power_measurement(self, message: PowerMeasurement): for power_measurement in message.values: commodity_quantity = power_measurement.commodity_quantity.value - # Without a sensor for it, a measurement is not recorded. - # It used to go to sensor 357 for ELECTRIC.POWER.L1, a sensor of whichever site on the server had that id. + # A measurement without a sensor is not recorded. + # An ELECTRIC.POWER.L1 measurement used to go to sensor 357, which could belong to any site on the server. sensor_id = (self.power_sensor_id or {}).get(commodity_quantity) if sensor_id is None: self._logger.debug( @@ -544,7 +549,8 @@ async def handle_power_measurement(self, message: PowerMeasurement): continue taken.append((sensor_id, power_measurement.value)) - # Every sensor is made ready before any value is taken, so that a message asked for again is not taken in part twice. + # Get every sensor ready before taking any value. + # Otherwise, part of a message could be taken twice, if the resource manager is asked to send it again. try: for sensor_id, _ in taken: await self._measurement_recorder.resolution(sensor_id) diff --git a/src/flexmeasures_client/s2/clock.py b/src/flexmeasures_client/s2/clock.py index 55646905..19aa6045 100644 --- a/src/flexmeasures_client/s2/clock.py +++ b/src/flexmeasures_client/s2/clock.py @@ -1,9 +1,9 @@ -"""The time an S2 CEM and its handlers go by. +"""Clocks that tell an S2 CEM and its handlers what time it is. -A CEM that runs live goes by the wall clock. -One that steers a co-simulation goes by the time the simulation says it is, -which a resource manager sends along with its messages (see CEM's accept_simulation_metadata), -so that what the CEM records and plans is placed at simulated moments rather than at the moment it ran. +A CEM that runs live uses the wall clock. +A CEM that steers a co-simulation uses the simulated time instead. +The resource manager sends that time along with its messages (see the CEM's accept_simulation_metadata). +This way, the CEM records and plans at simulated moments, not at the moments the code actually ran. """ from __future__ import annotations @@ -14,24 +14,26 @@ class Clock(Protocol): def now(self) -> datetime: - """The current moment, timezone-aware.""" + """Return the current moment, timezone-aware.""" ... class WallClock: - """The time it is.""" + """A clock that returns the current wall-clock time, in UTC.""" def now(self) -> datetime: return datetime.now(timezone.utc) class SimulatedClock: - """The time a simulation says it is, as last set. - - It stands still between settings, as a simulation's time does between the messages that carry it. - Until it is first set it goes by the wall clock, for a resource manager that does not wrap its first messages; - one that wraps every message from its handshake on, as the co-simulation's does, sets it with the first. - Background work, such as a schedule being planned, reads it as it stands, which may be a later message's time. + """A clock that returns the simulated time it was last set to. + + The clock stands still between settings, just as simulated time stands still between the messages that carry it. + Until it is first set, it returns the wall-clock time. + This suits a resource manager that does not wrap its first messages. + A resource manager that wraps every message from its handshake on (as the co-simulation's does) sets the clock with its first message. + Background work, such as planning a schedule, reads the clock at the moment it runs. + By then, a later message may have set it to a later time. """ def __init__(self, start: datetime | None = None) -> None: diff --git a/src/flexmeasures_client/s2/control_types/FRBC/__init__.py b/src/flexmeasures_client/s2/control_types/FRBC/__init__.py index 39ef719e..70857762 100644 --- a/src/flexmeasures_client/s2/control_types/FRBC/__init__.py +++ b/src/flexmeasures_client/s2/control_types/FRBC/__init__.py @@ -55,6 +55,8 @@ class FRBC(ControlTypeHandler): _timer_status_history: SizeLimitOrderedDict[str, FRBCTimerStatus] _actuator_status_history: SizeLimitOrderedDict[str, FRBCActuatorStatus] + #: Per actuator, the latest status that arrived before any system description, to be handled once one arrives. + _statuses_awaiting_a_description: dict[str, FRBCActuatorStatus] _storage_status_history: SizeLimitOrderedDict[str, FRBCStorageStatus] background_tasks: set _sent_instructions: dict[str, FRBCInstruction] @@ -75,6 +77,7 @@ def __init__(self, max_size: int = 100) -> None: self._timer_status_history = SizeLimitOrderedDict(max_size=max_size) self._actuator_status_history = SizeLimitOrderedDict(max_size=max_size) + self._statuses_awaiting_a_description = {} self._storage_status_history = SizeLimitOrderedDict(max_size=max_size) self._system_description_history = SizeLimitOrderedDict(max_size=max_size) @@ -209,14 +212,27 @@ def system_description_fingerprint(message: FRBCSystemDescription) -> int: def handle_system_description( self, message: FRBCSystemDescription ) -> pydantic.BaseModel: - # A resend is acknowledged, and otherwise ignored: storing it again would push a - # description out of a bounded history for no reason, and a handler that did act on a - # description would act on it twice. - # Acknowledging it still matters, or the resource manager retries. - # Ignoring a resend is safe now that planning hangs off storage statuses rather than - # descriptions, and not before: it would otherwise have dropped a planning request. - # The TUNES handler has fingerprinted its descriptions this way all along, and can adopt - # this shared helper separately. + # A resend is acknowledged, and otherwise ignored. + # Storing it again would push an older description out of the bounded history for no reason. + # Also, a handler that acts on a description would act on it twice. + # The acknowledgement still matters, because without it the resource manager retries. + # Ignoring a resend is safe because planning is normally triggered by storage statuses, and a resend never differs from the previous description. + # Before that was the case, ignoring a resend would have dropped a planning request. + # The TUNES handler has always fingerprinted its descriptions this way. + # It can adopt this shared helper separately. + # Every plan is made for exactly one actuator. + # So a description with several actuators is refused on arrival. + # Otherwise it would be acknowledged, and then fail in the background on every round, after a solve. + if len(message.actuators) != 1: + self._logger.error( + "Refusing an FRBC.SystemDescription with %s actuators: this CEM plans for exactly one.", + len(message.actuators), + ) + return get_reception_status( + message, + status=ReceptionStatusValues.INVALID_CONTENT, + diagnostic_label=f"This CEM plans for exactly one actuator, and the description has {len(message.actuators)}.", + ) fingerprint = self.system_description_fingerprint(message) if fingerprint == self._last_system_description_fingerprint: self._logger.debug( @@ -230,14 +246,35 @@ def handle_system_description( # store system_description message for later self._system_description_history[system_description_id] = message - # A description says what the device can do, and a storage status what it holds now, - # and a schedule needs both. Planning normally hangs off a storage status (see handle_storage_status), - # but a description that arrives after the latest status, and differs from the description before it, - # is answered with a schedule against that latest status. - # Anything but the message id counts as a difference, its validity included, - # so a resource manager that re-dates its description on every resend asks for a schedule every time. - # Otherwise a resource manager that sends its status first, or whose fill level holds steady - # so that no new status follows, is never planned for, or keeps being planned for with what it can no longer do. + # A status that arrived before any description could not be handled then, since only a description says what its operation mode stands for. + # A status describes the actuator's present state, so the latest per actuator is handled now. + awaiting = list(self._statuses_awaiting_a_description.values()) + self._statuses_awaiting_a_description.clear() + for status in awaiting: + task = asyncio.create_task(self.send_actuator_status(status)) + self.background_tasks.add(task) + task.add_done_callback(self.background_tasks.discard) + + # Warn once per description, because nothing else here would mention it. + # An instruction can ask for a switch that a running timer or a missing transition forbids, and then the device refuses it. + actuator = message.actuators[0] + if _transitions_restrict(actuator): + self._logger.warning( + "The actuator declares %s timers and %s transitions that rule out some switches between its operation modes, which this CEM does not honour: an instruction may ask for a switch the device refuses.", + len(actuator.timers), + len(actuator.transitions), + ) + + # A description says what the device can do, and a storage status says what it holds now. + # A schedule needs both. + # Planning is normally triggered by a storage status (see handle_storage_status). + # But suppose a description arrives after the latest status, and differs from the previous description. + # Then the CEM plans a schedule from that latest status. + # Any difference other than the message id counts, including the validity period. + # So a resource manager that re-dates its description on every resend triggers a schedule every time. + # Without this, two kinds of resource managers would be served badly. + # One that sends its status first would never be planned for. + # One whose fill level holds steady sends no new status, so it would keep being planned for with capabilities it no longer has. if self._storage_status_history: request_number = self._count_request(STORAGE_STATUS_REQUESTS) task = asyncio.create_task( @@ -332,6 +369,18 @@ def handle_fill_level_target_profile( @register(FRBCTimerStatus) def handle_frbc_timer_status(self, message: FRBCTimerStatus) -> pydantic.BaseModel: + # Store the timer status, so that it can be inspected, even though no planning uses it yet. + # Warn only once per timer, because a resource manager sends a status every time any timer runs. + warned = any( + known.timer_id == message.timer_id + for known in self._timer_status_history.values() + ) + self._timer_status_history[str(message.message_id)] = message + (self._logger.debug if warned else self._logger.warning)( + "Timer %s runs until %s, which this CEM does not honour: an instruction may ask for a switch it blocks.", + message.timer_id, + message.finished_at, + ) return get_reception_status(message, status=ReceptionStatusValues.OK) async def send_storage_status( @@ -392,14 +441,46 @@ async def trigger_schedule( actuator = system_description.actuators[0] + # Import here, because the utilities module imports from this package. + from flexmeasures_client.s2.control_types.FRBC.utils import ( + normal_operation_modes, + ) + instruction = FRBCInstruction( message_id=get_unique_id(), id=get_unique_id(), actuator_id=actuator.id, - operation_mode=actuator.operation_modes[0].id, + operation_mode=normal_operation_modes(actuator)[0].id, operation_mode_factor=0.5, execution_time=system_description.valid_from, abnormal_condition=False, ) await self.send_message(instruction) + + +def _transitions_restrict(actuator) -> bool: + """Return whether an actuator's timers and transitions rule out a switch between two of its operation modes. + + Some transitions rule out nothing. + That is the case when they allow every switch between the modes usable under normal conditions, and there is no timer to wait for. + An example is an on/off device that declares both of its switches. + This CEM does not honour timers and transitions, but that only matters if they rule something out. + """ + # A transition can only be blocked by a timer that the actuator declares (s2python checks this). + # So checking for timers covers blocked transitions too. + if actuator.timers: + return True + if not actuator.transitions: + return False + modes = [ + str(mode.id) + for mode in actuator.operation_modes + if not mode.abnormal_condition_only + ] + allowed = { + (str(t.from_), str(t.to)) + for t in actuator.transitions + if not t.abnormal_condition_only + } + return any((a, b) not in allowed for a in modes for b in modes if a != b) diff --git a/src/flexmeasures_client/s2/control_types/FRBC/frbc_simple.py b/src/flexmeasures_client/s2/control_types/FRBC/frbc_simple.py index ffa491a6..1331a499 100644 --- a/src/flexmeasures_client/s2/control_types/FRBC/frbc_simple.py +++ b/src/flexmeasures_client/s2/control_types/FRBC/frbc_simple.py @@ -25,16 +25,23 @@ import pandas as pd +from aiohttp import ClientError, ClientResponseError from flexmeasures_client.client import convert_units -from flexmeasures_client.exceptions import ContentTypeError, JobFailedError +from flexmeasures_client.exceptions import ( + ContentTypeError, + JobFailedError, + JobTimeoutError, +) from flexmeasures_client.s2.control_types.FRBC import FRBC, STORAGE_STATUS_REQUESTS from flexmeasures_client.s2.control_types.FRBC.utils import ( WATTS_PER_ENERGY_UNIT_PER_SECOND, + WATTS_PER_FILL_RATE_UNIT, clip_fill_level_target_profile, element_for_fill_level, fm_schedule_to_instructions, get_soc_min_max, + normal_operation_modes, om_label, operation_mode_conversion_efficiency, ) @@ -44,60 +51,114 @@ translate_usage_forecast_to_fm, ) -#: The unit S2 states a power value in, and so the unit of an operation mode's power ranges. -#: A schedule is read back in this unit, whatever unit the power sensor holding it is recorded in, -#: and what goes out to FlexMeasures in it is what came in from a resource manager in it. +#: The unit of power values in S2, and so the unit of an operation mode's power ranges. +#: A schedule is read back in this unit, whatever unit its power sensor records in. +#: Values sent to FlexMeasures in this unit are passed on as they came in from the resource manager. S2_POWER_UNIT = "W" -#: How many times a schedule is asked for before the storage is left without an instruction for its window. -#: Every attempt costs the server a solve, so this is a handful rather than a vigil. +#: How many times to request a schedule before leaving the storage without an instruction for its window. +#: Every attempt costs the server a solve, so this number is kept small. SCHEDULE_ATTEMPTS = 5 #: How long to wait between two attempts at a schedule. -#: Flat rather than backed off: what it waits out is a post landing or a worker taking the job up, -#: and a resource manager is waiting on an instruction for the whole of it. +#: The wait is fixed, without backoff. +#: It only needs to cover a post landing or a worker picking up the job, and meanwhile a resource manager is waiting for an instruction. SCHEDULE_RETRY_INTERVAL = timedelta(seconds=2) -#: Causes of a failed scheduling job that fail the same way however often the job is asked for again. -#: Asking again only multiplies failed jobs over the server's workers, so these are never asked again. +#: Causes of a failed scheduling job that fail the same way, however often the job is requested again. +#: Requesting again only multiplies failed jobs across the server's workers, so these are never retried. NON_RETRYABLE_SCHEDULE_FAILURES = ( - # A scheduler that records a belief twice collides with its own first write on every attempt. + # A scheduler that records a belief twice collides with its own first write, on every attempt. "UniqueViolation", "duplicate key value violates unique constraint", # An infeasible problem is planned from the same flex model and context every time, so it stays infeasible. - # FlexMeasures answered one with a fallback scheduler until v1.0.0 retired it, and fails the job itself since. + # Until v1.0.0, FlexMeasures handled one with a fallback scheduler. + # Since v1.0.0 retired that fallback, the job itself fails. "InfeasibleProblemException", ) -#: The refusal of a trigger, rather than a failure of the job behind it, that is worth asking again about. -#: A schedule can be asked for before the state of charge it is to be planned from has landed, -#: and FlexMeasures refuses to plan from one it considers too old. +#: Refusals of a trigger (not failures of the job behind it) that are worth retrying. +#: A schedule can be requested before the state of charge to plan from has landed. +#: FlexMeasures refuses to plan from a state of charge that it considers too old. RETRYABLE_TRIGGER_REFUSALS = ("No recent state-of-charge value found",) +#: The exceptions a request to FlexMeasures can raise when there is no defect in this code. +#: The request may be refused (ValueError), the server unreachable (ConnectionError, TimeoutError, ClientError), +#: or the response unexpected (ContentTypeError). +#: A queued job may also fail, or never be picked up (JobFailedError, JobTimeoutError). +REQUEST_FAILURES = ( + JobFailedError, + JobTimeoutError, + ValueError, + ConnectionError, + TimeoutError, + ContentTypeError, + ClientError, +) + +#: How FlexMeasures before v0.33.0 reports a failed scheduling job, since it has no job-status API. +#: It responds with an HTTP 400, which the client raises as a ValueError with only this message. +BARE_REFUSAL = "Request failed with status code 400" + + +def _merged(ranges: list[tuple[float, float]]) -> list[tuple[float, float]]: + """Return the union of closed ranges, as the fewest ranges that cover it, in order.""" + merged: list[tuple[float, float]] = [] + for lower, upper in sorted(ranges): + if merged and lower <= merged[-1][1]: + merged[-1] = (merged[-1][0], max(merged[-1][1], upper)) + else: + merged.append((lower, upper)) + return merged + + +def _without_negative_usage(usage_forecast): + """Return a usage forecast in which each negative expected rate is replaced by zero. + + This is how a usage forecast is recorded and planned against. + """ + if usage_forecast is None or all( + element.usage_rate_expected >= 0 for element in usage_forecast.elements + ): + return usage_forecast + return usage_forecast.model_copy( + update={ + "elements": [ + element.model_copy( + update={ + "usage_rate_expected": max(0.0, element.usage_rate_expected) + } + ) + for element in usage_forecast.elements + ] + } + ) + def _operation_mode_power_bands( power_ranges: list[tuple[float, float]], power_unit: str ) -> list[dict[str, list[str]]]: - """Turn the power ranges an actuator states into the bands FlexMeasures plans within. - - A device with operation modes may only run inside one of its bands, - so an on/off heater is asked for as {0 W} and {883.7 W} rather than as everything in between, - which the resource manager would have to round into a full-on block, - overshooting whatever capacity the site is held to. - - S2 states one signed range, where positive means consumption, - while FlexMeasures asks for the two directions separately, each non-negative. - So a range is split at zero: its non-negative part is the consumption range, - its negative part is the production range with the sign flipped, - and a range spanning zero becomes both, each starting at zero. - - :param power_ranges: lower and upper bound of each range the actuator states, signed as S2 states them. - :param power_unit: the unit those bounds are in. - :returns: one band per distinct range, in the order the actuator stated them. + """Turn the power ranges an actuator declares into the bands FlexMeasures plans within. + + A device with operation modes may only run inside one of its bands. + So an on/off heater is requested as {0 W} and {883.7 W}, not as everything in between. + Otherwise, the resource manager would have to round a value in between into a full-on block. + That block would overshoot any capacity limit the site is held to. + + S2 gives one signed range, where positive means consumption. + FlexMeasures expects the two directions separately, each non-negative. + So a range is split at zero. + Its non-negative part becomes the consumption range. + Its negative part, with the sign flipped, becomes the production range. + A range that spans zero becomes both, each starting at zero. + + :param power_ranges: lower and upper bound of each range the actuator declares, signed as in S2. + :param power_unit: the unit of those bounds. + :returns: one band per distinct range, in the order the actuator declared them. """ - # Zero is the first argument of each max() below, so that a bound of exactly zero is stated as zero, - # rather than as the negative zero that flipping a sign leaves behind. + # Zero is the first argument of each max() below, so that a bound of exactly zero is written as zero, + # not as the negative zero that flipping a sign leaves behind. bands: list[dict[str, list[str]]] = [] for lower, upper in power_ranges: band: dict[str, list[str]] = {} @@ -112,7 +173,7 @@ def _operation_mode_power_bands( f"{-lower} {power_unit}", ] if not band: - # A range of exactly {0}: the mode in which the actuator draws nothing either way. + # A range of exactly {0}: the mode in which the actuator neither consumes nor produces. band["consumption-range"] = [f"0 {power_unit}", f"0 {power_unit}"] if band not in bands: bands.append(band) @@ -127,14 +188,14 @@ class FRBCSimple(FRBC): _soc_sensor_id: int _rm_discharge_sensor_id: int _fill_rate_sensor_id: int | None - #: Where what a resource manager declares about its storage is recorded, each optional: - #: without one, that declaration is not recorded, rather than being posted somewhere else. + #: The sensors to record the resource manager's declarations about its storage on. Each one is optional. + #: Without a sensor, that declaration is not recorded at all, instead of being posted somewhere else. _soc_minima_sensor_id: int | None _soc_maxima_sensor_id: int | None _usage_forecast_sensor_id: int | None _leakage_behaviour_sensor_id: int | None - #: Where what the operation modes say about converting drawn power into stored energy is recorded, - #: for the scheduler to plan the storage's conversion with. + #: The sensor to record the operation modes' conversion of drawn power into stored energy on. + #: The scheduler uses it to plan the storage's conversion. _charging_efficiency_sensor_id: int | None _resolution: timedelta _schedule_duration: timedelta @@ -142,16 +203,16 @@ class FRBCSimple(FRBC): _fill_level_scale: float _hard_bound_factor: float #: Whether the power sensor records consumption as positive, as S2 does. - #: Read from the sensor itself the first time a power value is posted, and remembered after that. - #: None until it has been read. + #: This is read from the sensor itself the first time a power value is posted, and remembered after that. + #: It is None until it has been read. _power_sensor_takes_consumption_as_positive: bool | None - #: When a power value was last recorded for the actuator, and for which moment, - #: so that a second status inside one event does not collide with the first. + #: For each moment, when a power value was last recorded for the actuator. + #: This prevents a second status within one event from colliding with the first. _power_recorded_at: dict[datetime, datetime] - #: The charging efficiency last recorded, and the window it was recorded for, - #: so that the same value for the same window is not recorded again on every schedule. + #: The last recorded charging efficiency, and the window it was recorded for. + #: This prevents recording the same value for the same window again on every schedule. _charging_efficiency_recorded: tuple[datetime, float] | None - #: How many times a schedule is asked for, and how long between two attempts. + #: How many times to request a schedule, and how long to wait between two attempts. _schedule_attempts: int _schedule_retry_interval: timedelta @@ -161,56 +222,60 @@ def __init__( soc_sensor_id: int, rm_discharge_sensor_id: int, price_sensor_id: int, - # Where feeding power back is priced differently from drawing it, as a feed-in tariff is. - # Without one, both directions are priced by price_sensor_id. + # Use this if feeding power back is priced differently from drawing it, as with a feed-in tariff. + # Without it, both directions are priced by price_sensor_id. production_price_sensor_id: int | None = None, # Whether the site's connection capacity may be breached at a price (True) or never (False). - # Unset, it is left to FlexMeasures, which since v1.0 relaxes it at a high default breach price, - # under its relax-constraints umbrella, and before that did not. - # A home whose connection is a physical limit can pass False; - # a site whose capacity is a target set by others, as in a community steering its members, can pass True. + # If unset, FlexMeasures decides. + # Since v1.0, FlexMeasures relaxes it at a high default breach price, under its relax-constraints umbrella. + # Before v1.0, it did not relax it. + # A home whose connection is a physical limit can pass False. + # A site whose capacity is a target set by others, as in a community that steers its members, can pass True. relax_site_capacity: bool | None = None, - # Optional so existing callers keep working; when absent the actuator's fill rate is - # simply not recorded, rather than being posted onto the power sensor. + # Optional, so that existing callers keep working. + # Without it, the actuator's fill rate is not recorded at all, instead of being posted onto the power sensor. fill_rate_sensor_id: int | None = None, - # Likewise optional, for the three things a resource manager declares about its storage. + # Also optional: the sensors for three declarations a resource manager makes about its storage. soc_minima_sensor_id: int | None = None, soc_maxima_sensor_id: int | None = None, usage_forecast_sensor_id: int | None = None, - # Its resolution has to be the handler's resolution: a storage efficiency is what remains after one step, - # computed here for a step of the handler's resolution, - # and FlexMeasures reads it as one step of the sensor's own resolution. + # The resolution of this sensor must equal the handler's resolution. + # A storage efficiency is what remains after one step, and this handler computes it for a step of its own resolution. + # FlexMeasures reads it as one step of the sensor's resolution. leakage_behaviour_sensor_id: int | None = None, - # Likewise optional: without a sensor to record it on, no charging efficiency is declared, - # and the scheduler plans the storage as if the actuator converted power into stored energy one for one. + # Also optional. + # Without a sensor to record it on, no charging efficiency is declared. + # Then the scheduler plans the storage as if the actuator converted power into stored energy one to one. charging_efficiency_sensor_id: int | None = None, timezone: str = "UTC", schedule_duration: timedelta = timedelta(hours=12), - # How many times a schedule is asked for before the storage is left without an instruction, - # and how long between two attempts. - # One attempt means a failure is never answered by another, as it was before this was bounded at all. + # How many times to request a schedule before leaving the storage without an instruction, + # and how long to wait between two attempts. + # One attempt means a failure is never retried, + # which is how this handler behaved before retries were added. schedule_attempts: int = SCHEDULE_ATTEMPTS, schedule_retry_interval: timedelta = SCHEDULE_RETRY_INTERVAL, - # The resolution a schedule is planned on, which a schedule's start has to line up with. + # The resolution that schedules are planned on; a schedule's start must line up with it. resolution: timedelta = timedelta(minutes=15), max_size: int = 100, valid_from_shift: timedelta = timedelta(days=1), fill_level_scale: float = 1, - # How far above the declared fill-level range the hard upper bound of a schedule sits. - # Half again as high, so that a storage overshooting its declared range stays feasible. - # A deployment whose storage cannot overshoot at all can bring this down to 1. + # How far above the declared fill-level range a schedule's hard upper bound sits. + # The default is 1.5 times as high, so that a storage that overshoots its declared range stays feasible. + # A deployment whose storage cannot overshoot at all can lower this to 1. hard_bound_factor: float = 1.5, - # The unit a power value this handler posts is labelled with. - # Those values come straight from the operation modes' power ranges, which S2 states in watts, - # and nothing converts them on the way out, - # so a deployment that sets another unit here has them recorded at the wrong magnitude. - # TODO: either convert a power value into this unit before posting it, or drop this and post watts, as S2 states them. + # The unit that this handler converts posted power values into, and labels them with. + # These values come from the operation modes' power ranges, which S2 gives in watts. power_unit: str = "W", - # The unit a fill level is recorded in: a state of charge and the minima and maxima - # bounding it all go out in this one unit, so that they can be compared at all. - # Kilowatt hours, because S2 is a residential protocol and a home battery or heat buffer - # is of that order; a deployment at another scale says so here. + # The unit that fill levels are recorded in. + # A state of charge, and the minima and maxima that bound it, are all sent in this one unit, so that they can be compared. + # The default is kilowatt-hours, because S2 is a residential protocol, and a home battery or heat buffer is of that order. + # A deployment at another scale can set a different unit here. energy_unit: str = "kWh", + # The unit that rates of fill level are recorded in: the fill rate and the usage forecast. + # S2 states them per second, which in kWh/s makes tiny numbers, so they are recorded in a power unit: W, kW or MW. + # The default is watts, in which a residential heat buffer's rates read as such. + fill_rate_unit: str = "W", ) -> None: super().__init__(max_size) self._power_sensor_id = power_sensor_id @@ -238,14 +303,14 @@ def __init__( f"resolution and schedule_duration have to be positive, but {resolution} and {schedule_duration} were given." ) if schedule_duration % resolution: - # FlexMeasures answers with whole steps of the power sensor's resolution only, - # so a duration between two whole numbers of steps comes back with fewer values than it spans, - # and the schedule's step, read as its duration over its number of values, would come out wrong. + # FlexMeasures only returns whole steps of the power sensor's resolution. + # So a duration that is not a whole number of steps comes back with fewer values than it spans. + # The schedule's step is computed as its duration divided by its number of values, so it would come out wrong. raise ValueError( f"schedule_duration has to be a whole number of resolution steps, but {schedule_duration} is not a multiple of {resolution}." ) if schedule_attempts < 1: - # A schedule is asked for at least once, or no schedule is asked for at all. + # A schedule must be requested at least once, or there is no schedule at all. raise ValueError( f"schedule_attempts has to be at least one, since it counts the attempts at a schedule rather than the retries, but {schedule_attempts} was given." ) @@ -253,9 +318,9 @@ def __init__( self._schedule_retry_interval = schedule_retry_interval self._timezone = ZoneInfo(timezone) if fill_level_scale <= 0: - # Every fill level this handler records is multiplied by the scale, - # and the hard bounds a schedule was planned against are divided back by it, - # so zero divides by zero and a negative scale turns the bounds upside down. + # Every fill level this handler records is multiplied by the scale. + # The hard bounds that a schedule was planned against are divided by it again. + # So a scale of zero would divide by zero, and a negative scale would turn the bounds upside down. raise ValueError( f"fill_level_scale has to be greater than zero, since it converts a fill level into the unit it is recorded in, but {fill_level_scale} was given." ) @@ -265,39 +330,49 @@ def __init__( self._soc_sensor_records_instants: bool | None = None self._planning = asyncio.Lock() if hard_bound_factor < 1: - # Below one, the hard upper bound sits inside the declared range, - # and FlexMeasures clips the band into the hard bounds, - # so the band would be narrowed to something the resource manager never asked for. + # Below one, the hard upper bound sits inside the declared range. + # FlexMeasures clips the band to the hard bounds. + # So the band would be narrowed to something the resource manager never asked for. raise ValueError( f"hard_bound_factor has to be at least 1, so that the hard upper bound sits at or above the declared fill-level range, but {hard_bound_factor} was given." ) self._hard_bound_factor = hard_bound_factor self.power_unit = power_unit if energy_unit not in WATTS_PER_ENERGY_UNIT_PER_SECOND: - # A fill rate is per second of this unit, and has to be compared with the watts S2 states a power in, - # so a unit that cannot be converted to watts cannot be planned with. + # A fill rate is in this unit per second, and must be compared with power values, which S2 gives in watts. + # So a unit that cannot be converted to watts cannot be planned with. raise ValueError( f"energy_unit has to be one of {', '.join(WATTS_PER_ENERGY_UNIT_PER_SECOND)}, but {energy_unit!r} was given." ) self.energy_unit = energy_unit + if fill_rate_unit not in WATTS_PER_FILL_RATE_UNIT: + raise ValueError( + f"fill_rate_unit has to be one of {', '.join(WATTS_PER_FILL_RATE_UNIT)}, but {fill_rate_unit!r} was given." + ) + self._fill_rate_unit = fill_rate_unit # delay the start of the schedule from the time `valid_from` # of the FRBC.SystemDescription. self._valid_from_shift = valid_from_shift def fill_rate_unit(self) -> str: - """The unit a rate of change of the fill level is recorded in. + """The unit a rate of change of the fill level is recorded in: the fill_rate_unit given, by default W. - S2 states a fill rate, and a usage forecast, as fill level per second, so this follows - whatever unit the fill level itself is recorded in. It is not the power unit: a fill rate - is what an actuator delivers into its buffer, which for a heat buffer is not electricity - at all, and which even for a battery differs from kilowatts by the seconds in an hour. + S2 states a fill rate and a usage forecast as fill level per second. + Per second, a residential heat buffer's rates are tiny numbers in kWh/s, so they are recorded in a power unit instead (see _per_second_in_fill_rate_unit). + It is not the power unit: a fill rate is what an actuator delivers into its buffer, which for a heat buffer is not electricity at all. - A method rather than a property, because S2 Handler's discover() walks dir(self) and - getattrs every name during __init__, before a subclass has assigned its own attributes. + A method rather than a property, because S2 Handler's discover() walks dir(self) and getattrs every name during __init__, before a subclass has assigned its own attributes. A property would be evaluated there and raise on the energy_unit it reads. """ - return f"{self.energy_unit}/s" + return self._fill_rate_unit + + def _per_second_in_fill_rate_unit(self, rate): + """Return a rate of fill level per second, as S2 states it, in the fill-rate unit; a number or a series.""" + return rate * ( + WATTS_PER_ENERGY_UNIT_PER_SECOND[self.energy_unit] + / WATTS_PER_FILL_RATE_UNIT[self._fill_rate_unit] + ) def align_to_resolution(self, moment: datetime) -> datetime: """The last resolution boundary at or before that moment. @@ -313,13 +388,14 @@ def align_to_resolution(self, moment: datetime) -> datetime: resolution_seconds = int(self._resolution.total_seconds()) if resolution_seconds <= 0: return moment - # Floored on the moment's own clock, in the offset it carries, which is how FlexMeasures floors a series posted with it, - # so an hourly step in a zone half an hour off UTC, or a daily one, starts where FlexMeasures' steps do. - # A boundary in the repeated hour of an autumn DST change exists twice, and the one meant is the later of the two - # that is not after the moment: taking the first pass every time put a boundary an hour or more too early. - # A boundary in a spring gap, which only a resolution of more than an hour can land on, moves to the end of the gap. - # Compared as pandas timestamps, which compare instants: two datetimes on one local clock compare by wall time, - # whichever pass each is on. + # Floor in the moment's own timezone offset, which is how FlexMeasures floors a series posted with that offset. + # So an hourly step in a timezone that is half an hour off UTC, or a daily step, starts where FlexMeasures' steps start. + # During the repeated hour of an autumn DST change, a boundary occurs twice. + # The intended boundary is the later of the two that is not after the moment. + # Always taking the first pass put the boundary an hour or more too early. + # A boundary in a spring DST gap (which only a resolution of more than an hour can hit) moves to the end of the gap. + # Compare as pandas timestamps, which compare instants. + # Two datetimes in the same local timezone compare by wall time, whichever pass each is on. at = pd.Timestamp(moment) candidates = [ at.floor( @@ -331,15 +407,16 @@ def align_to_resolution(self, moment: datetime) -> datetime: return boundary.to_pydatetime() def schedule_start(self, valid_from: datetime) -> datetime: - """Where a schedule asked for at valid_from starts: shifted and floored to a boundary, but never before now. - - Never before now, since planning happens on every storage status while a description may be days old: - anchored to that description alone, every status in a long session was answered with the same window, - which slid into the past, so its instructions were all due already and arrived as one burst. - Where now is past that boundary, the schedule starts at the next one, not at the start of the step now is in. - A schedule is planned from the fill level reported now, and FlexMeasures takes that to be the level at its start, - so a schedule starting at a step already under way was planned as if the whole step were left, - and every replan within it asked for less power over what remained of it, delivering less than planned. + """Return the start of a schedule requested for valid_from: shifted and floored to a boundary, but never before now. + + Planning happens on every storage status, while a description may be days old. + If the start were based on that description alone, every status in a long session would get the same window. + That window would slide into the past, so all its instructions would already be due, and arrive as one burst. + Hence the start is never before now. + If now is past that boundary, the schedule starts at the next boundary, not at the start of the current step. + A schedule is planned from the fill level reported now, and FlexMeasures takes that as the fill level at the schedule's start. + So a schedule that started at a step already under way was planned as if the whole step remained. + Every replan within that step then asked for less power over the rest of it, and delivered less than planned. The step under way keeps the instruction it is already following. """ now = self.now() @@ -350,12 +427,13 @@ def schedule_start(self, valid_from: datetime) -> datetime: return current if current == now else current + self._resolution async def _soc_duration(self) -> timedelta: - """What a reported fill level is posted as lasting: an instant, where the state-of-charge sensor records instants. + """Return the duration to post a reported fill level with. - Posted over the minute after it, a reading on such a sensor was taken for a forecast of that minute. - FlexMeasures refuses a single value of no duration on a sensor with a resolution, though, - so on any other sensor, or one that cannot be read, it is posted over a minute, as before. - The sensor is asked until it answers, and its answer kept, since its resolution is a property of how it was set up. + If the state-of-charge sensor records instants, the duration is zero. + On such a sensor, a reading posted over the following minute was taken as a forecast of that minute. + However, FlexMeasures refuses a single value without duration on a sensor that has a resolution. + So on any other sensor, or on a sensor that cannot be read, the reading is posted over a minute, as before. + The sensor is queried until it responds, and the response is kept, because its resolution is fixed when the sensor is set up. """ if self._soc_sensor_records_instants is None: try: @@ -388,25 +466,35 @@ async def _soc_duration(self) -> timedelta: async def send_storage_status( self, status: FRBCStorageStatus, request_number: int | None = None ): - # Read by this handler's clock, and known as of then, which in a co-simulation is the simulated time: - # without a prior, FlexMeasures took the moment the request reached it, - # and a simulated reading was recorded as known years after it was taken. + # Use this handler's clock for both the reading time and the belief time (in a co-simulation, that is the simulated time). + # Without a prior, FlexMeasures used the moment the request reached it as the belief time. + # A simulated reading was then recorded as known years after it was taken. now = self.now() - await self._fm_client.post_measurements( - self._soc_sensor_id, - start=now, - # Scaled like every other fill level this handler records, or the level and the - # band around it would be in different scales wherever fill_level_scale is not 1. - values=[status.present_fill_level * self._fill_level_scale], - unit=self.energy_unit, - duration=await self._soc_duration(), - prior=now, - ) + # Catch failures here, because otherwise a failed post would exit this method before the planning below. + # Planning does not depend on this post: a schedule starts from the state of charge in this status, which is sent inline. + try: + await self._fm_client.post_measurements( + self._soc_sensor_id, + start=now, + # Scale this like every other fill level this handler records. + # Otherwise, if fill_level_scale is not 1, the fill level and the band around it would be on different scales. + values=[status.present_fill_level * self._fill_level_scale], + unit=self.energy_unit, + duration=await self._soc_duration(), + prior=now, + ) + except REQUEST_FAILURES: + self._logger.exception( + "Could not record the state of charge %s on sensor %s for %s; planning from it all the same.", + status.present_fill_level, + self._soc_sensor_id, + now, + ) - # The state of charge this status reports is what a schedule starts from, so this is - # where a schedule is asked for. - # A system description is needed to turn that schedule into instructions. - # Without one yet, nothing is planned here, and the description plans against this status when it arrives. + # A schedule starts from the state of charge in this status, so this is where a schedule is requested. + # Turning that schedule into instructions requires a system description. + # If none has arrived yet, nothing is planned here. + # Then the description triggers planning from this status when it arrives. if not self._system_description_history: self._logger.warning( "Cannot ask for a schedule: no FRBC.SystemDescription has arrived yet." @@ -503,11 +591,87 @@ def _in_the_power_unit(self, power: float) -> float: @staticmethod def _latest(history: dict): - """The message a history holds that arrived last, or None while it holds none.""" + """Return the last message that arrived in a history, or None if the history is empty.""" return list(history.values())[-1] if history else None + def _fill_level_band( + self, + start: datetime, + system_description: FRBCSystemDescription, + soc_unit: str, + to_soc_unit: float, + ) -> tuple[list[dict], list[dict]]: + """Return the fill levels to keep the storage between over the planned window, as time segments for FlexMeasures. + + The declared fill-level range applies to every step. + The latest fill-level target profile, held inside that range, applies to the steps it covers. + The range used to reach FlexMeasures only through the band sensors, and only where a target profile was recorded on them. + So a storage without those sensors, or past the profile's end, was planned without any band. + It could then be planned down to empty, and up to the hard upper bound (by default, half again above its range). + The values are clipped at zero, because FlexMeasures does not accept negative minima. + """ + steps = int( + pd.Timedelta(self._schedule_duration) / pd.Timedelta(self._resolution) + ) + index = pd.date_range(start, periods=steps, freq=self._resolution) + range_bottom, range_top = get_soc_min_max( + system_description, self._fill_level_scale + ) + minima = pd.Series(range_bottom, index=index, dtype=float) + maxima = pd.Series(range_top, index=index, dtype=float) + profile = self._latest(self._fill_level_target_profile_history) + if profile is not None: + profile_minima, profile_maxima = translate_fill_level_target_profile( + fill_level_target_profile=profile, + resolution=self._resolution, + fill_level_scale=self._fill_level_scale, + ) + profile_minima, profile_maxima, _ = clip_fill_level_target_profile( + profile_minima, profile_maxima, range_bottom, range_top + ) + minima.update(self._on_the_window(profile_minima, index)) + maxima.update(self._on_the_window(profile_maxima, index)) + return ( + self._as_segments(minima.clip(lower=0) * to_soc_unit, soc_unit), + self._as_segments(maxima.clip(lower=0) * to_soc_unit, soc_unit), + ) + + def _on_the_window(self, profile: pd.Series, index: pd.DatetimeIndex) -> pd.Series: + """Return a target profile's values at the window's steps, for the steps that the profile covers. + + Steps are matched by the instant at which they start. + The profile is floored to steps in its own timezone, and the window in the description's timezone. + If the two timezones are half an hour apart, the two sets of steps never share an instant. + Matching them by label then silently dropped the whole profile. + """ + if profile.empty: + return profile + covered_until = profile.index[-1] + pd.Timedelta(self._resolution) + at_steps = profile.reindex(index.union(profile.index)).ffill().reindex(index) + return at_steps[(index >= profile.index[0]) & (index < covered_until)] + + def _as_segments(self, series: pd.Series, unit: str) -> list[dict]: + """Convert a series on this handler's resolution into FlexMeasures time segments, one per run of equal values.""" + segments: list[dict] = [] + for moment, value in series.items(): + end = moment + pd.Timedelta(self._resolution) + if segments and segments[-1]["_value"] == value: + segments[-1]["end"] = end.isoformat() + continue + segments.append( + { + "start": moment.isoformat(), + "end": end.isoformat(), + "value": f"{value} {unit}", + "_value": value, + } + ) + for segment in segments: + del segment["_value"] + return segments + def _soc_unit(self) -> tuple[str, float]: - """The unit a state of charge goes to FlexMeasures in, and what a fill level is multiplied by to be in it.""" + """Return the unit for sending a state of charge to FlexMeasures, and the factor that converts a fill level into it.""" if self.energy_unit in ("kWh", "MWh"): return self.energy_unit, 1.0 return "kWh", ( @@ -516,7 +680,10 @@ def _soc_unit(self) -> tuple[str, float]: ) def _latest_fill_level(self) -> float | None: - """The fill level the latest storage status reported, in the resource manager's own units, if it reported a number.""" + """Return the fill level from the latest storage status, in the resource manager's own units. + + Return None if there is no storage status, or if its fill level is not a number. + """ if not self._storage_status_history: return None fill_level = list(self._storage_status_history.values())[-1].present_fill_level @@ -524,6 +691,15 @@ def _latest_fill_level(self) -> float | None: async def send_actuator_status(self, status: FRBCActuatorStatus): factor = status.operation_mode_factor + # After a reconnection, an actuator status can arrive before any description. + # This method needs a description to tell what power the status stands for. + if not self._system_description_history: + # Kept instead, and recorded once a description arrives (see handle_system_description). + self._statuses_awaiting_a_description[str(status.actuator_id)] = status + self._logger.info( + "Not recording the power of an FRBC.ActuatorStatus yet: no FRBC.SystemDescription has arrived to tell what its operation mode draws, so it is recorded once one does." + ) + return sd: FRBCSystemDescription = list(self._system_description_history.values())[-1] # Report the mode the actuator is ACTUALLY in, not whichever the system description happens @@ -556,13 +732,13 @@ async def send_actuator_status(self, status: FRBCActuatorStatus): ) operation_mode = actuator.operation_modes[0] - # The element the actuator runs on depends on the fill level, as a heat pump derates while its buffer warms, - # so it is the one that applies at the latest fill level reported, not whichever the mode happens to list first. + # The element the actuator runs on depends on the fill level; for example, a heat pump derates while its buffer warms up. + # So use the element that applies at the latest reported fill level, not the element that the mode happens to list first. fill_level = self._latest_fill_level() if fill_level is not None: element = element_for_fill_level(operation_mode.elements, fill_level) else: - # As if at the bottom of the range, which is as good a guess as any, and is said to be one below. + # Assume the bottom of the range, which is as good a guess as any, and log below that it is a guess. element = element_for_fill_level(operation_mode.elements, float("-inf")) if len(operation_mode.elements) > 1: self._logger.warning( @@ -587,21 +763,19 @@ async def send_actuator_status(self, status: FRBCActuatorStatus): start = status.transition_timestamp or self.now() - # A belief time that supersedes the one before it, rather than colliding with it. - # FlexMeasures keys a belief on its sensor, its source, the event and the moment it was formed, - # and it floors the event to the sensor's resolution. - # Two statuses inside one resolution step, milliseconds apart, were refused in practice, - # with a duplicate key whose belief time had lost its sub-second part. - # Posting two such beliefs by hand does not reproduce that, - # so the exact rule is the server's to state. - # Either way, a second apart is unambiguous: the later status supersedes the earlier one, - # which is what a resource manager reporting twice in a second means. - # A resource manager sends a status for each instruction it applies, - # and this handler sends several at once, so that is the ordinary case rather than an edge. - # The comparison is on the event as FlexMeasures keys it, - # since two statuses seconds apart still land on the same resolution step. - # Kept per event, since statuses for different steps can interleave within a second, - # and a comparison with the latest status alone then missed the one before it about the same step. + # Pick a belief time that supersedes the previous one, instead of colliding with it. + # FlexMeasures keys a belief on its sensor, its source, the event, and the moment the belief was formed. + # It floors the event to the sensor's resolution. + # In practice, two statuses within one resolution step, milliseconds apart, were refused. + # The error was a duplicate key, with a belief time that had lost its sub-second part. + # Posting two such beliefs by hand does not reproduce that, so only the server can say what the exact rule is. + # Either way, belief times a second apart are unambiguous: the later status supersedes the earlier one. + # That is what a resource manager means when it reports twice within a second. + # A resource manager sends a status for each instruction it applies, and this handler sends several instructions at once. + # So this is the ordinary case, not an edge case. + # The comparison uses the event as FlexMeasures keys it, because two statuses seconds apart still land on the same resolution step. + # Keep the times per event, because statuses for different steps can interleave within a second. + # Comparing with the latest status alone then missed the earlier status for the same step. event = self.align_to_resolution(start) recorded_at = self.now() last_recorded_at = self._power_recorded_at.get(event) @@ -610,7 +784,7 @@ async def send_actuator_status(self, status: FRBCActuatorStatus): ): recorded_at = last_recorded_at + timedelta(seconds=1) self._power_recorded_at[event] = recorded_at - # Steps more than a day older than this one are forgotten, to keep this bounded: a status that late is not expected. + # Forget steps more than a day older than this one, to keep this bounded. A status that late is not expected. for old_event in [ e for e in self._power_recorded_at if e < event - timedelta(days=1) ]: @@ -651,18 +825,30 @@ async def send_actuator_status(self, status: FRBCActuatorStatus): # useful quantity - what the actuator delivers into its buffer - it simply is not power. if self._fill_rate_sensor_id is not None: fill_rate = element.fill_rate - rate = ( - fill_rate.start_of_range - + (fill_rate.end_of_range - fill_rate.start_of_range) * factor - ) * self._fill_level_scale - await self._fm_client.post_sensor_data( - self._fill_rate_sensor_id, - start=start, - prior=self.now(), - values=[rate], - unit=self.fill_rate_unit(), - duration=self._resolution, + rate = self._per_second_in_fill_rate_unit( + ( + fill_rate.start_of_range + + (fill_rate.end_of_range - fill_rate.start_of_range) * factor + ) + * self._fill_level_scale ) + # Catch failures, like for the power above, because this runs in a task that nobody awaits. + try: + await self._fm_client.post_sensor_data( + self._fill_rate_sensor_id, + start=start, + prior=self.now(), + values=[rate], + unit=self.fill_rate_unit(), + duration=self._resolution, + ) + except REQUEST_FAILURES: + self._logger.exception( + "Could not record the fill rate %s on sensor %s for %s.", + rate, + self._fill_rate_sensor_id, + start, + ) async def _record_charging_efficiency( self, @@ -670,18 +856,18 @@ async def _record_charging_efficiency( start: datetime, fill_level: float, ) -> bool: - """Record what the actuator converts drawn power into stored energy at, over the window a schedule covers. - - FlexMeasures reads a charging efficiency from a sensor rather than taking a value in the flex model, - so what the operation modes say about it is recorded here, by the call that is about to plan with it. - Recorded before the schedule is asked for, and not alongside it: - a schedule planned against a sensor holding nothing for its window is planned as if the actuator converted power one for one. - - :param start: where the schedule starts, which is where the value is recorded from. - :param fill_level: the fill level the schedule starts from, in the resource manager's own units, - which picks the element of each mode that answers. - :returns: whether that schedule may be planned against the sensor, - which it may not where nothing is recorded on it for this window. + """Record the efficiency at which the actuator converts drawn power into stored energy, over the window a schedule covers. + + FlexMeasures reads a charging efficiency from a sensor, instead of taking a value in the flex model. + So the efficiency that the operation modes imply is recorded here, by the call that is about to plan with it. + It is recorded before the schedule is requested, not at the same time. + A schedule planned against a sensor that holds nothing for its window is planned as if the actuator converted power one to one. + + :param start: where the schedule starts; the value is recorded from there. + :param fill_level: the fill level the schedule starts from, in the resource manager's own units. + It decides which element of each mode is used. + :returns: whether the schedule may be planned against the sensor. + It may not, if nothing is recorded on the sensor for this window. """ if self._charging_efficiency_sensor_id is None: return False @@ -689,8 +875,9 @@ async def _record_charging_efficiency( # One actuator, as everywhere else in this handler. actuator = system_description.actuators[0] efficiencies: dict[str, float] = {} - # The energy unit is one fill_rate_in_watts knows, which the constructor saw to. - for operation_mode in actuator.operation_modes: + # The constructor already checked that fill_rate_in_watts knows the energy unit. + # Only use the modes the actuator may be instructed into under normal conditions, because the schedule is planned in those. + for operation_mode in normal_operation_modes(actuator): efficiency = operation_mode_conversion_efficiency( operation_mode, self._fill_level_scale, self.energy_unit, fill_level ) @@ -733,8 +920,11 @@ async def _record_charging_efficiency( # and for a heat pump, whose fill rate is the heat it delivers, it is well above one. unit="dimensionless", duration=self._schedule_duration, + # Below, the value is remembered as recorded, and is then never posted again for this window. + # So an ingestion that is still pending at the timeout must not count as recorded. + confirm_ingestion=True, ) - except (JobFailedError, ValueError, ConnectionError, NotImplementedError): + except (*REQUEST_FAILURES, NotImplementedError): # A schedule planned as if the actuator converted power one for one beats no schedule at all, # as long as it is said out loud: # that plan overstates what a lossy storage gains from a given power, and understates what a heat pump puts in. @@ -749,7 +939,9 @@ async def _record_charging_efficiency( self._charging_efficiency_recorded = (start, efficiency) return True - def _is_worth_asking_again(self, failure: Exception) -> bool: + async def _is_worth_asking_again( + self, failure: Exception, attempt: int = 1 + ) -> bool: """Whether a failed attempt at a schedule is worth making again. A job that reached the server and failed inside it is, @@ -765,9 +957,33 @@ def _is_worth_asking_again(self, failure: Exception) -> bool: message = str(failure) if any(marker in message for marker in NON_RETRYABLE_SCHEDULE_FAILURES): return False - if isinstance(failure, JobFailedError): + # A job that nobody picked up in time, or a server that could not be reached, may both work on the next attempt. + # The likeliest causes are a scheduling queue without a live worker, and a gateway that is restarting. + # A timeout has already waited for the job's whole timeout, and a dead queue will not come back two seconds later. + # So after a timeout, retry only once, not on every allowed attempt. + if isinstance(failure, JobTimeoutError): + return attempt == 1 + # The client wraps every HTTP error in a ConnectionError. + # A 4xx response, such as for an endpoint that an old server lacks, will be the same next time. + cause = failure.__cause__ + if isinstance(cause, ClientResponseError) and cause.status < 500: + return False + if isinstance( + failure, + (JobFailedError, ConnectionError, TimeoutError, ClientError), + ): + return True + if any(marker in message for marker in RETRYABLE_TRIGGER_REFUSALS): return True - return any(marker in message for marker in RETRYABLE_TRIGGER_REFUSALS) + # FlexMeasures before v0.33.0 reports a failed job only as a bare HTTP 400. + # That cannot be told apart from a refused trigger, so retry it, within the allowed number of attempts. + if message != BARE_REFUSAL: + return False + # This lookup happens while handling another failure, so a failure of the lookup itself must not replace that one. + try: + return not await self._fm_client.server_version_at_least("0.33.0") + except REQUEST_FAILURES: + return False async def _ask_for_a_schedule( self, @@ -776,20 +992,22 @@ async def _ask_for_a_schedule( flex_model: dict, request_number: int | None = None, ) -> dict | list[dict] | None: - """Ask FlexMeasures for a schedule, and ask again where the attempt is worth making again. - - One failed scheduling job otherwise leaves the storage without an instruction for its window. - The resource manager is told nothing, and goes on polling, - and since this runs as a task whose exception is logged and dropped, nothing else notices either. - That is what one failed job did in a co-simulation: - a worker crashed inside its solver, - and the resource manager it belonged to polled until its round timed out, every round, for the rest of the run. - - :param start: the window the schedule is asked for, which every attempt asks for again. - :param request_number: the planning request this answers; where a newer one arrives between attempts, - no further attempt is made and None is returned, so that the newer one is planned sooner. - :raises JobFailedError: where the last attempt worth making failed inside the server. - :raises ValueError: where the trigger itself was refused. + """Ask FlexMeasures for a schedule, and retry if the failure is worth retrying. + + Without retries, one failed scheduling job leaves the storage without an instruction for its window. + The resource manager is not told, and keeps polling. + This runs as a task whose exception is logged and dropped, so nothing else notices either. + That happened in a co-simulation, after a worker crashed inside its solver. + The affected resource manager polled until its round timed out, in every round, for the rest of the run. + + :param start: the start of the window to request the schedule for; every attempt requests the same window. + :param request_number: the planning request this call serves. + If a newer request arrives between attempts, no further attempt is made, and None is returned. + That way, the newer request is planned sooner. + :raises JobFailedError: if the last attempt worth making failed inside the server. + :raises JobTimeoutError: if the last attempt worth making was not picked up in time. + :raises ConnectionError: if the server could not be reached on the last attempt worth making. + :raises ValueError: if the trigger itself was refused. """ attempt = 0 while True: @@ -809,10 +1027,10 @@ async def _ask_for_a_schedule( # and not in the power sensor's own unit, which is what a schedule comes back in by default. unit=S2_POWER_UNIT, ) - except (JobFailedError, ValueError) as failure: + except (*REQUEST_FAILURES, NotImplementedError) as failure: if ( attempt >= self._schedule_attempts - or not self._is_worth_asking_again(failure) + or not await self._is_worth_asking_again(failure, attempt) ): self._logger.exception( "The storage is left without an instruction for the window starting %s: asking for a schedule failed on attempt %s of %s.", @@ -841,27 +1059,26 @@ async def _ask_for_a_schedule( def _warn_where_a_fallback_scheduler_answered( self, schedule: dict | list[dict], start: datetime ) -> None: - """Say so where a schedule was planned by a scheduler this handler did not ask for. - - FlexMeasures answered an infeasible problem with a fallback scheduler until v1.0.0 retired it. - Against an older server, the schedule endpoint follows that fallback job rather than reporting the failure, - or redirects to it where the server sets FLEXMEASURES_FALLBACK_REDIRECT, which this client follows, - so either way a schedule nobody asked for arrives looking like the one that was. - It is a coarse charging policy rather than a plan, - and it records no state of charge, so a later window has nothing to resolve its soc-at-start against: - that is how a house in a co-simulation starved mid-run. - - :param schedule: what came back, which names the scheduler that planned it. - :param start: the window it was asked for. + """Log an error if a schedule was planned by a scheduler that this handler did not ask for. + + Until v1.0.0 retired it, FlexMeasures handled an infeasible problem with a fallback scheduler. + On an older server, the schedule endpoint follows that fallback job instead of reporting the failure. + Or, if the server sets FLEXMEASURES_FALLBACK_REDIRECT, it redirects to the fallback job, and this client follows the redirect. + Either way, a schedule that nobody asked for arrives looking like the requested one. + It is a coarse charging policy, not a plan. + It ignores the band, the efficiencies and the power bands that this handler sent with its request. + + :param schedule: the response, which names the scheduler that planned it. + :param start: the start of the window it was requested for. """ if not isinstance(schedule, dict): - # A schedule per device, which only an asset-wide trigger is answered with. + # A schedule per device, which is only returned for an asset-wide trigger. return scheduler = str((schedule.get("scheduler_info") or {}).get("scheduler", "")) if "fallback" not in scheduler.lower(): return self._logger.error( - "The schedule for the window starting %s was planned by %s, which is a fallback scheduler: the problem this handler asked about was infeasible. The schedule is a coarse charging policy, and no state of charge was recorded for a later window to be planned from.", + "The schedule for the window starting %s was planned by %s, which is a fallback scheduler: the problem this handler asked about was infeasible. The schedule is a coarse charging policy, not a plan.", start, scheduler, ) @@ -869,19 +1086,21 @@ def _warn_where_a_fallback_scheduler_answered( async def trigger_schedule( self, system_description_id: str, request_number: int | None = None ): - """Plan, one schedule at a time, and only for the newest request once it is this one's turn. + """Plan one schedule at a time, and when it is this request's turn, only plan if it is the newest request. Each storage status is a planning request. - One arriving while a schedule was being planned used to start a second solve beside it, whose answer replaced the first's, - so a resource manager reporting faster than FlexMeasures plans kept several solves running for nothing. - Now a request waits for the one in flight, and is dropped where a newer one has arrived meanwhile; - the one in flight, once out of date, makes no further attempt after a failed one. - So a burst of statuses costs at most the attempt in flight and one plan more, for the newest, - which starts once that attempt is over rather than beside it. + A request that arrived while a schedule was being planned used to start a second solve in parallel. + The second result then replaced the first. + So a resource manager that reported faster than FlexMeasures could plan kept several solves running for nothing. + Now a request waits for the one in flight. + It is dropped if a newer request has arrived in the meantime. + Once the request in flight is out of date, it makes no further attempt after a failed one. + So a burst of statuses costs at most the attempt in flight, plus one more plan for the newest request. + That plan starts after the attempt in flight is over, not in parallel. :param system_description_id: system description to base the schedule on - :param request_number: which planning request this schedule answers, which is checked when its turn comes, - between attempts, and before the schedule is translated and sent + :param request_number: the planning request this schedule serves. + It is checked when its turn comes, between attempts, and before the schedule is translated and sent. """ async with self._planning: if request_number is not None and not self._is_latest_request( @@ -900,7 +1119,7 @@ async def _plan( """Translates S2 System Description into FM API calls. :param system_description_id: system description to base the schedule on - :param request_number: which planning request this schedule answers + :param request_number: the planning request this schedule serves """ system_description: FRBCSystemDescription = self._system_description_history[ @@ -926,6 +1145,11 @@ async def _plan( # One actuator, as everywhere else in this handler. actuator = system_description.actuators[0] + if not normal_operation_modes(actuator): + self._logger.warning( + "Not asking for a schedule: the actuator states no operation mode it may be instructed into under normal conditions." + ) + return # Every power range the actuator states, over all of its operation modes and their elements. # An S2 power range runs from its lower bound to its upper one, and S2 fixes one sign convention, # where positive means consumption, @@ -937,40 +1161,50 @@ async def _plan( min(power_range.start_of_range, power_range.end_of_range), max(power_range.start_of_range, power_range.end_of_range), ) - for operation_mode in actuator.operation_modes + # Only use the modes the actuator may be instructed into under normal conditions. + # This is because the actuator may then be instructed into any band that the scheduler plans in. + for operation_mode in normal_operation_modes(actuator) for element in operation_mode.elements for power_range in element.power_ranges ] - # The declared range is where a resource manager wants its storage kept, not where its storage can be: - # S2 calls it the range in which the fill level should remain, - # and has a resource manager free to ignore instructions while the fill level is outside it. - # So the range goes out as a band whose breach FlexMeasures prices, - # rather than as the hard soc-min and soc-max, which made states a storage really reaches infeasible, - # and an infeasible job is answered by the fallback scheduler, which ignores the band altogether. - # Hard bounds are still sent, but wide ones: - # a fill-level target profile need not cover the whole schedule, - # and over the part it does not cover there is nothing to price a breach of, - # so nothing else keeps a plan within reach of the storage it is planned for. + # The declared range is where a resource manager wants its storage kept, not the limits of what its storage can reach. + # S2 calls it the range in which the fill level should remain. + # While the fill level is outside it, S2 lets a resource manager ignore instructions. + # So the range is sent as a band, and FlexMeasures puts a price on breaching it (see _fill_level_band). + # Sending the range as the hard soc-min and soc-max made states that a storage really reaches infeasible. + # Hard bounds are still sent, but wide ones. _, range_top = get_soc_min_max(system_description, self._fill_level_scale) # Zero below, since a stock cannot be negative and FlexMeasures accepts no lower floor. soc_min = 0.0 + # S2 allows a top below zero, and multiplying it would pull it further down. + # But below, the bound widens to include the state of charge at the start, which is never below zero. soc_max = range_top * self._hard_bound_factor - # A state of charge outside the hard bounds is infeasible before the first step, - # so the bounds widen to take it in. - # A storage outside its declared range is not an error: - # a resource manager is free to report one, and a schedule still has to be planned from where it says its storage is. - soc_min = min(soc_min, soc_at_start) + # A state of charge outside the hard bounds is infeasible before the first step. + # So the bounds widen to include it. + # A storage outside its declared range is not an error. + # A resource manager is free to report one, and a schedule must still be planned from the reported fill level. + # The bounds only widen upward, because FlexMeasures does not accept a negative soc-min. + # A storage reported below zero is planned from zero, with a warning, instead of being refused without a schedule. + if soc_at_start < 0: + self._logger.warning( + "The storage is reported at %s, below zero, which FlexMeasures cannot plan from; planning from zero instead.", + soc_at_start, + ) + soc_at_start = 0.0 + # Also translate the schedule from zero, so that the instructions follow the plan from where it started. + fill_level_at_start = 0.0 soc_max = max(soc_max, soc_at_start) - # Where this schedule starts, which is both what it is asked for, - # and the window a charging efficiency has to be recorded over to be planned against. + # The start of this schedule. + # It is both the start of the requested window and the start of the window that a charging efficiency must be recorded over. start = self.schedule_start(system_description.valid_from) - # FlexMeasures takes a state of charge in kWh or MWh only. - # So a fill level recorded in another energy unit is converted here, at the edge, rather than refused, - # and the bounds read back below stay in the recorded unit. - # The band, the usage and the leakage are read from sensors, which FlexMeasures converts itself. + # FlexMeasures only accepts a state of charge in kWh or MWh. + # So a fill level recorded in another energy unit is converted here, at the edge, instead of being refused. + # The bounds used for translating the schedule below stay in the recorded unit. + # The usage and the leakage are read from sensors, which FlexMeasures converts itself. + # The band is converted here. soc_unit, to_soc_unit = self._soc_unit() flex_model: dict = { "soc-unit": soc_unit, @@ -979,28 +1213,34 @@ async def _plan( "soc-min": soc_min * to_soc_unit, "soc-max": soc_max * to_soc_unit, } - # What the resource manager declared about its storage, each entry only where a sensor holds it: - # an entry naming a sensor that was never configured has the scheduler plan against nothing at all. - # Both band sensors or neither, as when the band is recorded: - # one side alone is an open-ended push in one direction, not the band the resource manager declared. - has_soc_band = ( - self._soc_minima_sensor_id is not None - and self._soc_maxima_sensor_id is not None + # Send the band the resource manager declared within the request itself, instead of referring to a sensor. + # That way, the band is included whether or not the sensors it is also recorded on exist. + soc_minima, soc_maxima = self._fill_level_band( + start, system_description, soc_unit, to_soc_unit ) - if has_soc_band: - flex_model["soc-minima"] = {"sensor": self._soc_minima_sensor_id} - flex_model["soc-maxima"] = {"sensor": self._soc_maxima_sensor_id} - elif ( - self._soc_minima_sensor_id is not None - or self._soc_maxima_sensor_id is not None - ): + # FlexMeasures has relax-soc-constraints, which makes this a band, from v0.26.0 on. + # A server refuses a flex context with a field it does not know, which would leave no schedule at all. + if await self._fm_client.server_version_at_least("0.26.0"): + flex_model["soc-minima"] = soc_minima + flex_model["soc-maxima"] = soc_maxima + else: self._logger.info( - "Not asking for a schedule against a fill-level band: a band needs both a soc-minima and a soc-maxima sensor, and this handler was given %s and %s.", - self._soc_minima_sensor_id, - self._soc_maxima_sensor_id, + "Not asking for a schedule against the declared fill-level band: FlexMeasures can relax it from v0.26.0, and this server reports %s.", + self._fm_client.server_version, ) if self._usage_forecast_sensor_id is not None: - flex_model["soc-usage"] = [{"sensor": self._usage_forecast_sensor_id}] + # Skip this if the resource manager says it provides no usage forecast, + # because then nothing is recorded on the sensor for the scheduler to read. + # But use it if the resource manager sent a forecast anyway, since sending one says it provides one, whatever its flag says. + if ( + system_description.storage.provides_usage_forecast + or self._usage_forecast_history + ): + flex_model["soc-usage"] = [{"sensor": self._usage_forecast_sensor_id}] + else: + self._logger.info( + "Planning without usage: the resource manager says it provides no usage forecast, and has sent none." + ) if await self._record_storage_efficiency(start, fill_level_at_start): flex_model["storage-efficiency"] = { "sensor": self._leakage_behaviour_sensor_id @@ -1029,18 +1269,20 @@ async def _plan( ) flex_model["production-capacity"] = f"{production_capacity} {S2_POWER_UNIT}" operation_modes = _operation_mode_power_bands(power_ranges, S2_POWER_UNIT) - # Only where the bands say more than the two capacities above already do. - # The capacities say the actuator can run at any power from minus its production capacity to its consumption capacity, - # so a single band running through zero is exactly that and says nothing new. - # Any other single band does: an actuator that runs at 200 W or not at all rules out everything in between, - # and so does one that only ever feeds back. - # And only to a server that has the field. - # FlexMeasures added operation-modes in v1.0.0, - # and a flex model naming a field the server does not know is refused outright, - # which would leave the resource manager with no schedule at all. - says_more_than_the_capacities = len(operation_modes) > 1 or not any( - lower <= 0 <= upper for lower, upper in power_ranges - ) + # Only send the bands if they add information to the two capacities above. + # The capacities say that the actuator can run at any power from minus its production capacity up to its consumption capacity. + # Bands that together cover exactly that range, without a gap, add nothing. + # An example is an idle mode at zero next to a mode running from zero up. + # Each band costs the solver a binary variable per step. + # Bands with a gap between them do add information. + # An actuator that runs at 200 W or not at all rules out everything in between. + # So does an actuator that only ever feeds back. + # Also, only send the bands to a server that has the field. + # FlexMeasures added operation-modes in v1.0.0. + # A server refuses a flex model with a field it does not know, which would leave the resource manager with no schedule at all. + says_more_than_the_capacities = _merged(power_ranges) != [ + (-production_capacity, consumption_capacity) + ] if says_more_than_the_capacities: if await self._fm_client.server_version_at_least("1.0.0"): flex_model["operation-modes"] = operation_modes @@ -1056,11 +1298,11 @@ async def _plan( "consumption-price": {"sensor": self._price_sensor_id}, "site-power-capacity": f"{3 * 25 * 230} VA", } - if has_soc_band: - # Without this, the minima and maxima above are hard bounds again, - # and the band is back to making reachable states infeasible. - # The breach price is left to FlexMeasures' own default, - # since what a breach of comfort is worth is the deployment's judgement, not the protocol's. + # The band is always sent, and without this setting its minima and maxima are hard bounds. + # Hard bounds make states that a storage really reaches infeasible. + # The breach price is left to FlexMeasures' own default. + # The value of a comfort breach is for the deployment to judge, not for the protocol. + if "soc-minima" in flex_model: flex_context["relax-soc-constraints"] = True if self._relax_site_capacity is not None: flex_context["relax-site-capacity-constraints"] = self._relax_site_capacity @@ -1072,8 +1314,8 @@ async def _plan( return self._warn_where_a_fallback_scheduler_answered(schedule, start) - # send_instruction_batch checks this once more, but translating a schedule follows every step of it, - # which is wasted on a request that is already out of date. + # send_instruction_batch checks this again, but translating a schedule goes through every step of it. + # That work is wasted on a request that is already out of date. if request_number is not None and not self._is_latest_request( STORAGE_STATUS_REQUESTS, request_number ): @@ -1096,10 +1338,13 @@ async def _plan( soc_min / self._fill_level_scale, soc_max / self._fill_level_scale, ), - # What the storage loses by itself and what is drawn from it, as the resource manager last said, - # so that the fill level each step is answered at falls as well as rises. + # Pass what the storage loses by itself and what is drawn from it, as the resource manager last reported. + # Then the fill level used for each step can fall as well as rise. leakage_behaviour=self._latest(self._leakage_behaviour_history), - usage_forecast=self._latest(self._usage_forecast_history), + # Leave out negative usage, as when the forecast was recorded and planned against (see send_usage_forecast). + usage_forecast=_without_negative_usage( + self._latest(self._usage_forecast_history) + ), ) await self.send_instruction_batch( @@ -1157,18 +1402,27 @@ async def send_fill_level_target_profile( # that comes back begins there, not at the start_time the resource manager sent. Posting # it at the raw start_time would shift every value by the difference. start = self.align_to_resolution(fill_level_target_profile.start_time) + # These are recorded for inspection only, because the band that a schedule is planned against is sent inline. + # So a failure is logged, and the other bound is still recorded, instead of raising in a task that nobody awaits. for sensor_id, values in ( (self._soc_minima_sensor_id, soc_minima), (self._soc_maxima_sensor_id, soc_maxima), ): - await self._fm_client.post_sensor_data( - sensor_id=sensor_id, - start=start, - prior=now, - values=values.tolist(), - unit=self.energy_unit, - duration=duration, - ) + try: + await self._fm_client.post_sensor_data( + sensor_id=sensor_id, + start=start, + prior=now, + values=values.tolist(), + unit=self.energy_unit, + duration=duration, + ) + except REQUEST_FAILURES: + self._logger.exception( + "Could not record a bound of the fill-level target profile on sensor %s from %s.", + sensor_id, + start, + ) async def send_usage_forecast(self, usage_forecast: FRBCUsageForecast): """Record what a resource manager expects its storage to be drawn down by.""" @@ -1178,12 +1432,27 @@ async def send_usage_forecast(self, usage_forecast: FRBCUsageForecast): ) return - translated = translate_usage_forecast_to_fm( - usage_forecast, - self._resolution, - strategy="mean", - fill_level_scale=self._fill_level_scale, + translated = self._per_second_in_fill_rate_unit( + translate_usage_forecast_to_fm( + usage_forecast, + self._resolution, + strategy="mean", + fill_level_scale=self._fill_level_scale, + ) ) + # A negative usage is a gain that the storage is forced to take, and FlexMeasures cannot relax it. + # Above the hard ceiling, it makes every schedule for the window infeasible, so the storage gets no instruction at all. + # S2 allows the negative sign, but no storage that this handler controls fills up by being drawn from. + # So a negative value is treated as an artefact of the forecast: it is counted, and recorded as no usage. + negative = translated < 0 + if negative.any(): + self._logger.warning( + "An FRBC.UsageForecast expects negative usage on %s of %s steps, down to %s; recorded as no usage there, since FlexMeasures would plan it as a gain the storage cannot refuse.", + int(negative.sum()), + len(translated), + float(translated.min()), + ) + translated = translated.clip(lower=0) await self._fm_client.post_sensor_data( sensor_id=self._usage_forecast_sensor_id, @@ -1192,16 +1461,17 @@ async def send_usage_forecast(self, usage_forecast: FRBCUsageForecast): start=self.align_to_resolution(usage_forecast.start_time), prior=self.now(), values=translated.tolist(), - # translate_usage_forecast_to_fm returns fill level per second, scaled. unit=self.fill_rate_unit(), duration=pd.Timedelta(self._resolution) * len(translated), ) async def send_leakage_behaviour(self, leakage_behaviour: FRBCLeakageBehaviour): - """Keep a resource manager's leakage behaviour for planning, which records it when it asks for a schedule. + """Keep a resource manager's leakage behaviour for planning; it is recorded when a schedule is requested. - Recorded on arrival instead, it covered the hours after the message rather than the window being planned, - at a fill level nobody had reported yet, and planning went on without it once those hours had passed. + It used to be recorded on arrival. + Then it covered the hours after the message, instead of the window being planned. + It was also computed at a fill level that nobody had reported yet. + And once those hours had passed, planning went on without it. """ if self._leakage_behaviour_sensor_id is None: self._logger.info( @@ -1211,9 +1481,9 @@ async def send_leakage_behaviour(self, leakage_behaviour: FRBCLeakageBehaviour): async def _record_storage_efficiency( self, start: datetime, fill_level: float ) -> bool: - """Record what the storage keeps of itself per step, over the window a schedule covers, from where it starts. + """Record the share of its stock that the storage keeps per step, over the window a schedule covers, from the schedule's start. - :returns: whether the schedule may be planned against the sensor, which it may not where nothing is recorded on it. + :returns: whether the schedule may be planned against the sensor. It may not, if nothing is recorded on the sensor. """ leakage_behaviour = self._latest(self._leakage_behaviour_history) if ( @@ -1222,18 +1492,19 @@ async def _record_storage_efficiency( or not self._system_description_history ): return False - # S2 states a leakage as a rate, and FlexMeasures takes a share of the stock, so one share stands for the rate - # only at one fill level, and the loss scales with the stock away from it. - # That fill level is where the schedule starts, as long as that is at least the middle of the storage's range: - # from a near-empty start, a share of the start level would let almost nothing remain per step, - # and the loss would be overstated many times over as soon as the plan filled the storage. - # Below the middle the loss is understated instead, by at most the ratio of the middle to the stock, which is the milder error. + # S2 gives leakage as a rate, but FlexMeasures takes a share of the stock. + # So one share matches the rate at only one fill level; at other fill levels, the loss scales with the stock. + # That fill level is where the schedule starts, as long as that is at least the middle of the storage's range. + # From a near-empty start, a share of the start level would let almost nothing remain per step. + # The loss would then be overstated many times over, as soon as the plan filled the storage. + # Below the middle, the loss is understated instead, by at most the ratio of the middle to the stock. + # That is the milder error. storage_range = list(self._system_description_history.values())[ -1 ].storage.fill_level_range middle = (storage_range.start_of_range + storage_range.end_of_range) / 2 - # What remains after one step of this handler's resolution, - # which FlexMeasures reads as one step of the leakage sensor's resolution, so the two have to agree. + # The share that remains after one step of this handler's resolution. + # FlexMeasures reads it as one step of the leakage sensor's resolution, so the two resolutions must be equal. efficiency = leakage_behaviour_to_storage_efficiency( message=leakage_behaviour, resolution=self._resolution, @@ -1251,8 +1522,10 @@ async def _record_storage_efficiency( values=[efficiency], unit="%", duration=self._schedule_duration, + # As for the charging efficiency: the value is remembered below as recorded, so it must really be recorded. + confirm_ingestion=True, ) - except (JobFailedError, ValueError, ConnectionError): + except REQUEST_FAILURES: self._logger.exception( "Not asking for a schedule against a storage efficiency: could not record %s %% on sensor %s for %s.", efficiency, diff --git a/src/flexmeasures_client/s2/control_types/FRBC/frbc_tunes.py b/src/flexmeasures_client/s2/control_types/FRBC/frbc_tunes.py index 23babaf3..5d2b0a14 100644 --- a/src/flexmeasures_client/s2/control_types/FRBC/frbc_tunes.py +++ b/src/flexmeasures_client/s2/control_types/FRBC/frbc_tunes.py @@ -169,11 +169,11 @@ def __init__( self._datastore = datastore if datastore is not None else {} def _is_timer_due(self, name: str) -> bool: - # By this handler's clock, so that a co-simulation's timers run on simulated time. + # Use this handler's clock, so that a co-simulation's timers run on simulated time. now = self.now() due_time = self._timers.get(name, now - self._minimum_measurement_period) if due_time.tzinfo is None: - # A timer handed in without a zone was compared with the host's local time, so it is read as that, as before. + # A timer without a timezone used to be compared with the host's local time, so keep reading it as local time. due_time = due_time.astimezone(self._timezone) if due_time <= now: # Get total seconds of the period @@ -191,8 +191,8 @@ def _is_timer_due(self, name: str) -> bool: next_due = now.replace(minute=0, second=0, microsecond=0) + timedelta( seconds=next_tick_seconds ) - # On a period boundary the ceiling is the moment itself, - # and a clock that stands still, as a simulated one does between messages, then found the timer due on every call. + # On a period boundary, the ceiling equals the current moment. + # A clock that stands still (as a simulated clock does between messages) would then find the timer due on every call. if next_due <= now: next_due += self._minimum_measurement_period self._timers[name] = next_due @@ -315,9 +315,9 @@ async def send_actuator_status(self, status: FRBCActuatorStatus): duration=timedelta(minutes=15), ) - # It takes neither argument the base class declares: TUNES plans from its latest description, - # and does not take part in dropping superseded requests. - # Accepting them and ignoring them would turn a caller's mistake into a silent one, so the mismatch stays. + # This override takes neither of the arguments that the base class declares. + # TUNES plans from its latest description, and does not take part in dropping superseded requests. + # Accepting the arguments and ignoring them would hide a caller's mistake, so the signatures stay different. async def trigger_schedule(self): # type: ignore[override] """ Ask FlexMeasures for a new schedule and create FRBC.Instructions to send back to the ResourceManager @@ -485,8 +485,9 @@ async def trigger_schedule(self): # type: ignore[override] except Exception as exc: self._logger.error(str(exc)) - # Posted per hour and scaled into FlexMeasures' units by send_usage_forecast, and read back as such, - # while the schedule translation takes a usage rate per second, in the resource manager's own units, as S2 states it. + # send_usage_forecast posts the usage forecast per hour, scaled to FlexMeasures' units, so that is how it is read back. + # The schedule translation needs a usage rate per second, in the resource manager's own units, as in S2. + # So convert it back here. usage_forecast = pd.Series( usage_forecast["values"], index=idx, name="usage_forecast" ) / ((pd.Timedelta(hours=1) / pd.Timedelta(seconds=1)) * self._fill_level_scale) @@ -512,7 +513,7 @@ async def trigger_schedule(self): # type: ignore[override] system_description, initial_fill_level=soc_at_start / self._fill_level_scale, logger=self._logger, - # What the storage loses by itself, so that the running fill level falls as well as rises. + # Pass what the storage loses by itself, so that the running fill level can fall as well as rise. leakage_behaviour=( list(self._leakage_behaviour_history.values())[-1] if self._leakage_behaviour_history @@ -693,9 +694,9 @@ async def send_usage_forecast(self, usage_forecast: FRBCUsageForecast): fill_level_scale=self._fill_level_scale, ) - # From per second, as S2 states a usage rate, to per hour, which is what a value labelled MW means: - # FlexMeasures reads soc-usage as a power and multiplies it by the step's share of an hour. - # Posted per quarter hour instead, it was planned as a quarter of the real draw. + # Convert from per second (as S2 gives a usage rate) to per hour, which is what a value in MW means. + # FlexMeasures reads soc-usage as a power, and multiplies it by the step's fraction of an hour. + # When this was posted per quarter hour instead, the scheduler planned for a quarter of the real draw. scale = timedelta(hours=1) / timedelta(seconds=1) scaled_usage_forecast = usage_forecast_series * scale @@ -703,7 +704,7 @@ async def send_usage_forecast(self, usage_forecast: FRBCUsageForecast): sensor_id=self._usage_forecast_sensor_id, start=start_time, values=scaled_usage_forecast.tolist(), - unit=POWER_UNIT, # a rate per hour, as MW is + unit=POWER_UNIT, # a rate per hour, like MW duration=str(pd.Timedelta(RESOLUTION) * len(usage_forecast_series)), ) diff --git a/src/flexmeasures_client/s2/control_types/FRBC/utils.py b/src/flexmeasures_client/s2/control_types/FRBC/utils.py index cec7fa08..0c30ca9f 100644 --- a/src/flexmeasures_client/s2/control_types/FRBC/utils.py +++ b/src/flexmeasures_client/s2/control_types/FRBC/utils.py @@ -37,16 +37,17 @@ def compute_next_fill_level( leakage_behaviour: FRBCLeakageBehaviour | None = None, usage_rate: float = 0.0, ) -> float: - """The fill level one step later. + """Return the fill level one step later. - This is the unit contract for every rate it takes, and it is the one S2 states: - each is a change in fill level per second, in the resource manager's own fill-level units. - A fill rate raises the fill level, while a usage rate and a leakage rate both lower it where they are positive. - The step is a duration, rather than a number of some unit, so that it cannot be taken in the wrong one. + This function sets the unit contract for every rate it takes, and that contract is the one S2 uses. + Each rate is a change in fill level per second, in the resource manager's own fill-level units. + A fill rate raises the fill level. + A positive usage rate or a positive leakage rate lowers it. + The step is a duration, not a number in some unit, so that it cannot be given in the wrong unit. - :param leakage_behaviour: what the storage loses by itself, from the element that applies at this fill level; - none where the resource manager has not said. - :param usage_rate: what is drawn from the storage, held over this step. + :param leakage_behaviour: what the storage loses by itself; the element that applies at this fill level is used. + None if the resource manager has not reported it. + :param usage_rate: what is drawn from the storage, constant over this step. """ leakage_rate = 0.0 if leakage_behaviour is not None: @@ -56,23 +57,25 @@ def compute_next_fill_level( return fill_level + (fill_rate - usage_rate - leakage_rate) * step.total_seconds() -#: How close two factors are to count as the same instruction. -#: It absorbs the float noise of converting a schedule from MW to W, which is not a new instruction. +#: How close two factors must be to count as the same instruction. +#: It absorbs the float noise from converting a schedule from MW to W, which should not lead to a new instruction. FACTOR_TOLERANCE = 1e-6 class UsageProfile: - """What a usage forecast expects, per second, at any moment: piecewise constant, looked up by bisection. + """The usage rate per second that a usage forecast expects at any moment. - A forecast is a run of elements, each lasting its own duration from where the one before it ended. - Outside that run, nothing is forecast, and nothing is assumed: the rate is zero. - Built once per schedule, since a forecast may hold hundreds of elements and a schedule asks about it often. + The rate is piecewise constant, and is looked up by bisection. + A forecast is a sequence of elements. + Each element starts where the previous one ended, and lasts its own duration. + Outside that sequence, nothing is forecast, and nothing is assumed: the rate is zero. + Build this once per schedule, because a forecast may hold hundreds of elements, and a schedule looks up the rate often. """ def __init__( self, usage_forecast: FRBCUsageForecast | None = None, rate: float | None = None ): - #: The moments the rate changes at, and the rate from each of them on; one constant rate has none. + #: The moments at which the rate changes, and the rate from each moment on. A constant rate has no changes. self._changes: list[pd.Timestamp] = [] self._rates: list[float] = [] self._before = 0.0 if rate is None else rate @@ -98,14 +101,15 @@ def next_change_after(self, moment: pd.Timestamp) -> pd.Timestamp | None: def usage_rate_at( usage_forecast: FRBCUsageForecast | None, moment: pd.Timestamp ) -> float: - """The usage rate a forecast expects at a moment, per second, as S2 states it (see UsageProfile).""" + """Return the usage rate that a forecast expects at a moment, per second as in S2 (see UsageProfile).""" return UsageProfile(usage_forecast).rate_at(moment) def _sides(elements, fill_level: float): - """The element that applies just below a fill level, and the one just above it. + """Return the element that applies just below a fill level, and the element that applies just above it. - Inside an element both are that element; at a boundary two elements share, they are the two neighbours. + Inside an element, both are that element. + At a boundary between two elements, they are the two neighbouring elements. """ above = element_for_fill_level(elements, fill_level) below_candidates = [ @@ -133,18 +137,20 @@ def follow_operation_mode( usage: UsageProfile | None = None, bounds: tuple[float, float] | None = None, ) -> float: - """The fill level after following one operation mode at one factor for a step, as a resource manager does. - - An instruction fixes the mode and the factor, not the element: - within the step the fill level can pass from one element into another, of the mode or of the leakage behaviour, - and the usage a forecast expects can change. - Every rate is constant between such moments, so the step is followed exactly, from one of them to the next. - A level that reaches a boundary its far side would push it back from, as where a heat pump cannot heat above it, - stays at that boundary, as a real storage does, rather than flipping across it. - So does a level that reaches one of the bounds. - - :param usage: what is drawn from the storage, per second; none where nothing is known. - :param bounds: the fill levels the storage is held between. + """Return the fill level after following one operation mode at one factor for one step, as a resource manager does. + + An instruction fixes the mode and the factor, but not the element. + Within the step, the fill level can pass from one element into another, of the mode or of the leakage behaviour. + Also, the usage that a forecast expects can change. + Every rate is constant between such moments. + So the step is followed exactly, from one such moment to the next. + Sometimes a fill level reaches a boundary where the rate on the far side would push it back. + For example, a heat pump may be unable to heat above that boundary. + Then the fill level stays at that boundary, as in a real storage, instead of flipping back and forth across it. + The same holds for a fill level that reaches one of the bounds. + + :param usage: what is drawn from the storage, per second; None if nothing is known. + :param bounds: the fill levels between which the storage is held. """ usage = usage or UsageProfile() leakage_elements = leakage_behaviour.elements if leakage_behaviour else [] @@ -183,7 +189,7 @@ def net_rate(mode_element, leakage_element, moment) -> float: elif falling < 0 and fill_level > lowest: rate = falling else: - rate = 0.0 # held where it is, at a boundary or a bound, until something changes + rate = 0.0 # stay at the boundary or bound until something changes horizon = end change = usage.next_change_after(moment) if change is not None and change < horizon: @@ -194,7 +200,7 @@ def net_rate(mode_element, leakage_element, moment) -> float: elif rate < 0: target = next((b for b in reversed(boundaries) if b < fill_level), None) if target is not None: - # Compared in seconds first: a slow rate can put a boundary further off than a timedelta can say. + # Compare in seconds first: at a slow rate, a boundary can be further away than a timedelta can represent. seconds_to_target = (target - fill_level) / rate if seconds_to_target < (horizon - moment).total_seconds(): fill_level = target @@ -205,6 +211,19 @@ def net_rate(mode_element, leakage_element, moment) -> float: return min(max(fill_level, lowest), highest) +def normal_operation_modes(actuator) -> list[FRBCOperationMode]: + """Return the operation modes an actuator may be instructed into under normal conditions. + + S2 marks a mode as abnormal_condition_only if it is reserved for an abnormal condition, such as an emergency boost. + A resource manager refuses an instruction into such a mode, unless the instruction itself is marked abnormal. + The instructions made from a schedule are never marked abnormal. + So such a mode is neither planned in nor instructed into. + """ + return [ + mode for mode in actuator.operation_modes if not mode.abnormal_condition_only + ] + + def fm_schedule_to_instructions( schedule: pd.DataFrame | dict, system_description: FRBCSystemDescription, @@ -216,31 +235,33 @@ def fm_schedule_to_instructions( ) -> List[FRBCInstruction]: """Turn a FlexMeasures schedule into the instructions that follow it. - Each step is answered with the operation mode, and the factor within it, that draws the power the schedule asks for - at the fill level the storage is then at, so a running fill level is kept alongside, under the contract of compute_next_fill_level. - An instruction is sent where that answer changes, which can happen at a constant power too, - where the fill level passes from one element of a mode into another. + For each step, this picks the operation mode, and the factor within it, that draw the power the schedule asks for. + That choice depends on the fill level the storage is at during the step. + So a running fill level is tracked alongside, under the unit contract of compute_next_fill_level. + An instruction is sent whenever the chosen mode or factor changes. + That can happen even at constant power, when the fill level passes from one element of a mode into another. :param schedule: a schedule as FlexMeasures returns it, or a frame indexed by time with a "schedule" column. - A frame may also carry a "usage_forecast" column, per second as S2 states it, - which then takes the place of usage_forecast. - :param fill_level_bounds: the fill levels this conversion holds its own running fill level between, + A frame may also have a "usage_forecast" column, per second as in S2. + If it does, that column is used instead of usage_forecast. + :param fill_level_bounds: the fill levels between which this conversion holds its own running fill level, in the resource manager's units. - These have to be the hard bounds the schedule was planned against, - or a step is answered with the operation mode for a fill level the scheduler never planned. - Without them the declared fill-level range is used, - which is right only for a caller that asked for a schedule against that same range. - :param leakage_behaviour: what the storage loses by itself, as the resource manager last said. + These must be the hard bounds that the schedule was planned against. + Otherwise, a step may get the operation mode for a fill level that the scheduler never planned. + Without them, the declared fill-level range is used. + That is only right for a caller that asked for a schedule against that same range. + :param leakage_behaviour: what the storage loses by itself, as the resource manager last reported it. :param usage_forecast: what is drawn from the storage, as the resource manager last forecast it. - The step is the schedule's own: a schedule from FlexMeasures states it as its duration over its number of values, - which is the resolution of the sensor it was asked for, whatever resolution the caller plans on. + The step comes from the schedule itself. + A schedule from FlexMeasures gives a duration and a number of values, and the step is the duration divided by that number. + That step is the resolution of the sensor the schedule was requested for, whatever resolution the caller plans on. """ if fill_level_bounds is not None and fill_level_bounds[0] > fill_level_bounds[1]: - # A mistake in the caller, which computes these, rather than anything a resource manager sent: - # held between a lower bound above its upper one, the running fill level would sit at the lower one, - # and every step would be answered with the operation mode for a fill level the storage is never at. + # This is a mistake by the caller, which computes these bounds, not something a resource manager sent. + # With a lower bound above the upper bound, the running fill level would stay at the lower bound. + # Then every step would get the operation mode for a fill level the storage is never at. raise ValueError( f"fill_level_bounds runs from its lower bound to its upper one, but {fill_level_bounds} was given." ) @@ -257,7 +278,7 @@ def fm_schedule_to_instructions( idx = pd.date_range(start=start, periods=len(values), freq=step) schedule = pd.Series(values, index=idx, name="schedule").to_frame() else: - # A frame of one row has no later step for its fill level to matter to. + # A frame with one row has no later step, so the fill level after it does not matter, and a zero step will do. step = ( pd.Timedelta(schedule.index[1] - schedule.index[0]) if len(schedule) > 1 @@ -270,12 +291,16 @@ def fm_schedule_to_instructions( f"{len(system_description.actuators)} were provided" ) actuator = system_description.actuators[0] - operation_modes: list[FRBCOperationMode] = actuator.operation_modes + operation_modes: list[FRBCOperationMode] = normal_operation_modes(actuator) + if not operation_modes: + raise ValueError( + "The actuator states no operation mode it may be instructed into under normal conditions." + ) modes_by_id = {str(mode.id): mode for mode in operation_modes} - # Held between the hard bounds the schedule was planned against, - # so that each step is answered with the operation mode for the fill level the scheduler planned. - # Falling back to the declared range keeps every other caller as it was. + # Hold the running fill level between the hard bounds that the schedule was planned against. + # Then each step gets the operation mode for the fill level that the scheduler planned. + # Falling back to the declared range keeps the behaviour unchanged for every other caller. if fill_level_bounds is None: fill_level_range: NumberRange = system_description.storage.fill_level_range fill_level_min = fill_level_range.start_of_range @@ -292,8 +317,8 @@ def fm_schedule_to_instructions( for timestamp, row in schedule.iterrows(): power = row["schedule"] - # The mode and factor for this power at this fill level, every step, - # since the element that applies can change while the power does not. + # Pick the mode and factor for this power at this fill level in every step, + # because the element that applies can change while the power stays the same. results = [ (om, *power_to_fill_rate_with_metrics(om, power, fill_level)) for om in operation_modes @@ -336,10 +361,10 @@ def fm_schedule_to_instructions( explain_choice(results, best), ) - # The factor is the one that draws the power asked for, taken from the power range itself. - # It used to be worked back out of the fill rate instead, - # which fails wherever the fill-rate range is a single value and the power range is not: - # asked for 1200 W from a mode running 800 to 1200 W at one fill rate, that asked for 800 W. + # Take the factor that draws the requested power from the power range itself. + # It used to be derived from the fill rate instead. + # That fails wherever the fill-rate range is a single value but the power range is not. + # For example, given a request for 1200 W from a mode that runs at 800 to 1200 W with a single fill rate, the old code instructed 800 W. answer = (str(best_operation_mode.id), operation_mode_factor) if last_instructed is None or not ( answer[0] == last_instructed[0] @@ -360,8 +385,9 @@ def fm_schedule_to_instructions( instructions.append(instruction) last_instructed = answer - # A gap in a usage column, as FlexMeasures returns for a moment it holds nothing for, is no usage known, not NaN: - # a NaN fill level would be held at the top of the bounds, and answered there from then on. + # FlexMeasures returns a gap in a usage column for a moment it holds no data for. + # Treat such a gap as no known usage, not as NaN. + # A NaN fill level would be held at the top of the bounds, and every later step would be chosen from there. usage = forecast_usage if has_usage_column: usage = UsageProfile( @@ -383,7 +409,7 @@ def fm_schedule_to_instructions( ) if logger.isEnabledFor(logging.DEBUG): - # The whole batch in one line, by mode name, which the per-step lines above do not give. + # Log the whole batch in one line, by mode name, which the per-step log lines above do not show. logger.debug( "Instructions: %s", [ @@ -412,9 +438,9 @@ def get_soc_min_max( return soc_min, soc_max -#: What one unit of energy per second is in watts, for the units a fill level can be recorded in. -#: A joule per second is a watt by definition, -#: and an hour's worth of a watt is 3600 times as much energy as a watt-second. +#: The number of watts in one unit of energy per second, for each unit a fill level can be recorded in. +#: A joule per second is a watt, by definition. +#: A watt-hour is 3600 times as much energy as a watt-second. WATTS_PER_ENERGY_UNIT_PER_SECOND = { "J": 1.0, "kJ": 1e3, @@ -425,16 +451,21 @@ def get_soc_min_max( } +#: The units a rate of fill level can be recorded in, and the number of watts in one of each. +#: S2 states rates per second, which in kWh/s puts a heat buffer's rates of some tens of watts near 1e-5, so they are recorded in a power unit instead, by default W. +WATTS_PER_FILL_RATE_UNIT: dict[str, float] = {"W": 1.0, "kW": 1e3, "MW": 1e6} + + def fill_rate_in_watts(fill_rate: float, energy_unit: str) -> float: - """A fill rate, stated per second in the unit a fill level is recorded in, as watts. + """Convert a fill rate to watts, given the unit that the fill level is recorded in. - S2 states a fill rate as fill level per second, so what it is in watts depends on the unit the fill level itself is in: - a kilowatt hour per second is 3.6 million watts. - The conversion is needed wherever a fill rate is compared against a power value, - which S2 always states in watts. + S2 gives a fill rate as fill level per second. + So its value in watts depends on the unit of the fill level itself. + For example, one kilowatt-hour per second is 3.6 million watts. + This conversion is needed wherever a fill rate is compared with a power value, which S2 always gives in watts. - :raises NotImplementedError: for an energy unit this does not know, - rather than returning a number that is wrong by a factor of thousands. + :raises NotImplementedError: for an unknown energy unit, + instead of returning a number that is wrong by a factor of thousands. """ try: return fill_rate * WATTS_PER_ENERGY_UNIT_PER_SECOND[energy_unit] @@ -451,41 +482,42 @@ def operation_mode_conversion_efficiency( energy_unit: str, fill_level: float, ) -> float | None: - """How much energy an operation mode puts into the storage for each unit of power it draws, at a fill level. - - An operation mode states a fill rate and a power range, - so the ratio between the two is what the mode converts drawn power into stored energy at: - a heat pump's coefficient of performance, or a battery's charging efficiency. - Both sides are taken in watts, since FlexMeasures asks for a charging efficiency as a dimensionless ratio, - and for a thermal actuator that ratio is well above one. - - A mode may convert differently at different fill levels, as a heat pump does when its buffer warms, - and FlexMeasures takes one charging efficiency for a whole schedule. - The element that applies at the fill level the schedule starts from answers, - since the start of a schedule is the part that runs before the next storage status asks for a new one. - Not the lowest element over the whole range: - that overstates what the actuator needs to fill the storage, so the plan overshoots its upper band and buys more than it uses, - and against a usage the plan cannot relax it can turn a problem the actuator can meet into one the scheduler finds infeasible. - - The ratio is taken at the top of that element's range, rather than as the slope over it. - An actuator may well declare degenerate ranges, - where a mode runs at one power and one fill rate only (the heaters in our own smoke tests state on as [883.7, 883.7] W), - and a slope is zero over such a range while the ratio is the efficiency the mode really runs at. + """Return how much energy an operation mode puts into the storage per unit of power it draws, at a fill level. + + An operation mode gives a fill rate and a power range. + The ratio between the two is the rate at which the mode converts drawn power into stored energy. + Examples are a heat pump's coefficient of performance and a battery's charging efficiency. + Both sides are taken in watts, because FlexMeasures expects a charging efficiency as a dimensionless ratio. + For a thermal actuator, that ratio is well above one. + + A mode may convert differently at different fill levels, as a heat pump does when its buffer warms up. + But FlexMeasures takes a single charging efficiency for a whole schedule. + So this uses the element that applies at the fill level where the schedule starts. + The start of a schedule is the part that runs before the next storage status triggers a new schedule. + The lowest element over the whole range would be the wrong choice. + It overstates what the actuator needs to fill the storage. + The plan then overshoots its upper band, and buys more than it uses. + And because the plan cannot relax the usage, the scheduler may find a problem infeasible that the actuator can actually meet. + + The ratio is taken at the top of that element's range, not as the slope over the range. + An actuator may well declare degenerate ranges, where a mode runs at only one power and one fill rate. + For example, the heaters in our own smoke tests declare their on mode as [883.7, 883.7] W. + Over such a range the slope is zero, while the ratio is the efficiency the mode really runs at. :param fill_level_scale: converts a fill level, and so a fill rate, from the resource manager's own units into the unit it is recorded in. - :param energy_unit: the unit a fill level is recorded in, which the fill rate is per second of. - :param fill_level: where the schedule starts, in the resource manager's own units, as the elements' ranges are. - :returns: the efficiency, or None where the element that applies converts no drawn power into stored energy. - It converts nothing where it draws no power, as an idle mode does, - where it only feeds power back, which FlexMeasures takes as a separate discharging efficiency, - or where it draws power while the storage does not fill, as a standby or circulation mode does: - no single ratio states that, and a ratio of zero would tell the scheduler the storage cannot be filled at all. + :param energy_unit: the unit a fill level is recorded in; the fill rate is in this unit per second. + :param fill_level: where the schedule starts, in the resource manager's own units, like the elements' ranges. + :returns: the efficiency, or None if the element that applies converts no drawn power into stored energy. + That is the case in three situations. + The element draws no power, as in an idle mode. + The element only feeds power back, which FlexMeasures handles with a separate discharging efficiency. + Or the element draws power while the storage does not fill, as in a standby or circulation mode. + No single ratio can express that, and a ratio of zero would tell the scheduler that the storage cannot be filled at all. """ element = element_for_fill_level(operation_mode.elements, fill_level) - # todo: an element states one power range per commodity quantity, - # and no distinction is made between them here, - # as the rest of this handler does not make one either (see power_to_fill_rate_with_metrics). + # todo: an element gives one power range per commodity quantity, and this makes no distinction between them. + # Neither does the rest of this handler (see power_to_fill_rate_with_metrics). power = element.power_ranges[0].end_of_range fill_rate = element.fill_rate.end_of_range * fill_level_scale if power <= 0 or fill_rate <= 0: @@ -545,7 +577,7 @@ def power_to_fill_rate_with_metrics( power_penalty: float efficiency: float element: FRBCOperationModeElement that produced fill_rate - factor: the operation mode factor that draws this power, which is what an instruction asks for + factor: the operation mode factor that draws this power; an instruction asks for this factor """ # --- 1. Select the element that applies at this fill level --- best_element = element_for_fill_level(operation_mode.elements, fill_level) @@ -569,10 +601,10 @@ def power_to_fill_rate_with_metrics( pr_end = power_range.end_of_range if pr_end == pr_start: - # Every factor draws this one power, so the power cannot say which factor is meant. - # The top of the fill-rate range is asked for, which is where operation_mode_conversion_efficiency - # declares the efficiency FlexMeasures plans with, so the plan and the instruction agree. - # Where the fill-rate range is a single value too, the factor makes no difference and stays at zero. + # Every factor draws this same power, so the power cannot tell which factor is meant. + # Ask for the top of the fill-rate range. + # That is where operation_mode_conversion_efficiency declares the efficiency that FlexMeasures plans with, so the plan and the instruction agree. + # If the fill-rate range is a single value too, the factor makes no difference, and it stays at zero. fr_start = best_element.fill_rate.start_of_range fr_end = best_element.fill_rate.end_of_range factor = 0.0 if fr_end == fr_start else 1.0 @@ -608,23 +640,24 @@ def clamp_distance(value: float, start: float, end: float) -> float: def element_for_fill_level( elements: Sequence[ElementWithFillLevelRange], fill_level: float ) -> ElementWithFillLevelRange: - """The element that applies at a fill level, whatever order the elements were listed in. + """Return the element that applies at a fill level, whatever order the elements are listed in. S2 lets the elements of an operation mode, or of a leakage behaviour, cover the fill-level range in pieces, in any order. - s2python checks that an operation mode's elements are contiguous, on a sorted copy, and leaves the list as it was sent; - it checks nothing of the kind for a leakage behaviour, whose elements may leave gaps or overlap. + s2python checks that an operation mode's elements are contiguous, using a sorted copy, and leaves the list in the order it was sent. + It does not check this for a leakage behaviour, whose elements may leave gaps or overlap. - Two neighbouring elements share a boundary, which belongs to the upper of the two: - of every element whose range holds the fill level, the one starting highest applies, and of two starting together, the one reaching higher. - A range that is a single point therefore applies only at the very top. - The standard says a range starts below where it ends, so such a range is outside it, though s2python lets it through. - Two elements with the same range are ambiguous, which overlapping elements always are, and the first listed answers. + Two neighbouring elements share a boundary, which belongs to the upper of the two. + Of all elements whose range holds the fill level, the one that starts highest applies. + If two of them start at the same level, the one that reaches higher applies. + So a range that is a single point only applies at the very top. + The standard says a range starts below where it ends, so such a range breaks the standard, but s2python lets it through. + Two elements with the same range are ambiguous (as overlapping elements always are), and then the first one listed applies. - A fill level outside the range the elements cover is answered as the nearest end of that range is, - since a storage may be outside its declared range, and the answer should not jump as it crosses that end. - A fill level in a gap between elements is answered by the nearer, and on a tie the lower. + A fill level outside the range the elements cover is treated as the nearest end of that range. + That is because a storage may be outside its declared range, and the result should not jump when the fill level crosses that end. + For a fill level in a gap between elements, the nearer element applies; on a tie, the lower one applies. - :raises ValueError: for a fill level that is not a number, which no element applies at. + :raises ValueError: for a fill level that is not a number, since no element applies there. """ if math.isnan(fill_level): raise ValueError("No element applies at a fill level that is not a number.") diff --git a/src/flexmeasures_client/s2/control_types/__init__.py b/src/flexmeasures_client/s2/control_types/__init__.py index c708c2b2..f2504ffa 100644 --- a/src/flexmeasures_client/s2/control_types/__init__.py +++ b/src/flexmeasures_client/s2/control_types/__init__.py @@ -44,18 +44,20 @@ async def send_message(self, message: BaseModel) -> None: await self._sending_queue.put(message) async def close(self): - """Stop this handler's work in flight, when the CEM closes as its websocket tears down. + """Stop this handler's work in flight. - Defined here so that CEM.close(), which calls close() on every registered handler, works for any of them: - a handler lacking close() raised AttributeError inside CEM.close(), - killing the CEM's request handler and hanging the resource manager. + The CEM calls this when it closes, which happens when its websocket tears down. + CEM.close() calls close() on every registered handler, so every handler needs this method. + It is defined here for that reason. + Before, a handler without close() raised AttributeError inside CEM.close(). + That killed the CEM's request handler and left the resource manager hanging. """ self._logger.debug(f"Closing {self.__class__.__name__} handler") - # Work still in flight would otherwise go on after the session has ended, - # planning for a resource manager that is no longer there and racing the session that replaces it. - # That includes recording what the resource manager last reported, which is stopped too: - # a reading cut off mid-post is reported again by a resource manager that reconnects. - # Not every handler keeps background tasks, so a handler without them has nothing to stop. + # Otherwise, work in flight would continue after the session has ended. + # It would plan for a resource manager that is gone, and race the session that replaces it. + # Recording the resource manager's last reports is stopped too. + # That is safe: a resource manager that reconnects reports a reading again if its post was cut off. + # Not every handler keeps background tasks; a handler without them has nothing to stop. for task in list(getattr(self, "background_tasks", ())): task.cancel() diff --git a/src/flexmeasures_client/s2/control_types/translations.py b/src/flexmeasures_client/s2/control_types/translations.py index e61744db..54bc0158 100644 --- a/src/flexmeasures_client/s2/control_types/translations.py +++ b/src/flexmeasures_client/s2/control_types/translations.py @@ -58,10 +58,10 @@ def leakage_behaviour_to_storage_efficiency( """ - # FlexMeasures takes one storage efficiency here, for the whole window it covers. - # Where the fill level is known, the element that applies at it answers, as for a charging efficiency. - # Where it is not, the leakiest answers, looked for rather than assumed to come last, - # since S2 does not fix the order the elements are listed in, and the leakiest is not always the top one. + # FlexMeasures takes a single storage efficiency here, for the whole window it covers. + # If the fill level is known, use the element that applies at that fill level, as for a charging efficiency. + # If not, use the leakiest element. + # Search for it instead of assuming it comes last: S2 does not fix the order of the elements, and the leakiest one is not always the top one. if fill_level is not None: element = element_for_fill_level(message.elements, fill_level) else: @@ -70,13 +70,15 @@ def leakage_behaviour_to_storage_efficiency( e.fill_level_range.end_of_range * fill_level_scale for e in message.elements ) leakage = element.leakage_rate * fill_level_scale - # S2 states a leakage rate per second, so what remains after one second is compounded over the seconds in a step. - # This used to compound it over the hours in a step instead, - # which declared a buffer losing 1.5 kWh of 10 a day as holding its heat to within 0.002% over twelve hours, - # where it loses about 7%, and had the scheduler pre-charge in a cheap hour on the strength of it. - # FlexMeasures applies a storage efficiency to the stock as it stands, so the loss is a share of a fill level: - # of share_of where given, as the fill level a schedule starts from is, - # and otherwise of the top of the range, which understates it at lower fill levels. + # S2 gives a leakage rate per second. + # So the share that remains after one second is compounded over the number of seconds in a step. + # This used to be compounded over the number of hours in a step instead. + # Take a buffer that loses 1.5 kWh of 10 kWh a day: over twelve hours, it loses about 7%. + # The old calculation declared that it lost only 0.002% over those twelve hours. + # Because of that, the scheduler pre-charged the buffer in a cheap hour. + # FlexMeasures applies a storage efficiency to the current stock, so the loss is a share of a fill level. + # That fill level is share_of, if given, such as the fill level a schedule starts from. + # Otherwise, it is the top of the range, which understates the loss at lower fill levels. reference = ( share_of * fill_level_scale if share_of is not None and share_of > 0 diff --git a/src/flexmeasures_client/s2/measurements.py b/src/flexmeasures_client/s2/measurements.py index 5f154514..b5c9f443 100644 --- a/src/flexmeasures_client/s2/measurements.py +++ b/src/flexmeasures_client/s2/measurements.py @@ -1,4 +1,7 @@ -"""Recording what a resource manager measures on FlexMeasures sensors, one value per step of each sensor's resolution.""" +"""Record what a resource manager measures on FlexMeasures sensors. + +Each sensor gets one value per step of its resolution. +""" from __future__ import annotations @@ -19,17 +22,17 @@ _LOGGER = getLogger(__name__) -#: The unit S2 states every power in, measured ones included. +#: The unit of every power value in S2, including measured power. S2_POWER_UNIT = "W" -#: How long a recorded step is remembered, so that a resource manager sending it again is not recorded twice. +#: How long a recorded step is remembered. A step that a resource manager sends again within this time is not recorded twice. REMEMBERED_FOR = timedelta(days=7) -#: How often a step is tried before it is given up on, and how long to wait between tries. +#: How many times to try recording a step before giving up, and how long to wait between tries. ATTEMPTS = 3 RETRY_AFTER = timedelta(seconds=30) -#: What a post can fail with, short of a defect in this code. +#: The exceptions a post can raise when there is no defect in this code. POST_FAILURES = ( ValueError, ConnectionError, @@ -41,24 +44,30 @@ class MeasurementRecorder: - """Records measured power on a sensor as the average of what was measured in each step of its resolution. - - A step is recorded once it is over by the CEM's clock, as it stands when the step's last measurement arrives, - with the moment it ended as the moment it became known, - so a measurement is a belief formed at the end of what it measures, rather than a forecast of it. - Steps are taken by the time each measurement is stamped with, not by when it arrives, - since a co-simulation's resource manager sends a window's measurements together once the window has passed, - each stamped at the start of its quarter hour, while a live one sends them as they are taken, - in which case a step is recorded with the first measurement after it is over. - - Recording happens in the background, so that a resource manager is not kept waiting for FlexMeasures - before its measurement is acknowledged, one post at a time per sensor. - A step that could not be recorded is tried again a while later, and given up on after a few tries. - - Made once per server and handed to each CEM, it outlives the reconnections of a resource manager, - which a co-simulation makes every round, and so remembers what was recorded: - a step sent again is not recorded again, and the first value recorded for a step stands. - A sensor is meant to be one resource manager's: one that several share has their measurements averaged together. + """Record measured power on a sensor, as the average of the measurements in each step of its resolution. + + A step is recorded once it is over, according to the CEM's clock when the step's last measurement arrives. + The recorded belief time is the moment the step ended. + So a measurement is recorded as a belief formed at the end of the period it measures, not as a forecast of it. + + Each measurement is assigned to a step by its timestamp, not by its arrival time. + This matters because resource managers send measurements in different ways. + A co-simulation's resource manager sends a window's measurements together, once the window has passed. + It stamps each one at the start of its quarter hour. + A live resource manager sends each measurement when it is taken. + In that case, a step is recorded when the first measurement after the step arrives. + + Recording happens in the background, so the resource manager does not wait for FlexMeasures before its measurement is acknowledged. + Per sensor, only one post runs at a time. + If a step could not be recorded, it is tried again a while later. + After a few failed tries, it is given up on. + + The server creates one recorder and hands it to each CEM. + So the recorder outlives the reconnections of a resource manager, which a co-simulation makes every round. + The recorder therefore remembers what it recorded. + A step that is sent again is not recorded again, and the first value recorded for a step stands. + Each sensor is meant to belong to one resource manager. + If several resource managers share a sensor, their measurements are averaged together. """ def __init__(self, fm_client: FlexMeasuresClient, logger: Logger | None = None): @@ -67,16 +76,17 @@ def __init__(self, fm_client: FlexMeasuresClient, logger: Logger | None = None): self._resolutions: dict[int, timedelta] = {} self._pending: dict[int, dict[datetime, list[float]]] = defaultdict(dict) self._attempts: dict[int, dict[datetime, int]] = defaultdict(dict) - #: By sensor, the steps recorded, in the order they were, which is roughly that of time. + #: The recorded steps per sensor, in recording order, which is roughly chronological. self._recorded: dict[int, dict[datetime, None]] = defaultdict(dict) self._locks: dict[int, asyncio.Lock] = defaultdict(asyncio.Lock) self._tasks: set[asyncio.Task] = set() async def resolution(self, sensor_id: int) -> timedelta: - """The sensor's resolution, asked once, since it is a property of how the sensor was set up. + """Return the sensor's resolution. - A caller taking several measurements at once asks this for each sensor first, - so that none is taken where one of them cannot be. + The resolution is fetched from FlexMeasures only once, because it is fixed when the sensor is set up. + A caller that takes several measurements at once should first call this for each sensor. + Then, if any one sensor fails, no measurement is taken at all. """ if sensor_id not in self._resolutions: sensor = await self._fm_client.get_sensor(sensor_id) @@ -93,7 +103,7 @@ async def resolution(self, sensor_id: int) -> timedelta: async def add( self, sensor_id: int, measured_at: datetime, watts: float, now: datetime ) -> None: - """Take one measurement, and record in the background every step of this sensor that is over by now.""" + """Take one measurement, then record every step of this sensor that is over by now, in the background.""" resolution = await self.resolution(sensor_id) step = step_of(measured_at, resolution) if step in self._recorded[sensor_id]: @@ -105,10 +115,11 @@ async def add( self._in_background(self.record(sensor_id, now)) async def record(self, sensor_id: int, now: datetime) -> None: - """Record every pending step of this sensor that is over by now, each in a request of its own. + """Record every pending step of this sensor that is over by now, each in its own request. - Each with its own end as the moment it became known, which one request for several steps cannot say. - One at a time per sensor, so a step is not posted twice, nor taken from under a post in flight. + Each step needs its own end as its belief time, and a single request for several steps cannot express that. + Per sensor, only one call runs at a time. + This way, a step is never posted twice, and no step is removed while a post for it is in flight. """ async with self._locks[sensor_id]: resolution = await self.resolution(sensor_id) @@ -159,7 +170,7 @@ async def _retry(self, sensor_id: int, now: datetime) -> None: await self.record(sensor_id, now) async def settle(self) -> None: - """Wait for every recording in the background to finish, retries included.""" + """Wait for every background recording to finish, including retries.""" while self._tasks: await asyncio.gather(*list(self._tasks), return_exceptions=True) @@ -176,7 +187,10 @@ def _done(self, task: asyncio.Task) -> None: ) def _forget_before(self, sensor_id: int, cutoff: datetime) -> None: - """Forget recorded steps from before the cutoff, oldest first, so that this costs little when nothing is old.""" + """Forget recorded steps from before the cutoff. + + Steps are checked oldest first, so this costs little when no step is old. + """ recorded = self._recorded[sensor_id] while recorded: oldest = next(iter(recorded)) @@ -186,10 +200,12 @@ def _forget_before(self, sensor_id: int, cutoff: datetime) -> None: def step_of(moment: datetime, resolution: timedelta) -> datetime: - """The start of the step a moment falls in, on the moment's own clock, as FlexMeasures floors it. + """Return the start of the step that a moment falls in. - In the repeated hour of an autumn DST change a boundary exists twice, and the step is the later of the two - that is not after the moment; taken from the moment's pass alone, flooring cannot always be done at all. + The step start is in the moment's own timezone, floored the way FlexMeasures floors it. + During the repeated hour of an autumn DST change, a step boundary occurs twice. + The step then starts at the later of the two boundaries that is not after the moment. + Flooring based only on which of the two passes the moment is in cannot always be done. """ if not resolution: return moment diff --git a/src/flexmeasures_client/s2/script/__init__.py b/src/flexmeasures_client/s2/script/__init__.py index c777fdcf..35f6224c 100644 --- a/src/flexmeasures_client/s2/script/__init__.py +++ b/src/flexmeasures_client/s2/script/__init__.py @@ -1 +1,4 @@ -"""Example scripts: a CEM server and a resource manager that talk S2 to each other, run by path as the CEM documentation shows.""" +"""Example scripts: a CEM server and a resource manager that talk S2 to each other. + +Run them by path, as the CEM documentation shows. +""" diff --git a/src/flexmeasures_client/s2/script/websockets_client.py b/src/flexmeasures_client/s2/script/websockets_client.py index b73ce4c9..6edbbf69 100644 --- a/src/flexmeasures_client/s2/script/websockets_client.py +++ b/src/flexmeasures_client/s2/script/websockets_client.py @@ -1,12 +1,12 @@ -"""An example resource manager: a heat pump keeping a hot-water buffer warm, speaking S2's FRBC to a CEM. +"""An example resource manager: a heat pump that keeps a hot-water buffer warm, and talks S2's FRBC to a CEM. Run it by path, next to the example CEM server, as the CEM documentation shows. -It declares what the heat pump can do and what its buffer needs, then simulates that buffer: -it follows each instruction from the moment it is due, and reports its fill level every so often, -which is what asks the CEM for a new schedule. +It declares what the heat pump can do and what its buffer needs, and then simulates that buffer. +The simulation follows each instruction from the moment it is due. +Every so often, it reports its fill level, which triggers the CEM to plan a new schedule. -The buffer is simulated from S2's own definitions, and not with the CEM's code, -so that it is a second opinion on where the instructions take the storage rather than a copy of the first. +The buffer is simulated from S2's own definitions, not with the CEM's code. +That way, the simulation is an independent check on where the instructions take the storage, not a copy of the CEM's own calculation. """ from __future__ import annotations @@ -58,20 +58,20 @@ #: The range the buffer's fill level should stay in, in kilowatt hours of stored heat. FILL_LEVEL_RANGE = (2.0, 10.0) -#: Below this fill level the water is cold enough for the heat pump to run at a COP of 4; above it, 3. +#: Below this fill level, the water is cold enough for the heat pump to run at a COP of 4. Above it, the COP is 3. COLD_BELOW = 6.0 -#: What the buffer holds when the simulation starts, and what it can physically hold at most. +#: The buffer's fill level when the simulation starts, and the most it can physically hold. STARTING_FILL_LEVEL = 5.0 PHYSICAL_CAPACITY = 12.0 #: How often the simulated buffer reports its fill level, in seconds of wall-clock time. STATUS_INTERVAL = float(os.getenv("S2_EXAMPLE_STATUS_INTERVAL", "300")) -#: How often the forecasts are sent again, each covering the day from then on, so that planning never runs out of them. +#: How often the forecasts are sent again, each covering the next day, so that planning never runs out of them. FORECAST_INTERVAL = float(os.getenv("S2_EXAMPLE_FORECAST_INTERVAL", "3600")) ELECTRIC_POWER = CommodityQuantity.ELECTRIC_POWER_3_PHASE_SYMMETRIC def heat(watts: float) -> float: - """A rate of heat in watts as a fill rate in kilowatt hours per second, which is how S2 states a fill rate.""" + """Convert a heat rate in watts into a fill rate in kilowatt-hours per second, which is how S2 expresses a fill rate.""" return watts / 3.6e6 @@ -101,7 +101,7 @@ def _element( def operation_modes() -> list[FRBCOperationMode]: - """Idle, or heating at anything from 400 to 1500 W, at a COP of 4 while the buffer is cold and 3 once it is warm.""" + """Return two modes: idle, and heating at 400 to 1500 W, with a COP of 4 while the buffer is cold and 3 once it is warm.""" idle = FRBCOperationMode( id=get_unique_id(), diagnostic_label="idle", @@ -123,8 +123,8 @@ def operation_modes() -> list[FRBCOperationMode]: def system_description( modes: list[FRBCOperationMode], valid_from: datetime ) -> FRBCSystemDescription: - # S2 lists the transitions an actuator can make, so the heat pump says it can go from either mode to the other, - # at once and at no cost. + # S2 lists the transitions an actuator can make. + # This heat pump declares that it can switch from either mode to the other, immediately and at no cost. transitions = [ Transition( id=get_unique_id(), @@ -165,7 +165,7 @@ def system_description( def leakage_behaviour(valid_from: datetime) -> FRBCLeakageBehaviour: - """A warmer buffer loses more: 0.8 kWh a day while cold, 1.5 kWh a day once warm.""" + """Return the leakage behaviour: a warmer buffer loses more, 0.8 kWh a day while cold, and 1.5 kWh a day once warm.""" return FRBCLeakageBehaviour( message_id=get_unique_id(), valid_from=valid_from, @@ -185,12 +185,12 @@ def leakage_behaviour(valid_from: datetime) -> FRBCLeakageBehaviour: def expected_usage(hour: int) -> float: - """Hot water drawn in a given hour of the day, in kWh: showers in the morning, washing up in the evening.""" + """Return the hot water drawn in a given hour of the day, in kWh: showers in the morning, washing up in the evening.""" return {7: 3.0, 19: 1.0, 20: 1.0}.get(hour, 0.1) def usage_forecast(start: datetime) -> FRBCUsageForecast: - """A day of hot-water use, hour by hour, from the hour start falls in.""" + """Return a forecast for a day of hot-water use, hour by hour, from the start of the hour that start falls in.""" start = start.replace(minute=0, second=0, microsecond=0) return FRBCUsageForecast( message_id=get_unique_id(), @@ -206,7 +206,7 @@ def usage_forecast(start: datetime) -> FRBCUsageForecast: def fill_level_target_profile(start: datetime) -> FRBCFillLevelTargetProfile: - """Kept within the declared range, and at least 6 kWh in the hour before the morning showers.""" + """Return a target profile within the declared range: at least 3 kWh, and at least 6 kWh in the hour before the morning showers.""" start = start.replace(minute=0, second=0, microsecond=0) return FRBCFillLevelTargetProfile( message_id=get_unique_id(), @@ -226,12 +226,12 @@ def fill_level_target_profile(start: datetime) -> FRBCFillLevelTargetProfile: @dataclass class Buffer: - """The simulated buffer: where its fill level goes, following S2's definitions. + """The simulated buffer, whose fill level changes according to S2's definitions. - S2 states every rate per second, in the fill level's own units: - a fill rate raises the fill level, and usage and leakage lower it. - An operation mode's fill rate is interpolated between the ends of its element's range by the factor, - from the element whose fill-level range holds the fill level. + S2 gives every rate per second, in the fill level's own units. + A fill rate raises the fill level, and usage and leakage lower it. + An operation mode's fill rate comes from the element whose fill-level range holds the fill level. + It is interpolated between the ends of that element's fill-rate range, using the factor. """ modes: dict[str, FRBCOperationMode] @@ -243,7 +243,11 @@ class Buffer: @staticmethod def _holding(elements, fill_level: float): - """The element whose range holds the fill level, the upper one at a shared boundary, the nearest outside.""" + """Return the element whose range holds the fill level. + + At a shared boundary, the upper element applies. + Outside all ranges, the nearest element applies. + """ lowest = min(e.fill_level_range.start_of_range for e in elements) highest = max(e.fill_level_range.end_of_range for e in elements) level = min(max(fill_level, lowest), highest) @@ -262,7 +266,7 @@ def _holding(elements, fill_level: float): ) def rates(self, moment: datetime) -> tuple[float, float, float]: - """Fill, usage and leakage, each per second, at this fill level and moment.""" + """Return the fill, usage and leakage rates, each per second, at this fill level and moment.""" fill = 0.0 if self.mode_id is not None: element = self._holding(self.modes[self.mode_id].elements, self.fill_level) @@ -273,7 +277,7 @@ def rates(self, moment: datetime) -> tuple[float, float, float]: return fill, usage, leakage def advance(self, until: datetime, step: timedelta = timedelta(minutes=1)) -> None: - """Move the fill level forward to a moment, a minute at a time, so that an element boundary is crossed in time.""" + """Move the fill level forward to a moment, a minute at a time, so that element boundaries are crossed at the right time.""" while self.at < until: dt = min(step, until - self.at) fill, usage, leakage = self.rates(self.at) @@ -290,7 +294,10 @@ def follow(self, instruction: FRBCInstruction, moment: datetime) -> None: async def send_instruction_status( ws, instruction_id, status: InstructionStatus ) -> None: - """Tell the CEM where an instruction stands, which is what it revokes by: only one not yet carried out.""" + """Send the CEM the status of an instruction. + + The CEM uses these statuses to decide what to revoke: it only revokes instructions that are not yet carried out. + """ await ws.send_str( InstructionStatusUpdate( message_id=get_unique_id(), @@ -310,14 +317,14 @@ async def apply_at_execution_time( ) -> None: """Enter the operation mode an instruction asks for, at the moment it asks for it. - An instruction carries an execution_time, - which is when the resource manager is to be in that mode, not when the instruction happens to arrive. - A resource manager that answers on arrival reports a mode it is not in yet, - and a customer energy manager recording that status has a measurement of a power the device is not drawing, + An instruction has an execution_time. + That is when the resource manager should be in that mode, not when the instruction happens to arrive. + A resource manager that acts on arrival reports a mode it is not in yet. + A customer energy manager that records that status then has a measurement of a power the device is not drawing, timestamped now. - The mode it reports leaving is the one the last instruction carried out put it in, kept in applied, - rather than whatever the instruction before it asked for, which may have been revoked and never carried out. + The mode it reports leaving is the mode the last executed instruction put it in, which is kept in applied. + It is not the mode the previous instruction asked for, because that instruction may have been revoked and never carried out. """ delay = (instruction.execution_time - datetime.now(timezone.utc)).total_seconds() @@ -325,7 +332,7 @@ async def apply_at_execution_time( print(f"WAITING: {delay:.0f}s until this instruction is due") await asyncio.sleep(delay) - # From here on it is being carried out, so a revocation arriving now is too late to stop it. + # From here on, the instruction is being carried out, so a revocation that arrives now is too late to stop it. if on_due is not None: on_due() now = datetime.now(timezone.utc) @@ -351,8 +358,9 @@ async def apply_at_execution_time( async def answer_instructions(ws, buffer: Buffer | None = None) -> None: """Acknowledge every instruction, and report the transition each one asks for when it is due. - An instruction the CEM revokes, because a newer plan replaces the one it belonged to, is not followed: - without that, the buffer would end up in whatever the last instruction to fire across every plan ever sent said. + The CEM revokes an instruction when a newer plan replaces the plan it belonged to. + A revoked instruction is not followed. + Otherwise, the buffer would end up in whatever mode the last instruction to fire said, across every plan ever sent. """ applied: dict = {"mode": None} @@ -405,7 +413,7 @@ def forget(done: asyncio.Task, key: str = key) -> None: async def report_fill_level( ws, buffer: Buffer, interval: float = STATUS_INTERVAL ) -> None: - """Report where the buffer is every interval, which is what asks the CEM for a new schedule.""" + """Report the buffer's fill level every interval, which triggers the CEM to plan a new schedule.""" while True: await asyncio.sleep(interval) buffer.advance(datetime.now(timezone.utc)) @@ -420,9 +428,9 @@ async def report_fill_level( async def refresh_forecasts( ws, leakage: FRBCLeakageBehaviour, interval: float = FORECAST_INTERVAL ) -> None: - """Send the leakage, the usage forecast and the comfort profile again every interval, each from now on. + """Send the leakage behaviour, the usage forecast and the comfort profile again every interval, each starting now. - Sent only once, they would cover the first day, and the CEM would plan without them from then on. + If they were sent only once, they would only cover the first day, and after that the CEM would plan without them. """ while True: await asyncio.sleep(interval) @@ -439,7 +447,7 @@ async def refresh_forecasts( async def receive_until(ws, message_type: str) -> dict: - """The next message of a type, past the reception statuses that arrive in between.""" + """Return the next message of a type, skipping the reception statuses that arrive in between.""" while True: message = (await ws.receive()).json() print("RECEIVED: ", message) @@ -450,10 +458,10 @@ async def receive_until(ws, message_type: str) -> dict: async def run_resource_manager( ws, interval: float = STATUS_INTERVAL, forecast_interval: float = FORECAST_INTERVAL ) -> Buffer: - """Introduce the heat pump, declare what it can do and what its buffer needs, then follow and report. + """Introduce the heat pump, declare what it can do and what its buffer needs, then follow instructions and report. - In the order S2 suggests: the system description first, then the storage's status. - Returns the simulated buffer, for whoever wants to look at where the instructions took it. + The messages follow the order that S2 suggests: the system description first, then the storage's status. + Returns the simulated buffer, so that a caller can inspect where the instructions took it. """ await ws.send_str( Handshake( @@ -469,7 +477,7 @@ async def run_resource_manager( message_id=get_unique_id(), resource_id=get_unique_id(), name="heat pump with a hot-water buffer", - # It draws electricity and stores heat, so it is a consumer of the commodity it is planned in. + # It draws electricity and stores heat, so it consumes the commodity it is planned in. roles=[ Role(role=RoleType.ENERGY_CONSUMER, commodity=Commodity.ELECTRICITY) ], @@ -478,8 +486,8 @@ async def run_resource_manager( ControlType.FILL_RATE_BASED_CONTROL, ControlType.NO_SELECTION, ], - # It sends no PowerForecast. It could measure its power, which is what the next field says, - # and S2 requires at least one type there. + # It sends no PowerForecast. + # It could measure its power, which is what the next field declares, and S2 requires at least one type there. provides_forecast=False, provides_power_measurement_types=[ELECTRIC_POWER], ).model_dump_json() diff --git a/src/flexmeasures_client/s2/script/websockets_server.py b/src/flexmeasures_client/s2/script/websockets_server.py index 591540bd..80d736ec 100644 --- a/src/flexmeasures_client/s2/script/websockets_server.py +++ b/src/flexmeasures_client/s2/script/websockets_server.py @@ -35,8 +35,8 @@ SITE_NAME = "My CEM" SITE_TIMEZONE = "Europe/Amsterdam" -# Enough days of demo prices to cover what the CEM plans for, which is twelve hours from now, -# with the seeding repeated every SEED_PRICES_EVERY so that a demo left running never runs out. +# Enough days of demo prices to cover the CEM's planning horizon, which is twelve hours from now. +# Seeding is repeated every SEED_PRICES_EVERY, so that a demo left running never runs out of prices. SEED_PRICE_DAYS = 3 SEED_PRICES_EVERY = timedelta(hours=12) @@ -47,17 +47,17 @@ "yes", ) -#: Each sensor the site needs, which is the library's table of what an FRBC site records. +#: The sensors the site needs, taken from the library's table of sensors for an FRBC site. SITE_SENSORS = FRBC_SITE_SENSORS -#: What every site asset is made with: a building, whose connection takes three phases of 25 A at 230 V. +#: The specs to create every site asset with: a building with a three-phase connection of 25 A per phase at 230 V. SITE_ASSET_SPECS: dict[str, Any] = dict( generic_asset_type_id=6, flex_model={"power-capacity": f"{3 * 25 * 230} VA"}, ) -#: Whether each resource manager gets a site of its own, named after it, as in a co-simulation of a community, -#: rather than all of them sharing the one site this server sets up at startup. +#: Whether each resource manager gets its own site, named after it, as in a co-simulation of a community. +#: If not, they all share the one site that this server sets up at startup. SITE_PER_RESOURCE_MANAGER = os.getenv("S2_SITE_PER_RESOURCE_MANAGER", "").lower() in ( "1", "true", @@ -92,8 +92,8 @@ async def rm_details_watchdog(ws, cem: CEM): cem._logger.debug(f"CONTROL TYPE: {cem._control_type}") - # From here on, a schedule is asked for on each storage status, - # and on a system description that arrives after one. + # From here on, each storage status triggers a schedule request. + # So does a system description that arrives after a storage status. async def websocket_producer(ws, cem: CEM): @@ -168,22 +168,22 @@ async def serve_resource_manager( ) -> None: """Act as the CEM for one resource manager connected over ws, until it disconnects. - On the site whose sensors are given, or, with a configurator, on a site of the resource manager's own, - which is set up once its details arrive. + The CEM uses the site whose sensors are given. + With a configurator, it uses the resource manager's own site instead, which is set up once its details arrive. """ cem = CEM( fm_client=fm_client, logger=LOGGER, site_configurator=configurator, - # The server's, which outlives a connection, so a resource manager that reconnects is not recorded twice. + # The server's recorder outlives a connection, so a resource manager that reconnects is not recorded twice. measurement_recorder=recorder, - # Where what the resource manager measures is recorded, on the fixed site; a configured site wires its own. + # The sensor to record the resource manager's measurements on, for the fixed site. A configured site wires its own. power_sensor_id=( {"ELECTRIC.POWER.L1": sensors["measured-power"]["id"]} if sensors is not None and "measured-power" in sensors else None ), - # A co-simulation's resource manager sends its messages wrapped with the simulated time, which this sets the clock to. + # A co-simulation's resource manager wraps its messages with the simulated time, and this sets the clock to that time. accept_simulation_metadata=ACCEPT_SIMULATION_METADATA, ) if sensors is not None: @@ -192,13 +192,13 @@ async def serve_resource_manager( def fixed_site_handler(sensors: dict[str, dict]) -> FRBCSimple: - """The handler for a resource manager on the one site this server sets up at startup.""" + """Return the handler for a resource manager on the one site that this server sets up at startup.""" frbc = FRBCSimple( power_sensor_id=sensors["power"]["id"], price_sensor_id=sensors["price"]["id"], soc_sensor_id=sensors["state of charge"]["id"], rm_discharge_sensor_id=sensors["RM discharge"]["id"], - # Where the resource manager's declarations are recorded, so that each can be seen next to the schedule planned against it. + # Record the resource manager's declarations, so that each can be inspected next to the schedule planned against it. fill_rate_sensor_id=sensors["fill rate"]["id"], soc_minima_sensor_id=sensors["soc minima"]["id"], soc_maxima_sensor_id=sensors["soc maxima"]["id"], @@ -206,20 +206,21 @@ def fixed_site_handler(sensors: dict[str, dict]) -> FRBCSimple: leakage_behaviour_sensor_id=sensors["storage efficiency"]["id"], charging_efficiency_sensor_id=sensors["charging efficiency"]["id"], timezone=SITE_TIMEZONE, - # Plan from now, rather than from a day after the description, so that a demo shows its instructions being followed. + # Plan from now, not from a day after the description, so that a demo shows its instructions being followed. valid_from_shift=timedelta(0), ) - # A newer plan revokes what is left of the one before, so the resource manager does not follow both. + # A newer plan revokes the remainder of the previous plan, so the resource manager does not follow both. frbc.send_revocations = True return frbc async def serve(ws, cem: CEM) -> None: """Serve one resource manager with this CEM, for as long as its connection lasts.""" - # The session lasts as long as the resource manager's messages do. - # When they end, the CEM is closed, which stops what its handlers still have in flight, - # and the producer and the watchdog stop with it: left running, they outlived the connection, - # and a resource manager that reconnected was served by two sessions planning for it at once. + # The session lasts as long as the resource manager keeps sending messages. + # When the messages end, the CEM is closed, which stops any work its handlers still have in flight. + # The producer and the watchdog stop too. + # When they were left running, they outlived the connection. + # A resource manager that reconnected was then served by two sessions, both planning for it at once. consumer = asyncio.create_task(websocket_consumer(ws, cem)) side_tasks = [ asyncio.create_task(websocket_producer(ws, cem)), @@ -244,7 +245,7 @@ async def configure_site( asset_specs=SITE_ASSET_SPECS, sensors=SITE_SENSORS, timezone=SITE_TIMEZONE, - # The demo's own graphs, on every start, so changes made to them in the UI do not outlive a restart. + # Reset the demo's graphs on every start, so that changes made to them in the UI do not survive a restart. reset_graphs=True, ) await seed_prices(fm_client, sensors["price"]["id"]) @@ -252,12 +253,12 @@ async def configure_site( async def seed_prices(fm_client: FlexMeasuresClient, price_sensor_id: int) -> None: - """Demo prices, so that a site has something to schedule against. + """Post demo prices, so that a site has something to schedule against. - One day's hourly shape, repeated over SEED_PRICE_DAYS from this morning, because the CEM - plans ahead of the moment a resource manager reports its storage: - a fixed date here goes stale the day after it is written, and the scheduler then fails the - job with UnknownPricesException rather than returning a schedule. + The prices repeat one day's hourly shape over SEED_PRICE_DAYS, starting this morning. + The CEM plans ahead of the moment a resource manager reports its storage. + A fixed date here would go stale the day after it was written. + The scheduler would then fail the job with UnknownPricesException, instead of returning a schedule. """ one_day = [ 0.10, 0.11, 0.12, 0.15, 0.18, 0.17, 0.11, 0.09, @@ -277,9 +278,10 @@ async def seed_prices(fm_client: FlexMeasuresClient, price_sensor_id: int) -> No async def keep_prices_seeded(app: web.Application): - """Seed the prices again every SEED_PRICES_EVERY for as long as the server runs, so a demo never runs out of them. + """Seed the prices again every SEED_PRICES_EVERY for as long as the server runs, so that a demo never runs out of them. - Not where each resource manager has a site of its own, whose prices are whoever runs them to post. + This is skipped if each resource manager has its own site. + Then whoever runs those resource managers is responsible for posting their prices. """ if SITE_PER_RESOURCE_MANAGER: yield @@ -324,16 +326,16 @@ async def configure_site_once(app: web.Application) -> None: app[FM_CLIENT] = fm_client app[RECORDER] = MeasurementRecorder(fm_client, logger=LOGGER) if SITE_PER_RESOURCE_MANAGER: - # One configurator for the server's lifetime, so a resource manager that reconnects finds its site again by id. + # Use one configurator for the server's lifetime, so that a resource manager that reconnects finds its site again by id. app[CONFIGURATOR] = FRBCSiteConfigurator( fm_client, asset_type_id=SITE_ASSET_SPECS["generic_asset_type_id"], asset_specs={"flex_model": SITE_ASSET_SPECS["flex_model"]}, timezone=SITE_TIMEZONE, - # The co-simulation's resource managers are heat pumps, which cannot feed power back: - # FlexMeasures then keeps their production at zero even where constraints are relaxed. + # The co-simulation's resource managers are heat pumps, which cannot feed power back. + # With this attribute, FlexMeasures keeps their production at zero, even where constraints are relaxed. power_sensor_attributes={"is_strictly_non_positive": True}, - # A community's capacity is a target set by its members' steering, breached at a price, as #188 asked for. + # Each member's site capacity is a target set by the community that steers it, so it may be breached at a price, as #188 asked for. handler_options={"relax_site_capacity": True}, logger=LOGGER, ) diff --git a/src/flexmeasures_client/s2/site.py b/src/flexmeasures_client/s2/site.py index f94185f0..b7648518 100644 --- a/src/flexmeasures_client/s2/site.py +++ b/src/flexmeasures_client/s2/site.py @@ -1,8 +1,9 @@ -"""A FlexMeasures site for an S2 resource manager: its asset, its sensors, its graphs, and the handler wired to them. +"""Set up a FlexMeasures site for an S2 resource manager. +A site consists of an asset, its sensors and its graphs, plus a handler wired to those sensors. The example server keeps one site for whichever resource manager connects (see ensure_site). -A CEM serving many resource managers, as a co-simulation of a community does, gives each one a site of its own, -named after it, with FRBCSiteConfigurator. +A CEM can also serve many resource managers, as in a co-simulation of a community. +Such a CEM uses FRBCSiteConfigurator to give each resource manager its own site, named after it. """ from __future__ import annotations @@ -21,6 +22,7 @@ ) from flexmeasures_client.client import FlexMeasuresClient +from flexmeasures_client.exceptions import ContentTypeError from flexmeasures_client.s2.control_types.FRBC.frbc_simple import FRBCSimple _LOGGER = getLogger(__name__) @@ -28,7 +30,7 @@ @dataclass(frozen=True) class SiteSensor: - """What a site's sensor is made with: its unit, its resolution, and any attributes.""" + """The settings for creating a site's sensor: its unit, its resolution, and any attributes.""" unit: str resolution: str @@ -36,38 +38,39 @@ class SiteSensor: #: The sensors an FRBC site needs, by name. -#: Their units are the ones the CEM records in: fill levels in kWh, rates of fill level per second, as S2 states them. -#: The names are a contract with whoever reads the site, a co-simulation among them, -#: which looks up "price", "production price", "power", "state of charge" and "measured-power" by name. +#: Their units suit a residential site, and match the handler's defaults: energy and fill levels in kWh, prices per kWh, and powers and rates of fill level in W. +#: The names are a contract with any reader of the site, such as a co-simulation. +#: A co-simulation looks up "price", "production price", "power", "state of charge" and "measured-power" by name. FRBC_SITE_SENSORS: dict[str, SiteSensor] = { "price": SiteSensor("EUR/kWh", "PT15M"), "production price": SiteSensor("EUR/kWh", "PT15M"), - # S2 states a power as positive when it is consumed, - # and this sensor holds both an actuator's power and the schedule planned for it. - # Left unsaid, FlexMeasures reads a power sensor the other way round, - # and flips a schedule's sign on the way in, which puts the two in opposite directions. - "power": SiteSensor("kW", "PT15M", {"consumption_is_positive": True}), - # What the resource manager measures its site drawing, kept apart from the schedule on "power". - "measured-power": SiteSensor("kW", "PT15M", {"consumption_is_positive": True}), + # In S2, consumed power is positive. + # This sensor holds both an actuator's power and the schedule planned for it. + # Without this attribute, FlexMeasures treats production as positive on a power sensor. + # It would then flip the sign of an incoming schedule, so the power and the schedule would point in opposite directions. + "power": SiteSensor("W", "PT15M", {"consumption_is_positive": True}), + # The power the resource manager measures its site drawing, kept separate from the schedule on "power". + "measured-power": SiteSensor("W", "PT15M", {"consumption_is_positive": True}), "state of charge": SiteSensor("kWh", "PT0M"), "RM discharge": SiteSensor("dimensionless", "PT15M"), - "fill rate": SiteSensor("kWh/s", "PT15M"), + "fill rate": SiteSensor("W", "PT15M"), "soc minima": SiteSensor("kWh", "PT15M"), "soc maxima": SiteSensor("kWh", "PT15M"), - "usage forecast": SiteSensor("kWh/s", "PT15M"), + "usage forecast": SiteSensor("W", "PT15M"), "storage efficiency": SiteSensor("%", "PT15M"), "charging efficiency": SiteSensor("dimensionless", "PT15M"), } def frbc_site_graphs(sensors: dict[str, dict]) -> list[dict]: - """A site's graphs, top to bottom, as FlexMeasures takes an asset's sensors_to_show. + """Return a site's graphs, top to bottom, in the format of an asset's sensors_to_show in FlexMeasures. - In the form with a title and a list of sensors per graph, which every FlexMeasures version reads, - rather than the newer one with plots, which needs v0.32 or later and draws the same graphs. - A graph whose sensors the site does not have is left out. + Each graph has a title and a list of sensors. + Every FlexMeasures version reads this format. + The newer format with plots draws the same graphs, but needs FlexMeasures v0.32 or later. + A graph is left out if the site has none of its sensors. """ - # Titled as #188 titled them, since a co-simulation's controller adds to the graph titled "Power". + # Keep the titles from #188, because a co-simulation's controller adds to the graph titled "Power". graphs = [ ("Prices", ["price", "production price"]), ("Power", ["power", "measured-power"]), @@ -92,16 +95,17 @@ async def ensure_site( known_asset_id: int | None = None, reset_graphs: bool = False, ) -> tuple[dict, dict[str, dict]]: - """Find or make the site asset of this name, with its specs and every one of these sensors, and its graphs. + """Find or create the site asset with this name, with its specs, all of these sensors, and its graphs. - It is safe to run again: what exists is kept, and what is missing is made. - A flex model in the specs is merged into the one the asset has, since others write to it too, - such as a co-simulation's controller setting what its stock is worth at the end. - Graphs are set where the asset has none, since others add to them too, unless asked to reset them. + Running this again is safe: existing parts are kept, and missing parts are created. + A flex model in the specs is merged into the asset's existing flex model, because others write to it too. + For example, a co-simulation's controller sets the value of the asset's stock at the end. + Graphs are only set if the asset has none, because others add to them too. + Pass reset_graphs to set them anyway. - :param known_asset_id: the asset's id where it was found before, which spares a listing of every asset. - :param reset_graphs: whether to set the graphs even where the asset has some. - :returns: the asset, and its sensors by name. + :param known_asset_id: The asset's id from an earlier lookup, which avoids listing every asset. + :param reset_graphs: Whether to set the graphs even if the asset already has some. + :returns: The asset, and its sensors by name. """ site_asset = None if known_asset_id is not None: @@ -109,12 +113,12 @@ async def ensure_site( candidate = await fm_client.get_asset( known_asset_id, parse_json_fields=True ) - except (ValueError, ConnectionError): + except (ValueError, ConnectionError, ContentTypeError): candidate = None if candidate is not None and candidate.get("name") == name: site_asset = candidate if site_asset is None: - # The newest of any of that name, as a co-simulation picks it too. + # Pick the newest asset with that name, as a co-simulation does too. named = [ asset for asset in await fm_client.get_assets(parse_json_fields=True) @@ -162,20 +166,20 @@ async def ensure_site( @dataclass class ConfiguredSite: - """What a CEM needs to serve one resource manager: its handler, and where its measured power goes.""" + """What a CEM needs to serve one resource manager: its handler, and the sensors to record its measured power on.""" handler: FRBCSimple - #: The sensor each commodity quantity's measured power is recorded on, as the CEM's power_sensor_id takes it. + #: Per commodity quantity, the sensor to record measured power on, in the format of the CEM's power_sensor_id. power_sensor_ids: dict[str, int] asset_id: int class FRBCSiteConfigurator: - """Gives each resource manager a site of its own, named after it, and an FRBC handler wired to its sensors. + """Give each resource manager its own site, named after it, and an FRBC handler wired to the site's sensors. - Made once per server and handed to each CEM it makes (see CEM's site_configurator), - so that a resource manager that reconnects finds its site by the id remembered here, - rather than by a listing of every asset on the server. + The server creates one configurator and hands it to each CEM it creates (see the CEM's site_configurator). + The configurator remembers each site's asset id. + So when a resource manager reconnects, its site is found by that id, without listing every asset on the server. """ def __init__( @@ -192,15 +196,15 @@ def __init__( logger: Logger | None = None, ) -> None: """ - :param asset_type_id: the FlexMeasures asset type each site is made with. - :param timezone: the time zone of each site's sensors, and of its handler. - :param asset_specs: anything else each site asset is made or updated with, such as a flex model. - :param sensors: the sensors each site gets; FRBC_SITE_SENSORS by default. - :param power_sensor_attributes: attributes added to the "power" sensor's, which a device's nature decides, + :param asset_type_id: The FlexMeasures asset type to create each site with. + :param timezone: The timezone of each site's sensors, and of its handler. + :param asset_specs: Other specs to create or update each site asset with, such as a flex model. + :param sensors: The sensors each site gets; FRBC_SITE_SENSORS by default. + :param power_sensor_attributes: Extra attributes for the "power" sensor that depend on the device, such as that it cannot produce. - :param handler_options: keyword arguments for each FRBCSimple beyond its sensors. - By default it plans from now, over 24 hours, as a co-simulation steps through days. - :param send_revocations: whether a newer plan revokes what is left of the one before. + :param handler_options: Keyword arguments for each FRBCSimple, other than its sensors. + By default, it plans from now over 24 hours, because a co-simulation steps through days. + :param send_revocations: Whether a newer plan revokes the remainder of the previous plan. """ self._fm_client = fm_client self._timezone = timezone diff --git a/src/flexmeasures_client/s2/utils.py b/src/flexmeasures_client/s2/utils.py index cfdcd55e..86e6f7d9 100644 --- a/src/flexmeasures_client/s2/utils.py +++ b/src/flexmeasures_client/s2/utils.py @@ -28,15 +28,25 @@ def __init__(self, *args, max_size=100, **kwargs): super(SizeLimitOrderedDict, self).__init__(*args, **kwargs) self._max_size = max_size - # deleting values to make the dictionary of length _max_size at most + # Delete the oldest entries until the dictionary holds at most _max_size entries. while len(self) > self._max_size: - self.popitem() + self.popitem(last=False) def __setitem__(self, __key: KT, __value: VT) -> None: - if len(self) == self._max_size: - self.popitem() - - return super().__setitem__(__key, __value) + # Evict the oldest entry, not the newest (which is what popitem() removes by default). + # Popping the newest instead made a full history keep a session's first entries, and lose all later ones except the last. + # A key that is already present needs no extra room. + if ( + self._max_size is not None + and __key not in self + and len(self) >= self._max_size + ): + self.popitem(last=False) + + super().__setitem__(__key, __value) + # An entry that is set again becomes the latest entry. + # Everywhere this class is used, reading the last entry means reading the latest one. + self.move_to_end(__key) def get_unique_id() -> str: @@ -63,7 +73,7 @@ def get_message_id(message: pydantic.BaseModel) -> str | None: class _HasMessageId(Protocol): - """Any S2 message: every one of them carries a message id.""" + """Any S2 message, since every S2 message carries a message id.""" message_id: Any @@ -71,13 +81,18 @@ class _HasMessageId(Protocol): def get_reception_status( subject_message: _HasMessageId, status: ReceptionStatusValues = ReceptionStatusValues.OK, + diagnostic_label: str | None = None, ): """ This function returns a ReceptionStatus message for the subject message `subject_message`. By default, the status ReceptionStatusValues.OK is sent. + + :param diagnostic_label: For a status other than OK, the reason, which the resource manager can log. """ return ReceptionStatus( - subject_message_id=str(subject_message.message_id), status=status + subject_message_id=str(subject_message.message_id), + status=status, + diagnostic_label=diagnostic_label, ) diff --git a/tests/client/test_sensor.py b/tests/client/test_sensor.py index d8181e96..8c1781de 100644 --- a/tests/client/test_sensor.py +++ b/tests/client/test_sensor.py @@ -16,6 +16,7 @@ from flexmeasures_client.exceptions import ( InsufficientServerVersionError, JobFailedError, + JobTimeoutError, ) @@ -1069,6 +1070,66 @@ async def test_post_sensor_data_await_ingestion_timeout(caplog): await client.close() +@pytest.mark.asyncio +async def test_post_sensor_data_confirm_ingestion_raises_on_timeout(caplog): + """A caller that remembers having recorded the data is told it may not have been, instead of carrying on.""" + with aioresponses() as m: + client = FlexMeasuresClient(email="test@test.test", password="test") + client.access_token = "test-token" + client.ingestion_polling_timeout = 0.05 + client.ingestion_polling_interval = 0.01 + sensor_id = 5 + m.post( + f"http://localhost:5000/api/v3_0/sensors/{sensor_id}/data", + status=202, + payload={"status": "ACCEPTED", "job": "job-4"}, + ) + m.get( + "http://localhost:5000/api/v3_0/jobs/job-4", + status=200, + payload={"status": "QUEUED", "message": "waiting", "result": None}, + repeat=True, + ) + + with pytest.raises(JobTimeoutError): + await client.post_sensor_data( + sensor_id=sensor_id, + start="2023-01-01T00:00+00:00", + duration="PT1H", + values=[1.0], + unit="MW", + confirm_ingestion=True, + ) + assert any("unconfirmed" in record.message for record in caplog.records) + await client.close() + + +@pytest.mark.asyncio +async def test_post_sensor_data_confirm_ingestion_raises_where_the_job_cannot_be_looked_up(): + with aioresponses() as m: + client = FlexMeasuresClient(email="test@test.test", password="test") + client.access_token = "test-token" + client.request_retry_interval = 0.01 + sensor_id = 5 + m.post( + f"http://localhost:5000/api/v3_0/sensors/{sensor_id}/data", + status=202, + payload={"status": "ACCEPTED", "job": "job-5"}, + ) + m.get("http://localhost:5000/api/v3_0/jobs/job-5", status=404, repeat=True) + + with pytest.raises((ConnectionError, ValueError)): + await client.post_sensor_data( + sensor_id=sensor_id, + start="2023-01-01T00:00+00:00", + duration="PT1H", + values=[1.0], + unit="MW", + confirm_ingestion=True, + ) + await client.close() + + @pytest.mark.asyncio async def test_post_sensor_data_synchronous_response_is_not_polled(): """A synchronous 200 OK answer names no job, so nothing is polled.""" @@ -1079,9 +1140,9 @@ async def test_post_sensor_data_synchronous_response_is_not_polled(): m.post( f"http://localhost:5000/api/v3_0/sensors/{sensor_id}/data", status=200, - payload={"status": "PROCESSED", "message": "ok"}, + # Naming a job all the same, so that only the status keeps it from being polled. + payload={"status": "PROCESSED", "message": "ok", "job": "job-9"}, ) - # No job endpoint is registered, so a poll would fail the test by raising here. response, status = await client.post_sensor_data( sensor_id=sensor_id, @@ -1092,6 +1153,7 @@ async def test_post_sensor_data_synchronous_response_is_not_polled(): ) assert status == 200 + assert not any(method == "GET" for method, _ in m.requests), "nothing is polled" await client.close() @@ -1112,7 +1174,8 @@ async def test_post_sensor_data_await_ingestion_false_is_not_polled(): "job": "job-4", }, ) - # No job endpoint is registered, so a poll would fail the test by raising here. + # An unregistered job endpoint would not fail the test by itself: + # the poll's ConnectionError is swallowed by the ingestion wait, so the requests are checked instead. response, status = await client.post_sensor_data( sensor_id=sensor_id, @@ -1124,6 +1187,7 @@ async def test_post_sensor_data_await_ingestion_false_is_not_polled(): ) assert status == 202 + assert not any(method == "GET" for method, _ in m.requests), "nothing is polled" await client.close() @@ -1246,6 +1310,79 @@ async def test_post_measurements_awaits_ingestion_and_can_opt_out(): unit="MW", await_ingestion=False, ) + assert not any( + method == "GET" and url.path == "/api/v3_0/jobs/job-8" + for method, url in m.requests + ), "the opt-out is passed on" + await client.close() + + +@pytest.mark.asyncio +async def test_post_sensor_data_awaits_a_job_named_the_way_0_33_named_it(): + """FlexMeasures 0.33.0 named the job ``job_id``, before 1.0.0 settled on ``job``.""" + with aioresponses() as m: + client = FlexMeasuresClient(email="test@test.test", password="test") + client.access_token = "test-token" + m.post( + "http://localhost:5000/api/v3_0/sensors/5/data", + status=202, + payload={"status": "ACCEPTED", "job_id": "job-10"}, + ) + m.get( + "http://localhost:5000/api/v3_0/jobs/job-10", + status=200, + payload={"status": "FINISHED", "message": "ok", "result": None}, + ) + + await client.post_sensor_data( + sensor_id=5, + start="2023-01-01T00:00+00:00", + duration="PT1H", + values=[1.0], + unit="MW", + ) + + assert any( + method == "GET" and url.path == "/api/v3_0/jobs/job-10" + for method, url in m.requests + ) + await client.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("await_ingestion", [True, False]) +async def test_a_file_upload_awaits_its_ingestion_unless_told_not_to( + await_ingestion, tmp_path +): + csv_path = tmp_path / "data.csv" + csv_path.write_text("datetime,value\n2023-01-01T00:00+00:00,1.0\n") + with aioresponses() as m: + client = FlexMeasuresClient(email="test@test.test", password="test") + client.access_token = "test-token" + client.server_version = "1.1.0" + m.post( + "http://localhost:5000/api/v3_0/sensors/1/data/upload", + status=202, + payload={"status": "ACCEPTED", "job": "job-11"}, + ) + m.get( + "http://localhost:5000/api/v3_0/jobs/job-11", + status=200, + payload={"status": "FINISHED", "message": "ok", "result": None}, + ) + + await client.post_sensor_data( + sensor_id=1, + file_path=str(csv_path), + unit="kW", + await_ingestion=await_ingestion, + ) + + polled = any( + method == "GET" and url.path == "/api/v3_0/jobs/job-11" + for method, url in m.requests + ) + assert polled is await_ingestion await client.close() diff --git a/tests/s2/test_example_end_to_end.py b/tests/s2/test_example_end_to_end.py index 12a64aa3..ae23ecb7 100644 --- a/tests/s2/test_example_end_to_end.py +++ b/tests/s2/test_example_end_to_end.py @@ -326,7 +326,7 @@ async def test_the_site_gets_a_sensor_for_every_declaration_and_prices_from_toda made = {call.kwargs["name"]: call.kwargs for call in fm.add_sensor.await_args_list} assert "price" not in made assert made["power"]["attributes"] == {"consumption_is_positive": True} - assert made["fill rate"]["unit"] == "kWh/s" + assert made["fill rate"]["unit"] == "W" assert made["soc minima"]["unit"] == "kWh" prices = fm.post_sensor_data.await_args.kwargs assert prices["sensor_id"] == 50 @@ -539,7 +539,7 @@ def test_the_site_sensors_are_what_the_cem_records_in(): server.SITE_SENSORS["state of charge"].resolution, ) == ("kWh", "PT0M") assert server.SITE_SENSORS["storage efficiency"].unit == "%" - assert server.SITE_SENSORS["usage forecast"].unit == "kWh/s" + assert server.SITE_SENSORS["usage forecast"].unit == "W" @pytest.mark.asyncio diff --git a/tests/s2/test_example_server_websocket.py b/tests/s2/test_example_server_websocket.py new file mode 100644 index 00000000..06aa76c0 --- /dev/null +++ b/tests/s2/test_example_server_websocket.py @@ -0,0 +1,139 @@ +"""How the example server passes messages between a websocket and its CEM.""" + +from __future__ import annotations + +from types import SimpleNamespace + +import aiohttp +import pytest + +from flexmeasures_client.s2.script import websockets_server as server + + +class FakeCEM: + """Just enough of a CEM: a queue of messages to send, and a record of what it handled.""" + + def __init__(self, to_send=(), handle_raises=None): + self._to_send = list(to_send) + self.handled = [] + self.closed = False + self._handle_raises = handle_raises + self._logger = server.LOGGER + + def is_closed(self): + return self.closed or not self._to_send + + async def get_message(self): + return self._to_send.pop(0) + + async def handle_message(self, message): + if self._handle_raises: + raise self._handle_raises + self.handled.append(message) + + async def close(self): + self.closed = True + + +class FakeWebSocket: + def __init__(self, incoming=(), send_fails_after=None): + self._incoming = list(incoming) + self.sent = [] + self.closed = False + self._send_fails_after = send_fails_after + + async def send_json(self, message): + if ( + self._send_fails_after is not None + and len(self.sent) >= self._send_fails_after + ): + raise aiohttp.ClientConnectionResetError("gone") + self.sent.append(message) + + async def close(self): + self.closed = True + + def exception(self): + return RuntimeError("broken pipe") + + def __aiter__(self): + return self + + async def __anext__(self): + if not self._incoming: + raise StopAsyncIteration + return self._incoming.pop(0) + + +def text(data: str): + return SimpleNamespace(type=aiohttp.WSMsgType.TEXT, data=data) + + +@pytest.mark.asyncio +async def test_the_producer_sends_what_the_cem_queues(): + cem = FakeCEM(to_send=[{"n": 1}, {"n": 2}]) + ws = FakeWebSocket() + + await server.websocket_producer(ws, cem) + + assert ws.sent == [{"n": 1}, {"n": 2}] + + +@pytest.mark.asyncio +async def test_the_producer_stops_when_the_connection_is_reset(): + cem = FakeCEM(to_send=[{"n": 1}, {"n": 2}, {"n": 3}]) + ws = FakeWebSocket(send_fails_after=1) + + await server.websocket_producer(ws, cem) + + assert ws.sent == [{"n": 1}] + assert cem._to_send == [ + {"n": 3} + ], "nothing more is taken from the CEM once the connection is gone" + + +@pytest.mark.asyncio +async def test_the_consumer_hands_each_message_to_the_cem(): + cem = FakeCEM() + ws = FakeWebSocket(incoming=[text('{"message_type": "Handshake"}')]) + + await server.websocket_consumer(ws, cem) + + assert cem.handled == [{"message_type": "Handshake"}] + + +@pytest.mark.asyncio +async def test_the_consumer_logs_a_message_that_is_not_json_and_carries_on(caplog): + cem = FakeCEM() + ws = FakeWebSocket( + incoming=[text("not json"), text('{"message_type": "Handshake"}')] + ) + + await server.websocket_consumer(ws, cem) + + assert cem.handled == [{"message_type": "Handshake"}] + assert "handle_message failed for 'not json'" in caplog.text + + +@pytest.mark.asyncio +async def test_close_closes_the_cem_and_the_connection(): + cem = FakeCEM() + ws = FakeWebSocket(incoming=[text("close")]) + + await server.websocket_consumer(ws, cem) + + assert cem.closed and ws.closed + assert cem.handled == [] + + +@pytest.mark.asyncio +async def test_a_connection_error_closes_the_cem(caplog): + cem = FakeCEM() + ws = FakeWebSocket( + incoming=[SimpleNamespace(type=aiohttp.WSMsgType.ERROR, data=None)] + ) + + await server.websocket_consumer(ws, cem) + + assert cem.closed + assert "ws connection closed with exception broken pipe" in caplog.text diff --git a/tests/s2/test_frbc_actuator_measurement.py b/tests/s2/test_frbc_actuator_measurement.py index 55319525..c2cbf431 100644 --- a/tests/s2/test_frbc_actuator_measurement.py +++ b/tests/s2/test_frbc_actuator_measurement.py @@ -11,6 +11,7 @@ from __future__ import annotations +import asyncio import logging from datetime import datetime, timedelta, timezone from unittest.mock import AsyncMock @@ -348,3 +349,26 @@ async def test_a_unit_nothing_converts_between_is_reported(system_description, c # so that is the unit it is reported in: a pair that disagrees is worse than none. assert f"{ON_W} W" in caplog.text assert f"{ON_W} hp" not in caplog.text + + +@pytest.mark.asyncio +async def test_an_actuator_status_before_any_description_waits_for_one( + system_description, caplog +): + """After a reconnection one can arrive first; indexing the empty history raised an IndexError in a task nobody awaited.""" + caplog.set_level(logging.INFO) + frbc = _frbc(system_description, consumption_as_positive=True) + frbc._system_description_history.clear() + + await frbc.send_actuator_status(_status(system_description, 0)) + await frbc.send_actuator_status(_status(system_description, 1)) + + assert _posts_to_the_power_sensor(frbc) == [] + assert "no FRBC.SystemDescription has arrived" in caplog.text + + await frbc.handle_system_description(system_description.to_dict()) + await asyncio.gather(*frbc.background_tasks) + + # Only the latest status describes the actuator's present state, so only that one is recorded. + assert [post["values"] for post in _posts_to_the_power_sensor(frbc)] == [[ON_W]] + assert frbc._statuses_awaiting_a_description == {} diff --git a/tests/s2/test_frbc_charging_efficiency.py b/tests/s2/test_frbc_charging_efficiency.py index 2a899d82..05239a6c 100644 --- a/tests/s2/test_frbc_charging_efficiency.py +++ b/tests/s2/test_frbc_charging_efficiency.py @@ -27,7 +27,7 @@ ) from flexmeasures_client.client import FlexMeasuresClient, _server_version_at_least -from flexmeasures_client.exceptions import JobFailedError +from flexmeasures_client.exceptions import JobFailedError, JobTimeoutError from flexmeasures_client.s2.control_types.FRBC.frbc_simple import FRBCSimple from flexmeasures_client.s2.control_types.FRBC.utils import ( fill_rate_in_watts, @@ -419,16 +419,17 @@ async def test_a_state_of_charge_goes_out_in_a_unit_flexmeasures_takes( @pytest.mark.asyncio -async def test_a_lower_bound_below_zero_goes_out_in_the_same_unit(): - """Reported below its range, a storage widens the lower bound to take it in, which then needs the conversion too.""" +async def test_a_storage_reported_below_zero_is_planned_from_zero(caplog): + """FlexMeasures refuses a negative soc-min, so widening the bound to take the storage in got it no schedule at all.""" system_description = _heater() frbc = _frbc(system_description, energy_unit="J") frbc._storage_status_history[get_unique_id()] = FRBCStorageStatus( message_id=get_unique_id(), present_fill_level=-3.6e6 ) flex_model = await _asked_for(frbc, system_description) - assert flex_model["soc-at-start"] == pytest.approx(-1.0) - assert flex_model["soc-min"] == pytest.approx(-1.0) + assert flex_model["soc-at-start"] == 0.0 + assert flex_model["soc-min"] == 0.0 + assert "below zero" in caplog.text @pytest.mark.asyncio @@ -563,3 +564,24 @@ def test_a_batch_of_instructions_is_logged_once_and_by_name(caplog): ] assert len(batches) == 1 assert "'on'" in batches[0] and "'off'" in batches[0] + + +@pytest.mark.asyncio +async def test_an_efficiency_whose_ingestion_is_unconfirmed_is_recorded_again(): + """Remembered as recorded, it was never posted again for its window, and the schedule named a sensor empty for it.""" + system_description = _heater() + frbc = _frbc(system_description) + frbc._fm_client.post_sensor_data = AsyncMock( + side_effect=[JobTimeoutError("still pending"), ({}, 200)] + ) + + first = await _asked_for(frbc, system_description) + second = await _asked_for(frbc, system_description) + + assert "charging-efficiency" not in first + assert "charging-efficiency" in second + assert frbc._fm_client.post_sensor_data.await_count == 2 + assert all( + c.kwargs.get("confirm_ingestion") is True + for c in frbc._fm_client.post_sensor_data.await_args_list + ) diff --git a/tests/s2/test_frbc_declared_profiles.py b/tests/s2/test_frbc_declared_profiles.py index 201ed323..f8dca1b5 100644 --- a/tests/s2/test_frbc_declared_profiles.py +++ b/tests/s2/test_frbc_declared_profiles.py @@ -40,6 +40,7 @@ ) from flexmeasures_client.client import FlexMeasuresClient +from flexmeasures_client.exceptions import JobTimeoutError from flexmeasures_client.s2.control_types.FRBC.frbc_simple import FRBCSimple from flexmeasures_client.s2.control_types.FRBC.utils import ( clip_fill_level_target_profile, @@ -259,7 +260,7 @@ async def test_usage_forecast_is_recorded_from_a_resolution_boundary( assert posted["start"] == frbc.align_to_resolution(usage_forecast.start_time) assert posted["start"].minute % 15 == 0 and posted["start"].second == 0 # A usage forecast is a rate of fill level, not a power. - assert posted["unit"] == frbc.fill_rate_unit() == "kWh/s" + assert posted["unit"] == frbc.fill_rate_unit() == "W" @pytest.mark.asyncio @@ -545,31 +546,44 @@ async def test_the_instruction_translation_stays_in_the_resource_managers_units( assert translate.call_args.kwargs["initial_fill_level"] == 30.0 -def test_a_fill_rate_is_not_recorded_as_a_power(): - """S2 states a fill rate as fill level per second, which is not the power unit. - - For a heat buffer the quantity is not electricity at all, and even for a battery kWh/s - differs from kW by the number of seconds in an hour. - """ - frbc = FRBCSimple( +def _handler(**units) -> FRBCSimple: + return FRBCSimple( power_sensor_id=1, price_sensor_id=2, soc_sensor_id=3, rm_discharge_sensor_id=4, - energy_unit="kWh", - power_unit="W", + **units, ) - assert frbc.fill_rate_unit() == "kWh/s" - assert frbc.fill_rate_unit() != frbc.power_unit - megawatt_hours = FRBCSimple( - power_sensor_id=1, - price_sensor_id=2, - soc_sensor_id=3, - rm_discharge_sensor_id=4, - energy_unit="MWh", - ) - assert megawatt_hours.fill_rate_unit() == "MWh/s" + +@pytest.mark.parametrize( + "units, fill_rate_unit, recorded", + [ + # 0.01 kWh/s is 36 kW. + ({}, "W", 36000.0), + ({"fill_rate_unit": "kW"}, "kW", 36.0), + ({"energy_unit": "MWh", "fill_rate_unit": "MW"}, "MW", 36.0), + # A joule per second is a watt. + ({"energy_unit": "kJ"}, "W", 10.0), + ({"energy_unit": "J", "fill_rate_unit": "kW"}, "kW", 0.00001), + ], +) +def test_a_fill_rate_is_recorded_in_the_fill_rate_unit_whatever_the_power_unit( + units, fill_rate_unit, recorded +): + """S2 states a fill rate as fill level per second, which in kWh/s makes tiny numbers, so it is recorded in a power unit, by default W. + + That is not the power unit, which says how an actuator's electric power is posted: for a heat buffer, a fill rate is not electricity at all. + """ + frbc = _handler(power_unit="kW", **units) + assert frbc.fill_rate_unit() == fill_rate_unit + assert frbc._per_second_in_fill_rate_unit(0.01) == pytest.approx(recorded) + + +@pytest.mark.parametrize("fill_rate_unit", ["kWh/s", "kWh", "GW"]) +def test_a_fill_rate_unit_that_is_no_power_unit_is_refused(fill_rate_unit): + with pytest.raises(ValueError, match="fill_rate_unit has to be one of W, kW, MW"): + _handler(fill_rate_unit=fill_rate_unit) @pytest.mark.asyncio @@ -608,8 +622,13 @@ async def test_an_actuator_s_fill_rate_is_posted_in_the_fill_rate_unit( ) ) - assert ( - _posted(frbc, fill_rate_sensor_id)["unit"] == frbc.fill_rate_unit() == "kWh/s" + assert _posted(frbc, fill_rate_sensor_id)["unit"] == frbc.fill_rate_unit() == "W" + # Halfway through the mode's fill rate, from kWh/s into W. + fill_rate = actuator.operation_modes[0].elements[0].fill_rate + halfway = (fill_rate.start_of_range + fill_rate.end_of_range) / 2 + assert halfway > 0 + assert _posted(frbc, fill_rate_sensor_id)["values"] == pytest.approx( + [halfway * 3.6e6] ) # The power sensor really does carry a power, and is untouched by this. assert _posted(frbc, 1)["unit"] == frbc.power_unit @@ -668,3 +687,85 @@ async def test_a_storage_efficiency_that_could_not_be_recorded_is_not_planned_ag assert await frbc._record_storage_efficiency(start, 50.0) is False assert frbc._storage_efficiency_recorded is None assert "could not record" in caplog.text + + +@pytest.mark.asyncio +async def test_a_storage_efficiency_whose_ingestion_is_unconfirmed_is_not_remembered( + system_description, +): + frbc = _frbc(system_description, with_sensors=True) + leakage = FRBCLeakageBehaviour( + message_id=get_unique_id(), + valid_from="2024-01-01T12:00:00+00:00", + elements=[ + FRBCLeakageBehaviourElement( + fill_level_range=NumberRange( + start_of_range=RANGE_BOTTOM, end_of_range=RANGE_TOP + ), + leakage_rate=0.01, + ) + ], + ) + frbc._leakage_behaviour_history[str(leakage.message_id)] = leakage + frbc._fm_client.post_sensor_data.side_effect = JobTimeoutError("still pending") + start = datetime(2024, 1, 1, 12, 15, tzinfo=timezone.utc) + assert await frbc._record_storage_efficiency(start, 50.0) is False + assert frbc._storage_efficiency_recorded is None + assert ( + frbc._fm_client.post_sensor_data.await_args.kwargs["confirm_ingestion"] is True + ) + + +@pytest.mark.asyncio +async def test_a_bound_that_could_not_be_recorded_leaves_the_other_recorded( + system_description, caplog +): + """The band is planned against inline, so these are for display, and one failing post is no reason to lose the other.""" + frbc = _frbc(system_description, with_sensors=True) + frbc._fm_client.post_sensor_data.side_effect = [ConnectionError("down"), ({}, 200)] + + await frbc.send_fill_level_target_profile(_target_profile(40.0, 60.0)) + + assert frbc._fm_client.post_sensor_data.await_count == 2 + second = frbc._fm_client.post_sensor_data.await_args_list[1].kwargs + assert second["sensor_id"] == frbc._soc_maxima_sensor_id + assert "Could not record a bound" in caplog.text + + +@pytest.mark.asyncio +async def test_a_fill_rate_that_could_not_be_recorded_is_logged( + system_description, caplog +): + """The post runs in a task nobody awaits, so an exception there was asyncio's to report, if anyone's.""" + frbc = FRBCSimple( + power_sensor_id=1, + price_sensor_id=2, + soc_sensor_id=3, + rm_discharge_sensor_id=4, + fill_rate_sensor_id=20, + ) + frbc._fm_client = AsyncMock(FlexMeasuresClient) + frbc._fm_client.get_sensor = AsyncMock( + return_value=power_sensor_recording(consumption_as_positive=True) + ) + # The power goes through; the fill rate does not. + frbc._fm_client.post_sensor_data = AsyncMock( + side_effect=[({}, 200), ConnectionError("down")] + ) + frbc._logger = logging.getLogger(__name__) + frbc._system_description_history = { + system_description.message_id: system_description + } + actuator = system_description.actuators[0] + + await frbc.send_actuator_status( + FRBCActuatorStatus( + message_id=get_unique_id(), + actuator_id=actuator.id, + active_operation_mode_id=actuator.operation_modes[0].id, + operation_mode_factor=0.5, + ) + ) + + assert frbc._fm_client.post_sensor_data.await_count == 2 + assert "Could not record the fill rate" in caplog.text diff --git a/tests/s2/test_frbc_operation_modes.py b/tests/s2/test_frbc_operation_modes.py index 747f7ad0..88d24411 100644 --- a/tests/s2/test_frbc_operation_modes.py +++ b/tests/s2/test_frbc_operation_modes.py @@ -170,12 +170,11 @@ async def test_a_band_that_spans_zero_is_split_at_zero(): @pytest.mark.asyncio async def test_a_band_that_spans_zero_reaches_both_directions_from_zero(): """FlexMeasures reads the two ranges of one band as a single band through zero.""" - system_description = _system_description((0.0, 0.0), (-500.0, 1000.0)) - frbc = _frbc(system_description) - - flex_model = await _asked_for(frbc, system_description) + from flexmeasures_client.s2.control_types.FRBC.frbc_simple import ( + _operation_mode_power_bands, + ) - assert flex_model["operation-modes"] == [ + assert _operation_mode_power_bands([(0.0, 0.0), (-500.0, 1000.0)], "W") == [ {"consumption-range": ["0 W", "0 W"]}, { "consumption-range": ["0.0 W", "1000.0 W"], @@ -316,12 +315,150 @@ async def test_the_bands_are_in_watts_whatever_the_handler_posts_power_in(): @pytest.mark.asyncio async def test_a_bound_of_zero_is_stated_as_zero(): """Flipping the sign of a range that reaches up to zero leaves a negative zero behind.""" + from flexmeasures_client.s2.control_types.FRBC.frbc_simple import ( + _operation_mode_power_bands, + ) + + assert _operation_mode_power_bands([(0.0, 0.0), (-2000.0, 0.0)], "W")[1] == { + "production-range": ["0.0 W", "2000.0 W"] + } system_description = _system_description((0.0, 0.0), (-2000.0, 0.0)) - frbc = _frbc(system_description) + flex_model = await _asked_for(_frbc(system_description), system_description) + assert flex_model["production-capacity"] == "2000.0 W" + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "ranges, bands_sent", + [ + ( + ((0.0, 0.0), (0.0, 1500.0)), + False, + ), # idle beside a mode from zero up: nothing the capacities lack + (((0.0, 0.0), (-500.0, 1000.0)), False), + ( + ((0.0, 0.0), (0.0, 800.0), (800.0, 1500.0)), + False, + ), # touching ranges leave no gap + (((0.0, 0.0), (400.0, 1500.0)), True), # nothing between 0 and 400 W + (((400.0, 1500.0),), True), # and nothing at all below 400 W + (((0.0, 0.0), (-2000.0, -500.0)), True), + ], +) +async def test_bands_go_out_only_where_they_rule_out_what_the_capacities_allow( + ranges, bands_sent +): + """Each band costs the solver a binary per step, so bands that only repeat the capacities turned an LP into a MILP.""" + system_description = _system_description(*ranges) + flex_model = await _asked_for(_frbc(system_description), system_description) + assert ("operation-modes" in flex_model) is bands_sent + +@pytest.mark.asyncio +async def test_a_mode_for_abnormal_conditions_only_is_neither_planned_in_nor_instructed_into(): + """An instruction into it is not marked abnormal, so a conforming resource manager refuses it.""" + system_description = _system_description((0.0, 0.0), (400.0, 1000.0), (0.0, 5000.0)) + actuator = system_description.actuators[0] + boost = actuator.operation_modes[2].model_copy( + update={"abnormal_condition_only": True} + ) + system_description = system_description.model_copy( + update={ + "actuators": [ + actuator.model_copy( + update={"operation_modes": [*actuator.operation_modes[:2], boost]} + ) + ] + } + ) + frbc = _frbc(system_description) flex_model = await _asked_for(frbc, system_description) + assert flex_model["consumption-capacity"] == "1000.0 W" + assert all( + band.get("consumption-range", ["", ""])[1] != "5000.0 W" + for band in flex_model["operation-modes"] + ) + from flexmeasures_client.s2.control_types.FRBC.utils import normal_operation_modes - assert flex_model["operation-modes"][1] == { - "production-range": ["0.0 W", "2000.0 W"] + assert str(boost.id) not in { + str(m.id) for m in normal_operation_modes(system_description.actuators[0]) } - assert flex_model["production-capacity"] == "2000.0 W" + + +def test_an_actuator_with_no_normal_mode_cannot_be_instructed(): + + from flexmeasures_client.s2.control_types.FRBC.utils import ( + fm_schedule_to_instructions, + ) + + system_description = _system_description((0.0, 1000.0)) + actuator = system_description.actuators[0] + only = actuator.operation_modes[0].model_copy( + update={"abnormal_condition_only": True} + ) + system_description = system_description.model_copy( + update={"actuators": [actuator.model_copy(update={"operation_modes": [only]})]} + ) + with pytest.raises(ValueError, match="no operation mode"): + fm_schedule_to_instructions( + { + "start": "2024-01-01T00:00:00+00:00", + "duration": "PT15M", + "values": [500.0], + }, + system_description, + initial_fill_level=50.0, + ) + + +@pytest.mark.asyncio +async def test_no_schedule_is_asked_for_an_actuator_with_no_normal_mode(caplog): + """A solve whose answer could only be instructed into modes the resource manager refuses is a solve for nothing.""" + system_description = _system_description((0.0, 1000.0)) + actuator = system_description.actuators[0] + only = actuator.operation_modes[0].model_copy( + update={"abnormal_condition_only": True} + ) + system_description = system_description.model_copy( + update={"actuators": [actuator.model_copy(update={"operation_modes": [only]})]} + ) + frbc = _frbc(system_description) + await frbc.trigger_schedule(system_description.message_id) + frbc._fm_client.trigger_and_get_schedule.assert_not_awaited() + assert "under normal conditions" in caplog.text + + +@pytest.mark.asyncio +async def test_a_mode_for_abnormal_conditions_only_does_not_set_the_charging_efficiency(): + """Less efficient than the normal modes, an emergency boost lowered the efficiency every plan was made against.""" + system_description = _system_description((0.0, 1000.0), (0.0, 5000.0)) + actuator = system_description.actuators[0] + boost = actuator.operation_modes[1].model_copy( + update={"abnormal_condition_only": True} + ) + system_description = system_description.model_copy( + update={ + "actuators": [ + actuator.model_copy( + update={"operation_modes": [actuator.operation_modes[0], boost]} + ) + ] + } + ) + frbc = _frbc(system_description, charging_efficiency_sensor_id=9) + seen = [] + from flexmeasures_client.s2.control_types.FRBC import frbc_simple + + real = frbc_simple.operation_mode_conversion_efficiency + + def spy(mode, *args): + seen.append(str(mode.id)) + return real(mode, *args) + + from unittest.mock import patch + + with patch.object( + frbc_simple, "operation_mode_conversion_efficiency", side_effect=spy + ): + await _asked_for(frbc, system_description) + assert str(boost.id) not in seen diff --git a/tests/s2/test_frbc_schedule_retry.py b/tests/s2/test_frbc_schedule_retry.py index b93a81c2..39faa561 100644 --- a/tests/s2/test_frbc_schedule_retry.py +++ b/tests/s2/test_frbc_schedule_retry.py @@ -27,7 +27,7 @@ ) from flexmeasures_client.client import FlexMeasuresClient, _server_version_at_least -from flexmeasures_client.exceptions import JobFailedError +from flexmeasures_client.exceptions import JobFailedError, JobTimeoutError from flexmeasures_client.s2.control_types.FRBC.frbc_simple import ( SCHEDULE_ATTEMPTS, FRBCSimple, @@ -349,7 +349,7 @@ async def test_a_schedule_from_a_fallback_scheduler_is_said_out_loud(caplog): await frbc.trigger_schedule(system_description.message_id) assert "StorageFallbackScheduler" in caplog.text - assert "no state of charge was recorded" in caplog.text + assert "coarse charging policy, not a plan" in caplog.text # The schedule is still followed: a coarse policy beats leaving the actuator to itself. frbc.send_instruction_batch.assert_awaited_once() @@ -388,3 +388,177 @@ def test_a_schedule_per_device_names_no_scheduler_to_check(caplog): frbc._warn_where_a_fallback_scheduler_answered([SCHEDULE, SCHEDULE], VALID_FROM) assert caplog.text == "" + + +@pytest.mark.asyncio +async def test_a_schedule_is_asked_for_five_times_by_default(): + """Pinned here, since every other test imports the default rather than stating it.""" + system_description = _heater() + frbc = _frbc(system_description) + frbc._fm_client.trigger_and_get_schedule.side_effect = A_FAILED_JOB + + with pytest.raises(JobFailedError): + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == 5 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("scale", [1.0, 10.0]) +async def test_the_bounds_a_schedule_is_read_back_against_are_in_the_resource_manager_s_units( + scale, +): + """At a scale of ten, read back unscaled, the translation clamped at ten times the true hard maximum.""" + from unittest.mock import patch + + system_description = _heater() + frbc = _frbc(system_description, fill_level_scale=scale) + with patch( + "flexmeasures_client.s2.control_types.FRBC.frbc_simple.fm_schedule_to_instructions", + return_value=[], + ) as translate: + await frbc.trigger_schedule(system_description.message_id) + + assert translate.call_args.kwargs["fill_level_bounds"] == ( + 0.0, + pytest.approx(RANGE_TOP * 1.5), + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "failure", + [ + JobTimeoutError("Job 1 did not finish within 600 seconds."), + ConnectionError("Error occurred while communicating with the API: 502"), + ], + ids=["no worker picked the job up", "the gateway restarted"], +) +async def test_the_likeliest_failures_in_production_are_asked_again(failure, caplog): + """Neither was caught, so each escaped on the first attempt, and the line saying the storage was left without an instruction never went out.""" + system_description = _heater() + frbc = _frbc(system_description) + frbc._fm_client.trigger_and_get_schedule.side_effect = [failure, SCHEDULE] + + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == 2 + frbc.send_instruction_batch.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_giving_up_on_an_unreachable_server_says_the_storage_is_left_without_an_instruction( + caplog, +): + system_description = _heater() + frbc = _frbc(system_description) + frbc._fm_client.trigger_and_get_schedule.side_effect = ConnectionError("down") + + with pytest.raises(ConnectionError): + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == frbc._schedule_attempts + assert "left without an instruction" in caplog.text + + +@pytest.mark.asyncio +@pytest.mark.parametrize("server_version, attempts", [("0.32.0", 3), ("0.33.0", 1)]) +async def test_a_bare_refusal_is_asked_again_only_where_the_server_cannot_say_a_job_failed( + server_version, attempts +): + """Before v0.33.0 a failed job comes back as a bare HTTP 400, which matched nothing, so the retry was inert there.""" + system_description = _heater() + frbc = _frbc(system_description, schedule_attempts=3) + frbc._fm_client.server_version_at_least = AsyncMock( + side_effect=lambda minimum: _server_version_at_least(server_version, minimum) + ) + frbc._fm_client.trigger_and_get_schedule.side_effect = ValueError( + "Request failed with status code 400" + ) + + with pytest.raises(ValueError): + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == attempts + + +@pytest.mark.asyncio +async def test_a_state_of_charge_that_could_not_be_recorded_is_planned_from_all_the_same( + caplog, +): + """The post went first and unguarded, so a failed ingestion job cancelled the planning it preceded.""" + system_description = _heater() + frbc = _frbc(system_description) + frbc._fm_client.post_measurements = AsyncMock( + side_effect=JobFailedError("Job 2 ended with status failed.") + ) + + await frbc.send_storage_status( + FRBCStorageStatus(message_id=get_unique_id(), present_fill_level=40.0) + ) + + assert _attempts(frbc) == 1 + assert "Could not record the state of charge" in caplog.text + + +def _wrapped_http_error(status: int) -> ConnectionError: + """What the client raises for an HTTP error: a ConnectionError, caused by aiohttp's own.""" + from aiohttp import ClientResponseError + + failure = ConnectionError("Error occurred while communicating with the API") + failure.__cause__ = ClientResponseError(None, (), status=status) + return failure + + +@pytest.mark.asyncio +@pytest.mark.parametrize("status, attempts", [(404, 1), (503, 3)]) +async def test_an_http_error_is_asked_again_only_where_it_may_be_passing( + status, attempts +): + """A 4xx answers the same next time; a 5xx may be a restart.""" + system_description = _heater() + frbc = _frbc(system_description, schedule_attempts=3) + frbc._fm_client.trigger_and_get_schedule.side_effect = _wrapped_http_error(status) + + with pytest.raises(ConnectionError): + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == attempts + + +@pytest.mark.asyncio +async def test_a_job_nobody_picked_up_is_asked_about_once_more_only(): + """Each attempt waits out the whole job timeout, and a dead queue is still dead after it.""" + system_description = _heater() + frbc = _frbc(system_description, schedule_attempts=5) + frbc._fm_client.trigger_and_get_schedule.side_effect = JobTimeoutError("pending") + + with pytest.raises(JobTimeoutError): + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == 2 + + +@pytest.mark.asyncio +async def test_a_failed_version_lookup_does_not_take_the_place_of_the_failure_it_was_for( + caplog, +): + system_description = _heater() + frbc = _frbc(system_description, schedule_attempts=3) + + def version_at_least(minimum): + # Only the lookup the retry makes fails; the flex model's own lookups go through. + if minimum == "0.33.0": + raise ConnectionError("version lookup failed") + return _server_version_at_least("1.1.0", minimum) + + frbc._fm_client.server_version_at_least = AsyncMock(side_effect=version_at_least) + frbc._fm_client.trigger_and_get_schedule.side_effect = ValueError( + "Request failed with status code 400" + ) + + with pytest.raises(ValueError): + await frbc.trigger_schedule(system_description.message_id) + + assert _attempts(frbc) == 1 + assert "left without an instruction" in caplog.text diff --git a/tests/s2/test_frbc_simple_planning.py b/tests/s2/test_frbc_simple_planning.py index 428c76ee..74a39641 100644 --- a/tests/s2/test_frbc_simple_planning.py +++ b/tests/s2/test_frbc_simple_planning.py @@ -651,7 +651,7 @@ async def test_a_request_out_of_date_makes_no_further_attempt_while_the_newest_w frbc = frbc_simple() frbc._schedule_attempts = 5 frbc._schedule_retry_interval = td(milliseconds=10) - frbc._is_worth_asking_again = lambda failure: True + frbc._is_worth_asking_again = AsyncMock(return_value=True) first_attempt = asyncio.Event() planned = frbc._fm_client.trigger_and_get_schedule.return_value diff --git a/tests/s2/test_frbc_soft_band.py b/tests/s2/test_frbc_soft_band.py index 64055d78..26462c9c 100644 --- a/tests/s2/test_frbc_soft_band.py +++ b/tests/s2/test_frbc_soft_band.py @@ -88,6 +88,23 @@ def system_description() -> FRBCSystemDescription: ) +def _system_description_with_range(bottom: float, top: float) -> FRBCSystemDescription: + """The fixture's description, declaring another fill-level range.""" + sd = system_description.__wrapped__() + return sd.model_copy( + update={ + "message_id": get_unique_id(), + "storage": sd.storage.model_copy( + update={ + "fill_level_range": NumberRange( + start_of_range=bottom, end_of_range=top + ) + } + ), + } + ) + + def _frbc(system_description, *, with_sensors: bool = True, **kwargs) -> FRBCSimple: """A handler that has had a system description and a storage status, ready to plan.""" sensors = ( @@ -139,13 +156,22 @@ async def _asked_for(frbc: FRBCSimple, system_description) -> tuple[dict, dict]: async def test_the_declared_band_is_what_a_schedule_is_asked_against( system_description, ): - """Recording the band changed nothing until the flex model named the sensors holding it.""" + """The declared range goes out with the request, over the whole window, as a band whose breach is priced.""" frbc = _frbc(system_description) flex_model, flex_context = await _asked_for(frbc, system_description) - assert flex_model["soc-minima"] == {"sensor": SOC_MINIMA_SENSOR_ID} - assert flex_model["soc-maxima"] == {"sensor": SOC_MAXIMA_SENSOR_ID} + assert [segment["value"] for segment in flex_model["soc-minima"]] == [ + f"{RANGE_BOTTOM} kWh" + ] + assert [segment["value"] for segment in flex_model["soc-maxima"]] == [ + f"{RANGE_TOP} kWh" + ] + (whole,) = flex_model["soc-minima"] + assert ( + pd.Timestamp(whole["end"]) - pd.Timestamp(whole["start"]) + == frbc._schedule_duration + ) # A minimum or maximum FlexMeasures may not breach is a hard bound by another name, # and the whole point of the band is that it is soft. assert flex_context["relax-soc-constraints"] is True @@ -254,24 +280,21 @@ async def test_how_far_the_hard_bounds_sit_from_the_range_is_the_deployments_to_ @pytest.mark.asyncio -async def test_one_side_of_a_band_is_not_a_band(system_description): - """One side alone is an open-ended push, not what the resource manager declared. - - A band needs both sides, which is also the rule the recording half goes by: - a fill-level target profile is recorded only where both sensors are there to hold it. - """ - frbc = _frbc( - system_description, - with_sensors=False, - soc_minima_sensor_id=SOC_MINIMA_SENSOR_ID, - ) +async def test_the_band_goes_out_without_the_sensors_it_is_recorded_on( + system_description, +): + """It reached FlexMeasures only through those sensors, so without them a storage was planned with no band at all.""" + frbc = _frbc(system_description, with_sensors=False) flex_model, flex_context = await _asked_for(frbc, system_description) - assert "soc-minima" not in flex_model - assert "soc-maxima" not in flex_model - # And nothing is relaxed either, since there is no band for a breach price to apply to. - assert "relax-soc-constraints" not in flex_context + assert [segment["value"] for segment in flex_model["soc-minima"]] == [ + f"{RANGE_BOTTOM} kWh" + ] + assert [segment["value"] for segment in flex_model["soc-maxima"]] == [ + f"{RANGE_TOP} kWh" + ] + assert flex_context["relax-soc-constraints"] is True def test_a_hard_bound_inside_the_declared_range_is_refused(): @@ -299,9 +322,8 @@ async def test_nothing_is_asked_against_a_sensor_that_was_never_configured( flex_model, flex_context = await _asked_for(frbc, system_description) - for absent in ("soc-minima", "soc-maxima", "soc-usage", "storage-efficiency"): + for absent in ("soc-usage", "storage-efficiency"): assert absent not in flex_model - assert "relax-soc-constraints" not in flex_context # The hard bounds, the start state and the unit are not a resource manager's to declare, # so they are asked for whatever the deployment configured. assert flex_model["soc-min"] == 0.0 @@ -309,6 +331,58 @@ async def test_nothing_is_asked_against_a_sensor_that_was_never_configured( assert flex_model["soc-at-start"] == pytest.approx(FILL_LEVEL_AT_START) +@pytest.mark.asyncio +async def test_a_target_profile_applies_where_it_covers_the_window_and_the_range_elsewhere( + system_description, +): + """Beyond the profile's end the band used to be whatever the sensors last held, or nothing.""" + from datetime import timedelta + + from s2python.common import Duration + from s2python.frbc import ( + FRBCFillLevelTargetProfile, + FRBCFillLevelTargetProfileElement, + ) + + frbc = _frbc(system_description) + start = frbc.schedule_start(system_description.valid_from) + profile = FRBCFillLevelTargetProfile( + message_id=get_unique_id(), + start_time=start, + elements=[ + FRBCFillLevelTargetProfileElement( + duration=Duration.from_timedelta(timedelta(hours=1)), + fill_level_range=NumberRange(start_of_range=50, end_of_range=60), + ) + ], + ) + frbc._fill_level_target_profile_history[str(profile.message_id)] = profile + + flex_model, _ = await _asked_for(frbc, system_description) + + minima = flex_model["soc-minima"] + assert [segment["value"] for segment in minima] == [ + "50.0 kWh", + f"{RANGE_BOTTOM} kWh", + ] + assert pd.Timestamp(minima[0]["end"]) - pd.Timestamp( + minima[0]["start"] + ) == timedelta(hours=1) + assert [segment["value"] for segment in flex_model["soc-maxima"]] == [ + "60.0 kWh", + f"{RANGE_TOP} kWh", + ] + + +@pytest.mark.asyncio +async def test_a_band_below_zero_is_clipped_at_zero(): + """FlexMeasures takes no negative minima, and refused the whole request over one.""" + sd = _system_description_with_range(-10.0, 80.0) + frbc = _frbc(sd) + flex_model, _ = await _asked_for(frbc, sd) + assert [segment["value"] for segment in flex_model["soc-minima"]] == ["0.0 kWh"] + + @pytest.mark.asyncio async def test_a_schedule_is_read_back_against_the_bounds_it_was_planned_against( system_description, @@ -456,3 +530,172 @@ def test_bounds_that_run_backwards_are_refused(): initial_fill_level=RANGE_BOTTOM, fill_level_bounds=(RANGE_TOP, RANGE_BOTTOM), ) + + +@pytest.mark.asyncio +async def test_a_negative_usage_forecast_is_recorded_as_no_usage_and_said_so( + system_description, caplog +): + """A forced gain FlexMeasures cannot relax made every schedule for the window infeasible.""" + from datetime import timedelta + + from s2python.common import Duration + from s2python.frbc import FRBCUsageForecast, FRBCUsageForecastElement + + frbc = _frbc(system_description) + forecast = FRBCUsageForecast( + message_id=get_unique_id(), + start_time="2024-01-01T00:00:00+00:00", + elements=[ + FRBCUsageForecastElement( + duration=Duration.from_timedelta(timedelta(minutes=15)), + usage_rate_expected=rate, + ) + for rate in (0.01, -0.02, 0.0, -0.005) + ], + ) + await frbc.send_usage_forecast(forecast) + posted = frbc._fm_client.post_sensor_data.await_args.kwargs + assert min(posted["values"]) == 0.0 + assert posted["values"][0] == pytest.approx(36000.0) # 0.01 kWh/s, in W + assert "negative usage on 2 of 4 steps" in caplog.text + + +@pytest.mark.asyncio +async def test_a_target_profile_beyond_the_declared_range_is_held_inside_it( + system_description, +): + from datetime import timedelta + + from s2python.common import Duration + from s2python.frbc import ( + FRBCFillLevelTargetProfile, + FRBCFillLevelTargetProfileElement, + ) + + frbc = _frbc(system_description) + start = frbc.schedule_start(system_description.valid_from) + profile = FRBCFillLevelTargetProfile( + message_id=get_unique_id(), + start_time=start, + elements=[ + FRBCFillLevelTargetProfileElement( + duration=Duration.from_timedelta(timedelta(hours=1)), + fill_level_range=NumberRange(start_of_range=10, end_of_range=90), + ) + ], + ) + frbc._fill_level_target_profile_history[str(profile.message_id)] = profile + flex_model, _ = await _asked_for(frbc, system_description) + assert [s["value"] for s in flex_model["soc-minima"]] == [f"{RANGE_BOTTOM} kWh"] + assert [s["value"] for s in flex_model["soc-maxima"]] == [f"{RANGE_TOP} kWh"] + + +@pytest.mark.asyncio +async def test_the_band_goes_out_in_the_unit_flexmeasures_takes(system_description): + frbc = _frbc(system_description, energy_unit="Wh") + flex_model, _ = await _asked_for(frbc, system_description) + assert flex_model["soc-unit"] == "kWh" + assert [s["value"] for s in flex_model["soc-minima"]] == [ + f"{RANGE_BOTTOM / 1000} kWh" + ] + + +def test_a_profile_on_a_half_hour_grid_reaches_the_steps_of_an_hourly_window( + system_description, +): + """Floored on its own clock, half an hour off the window's, no step shared an instant, and the profile was dropped.""" + from datetime import datetime, timedelta, timezone + from zoneinfo import ZoneInfo + + from s2python.common import Duration + from s2python.frbc import ( + FRBCFillLevelTargetProfile, + FRBCFillLevelTargetProfileElement, + ) + + frbc = _frbc(system_description, resolution=timedelta(hours=1)) + start = datetime(2024, 1, 1, 23, tzinfo=timezone.utc) + profile = FRBCFillLevelTargetProfile( + message_id=get_unique_id(), + start_time=datetime(2024, 1, 2, 4, 0, tzinfo=ZoneInfo("Asia/Kolkata")), + elements=[ + FRBCFillLevelTargetProfileElement( + duration=Duration.from_timedelta(timedelta(hours=6)), + fill_level_range=NumberRange(start_of_range=50, end_of_range=60), + ) + ], + ) + frbc._fill_level_target_profile_history[str(profile.message_id)] = profile + minima, _ = frbc._fill_level_band(start, system_description, "kWh", 1.0) + assert "50.0 kWh" in [segment["value"] for segment in minima] + + +@pytest.mark.asyncio +async def test_the_band_is_left_out_for_a_server_that_cannot_relax_it( + system_description, caplog +): + """FlexMeasures before v0.26.0 refuses relax-soc-constraints, and with it the whole request.""" + import logging + + caplog.set_level(logging.INFO) + frbc = _frbc(system_description) + frbc._fm_client.server_version_at_least = AsyncMock(return_value=False) + flex_model, flex_context = await _asked_for(frbc, system_description) + assert "soc-minima" not in flex_model + assert "relax-soc-constraints" not in flex_context + assert "v0.26.0" in caplog.text + + +@pytest.mark.asyncio +async def test_the_schedule_is_read_back_without_the_negative_usage_it_was_planned_without( + system_description, +): + """Read back with it, the instructions followed fill levels the plan never had.""" + from datetime import timedelta + from unittest.mock import patch + + from s2python.common import Duration + from s2python.frbc import FRBCUsageForecast, FRBCUsageForecastElement + + frbc = _frbc(system_description) + forecast = FRBCUsageForecast( + message_id=get_unique_id(), + start_time="2024-01-01T00:00:00+00:00", + elements=[ + FRBCUsageForecastElement( + duration=Duration.from_timedelta(timedelta(hours=1)), + usage_rate_expected=-0.5, + ) + ], + ) + frbc._usage_forecast_history[str(forecast.message_id)] = forecast + with patch( + "flexmeasures_client.s2.control_types.FRBC.frbc_simple.fm_schedule_to_instructions", + return_value=[], + ) as translate: + await frbc.trigger_schedule(system_description.message_id) + read_back = translate.call_args.kwargs["usage_forecast"] + assert [e.usage_rate_expected for e in read_back.elements] == [0.0] + + +@pytest.mark.asyncio +async def test_a_storage_planned_from_zero_is_read_back_from_zero(system_description): + from unittest.mock import patch + + frbc = _frbc(system_description) + frbc._storage_status_history[get_unique_id()] = FRBCStorageStatus( + message_id=get_unique_id(), present_fill_level=-5.0 + ) + with patch( + "flexmeasures_client.s2.control_types.FRBC.frbc_simple.fm_schedule_to_instructions", + return_value=[], + ) as translate: + await frbc.trigger_schedule(system_description.message_id) + assert translate.call_args.kwargs["initial_fill_level"] == 0.0 + + +def test_an_empty_profile_covers_no_step(system_description): + frbc = _frbc(system_description) + index = pd.date_range("2024-01-01T00:00:00+00:00", periods=4, freq="15min") + assert frbc._on_the_window(pd.Series(dtype=float), index).empty diff --git a/tests/s2/test_frbc_time_base.py b/tests/s2/test_frbc_time_base.py index eb940673..89c22530 100644 --- a/tests/s2/test_frbc_time_base.py +++ b/tests/s2/test_frbc_time_base.py @@ -694,11 +694,14 @@ def test_a_handler_needs_a_positive_resolution_and_duration(resolution, duration def test_a_long_schedule_against_a_detailed_forecast_is_translated_quickly(): - """48 hours, at quarter hours, against the largest forecast S2 allows: well under the 1.7 s that stepping each minute took.""" + """48 hours, at quarter hours, against the largest forecast S2 allows: well under the 1.7 s that stepping each minute took. + + Timed in CPU time, so that other processes on a busy machine do not count against it. + """ import time detailed = _usage(*[(timedelta(minutes=10), 1e-4 * (i % 3)) for i in range(288)]) - started = time.perf_counter() + started = time.process_time() fm_schedule_to_instructions( _schedule([500.0, 0.0] * 96), _system_description(LOW, HIGH), @@ -706,7 +709,7 @@ def test_a_long_schedule_against_a_detailed_forecast_is_translated_quickly(): usage_forecast=detailed, leakage_behaviour=_leakage((0, 50, 1e-4), (50, 100, 2e-4)), ) - assert time.perf_counter() - started < 0.5 + assert time.process_time() - started < 0.5 def test_a_level_held_at_its_lower_bound_rises_from_there_once_filling_wins(): diff --git a/tests/s2/test_frbc_unsupported_is_said.py b/tests/s2/test_frbc_unsupported_is_said.py new file mode 100644 index 00000000..ad71e746 --- /dev/null +++ b/tests/s2/test_frbc_unsupported_is_said.py @@ -0,0 +1,226 @@ +"""What this CEM does not support is said when it arrives, rather than discovered after a solve, or never.""" + +from __future__ import annotations + +import logging +from datetime import datetime, timezone + +import pytest +from s2python.common import Duration, ReceptionStatusValues, Timer +from s2python.frbc import FRBCTimerStatus + +from flexmeasures_client.s2.utils import get_unique_id + +from .test_frbc_schedule_retry import _frbc, _heater + + +def _with(system_description, **actuator_update): + actuator = system_description.actuators[0].model_copy(update=actuator_update) + return system_description.model_copy(update={"actuators": [actuator]}) + + +@pytest.mark.asyncio +async def test_a_description_of_several_actuators_is_refused_when_it_arrives(caplog): + """Acknowledged, it failed in a task nobody awaited, after a solve, on every round.""" + system_description = _heater() + frbc = _frbc(system_description) + frbc._system_description_history.clear() + two = system_description.model_copy( + update={ + "message_id": get_unique_id(), + "actuators": [ + system_description.actuators[0], + system_description.actuators[0].model_copy( + update={"id": get_unique_id()} + ), + ], + } + ) + + status = await frbc.handle_message(two.to_dict()) + + assert status.status == ReceptionStatusValues.INVALID_CONTENT + assert "exactly one actuator" in status.diagnostic_label + assert not frbc._system_description_history + assert not frbc.background_tasks + + +@pytest.mark.asyncio +async def test_a_description_with_timers_or_transitions_is_taken_with_a_warning(caplog): + system_description = _heater() + frbc = _frbc(system_description) + frbc._storage_status_history.clear() + timer = Timer( + id=get_unique_id(), + diagnostic_label="minimum on time", + duration=Duration.from_milliseconds(60_000), + ) + described = _with(system_description, timers=[timer]) + described = described.model_copy(update={"message_id": get_unique_id()}) + + with caplog.at_level(logging.WARNING): + status = await frbc.handle_message(described.to_dict()) + + assert status.status == ReceptionStatusValues.OK + assert "1 timers and 0 transitions" in caplog.text + + +@pytest.mark.asyncio +async def test_a_description_without_timers_says_nothing_of_them(caplog): + system_description = _heater() + frbc = _frbc(system_description) + frbc._storage_status_history.clear() + with caplog.at_level(logging.WARNING): + status = await frbc.handle_message( + system_description.model_copy( + update={"message_id": get_unique_id()} + ).to_dict() + ) + # The description was taken, so silence is the handler's and not a description that never reached it. + assert status.status == ReceptionStatusValues.OK + assert len(frbc._system_description_history) == 2 + assert "does not honour" not in caplog.text + + +@pytest.mark.asyncio +async def test_a_timer_status_is_kept_and_said_to_be_unhonoured(caplog): + system_description = _heater() + frbc = _frbc(system_description) + timer_status = FRBCTimerStatus( + message_id=get_unique_id(), + timer_id=get_unique_id(), + actuator_id=system_description.actuators[0].id, + finished_at=datetime(2024, 1, 1, 0, 5, tzinfo=timezone.utc), + ) + + with caplog.at_level(logging.WARNING): + status = await frbc.handle_message(timer_status.to_dict()) + + assert status.status == ReceptionStatusValues.OK + assert str(timer_status.message_id) in frbc._timer_status_history + assert "does not honour" in caplog.text + + +async def _flex_model(frbc, system_description) -> dict: + await frbc.trigger_schedule(system_description.message_id) + return frbc._fm_client.trigger_and_get_schedule.await_args.kwargs["flex_model"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("provided", [True, False]) +async def test_usage_is_planned_with_only_where_the_resource_manager_says_it_sends_it( + provided, +): + """Named regardless, the scheduler read a sensor nothing was ever recorded on.""" + system_description = _heater() + storage = system_description.storage.model_copy( + update={"provides_usage_forecast": provided} + ) + system_description = system_description.model_copy(update={"storage": storage}) + frbc = _frbc(system_description, usage_forecast_sensor_id=7) + + flex_model = await _flex_model(frbc, system_description) + + assert ("soc-usage" in flex_model) is provided + + +@pytest.mark.asyncio +async def test_a_usage_forecast_sent_despite_the_flag_is_planned_with(): + """What a resource manager sends says more than the flag it set once.""" + from datetime import timedelta + + from s2python.frbc import FRBCUsageForecast, FRBCUsageForecastElement + + system_description = _heater() # provides_usage_forecast=False + frbc = _frbc(system_description, usage_forecast_sensor_id=7) + forecast = FRBCUsageForecast( + message_id=get_unique_id(), + start_time=datetime(2024, 1, 1, tzinfo=timezone.utc), + elements=[ + FRBCUsageForecastElement( + duration=Duration.from_timedelta(timedelta(hours=1)), + usage_rate_expected=0.1, + ) + ], + ) + frbc._usage_forecast_history[str(forecast.message_id)] = forecast + + flex_model = await _flex_model(frbc, system_description) + + assert "soc-usage" in flex_model + + +@pytest.mark.asyncio +async def test_a_timer_is_warned_about_once(caplog): + """A resource manager sends a status on every run of every timer.""" + system_description = _heater() + frbc = _frbc(system_description) + timer_id = get_unique_id() + + def status(): + return FRBCTimerStatus( + message_id=get_unique_id(), + timer_id=timer_id, + actuator_id=system_description.actuators[0].id, + finished_at=datetime(2024, 1, 1, 0, 5, tzinfo=timezone.utc), + ).to_dict() + + with caplog.at_level(logging.WARNING): + await frbc.handle_message(status()) + await frbc.handle_message(status()) + + assert caplog.text.count("does not honour") == 1 + assert len(frbc._timer_status_history) == 2 + + +def _two_modes_with(transitions_between, blocking_timers=()): + """The heater, with an idle mode beside it, and transitions between the two as given.""" + from s2python.common import Transition + + system_description = _heater() + actuator = system_description.actuators[0] + on = actuator.operation_modes[0] + off = on.model_copy(update={"id": get_unique_id(), "diagnostic_label": "off"}) + transitions = [ + Transition( + id=get_unique_id(), + from_=a.id, + to=b.id, + start_timers=[], + blocking_timers=list(blocking_timers), + abnormal_condition_only=False, + ) + for a, b in transitions_between(on, off) + ] + return _with(system_description, operation_modes=[on, off], transitions=transitions) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "transitions_between, blocking_timers, warned", + [ + (lambda on, off: [(on, off), (off, on)], (), False), + (lambda on, off: [(off, on)], (), True), + ], + ids=[ + "every switch allowed", + "switching off ruled out", + ], +) +async def test_transitions_are_warned_about_only_where_they_rule_a_switch_out( + transitions_between, blocking_timers, warned, caplog +): + """An on/off device that declares both of its switches rules nothing out, as the example resource manager does.""" + system_description = _two_modes_with(transitions_between, blocking_timers) + frbc = _frbc(system_description) + frbc._storage_status_history.clear() + + with caplog.at_level(logging.WARNING): + status = await frbc.handle_message( + system_description.model_copy( + update={"message_id": get_unique_id()} + ).to_dict() + ) + + assert status.status == ReceptionStatusValues.OK + assert ("does not honour" in caplog.text) is warned diff --git a/tests/s2/test_handler_base.py b/tests/s2/test_handler_base.py new file mode 100644 index 00000000..3e9af23f --- /dev/null +++ b/tests/s2/test_handler_base.py @@ -0,0 +1,159 @@ +"""What every S2 handler does with the messages it is given, whatever its control type.""" + +from __future__ import annotations + +import json + +import pandas as pd +import pytest +from s2python.common import ControlType, Handshake, ReceptionStatusValues + +from flexmeasures_client.s2 import Handler, register +from flexmeasures_client.s2.cem import CEM +from flexmeasures_client.s2.control_types.FRBC import FRBCTest +from flexmeasures_client.s2.utils import get_reception_status + + +class HandshakeHandler(Handler): + """A handler for one message type, which counts what reaches it.""" + + def __init__(self): + super().__init__() + self.handled = [] + + @register(Handshake) + def handle_handshake(self, message: Handshake): + self.handled.append(message) + return get_reception_status(message) + + +@pytest.mark.asyncio +async def test_an_invalid_message_is_answered_as_invalid_data(rm_handshake): + handler = HandshakeHandler() + message = rm_handshake.to_dict() + message["supported_protocol_versions"] = "not a list" + + answer = await handler.handle_message(message) + + assert answer.status == ReceptionStatusValues.INVALID_DATA + assert str(answer.subject_message_id) == str(message["message_id"]) + assert answer.diagnostic_label, "the answer says what was invalid" + assert handler.handled == [], "the handler itself never sees an invalid message" + assert answer in handler.outgoing_messages.values(), "the answer is kept as sent" + + +@pytest.mark.asyncio +async def test_a_valid_message_reaches_its_handler(rm_handshake): + """The positive control for the test above: the same message, unaltered, is handled.""" + handler = HandshakeHandler() + + answer = await handler.handle_message(rm_handshake.to_dict()) + + assert answer.status == ReceptionStatusValues.OK + assert len(handler.handled) == 1 + + +@pytest.mark.asyncio +async def test_a_message_can_be_given_as_json(rm_handshake): + handler = HandshakeHandler() + + answer = await handler.handle_message(rm_handshake.to_json()) + + assert answer.status == ReceptionStatusValues.OK + assert len(handler.handled) == 1 + + +def test_a_failure_callback_is_stored_with_its_arguments(): + handler = HandshakeHandler() + called = [] + + handler.register_failure_callbacks( + "some-id", lambda **kwargs: called.append(kwargs), why="test" + ) + handler.failure_callbacks["some-id"]() + + assert called == [{"why": "test"}] + assert handler.success_callbacks == {} + + +def test_a_message_given_as_json_is_recognised_as_supported(rm_handshake): + handler = HandshakeHandler() + + assert handler.supports_message(rm_handshake.to_json()) + assert not handler.supports_message( + json.dumps({"message_type": "FRBC.StorageStatus"}) + ) + + +async def _cem_with_details(resource_manager_details, rm_handshake) -> CEM: + cem = CEM(fm_client=None) + cem.register_control_type(FRBCTest()) + await cem.handle_message(rm_handshake) + await cem.handle_message(resource_manager_details) + while not cem._sending_queue.empty(): + await cem.get_message() + return cem + + +@pytest.mark.asyncio +async def test_activating_the_control_type_in_force_sends_nothing( + resource_manager_details, rm_handshake +): + cem = await _cem_with_details(resource_manager_details, rm_handshake) + assert cem.control_type == ControlType.NO_SELECTION + + answer = await cem.activate_control_type(ControlType.NO_SELECTION) + + assert answer is None + assert cem._sending_queue.empty() + + +@pytest.mark.asyncio +async def test_a_control_type_the_resource_manager_lacks_is_not_activated( + resource_manager_details, rm_handshake +): + cem = await _cem_with_details(resource_manager_details, rm_handshake) + + answer = await cem.activate_control_type(ControlType.POWER_ENVELOPE_BASED_CONTROL) + + assert answer is None + assert cem._sending_queue.empty() + assert cem.control_type == ControlType.NO_SELECTION + + +def test_registering_a_control_type_again_replaces_its_handler(caplog): + cem = CEM(fm_client=None) + first, second = FRBCTest(), FRBCTest() + cem.register_control_type(first) + + cem.register_control_type(second) + + assert cem._control_types_handlers[ControlType.FILL_RATE_BASED_CONTROL] is second + assert "already registered" in caplog.text + + +def test_an_empty_schedule_gives_no_instructions(frbc_system_description): + from flexmeasures_client.s2.control_types.FRBC.utils import ( + fm_schedule_to_instructions, + ) + + no_values = { + "start": "2026-01-01T00:00:00+00:00", + "duration": "PT0H", + "values": [], + "unit": "W", + } + no_frame = pd.DataFrame({"schedule": []}, index=pd.DatetimeIndex([], tz="UTC")) + + assert ( + fm_schedule_to_instructions( + no_values, frbc_system_description, initial_fill_level=0.0 + ) + == [] + ) + assert ( + fm_schedule_to_instructions( + no_frame, frbc_system_description, initial_fill_level=0.0 + ) + == [] + ) diff --git a/tests/s2/test_site.py b/tests/s2/test_site.py index c67dbad9..cb0ac08e 100644 --- a/tests/s2/test_site.py +++ b/tests/s2/test_site.py @@ -10,6 +10,7 @@ from s2python.common import ControlType, ReceptionStatus, ReceptionStatusValues from flexmeasures_client.client import FlexMeasuresClient +from flexmeasures_client.exceptions import ContentTypeError from flexmeasures_client.s2.cem import CEM from flexmeasures_client.s2.site import ( FRBC_SITE_SENSORS, @@ -83,6 +84,25 @@ async def test_an_existing_site_keeps_its_sensors_and_gets_the_missing_ones(): assert specs_update.args == (7, {"generic_asset_type_id": 6}) +def test_a_site_s_sensors_suit_a_residential_site(): + """Powers and rates of fill level in W, energy in kWh: in kWh/s, a heat buffer's usage of some tens of watts would read about 0.00001.""" + units = {name: spec.unit for name, spec in FRBC_SITE_SENSORS.items()} + assert units == { + "price": "EUR/kWh", + "production price": "EUR/kWh", + "power": "W", + "measured-power": "W", + "state of charge": "kWh", + "RM discharge": "dimensionless", + "fill rate": "W", + "soc minima": "kWh", + "soc maxima": "kWh", + "usage forecast": "W", + "storage efficiency": "%", + "charging efficiency": "dimensionless", + } + + @pytest.mark.asyncio async def test_a_known_site_is_fetched_by_id_rather_than_found_in_a_listing(): """A resource manager reconnects every round of a co-simulation; a listing of every asset each time is slow.""" @@ -331,10 +351,14 @@ async def forever(details): @pytest.mark.asyncio -async def test_a_known_site_that_cannot_be_fetched_is_looked_up_again(): +@pytest.mark.parametrize( + "failure", + [ValueError("404"), ContentTypeError("Expected an asset, but got a list")], +) +async def test_a_known_site_that_cannot_be_fetched_is_looked_up_again(failure): """Deleted since, say: the listing finds it by name, or it is made again.""" fm = _flexmeasures([{"id": 9, "name": "home A", "sensors": []}]) - fm.get_asset = AsyncMock(side_effect=ValueError("404")) + fm.get_asset = AsyncMock(side_effect=failure) asset, _ = await ensure_site( fm, "home A", asset_specs={}, sensors={}, timezone="UTC", known_asset_id=7 ) diff --git a/tests/s2/test_size_limit_ordered_dict.py b/tests/s2/test_size_limit_ordered_dict.py new file mode 100644 index 00000000..be2de5d3 --- /dev/null +++ b/tests/s2/test_size_limit_ordered_dict.py @@ -0,0 +1,22 @@ +from flexmeasures_client.s2.utils import SizeLimitOrderedDict + + +def test_a_full_history_keeps_the_newest_entries(): + """popitem() takes the newest by default, so a full history kept a session's first entries instead.""" + history = SizeLimitOrderedDict(max_size=3) + for n in range(5): + history[n] = n + assert list(history) == [2, 3, 4] + + +def test_setting_a_key_already_present_evicts_nothing_and_makes_it_the_latest(): + history = SizeLimitOrderedDict(max_size=2) + history["a"] = 1 + history["b"] = 2 + history["a"] = 3 + assert list(history.items()) == [("b", 2), ("a", 3)] + + +def test_made_from_more_than_fits_it_keeps_the_last(): + history = SizeLimitOrderedDict([(n, n) for n in range(5)], max_size=2) + assert list(history) == [3, 4]