diff --git a/.github/instructions/README.md b/.github/instructions/README.md new file mode 100644 index 0000000..89c2214 --- /dev/null +++ b/.github/instructions/README.md @@ -0,0 +1,30 @@ +# `.github/instructions/` — Path-Specific Custom Instructions + +This directory contains [path-specific custom instructions](https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/add-custom-instructions/add-repository-instructions#creating-path-specific-custom-instructions) for GitHub Copilot. + +Each `.instructions.md` file covers a cross-cutting convention that applies to multiple areas of the codebase. Copilot automatically applies these instructions when working on files that match the `applyTo:` glob pattern in the file's frontmatter. + +## Files in this directory + +| File | `applyTo` | Topic | +|------|-----------|-------| +| `atomic-commits.instructions.md` | `**` | Commit discipline — one logical change per commit | +| `changelog.instructions.md` | `**` | Changelog entry format and location | +| `docstrings.instructions.md` | `**/*.py` | RST docstring format, doctests | +| `error-handling.instructions.md` | `**/*.py` | Catching specific exceptions | +| `marshmallow-schemas.instructions.md` | `flexmeasures/data/schemas/**/*.py` | `data_key`, field naming, `load_default` | +| `pre-commit-hooks.instructions.md` | `**` | Running hooks before every commit | +| `testing.instructions.md` | `flexmeasures/**/tests/**/*.py` | Full suite, `db` vs `fresh_db`, `requesting_user` | +| `timezone-awareness.instructions.md` | `**/*.py` | Always timezone-aware datetimes | +| `ui-terminology.instructions.md` | `flexmeasures/ui/**` | "organisation" not "account" in user-facing text | + +## Adding new instruction files + +When a convention appears in two or more agent files or is frequently missed, extract it here: + +1. Create `.instructions.md` in this directory. +2. Add a YAML frontmatter block with `applyTo:` pointing to the relevant files. +3. Write concise, actionable instructions in natural language Markdown. +4. Update the table above. +5. Reference the new file from the relevant agent files in `.github/agents/`. +6. Go through the agent files to remove sections that have been made redundant by the new file. diff --git a/.github/instructions/atomic-commits.instructions.md b/.github/instructions/atomic-commits.instructions.md new file mode 100644 index 0000000..4037c13 --- /dev/null +++ b/.github/instructions/atomic-commits.instructions.md @@ -0,0 +1,56 @@ +--- +applyTo: "**" +--- +# Atomic Commits + +Each commit must represent exactly one logical change. Never mix different types of changes. + +## What belongs in separate commits + +- Production code changes +- Test changes +- Documentation updates +- Agent instruction updates +- Changelog entries +- Configuration / CI changes + +## Forbidden in commits + +Never commit temporary planning or analysis files: +- `ARCHITECTURE_ANALYSIS.md`, `TASK_SUMMARY.md`, `TEST_PLAN.md`, `DOCUMENTATION_CHANGES.md` +- Any `.md` file created for understanding or planning (use `/tmp/` if you need scratch space) + +## Commit message format + +``` +: + +Context: +- What triggered this change + +Change: +- What was adjusted and why +``` + +Examples: +``` +utils/time: fix duration parsing to respect timezone + +Context: +- Bug #1234: PT2H parsed incorrectly in CET timezone + +Change: +- Pass timezone through to isodate.parse_duration +``` + +``` +agents/test-specialist: add full-suite requirement after governance failure + +Context: +- Session 2026-02-10 revealed partial test execution as root cause + +Change: +- Added explicit requirement to run complete test suite +``` + +Use the file path or area as the prefix (e.g. `api/v3_0`, `data/models`, `cli`, `docs`, `agents/`). diff --git a/.github/instructions/changelog.instructions.md b/.github/instructions/changelog.instructions.md new file mode 100644 index 0000000..5d8aa27 --- /dev/null +++ b/.github/instructions/changelog.instructions.md @@ -0,0 +1,68 @@ +--- +applyTo: "**" +--- +# Changelog Entries + +Each PR generally requires a changelog entry. + +## Changelogs + +| File | Audience | +|------|----------| +| `documentation/changelog.rst` | Main changelog — API/CLI/UI users and plugin developers | +| `documentation/api/changelog.rst` | API-specific changes | +| `documentation/cli/changelog.rst` | CLI-specific changes | + +## Entry format + +Always follow the formatting convention already present in the target changelog file — the style may evolve over time. The canonical guide is the file itself. + +Current conventions: +- **Main changelog**: one sentence, no period at the end, written for end users (abstract away technical details when possible) +- **API and CLI changelogs**: slightly more technical audience; use a period at the end + +Example for the main changelog: + +```rst +* Add cross-cutting Copilot instruction files to `.github/instructions/` [see `PR #XXXX `_] +``` + +Replace `XXXX` with the actual PR number. If the PR number is not yet known, alert the maintainer. + +## Sections in `documentation/changelog.rst` + +Always use this order: + +- **New features** — new capabilities visible to API/CLI/UI users +- **Infrastructure / Support** — changes targeting plugin developers and hosts +- **Bugfixes** — corrections to incorrect behavior + +## PR description checkbox + +The PR template includes a changelog checkbox: + +```markdown +- [ ] Added changelog item in `documentation/changelog.rst` +``` + +Tick it once the changelog entry is committed. + +## What requires a changelog entry + +Each PR generally requires an entry. Examples of what warrants one: + +- New API endpoints or parameters +- Changed or removed API behavior +- New CLI commands or options +- UI feature additions or changes +- Performance improvements that affect users +- Breaking changes (always require a changelog entry, plus migration notes) +- Changes to plugin interfaces +- Significant changes to the repo's AI infrastructure + +## What does not require a changelog entry + +- Internal refactoring with no user-visible effect +- Test-only changes +- Small agent instruction updates (typo fixes, minor wording changes) +- Pure documentation typo fixes diff --git a/.github/instructions/docstrings.instructions.md b/.github/instructions/docstrings.instructions.md new file mode 100644 index 0000000..689d014 --- /dev/null +++ b/.github/instructions/docstrings.instructions.md @@ -0,0 +1,67 @@ +--- +applyTo: "**/*.py" +--- +# Python Docstrings + +All public functions, methods, and classes must have docstrings. Use RST (reStructuredText) format for Sphinx compatibility. + +## Standard function docstring + +```python +def function_name(param1: str, param2: int) -> bool: + """One-line summary ending with a period. + + Longer description explaining purpose and usage when needed. + Parameter descriptions are aligned and separated from the colon + with at least one space. + + :param param1: Description of param1. + :param param2: Description of param2. + :returns: Description of return value. + :raises ValueError: When param1 is empty. + + Example:: + + >>> function_name("test", 42) + True + """ +``` + +## Key conventions + +- One-line summary ends with a period and fits on one line. +- Leave one blank line between the summary and the extended description. +- Align `:param name:` descriptions using spaces (not tabs) for readability. +- Use `Example::` (double colon) to introduce a doctest block. +- Complement type hints — don't duplicate them in the docstring text. +- Use exactly one space after punctuation (no double spaces after periods). +- Use line breaks only after punctuation (this facilitates review commenting and text searching). This applies to in-line comments as well, not only docstrings; keep in-line comments short. So never wrap a docstring or comment in the middle of a phrase; each physical line must end at a comma, semicolon, colon, or period. (E501 is ignored and `max-line-length` is 160, so a long clause on one line is preferred over a mid-phrase break.) + +## For code review (incl. automated reviewers) + +The line-break-after-punctuation rule above is a deliberate, load-bearing convention (stable review comments and text search), not a stylistic nicety. When reviewing a change, always report a docstring or comment that wraps mid-phrase as a review comment; do not treat such a finding as low-confidence, and do not omit or suppress it. + +## Click CLI commands + +```python +@click.command() +def my_command(): + """One-line summary of what the command does. + + Longer description with usage context and examples if helpful. + """ +``` + +## Doctests + +Include doctests in docstrings for pure utility functions. Run them with: + +```bash +pytest --doctest-modules +``` + +Doctests in `documentation/` are also executed. Keep examples accurate and test them before committing. + +## Comments + +Comments explain *why*, not *what*. Avoid redundant comments that just restate the code. Include issue numbers or external links in TODO comments when relevant. diff --git a/.github/instructions/error-handling.instructions.md b/.github/instructions/error-handling.instructions.md new file mode 100644 index 0000000..97672e9 --- /dev/null +++ b/.github/instructions/error-handling.instructions.md @@ -0,0 +1,81 @@ +--- +applyTo: "**/*.py" +--- +# Error Handling + +Catch specific exceptions, not bare `Exception`. Broad catches hide bugs and make debugging difficult. + +## What to catch and when + +```python +from sqlalchemy.exc import SQLAlchemyError + +# ✅ Catch specific, expected failures +try: + obj = create_or_fetch(...) +except SQLAlchemyError as e: + db.session.rollback() + return error_response("Database error", 500) +except ValueError as e: + return error_response(str(e), 400) +# Let unexpected exceptions (AttributeError, TypeError, etc.) propagate +# — they indicate bugs that should fail loudly +``` + +## Never do this + +```python +# ❌ Too broad: catches programming errors +try: + result = some_operation() +except Exception as e: + return error_response("Something went wrong") +``` + +## Exception categories + +| Exception type | When to catch | +|----------------|---------------| +| `SQLAlchemyError` | Database connection, integrity, or query errors | +| `ValueError` | Invalid input that was expected and validated | +| `KeyError` | Missing required data in a known structure | +| `AttributeError` | **Never** — indicates a bug in your code | +| `TypeError` | **Never** — indicates a bug in your code | +| `NameError` | **Never** — indicates a bug in your code | + +## Error messages + +Good error messages state what went wrong, explain why, and suggest a fix: + +```python +raise ValueError( + f"Invalid sensor ID {sensor_id}. " + f"Sensor must belong to account {account.name}. " + f"See https://flexmeasures.readthedocs.io/..." +) +``` + +## HTTP status codes in API endpoints + +- `400 Bad Request` — invalid client input (validation failure) +- `401 Unauthorized` — authentication required +- `403 Forbidden` — authenticated but not permitted +- `404 Not Found` — resource does not exist +- `500 Internal Server Error` — unexpected server-side failure + +## get-or-create idempotency + +Never rely on `obj.id is None` to detect if an object is newly created (SQLAlchemy may not assign IDs until commit). Return an explicit boolean instead: + +```python +def get_or_create_thing(...): + existing = db.session.query(Thing).filter_by(...).first() + if existing: + return existing, False # (object, was_created) + new_obj = Thing(...) + db.session.add(new_obj) + return new_obj, True + +thing, was_created = get_or_create_thing(...) +status_code = 201 if was_created else 200 +``` diff --git a/.github/instructions/feature-branch-sync.instructions.md b/.github/instructions/feature-branch-sync.instructions.md new file mode 100644 index 0000000..97abfb9 --- /dev/null +++ b/.github/instructions/feature-branch-sync.instructions.md @@ -0,0 +1,42 @@ +--- +applyTo: "**" +--- +# Feature Branch Synchronization + +Feature branches must be kept synchronized with `origin/main` before implementing code changes. + +## Check branch status + +Before starting implementation work, verify the branch is up to date: + +```bash +git log --oneline origin/main...HEAD --left-right +``` + +If you see < markers, origin/main has commits the branch lacks — a fresh merge is needed. + +```bash +# ❌ Don't just check git status (it only tells you about uncommitted changes) +git status # shows "nothing to commit" even if behind main + +# ✅ Do check the commit graph +git log --left-right origin/main...HEAD +``` + +## Merge before implementation + +```bash +git fetch origin +git merge origin/main +# Resolve any conflicts +git add . +git commit -m "Merge origin/main into feature branch" +``` + +This ensures your implementation starts from the latest state of the repository. + +## Why this matters + +- Merging later causes merge conflicts to compound +- Large late merges are harder to review +- Feature work should build on current main, not diverge diff --git a/.github/instructions/marshmallow-schemas.instructions.md b/.github/instructions/marshmallow-schemas.instructions.md new file mode 100644 index 0000000..44f4bd8 --- /dev/null +++ b/.github/instructions/marshmallow-schemas.instructions.md @@ -0,0 +1,69 @@ +--- +applyTo: "flexmeasures/data/schemas/**/*.py" +--- +# Marshmallow Schema Conventions + +## `data_key` defines the actual dictionary key + +When a Marshmallow field has a `data_key` attribute, the serialized/deserialized dictionary uses the `data_key` value, **not** the Python attribute name. All code that accesses, cleans, stores, or compares such dictionaries must use the `data_key` values. + +```python +class ForecasterParametersSchema(Schema): + as_job = fields.Boolean(data_key="as-job") # dict key: "as-job" + sensor_to_save = SensorIdField(data_key="sensor-to-save") # dict key: "sensor-to-save" +``` + +```python +# ✅ Correct: use data_key format +params.pop("as-job", None) +value = params.get("sensor-to-save") + +# ❌ Wrong: Python attribute name — key does not exist in the dict +params.pop("as_job", None) +value = params.get("sensor_to_save") +``` + +## Audit all code paths that touch schema output + +When a schema changes format (e.g., snake_case → kebab-case migration), update **every** code path that accesses those dictionaries: + +1. Parameter cleaning (`.pop()`, `del`, `fields_to_remove`) +2. Parameter access (`.get()`, `[]`) +3. DataSource attributes storage +4. RQ job metadata (`job.meta`) +5. Schema parity across related schemas (see below) + +## Schema parity: `Input` and `BeliefsSearchConfigSchema` + +Two separate Marshmallow schemas expose `Sensor.search_beliefs` parameters: + +- `flexmeasures/data/schemas/io.py` → `Input` (used by reporters and forecasters) +- `flexmeasures/data/schemas/reporting/__init__.py` → `BeliefsSearchConfigSchema` (used by sensor status config) + +When adding a parameter to `Sensor.search_beliefs`, add it to **both** schemas. Omitting one creates a silent gap where documented features fail at schema validation. + +## API endpoint schemas + +Use one schema that separates load vs. dump behaviour with `dump_only` and `load_only`: + +```python +class AnnotationSchema(Schema): + id = fields.Int( + dump_only=True, # server-generated; excluded from request validation + metadata=dict(description="The annotation's ID.", example=19), + ) + content = fields.Str( + required=True, + validate=Length(max=1024), + metadata={"description": "Text content (max 1024 chars).", "examples": ["Maintenance"]}, + ) + source_id = fields.Int( + data_key="source", # serialized as "source" in JSON + dump_only=True, + metadata=dict(description="Data source ID.", example=21), + ) +``` + +- Always include `metadata` with `description` and at least one `example`. +- Use single-word `data_key` values where possible (prefer `"source"` over `"source_id"`). +- `dump_only=True` fields are excluded from request body validation and appear only in responses. diff --git a/.github/instructions/pre-commit-hooks.instructions.md b/.github/instructions/pre-commit-hooks.instructions.md new file mode 100644 index 0000000..8f49f5c --- /dev/null +++ b/.github/instructions/pre-commit-hooks.instructions.md @@ -0,0 +1,55 @@ +--- +applyTo: "**" +--- +# Pre-commit Hooks + +Every commit must pass pre-commit hooks. Committing code that fails pre-commit is a process failure. + +## Setup + +```bash +uv tool install pre-commit +pre-commit install # optional: install git hooks locally +``` + +## Run before every commit + +```bash +pre-commit run --all-files +``` + +All hooks must pass with zero failures before committing. + +## Hooks enforced + +| Hook | What it checks | +|------|----------------| +| **flake8** | Linting: unused imports, line complexity, style violations | +| **black** | Code formatting (auto-fixes; commit the result) | +| **mypy** | Type annotations (via `ci/run_mypy.sh`) | +| **generate-openapi-specs** | OpenAPI spec generation (local only, skipped in CI) | + +## Flake8 configuration + +Max line length: 160. Max complexity: 13. Ignored rules: E501, W503, E203. + +## Fixing failures + +```bash +# Black: auto-fix formatting +black path/to/file.py + +# Flake8: fix manually (unused imports, complexity) +flake8 path/to/file.py + +# Mypy: add type hints or justified `# type: ignore` comments +ci/run_mypy.sh +``` + +## openapi-specs hook caution + +The `generate-openapi-specs` hook can introduce unintended regressions (version string becoming a dev string, timezone list changing). After running it, inspect the diff and revert any changes that are not motivated by your PR: + +```bash +git diff flexmeasures/ui/static/openapi-specs.json +``` diff --git a/.github/instructions/testing.instructions.md b/.github/instructions/testing.instructions.md new file mode 100644 index 0000000..7df2b70 --- /dev/null +++ b/.github/instructions/testing.instructions.md @@ -0,0 +1,107 @@ +--- +applyTo: "flexmeasures/**/tests/**/*.py" +--- +# Testing Conventions + +## Run the full test suite + +Before finishing a session and requesting a review, run the complete test suite — partial test runs are insufficient because FlexMeasures has interconnected systems where changes in one area affect others: + +```bash +uv sync --group test +uv run poe test +``` + +Intermediate full test runs are encouraged when appropriate (e.g., after a significant refactor). During active development, targeted runs (`pytest path/to/test_module.py`) are acceptable for quick iteration but must not substitute for the full suite at session close. + +## Run the full module after changing any test + +When you fix or add a single test, always run the entire test module before closing: + +```bash +pytest flexmeasures/path/to/test_module.py -v +``` + +Fixing one test can break adjacent tests in the same module when they share mutable module-scoped fixtures. + +## Database fixture selection + +| Fixture | When to use | +|---------|-------------| +| `db` | Read-only tests — queries only, no mutations | +| `fresh_db` | Tests that create, update, or delete data | + +Using `db` when data is mutated causes `DetachedInstanceError` and flaky cross-test contamination. + +### Never mix `fresh_db` and `db` in the same module + +`fresh_db` is function-scoped and calls `_db.drop_all()` before and after each test. If a `db` test (module-scoped) is open at the same time, `drop_all()` will block forever waiting for the module-scoped connection to release its locks — **hanging CI indefinitely**. + +**Rule:** every test module must use either `db` OR `fresh_db` — never both. + +Put `fresh_db` tests in a dedicated `*_fresh_db.py` sibling module, following the established convention: + +``` +# ✅ Correct structure +test_api_v1_1.py ← uses `db` +test_api_v1_1_fresh_db.py ← uses `fresh_db` + +test_utils.py ← uses `db` +test_utils_fresh_db.py ← uses `fresh_db` + +# ❌ Wrong: mixes both fixtures in one file +test_utils.py ← uses `db` AND `fresh_db` ← CI will hang +``` + +## API test isolation + +```python +# ✅ Correct: use the requesting_user fixture +def test_my_endpoint(client, requesting_user): + response = client.get("/api/v3_0/...") + +# ❌ Wrong: manually patching _check_token breaks the auth flow +with patch("flexmeasures.api.common._check_token"): + ... +``` + +## Test design intent + +Before changing a test that fails, investigate whether the test is intentionally designed to catch a production bug: + +1. Read what the test is doing and why. +2. Check the production code for the real bug. +3. Only modify a test if you can prove the test design is wrong. + +A failing test often reveals a production bug, not a test bug. + +## Prove a new test can fail + +A test that never fails asserts nothing, and reads exactly like one that works. +Before calling a new test done, break what it covers — comment out the constraint, invert the condition — and confirm it goes red, then restore. + +Watch for assertions that depend on the problem having a unique answer. +An optimisation test over devices with no incentive to move has the same optimum with or without the constraint, so it passes either way. +In `test_highspy_equivalence.py`, where two backends are compared, also disable the new code path on one side: a scenario that survives that is comparing two no-ops. +Both traps have been hit — a balance-group scenario there passed with the constraint it was named after entirely disabled. + +Say in the PR description what you broke to prove it — a reviewer cannot tell a binding test from a vacuous one by reading it. + +## Module-scoped fixture state + +Module-scoped fixtures are shared across tests. When modifying shared objects (e.g. `asset.sensors_to_show`), reset them to the column default — not to `None` — in teardown: + +```python +# ✅ Reset to column default (empty list) +asset.sensors_to_show = [] + +# ❌ Reset to None (may cause unexpected ValidationError downstream) +asset.sensors_to_show = None +``` + +## Authentication failures in tests + +If you see unexpected `401 Unauthorized` in tests: +- Check that the `requesting_user` fixture is used. +- Verify `patch_check_token` is applied (it should be automatic via conftest). +- Do not manually patch authentication mechanisms. diff --git a/.github/instructions/timezone-awareness.instructions.md b/.github/instructions/timezone-awareness.instructions.md new file mode 100644 index 0000000..1dd8b65 --- /dev/null +++ b/.github/instructions/timezone-awareness.instructions.md @@ -0,0 +1,68 @@ +--- +applyTo: "**/*.py" +--- +# Timezone-Aware Datetimes + +FlexMeasures treats naive datetimes as an error. All datetime objects must be timezone-aware. + +## Creating datetimes + +```python +# ✅ Correct: always timezone-aware +from datetime import datetime, timezone +dt = datetime(2024, 1, 15, 10, 0, 0, tzinfo=timezone.utc) + +# ❌ Wrong: naive datetime +dt = datetime(2024, 1, 15, 10, 0, 0) +``` + +## Pandas DatetimeIndex + +```python +# ✅ Correct: localize naive index +index = index.tz_localize("Europe/Amsterdam") + +# ✅ Correct: convert aware index to different timezone +index = index.tz_convert("Europe/Amsterdam") + +# ❌ Wrong: arithmetic on timezone-naive index near DST transitions +``` + +## Timezone configuration + +| Setting | Purpose | +|---------|---------| +| `FLEXMEASURES_TIMEZONE` | Platform-wide default (default: `"Asia/Seoul"`) | +| `current_user.timezone` | User-specific timezone | +| `sensor.timezone` | Sensor-specific timezone for event timestamps | + +## Nominal durations (ISO 8601) + +Nominal durations like `P1M` (one month) cannot be converted to a `timedelta` without a reference datetime. Always pass a start datetime when grounding nominal durations: + +```python +# ❌ Wrong: P1M has no fixed length +delta = isodate.parse_duration("P1M") # returns isodate.Duration, not timedelta + +# ✅ Correct: ground nominal duration to a calendar date +start = datetime(2024, 1, 1, tzinfo=timezone.utc) +end = start + isodate.parse_duration("P1M") # Jan 1 + 1 month = Feb 1 +``` + +## DST transitions + +- Spring forward / fall back transitions cause off-by-one-hour bugs. +- When iterating over hourly slots, use `pd.date_range` with `freq` rather than adding `timedelta(hours=1)` in a loop. +- Test DST boundary cases explicitly (e.g., `Europe/Amsterdam` on last Sunday of March and October). + +## Documentation examples + +All code examples in documentation must use timezone-aware datetimes: + +```python +# ✅ In docs +from datetime import datetime, timezone +start_time = datetime(2024, 1, 15, 10, 0, 0, tzinfo=timezone.utc) +``` + +Never show a naive `datetime(...)` call in user-facing documentation or in log statements. diff --git a/.github/instructions/ui-terminology.instructions.md b/.github/instructions/ui-terminology.instructions.md new file mode 100644 index 0000000..8389ed0 --- /dev/null +++ b/.github/instructions/ui-terminology.instructions.md @@ -0,0 +1,44 @@ +--- +applyTo: "flexmeasures/ui/**" +--- +# UI Terminology + +Use **"organisation"** (not "account") in all user-visible text. The word "account" is too easily confused with a user login account. + +## The rule + +| Context | ✅ Use | ❌ Avoid | +|---------|--------|---------| +| Button labels | "Copy to my organisation" | "Copy to my account" | +| Tooltips | "contact your organisation admin" | "contact your account admin" | +| Flash messages | "Organisation not found" | "Account not found" | +| Template strings | "Select organisation" | "Select account" | +| Error messages shown to users | "You must be an organisation admin" | "You must be an account-admin" | + +## What stays unchanged + +The Python backend model is still named `Account`. Database columns, Python identifiers, URLs, and API field names all stay as-is. Only the text **shown to end users** changes. + +## Internal role names + +Never expose internal role names (e.g. `account-admin`, `admin`) in UI text. Use plain language instead: + +- ✅ "organisation admin" +- ❌ "account-admin" +- ❌ "ACCOUNT_ADMIN" + +## Jinja2 template safety + +- Sensor/asset IDs embedded in JavaScript use `{{ sensor.id }}` (integer — safe), not `.name` or freeform text. +- User-supplied values displayed in HTML use `{{ value | e }}` for auto-escaping. +- Only use `{{ value | safe }}` for pre-sanitised server values (e.g. `sensor._ui_unit | safe`). + +## Toast vs. inline alert + +| Use case | Pattern | +|----------|---------| +| Action result (save, delete, copy) | `showToast("...", "success"/"error")` | +| Background job progress | `showToast("...", "info")` | +| Persistent field description users need while interacting | `
` | + +Do not migrate inline `alert-info` divs showing persistent information to toasts — they could disappear before the user reads them. diff --git a/examples/lf-energy-demo/.gitignore b/examples/lf-energy-demo/.gitignore new file mode 100644 index 0000000..c162d31 --- /dev/null +++ b/examples/lf-energy-demo/.gitignore @@ -0,0 +1,5 @@ +fm-walkthrough-instance/* +!fm-walkthrough-instance/flexmeasures.cfg + +# Plans saved by python/compare_schedules.py, and the comparison.html diagram it draws from them. +schedule-runs/ diff --git a/examples/lf-energy-demo/LIVE_DEMO.md b/examples/lf-energy-demo/LIVE_DEMO.md new file mode 100644 index 0000000..24338f0 --- /dev/null +++ b/examples/lf-energy-demo/LIVE_DEMO.md @@ -0,0 +1,180 @@ +# Running this walkthrough as a live demo + +A leaner companion to the [main walkthrough](README.md), reordered for presenting it live: +seed everything that has no UI equivalent up front, then do as much as possible in the +FlexMeasures UI in front of an audience. Each step below is tagged **[UI]** or **[CLI]**. +Field values, form labels and troubleshooting are not repeated here — see the main README +for those; this doc is about *what to click versus what to run*, and in what order. + +## At a glance + +| # | Step | Where | +|---|------|-------| +| 1 | Seed assets + a week of forecasts | **[CLI]** | +| 2 | Trigger the baseline ("before") schedule | **[CLI]** | +| 3 | Seed the capacity-limit event on the VTN | **[CLI]** — no UI equivalent | +| 4 | Log in, open OpenADR 3 Configuration | **[UI]** | +| 5 | Create the VEN client | **[UI]** | +| 6 | Add the polling schedule, then trigger the fetch | **[UI]** + **[CLI]** — no UI trigger | +| 7 | Show the fetched OpenADR sensor values | **[UI]** | +| 8 | Nest the VEN asset under the campus | **[UI]** (or **[CLI]**) | +| 9 | Wire the flex-context live | **[UI]** | +| 10 | Trigger the after-OpenADR schedule | **[CLI]** — no UI equivalent | +| 11 | Show the shift in power usage | **not available in the FlexMeasures UI** — custom HTML report | + +## 1. Seed assets and forecasts — [CLI] + +Same as the main README's Steps 2-3: + +```bash +cd examples/lf-energy-demo +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_forecasts.py +``` + +## 2. Trigger the baseline ("before") schedule — [CLI] + +```bash +cd python +uv run compare_schedules.py +``` + +**Do this now, before wiring anything.** This is easy to get wrong when reordering steps +for a live flow: there is no "before" plan to compare against later unless you trigger it +*before* the flex-context is wired in Step 9. If you skip this and only trigger once at the +end, Step 11's comparison report will refuse to run (it needs both a `before-openadr` and +an `after-openadr` saved run). + +## 3. Seed the capacity-limit event on the VTN — [CLI], no UI equivalent + +Still in `python/`: + +```bash +uv run seed_events.py +``` + +The plugin has no VTN-administration CLI or UI — this talks to the OpenLEADR-rs VTN +directly via the Business Logic OAuth client. There is nothing to show on screen here +beyond the script's own printed summary. + +## 4. Log in, open OpenADR 3 Configuration — [UI] + +`http://localhost:5002`, sign in with the toy account, then **OpenADR 3 Configuration** in +the main menu (or `/flexmeasures-openadr3/dashboard` directly). + +## 5. Create the VEN client — [UI] + +**Add new VEN client**, fill in the form (name `demo-ven`, VTN URL, OAuth client +id/secret/token URL — see the main README's Step 7 table for exact values), **Create VEN +client**. + +## 6. Add the polling schedule — [UI], then trigger the fetch — [CLI] + +On the dashboard, **demo-ven** → **Add schedule**: name `demo-poll`, check both import and +export capacity limits, **Save polling schedule**. + +There's no UI button to run the fetch immediately — trigger it from the host, `cd ..` back +to `examples/lf-energy-demo` first: + +```bash +docker compose exec server uv run --active --no-project /walkthrough/python/trigger_fetch.py +``` + +## 7. Show the fetched OpenADR sensor values — [UI] + +Genuinely built into FlexMeasures, nothing custom here: **Polling schedules** for +`demo-ven` → **Import sensor** or **Export sensor** next to `demo-poll` → the sensor's +chart/beliefs view. You should see 96 beliefs, sourced from **OpenADR 3 VTN**. + +## 8. Nest the VEN asset under the campus — [UI] (or [CLI]) + +Without this step, Step 9's sensor search cannot find `demo-ven`'s sensors: FlexMeasures' +"Edit flex-context" picker only searches an asset's own descendants, and `demo-ven` is +otherwise created as an unrelated top-level asset. + +**Option A (UI) — move it live:** every asset page has a view switcher next to its +breadcrumb (**Context** / **Graphs** / **Properties** / **Audit Log** / **Status**). +Open `demo-ven`'s asset page — find it via **Assets** in the main nav, since the OpenADR +plugin's own pages don't link out to it — switch to **Properties**, and in the "Edit asset" panel on +the left set **Parent Asset Id** to `demo-campus (ID: ...)` — it appears in that dropdown +automatically, since it's an ordinary asset-edit field listing every other asset in the +account, not something specific to VEN clients. Click **Save**. + +**Option B (CLI):** + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py --skip-openadr-wiring +``` + +Nests `demo-ven` under `demo-campus` the same way, and additionally leaves +`site-consumption-capacity`/`site-production-capacity` unset (with `--skip-openadr-wiring`) +so Step 9 has something real to do — useful for a non-interactive run, or as a fallback if +you'd rather not click through Option A live. + +Either way, you'll see `demo-ven` appear in `demo-campus`'s "Structure" tab from now on. + +## 9. Wire the flex-context live — [UI] + +Open `demo-campus`'s asset page, **Edit flex-context**: + +1. Select `site-consumption-capacity` → **Add field** → search for and pick + `import-capacity-limit-demo-poll`. +2. Select `site-production-capacity` → **Add field** → search for and pick + `export-capacity-limit-demo-poll`. +3. **Save**. + +This is the step the whole demo has been building toward: the DR signal fetched from the +VTN is now a real scheduling constraint, wired in without touching a script. It's also +safe: the modal loads `demo-campus`'s *entire* existing flex-context (day-ahead prices, +`site-power-capacity`, breach prices, and so on) into the editor first, and **Add field** +only adds to that in-memory object — **Save** always PATCHes the whole thing back, so it +cannot clobber the fields `seed_assets.py` already set. (That's not automatic if you ever +drive the same API by hand instead of the UI — a partial `PATCH .../assets/8` with just the +two new fields silently wipes the rest of the flex-context. The UI form doesn't have that +failure mode.) + +## 10. Trigger the after-OpenADR schedule — [CLI], no UI equivalent + +```bash +cd python +uv run compare_schedules.py +``` + +FlexMeasures has **no "Trigger schedule" button anywhere in its UI** — the only trigger +button in the whole product is "Trigger forecast" on a sensor page, which is a different +thing. Triggering a schedule is an API call (`POST .../schedules/trigger`) plus a queued +job, full stop; this script is the only way to do it in this walkthrough. It reads +`demo-campus`'s flex-context again, finds the fields from Step 9, and labels this run +`after-openadr`. + +This absence isn't specific to `demo-campus` — there is no "Trigger schedule" button on +*any* asset's page, campus or child. You also don't need one: triggering the top-level +`demo-campus` already schedules the whole hierarchy in one pass. The script's own output +shows this — one trigger call produces a plan for the campus's own grid connection *and* +its children: + +``` +Campus grid connection: 96 values on demo-campus/power +EVSE charging hub: 96 values on demo-evse-hub/power +Office heat pump: 96 values on demo-office-heat-pump/power +``` + +## 11. Show the shift in power usage — not available in the FlexMeasures UI + +This is the one thing you cannot show natively in FlexMeasures, for a structural reason +worth explaining to an audience rather than working around: both triggered runs write +their schedules as beliefs to the *same* sensors, from the *same* `scheduler` data source. +Any FlexMeasures chart — or the schedule-fetch API endpoint itself — only ever shows the +most recent one; there's no concept of "run A" vs "run B" to overlay. (See +`python/schedule_comparison.py`'s module docstring for the full explanation.) + +That's why `compare_schedules.py` saves each run to its own file +(`../schedule-runs/{before,after}-openadr.json`) the moment it's triggered, and why, once +both exist (after this step), it automatically prints a before-and-after report and writes +`../schedule-runs/comparison.html` — open that in a browser for the multi-panel diagram +with a toggle between "Before OpenADR" and "With OpenADR". This is a custom artifact of +this repo, not a FlexMeasures feature. + +If you still want *something* on screen inside FlexMeasures itself: `demo-campus`'s own +asset dashboard plots the connection capacity alongside the wired OpenADR sensors over the +seeded week — a real UI feature, just without a before/after toggle. diff --git a/examples/lf-energy-demo/README.md b/examples/lf-energy-demo/README.md new file mode 100644 index 0000000..890d712 --- /dev/null +++ b/examples/lf-energy-demo/README.md @@ -0,0 +1,538 @@ +# LF Energy demo walkthrough + +Hands-on example combining two [LF Energy](https://lfenergy.org/) projects — FlexMeasures +and OpenLEADR — into one demo stack. It mirrors the end-to-end test in +`tests/test_ven_fetch_e2e.py`, extended with a realistic three-level FlexMeasures asset +hierarchy so the fetched DR signal ends up constraining a real schedule instead of just +landing on a sensor: + +1. Start an OpenADR 3.1 VTN (OpenLEADR-rs) with Keycloak OAuth, plus a FlexMeasures instance +2. Seed a demo campus / EVSE hub / office asset hierarchy in FlexMeasures +3. Seed a week of prices, forecasts and typical-usage profiles onto that hierarchy's sensors +4. Trigger a baseline schedule for the campus, before any OpenADR wiring +5. Seed capacity-limit events on the VTN via Python (Business Logic client) +6. Configure a VEN client and polling schedule in the FlexMeasures UI +7. Fetch events, then wire the resulting capacity-limit sensors into the site as a scheduling constraint +8. Trigger the same schedule again and compare it with the baseline, to see what OpenADR changed +9. Inspect beliefs on the linked sensors + +## Stack + +| Service | URL (host) | Purpose | +|---------|------------|---------| +| FlexMeasures | http://localhost:5002 | Web UI + plugin dashboard | +| OpenLEADR-rs VTN | http://localhost:3000 | Publishes DR events | +| Keycloak | http://localhost:8080 | OAuth for BL and VEN clients | +| PostgreSQL (FM) | localhost:5433 | FlexMeasures database | +| Redis | localhost:6380 | RQ job queue | + +Three more services run alongside these with no exposed port, so they only show up in +`docker compose ps`, not in the table above: `worker` (processes the OpenADR fetch queue, +`ingestion`), `scheduling-worker` (processes triggered scheduling jobs, on its own +`scheduling` queue — Steps 4 and 10 need it), and `cron-scheduler` (evaluates polling +schedules and enqueues fetch jobs at their configured time). `docker compose up -d` starts +all three automatically; if a triggered schedule sits `QUEUED` forever, it's usually because +`scheduling-worker` isn't running (see [Troubleshooting](#troubleshooting)). + +## Prerequisites + +- Docker and Docker Compose +- [uv](https://docs.astral.sh/uv/) (for running the Python scripts, on your host and inside the container) + +Presenting this live? [`LIVE_DEMO.md`](LIVE_DEMO.md) reorders these steps to do as much as +possible in the FlexMeasures UI, with each step tagged as UI or CLI. + +--- + +## Step 1 — Start the stack + +```bash +cd examples/lf-energy-demo +docker compose up -d --build +``` + +Wait until FlexMeasures is ready (first start runs DB migrations and creates a toy account): + +```bash +docker compose logs -f server +``` + +Look for the gunicorn startup message, then open `http://localhost:5002` to verify the dashboard is visible (this can take up to 2 minutes during initial startup). + +--- + +## Step 2 — Seed the FlexMeasures asset hierarchy + +`seed_assets.py` builds a realistic three-level demo site — a campus with an EVSE charging +hub and a grid-friendly office — as real `GenericAsset` and `Sensor` rows, so the capacity +limits fetched later in this walkthrough have a site to constrain. Like `trigger_fetch.py` +(Step 8), it runs inside the FlexMeasures container against the ORM directly, reusing the +container's own uv-managed venv, so it needs no API token: + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py +``` + +This needs the `Walkthrough Toy Account`, which the `server` container's entrypoint creates +on first boot (Step 1) — if it isn't there yet, wait for the gunicorn startup message and +try again. The script is idempotent: assets and sensors are looked up by name before being +created, so it's safe to re-run any time, including once more in Step 9. + +It builds: + +``` +demo-campus building, 1 MVA grid connection point +├── demo-evse-hub building, 400 kVA shared sub-connection +│ ├── demo-evse-01…06 one-way_evse, 22 kW +│ └── demo-evse-07…08 two-way_evse, 11 kW (V2G) +└── demo-office building, 250 kVA sub-connection + ├── demo-office-pv solar, must-take production + ├── demo-office-baseload process, inflexible consumption + └── demo-office-heat-pump heat-storage, thermal buffer +``` + +`demo-office-heat-pump` models a heat pump plus the building's thermal mass as a +`heat-storage` device, since FlexMeasures has no dedicated heat-pump asset type. `demo-campus` +carries the flex-context (day-ahead prices, site capacity, breach prices), which FlexMeasures +reads upwards through the tree; every device below it carries its own flex-model entry, which +FlexMeasures gathers downwards, so triggering a schedule on `demo-campus` optimises the whole +site in one pass. Open `http://localhost:5002` and browse to the campus asset to see it — the +script's closing line prints a direct link. + +See [`flexmeasures/README.md`](flexmeasures/README.md) for more detail on the hierarchy. + +--- + +## Step 3 — Seed a week of prices, forecasts and usage profiles + +The hierarchy from Step 2 has sensors but no data, so its charts are empty and a schedule +would have nothing to plan around — not even a price to optimise against. +`seed_forecasts.py` fills the coming seven days with synthetic but plausible data, and runs +in the container just like Step 2: + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_forecasts.py +``` + +It writes three different kinds of belief, from two clearly-labelled data sources, because +they do not mean the same thing: + +| Sensors | Data source | Meaning | +|---------|-------------|---------| +| `day-ahead prices`, `demo-office-baseload/power`, `demo-office-pv/power` | `LF Energy demo forecaster` (`forecaster`) | Real forecasts. The prices are what the scheduler optimises against; the other two are the site's `inflexible-consumption` and `inflexible-production`, which it reads as given and plans around. | +| `demo-evse-01…08/power`, `demo-office-heat-pump/power` | `LF Energy demo profiles` (`demo script`) | Reference profiles: what the site would do if nobody optimised it. These devices are dispatched by the scheduler, which writes its own beliefs over the same window from its own source. | +| `demo-evse-01…08/state of charge`, `demo-office-heat-pump/state of charge` | `LF Energy demo profiles` (`demo script`) | One current reading per storage device, so `soc-at-start` resolves and a schedule can be triggered straight away. | + +The shapes are what you would expect of an office campus on the Dutch market: prices trough +overnight, peak as the country wakes up, dip in the middle of the day while the sun is on +the roofs and peak again in the early evening; the background load sits on a night floor of +servers and standby, climbs over the hour before opening, dips over lunch and falls away +after 18:00, with a much lower weekend; the PV follows a daylight bell curve whose height +varies from day to day with cloud cover; the heat pump pre-heats the building in the small +hours and only tops up while it is occupied; and the charge points fill up over the office +day, most of them plugging in around 08:00 and unplugging around 17:00, with the occasional +shorter stay for an early bird or a late worker. + +The price curve is written to the public `day-ahead prices` sensor on FlexMeasures' own +`NL transmission zone` asset, which is what `demo-campus`'s flex-context already points at. +It is also the one thing the scheduler refuses to run without: with an empty price sensor it +stops at `Prices unknown for planning window` before it looks at anything else. + +The values are deterministic — each profile is drawn from a generator seeded on the sensor +and the local calendar day — so a re-run merges the same rows back over themselves instead +of producing a different week. Re-run it whenever the week has moved on, or pass +`--days` to seed a shorter or longer horizon. + +The aggregate `power` sensors of `demo-campus`, `demo-evse-hub` and `demo-office` are left +empty on purpose: those are the scheduler's output, and a synthetic aggregate there would +simply disagree with the schedule once you trigger one. + +--- + +## Step 4 — Trigger a baseline schedule, before OpenADR + +With a week of prices and forecasts in place but no OpenADR wiring yet, trigger one +schedule for `demo-campus` and save the plan it comes up with on its own — the "before" +half of the before-and-after comparison this walkthrough builds towards. + +`compare_schedules.py` is a [uv script](https://docs.astral.sh/uv/guides/scripts/) like +`seed_events.py`: it runs on your host rather than in a container, and only ever talks HTTP +to FlexMeasures at `http://localhost:5002`. + +```bash +cd python + +uv run compare_schedules.py +``` + +From `examples/lf-energy-demo`, this moves you into `python/`; this and the other Python +steps below assume you're still there, and any later `docker compose` command needs you +back at `examples/lf-energy-demo` (`cd ..`). + +The script triggers a schedule, waits for the scheduling job to finish, and saves the +result to `../schedule-runs/before-openadr.json`. It works out that this is the "before" +run itself, by reading `demo-campus`'s current flex-context: since the OpenADR +capacity-limit fields aren't wired in yet (that happens in Step 9), it labels the run +`before-openadr`. It then prints a note that there's nothing to compare yet — that's +expected, and gets resolved once you run it again in Step 10. + +If this fails because a triggered schedule never leaves `QUEUED`, see +[Troubleshooting](#troubleshooting) — it usually means the `scheduling-worker` service +isn't running. + +--- + +## Step 5 — Seed capacity-limit events on the VTN + +The plugin has no CLI for VTN administration. Use the Business Logic OAuth client +(`test-client-id`) to create a 24-hour event with 96 fifteen-minute intervals with an import and export capacity limit event from OpenADR 3.1 for this example scenario. + +`seed_events.py` is a [uv script](https://docs.astral.sh/uv/guides/scripts/) — its +dependencies are declared inline, so `uv run` resolves and caches them on first use +without any manual venv setup. It's still in `python/` from Step 4: + +```bash +uv run seed_events.py +``` + +Expected output: + +``` +Created event id=… with 96 intervals. + First interval start (UTC): … + Last interval start (UTC): … + Import capacity is 400 kW, dropping to 70 kW from … for 10:00:00. + That is Tue 08:00 to 18:00 local time (Europe/Amsterdam). +``` + +The event spans 24 hours. Import capacity stays generous (400 kW) for most of that day and +drops to a hard **70 kW** for a 10-hour window; export capacity stays generous throughout. + +Unlike the event start, the curtailment window itself is **pinned to a fixed local +wall-clock window — 08:00 to 18:00 in `Europe/Amsterdam`, the demo site's whole office +day** — rather than offset from whenever the script happens to run, and rather than some +narrower slice of it. That's deliberate, not incidental: taking the office day whole +guarantees the curtailment overlaps the EVSE fleet's entire presence (arrivals centred just +after opening, the earliest departures mid-afternoon) and the heat pump's whole occupied +period, so nothing flexible is left sitting outside the ask with somewhere easy to hide. It +also reaches past the day-ahead price curve's midday solar dip at both ends — into the +morning ramp near opening and the evening ramp near closing — hours where the price signal +alone would not have discouraged load, and in the morning would have positively encouraged +filling up before the peak. A flat 70 kW cap held across all of that is unmistakably the +capacity limit's doing rather than price optimisation happening to agree with it. + +One honest caveat: because the window now opens right at office opening, before the site's +rooftop PV has come up under a baseload that has already switched to its day level, a cloudy +morning can leave the site's own inflexible load at or just above 70 kW for the first +interval or two — load the scheduler cannot move out of the way no matter how it plans. That +residual breach is honest rather than a misconfiguration; a real congestion ask does not stop +at what the site finds convenient. So Step 10's report should normally show the intervals +over the limit falling to a low number, but not always cleanly to zero — how low depends on +the day's seeded cloud cover. In the representative run below, seeded weather happened to be +clear enough that it did fall to zero. + +It's one clear, deliberate curtailment window rather than noise, precisely so that Step 10's +before-and-after comparison has one obvious thing to show. Re-run `seed_events.py` any time +you want fresh data, but remove the previous event from the VTN first — see +[Troubleshooting](#troubleshooting). + +**Edge case:** if local time is already inside 08:00-18:00 when you run the script, the +pinned window would no longer fit inside the 24-hour horizon Steps 4 and 10 schedule +against, so the script falls back to a short-notice curtailment starting almost immediately +instead, and prints a note explaining why. Because the window is now a full 10 hours rather +than a narrow 4-hour slice, this fallback path is common rather than a rare edge case: the +office day only fits inside the coming 24-hour schedule horizon when the script happens to +run outside office hours, so a walkthrough run *during* the day typically takes the fallback. +The comparison still works, it just no longer +tells the "reaches into the price curve's morning and evening peaks" story above — the +short-notice window instead runs from roughly now for an office day's worth of hours, still +plugged-in-fleet-first but no longer aligned with the site's actual opening and closing +times. + +--- + +## Step 6 — Log in to FlexMeasures + +Open `http://localhost:5002` and sign in with the toy account created by Docker: + +| Field | Value | +|-------|-------| +| Email | `toy-user@flexmeasures.io` | +| Password | `toy-password` | + +Open **OpenADR 3 Configuration** in the main menu, or go directly to: + +`http://localhost:5002/flexmeasures-openadr3/dashboard` + +--- + +## Step 7 — Create a VEN client + +Click **Add new VEN client** and fill in the form. + +> Use **Docker-internal hostnames** for VTN and OAuth URLs. FlexMeasures resolves +> these from inside the `server` and `worker` containers. + +| Field | Value | +|-------|-------| +| **Name** | `demo-ven` | +| **VTN URL** | `http://openleadr-vtn:3000` | +| **OAuth client ID** | `test-ven-1` | +| **OAuth client secret** | `my-client-secret` | +| **OAuth token URL** | `http://keycloak:8080/realms/integration-test-realm/protocol/openid-connect/token` | +| **Scopes** | *(leave empty — Keycloak default scopes are used)* | + +Click **Create VEN client**. + +These credentials come from `keycloak/realm.json`, this file is human readable and can be used to view the exact configuration of the configured OAUTH clients which communicate with an OpenADR 3.1 VTN. + +--- + +## Step 8 — Add a polling schedule + +On the dashboard, open **demo-ven** → **Add schedule** (or **Polling schedules** → **Add schedule**). + +| Field | Value | +|-------|-------| +| **Name** | `demo-poll` | +| **Targets** | *(leave empty — fetch all events)* | +| **Daily trigger time (UTC)** | See options below | +| **Import capacity limits** | ✓ checked | +| **Export capacity limits** | ✓ checked | + +Click **Save polling schedule**. The plugin creates two sensors on the VEN asset: + +- `import-capacity-limit-demo-poll` +- `export-capacity-limit-demo-poll` + +### When to run the fetch + +The schedule runs **once per day** at the configured UTC time. For this walkthrough you have two options: + +#### Option A — Trigger immediately (recommended)** + +Skip waiting for the daily schedule and run the fetch now. If you're still in `python/` +from Step 4 or 5, `cd ..` first — this runs in the FlexMeasures container, from +`examples/lf-energy-demo`. `--active --no-project` tells `uv run` to reuse the container's +own venv (`/app/.venv`, where `flexmeasures` and `flexmeasures_openadr3` are already +installed) instead of resolving a fresh one: + +```bash +docker compose exec server uv run --active --no-project /walkthrough/python/trigger_fetch.py +``` + +#### Option B — Wait for the scheduled job** + +Set **Daily trigger time** to a UTC time 2–3 minutes from now, then save. Ensure the +`cron-scheduler` container is running — it's the dedicated process that evaluates +polling schedules and enqueues fetch jobs onto the `ingestion` queue at the +configured trigger time. The `worker` container must also be running — it's what +processes that queue. (This is a different queue, and a different container, from the +`scheduling-worker` that Steps 4 and 10 need for triggered schedules.) + +--- + +## Step 9 — Wire the OpenADR capacity limits into the site + +Now that `demo-ven`'s `demo-poll` schedule exists and has created its import/export +capacity-limit sensors, re-run the asset-seeding script from Step 2 (from +`examples/lf-energy-demo`; `cd ..` first if you're still in `python/`): + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py +``` + +The asset hierarchy itself is unchanged by a re-run, but the script always re-evaluates the +OpenADR wiring: it points `demo-campus`'s `site-consumption-capacity` and +`site-production-capacity` flex-context fields at the import/export sensors it finds, turning +the fetched DR signal into an actual scheduling constraint. It also nests `demo-ven` under +`demo-campus` in the asset tree — harmless to the schedule itself (FlexMeasures only pulls a +descendant into scheduling if that asset has its own flex-model, which the VEN asset never +sets), but it's now what makes `demo-campus`'s **Edit flex-context** sensor search able to +find the VEN's sensors at all, since that search only looks at an asset's own descendants. +You'll see `demo-ven` appear inside `demo-campus`'s "Structure" tab in the UI from now on — +that's expected. The script's summary line reports both outcomes, e.g.: + +``` +OpenADR VEN asset: nested 'demo-ven' under 'demo-campus' +OpenADR site capacity limits: site-consumption-capacity -> import-capacity-limit-demo-poll, site-production-capacity -> export-capacity-limit-demo-poll +``` + +If you run this before the VEN client or polling schedule exist (or before Step 2), it +reports the wiring as `not wired: …` instead of failing — that's expected until you reach +this step. + +**Wiring it live through the UI instead:** pass `--skip-openadr-wiring` to still nest +`demo-ven` under `demo-campus` but leave both flex-context fields blank: + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py --skip-openadr-wiring +``` + +Then, in the FlexMeasures UI, open `demo-campus`'s asset page and click **Edit +flex-context**: + +1. Select `site-consumption-capacity` from the field dropdown, click **Add field**, then + search for and pick `import-capacity-limit-demo-poll`. +2. Select `site-production-capacity`, click **Add field**, then search for and pick + `export-capacity-limit-demo-poll`. +3. Click **Save**. + +--- + +## Step 10 — Trigger the after-OpenADR schedule, see the comparison + +With the capacity limits wired in, trigger `demo-campus` a second time with the same +script, so you can see exactly what changed: + +```bash +cd python + +uv run compare_schedules.py +``` + +The script reuses the exact time window from Step 4's run automatically, so the two plans +are comparable interval by interval even if you spent a while working through Steps 5–9 in +between — pass `--fresh-window` if you'd rather start a new comparison from now. It reads +`demo-campus`'s flex-context again, this time finds the OpenADR fields wired in from Step 9, +and labels this run `after-openadr`. + +Once both runs exist, the script automatically prints a before-and-after report — peak +import/export, energy imported/exported, cost at day-ahead prices, how many intervals +exceed the OpenADR import limit, and the largest interval-by-interval changes at the grid +connection — and writes `../schedule-runs/comparison.html`: open it in a browser for a +multi-panel diagram overlaying both plans (the campus grid connection, the EVSE hub, and +the heat pump), with the day-ahead price and the OpenADR-curtailed window (dashed capacity +limit line, always visible) underneath. The diagram opens showing only the **Before +OpenADR** plan; a small control bar of checkboxes above the chart lets you bring in **With +OpenADR** as well (or instead — every combination of the two, including neither, is +reachable), and the toggle applies to every panel at once. That diagram is the clearest +single artifact of this whole walkthrough: switch on "With OpenADR" and, outside the +curtailment window, the two plans are drawn exactly on top of each other — they're +genuinely identical, the same price-driven optimum, not a rendering artifact — while inside +it you can see load pulled off the grid connection to respect the 70 kW import limit from +Step 5, with the heat pump's own panel showing it shifting its draw on the thermal buffer +rather than sitting idle. Because the window now spans the whole office day, that +"identical" stretch is confined to the hours outside 08:00-18:00, not most of the day. + +A representative run against the seeded demo data looks like this: + +``` +peak import 129 kW -> 70.0 kW (-59 kW, pinned exactly to the new limit) +intervals over limit 12 -> 0 (of 96) +worst breach 58.7 kW -> 0.0 kW +energy above limit 55.0 kWh -> 0.0 kWh +energy moved 51.1 kWh (site) + 36.9 kWh (EVSE hub) + 34.4 kWh (heat pump) +cost at day-ahead prices 102.7 EUR -> 104.4 EUR (+1.7 EUR) +``` + +The "before" plan arches up to ~129 kW through the curtailment window because, absent any +constraint, that's when the site's price-optimal plan happens to want to charge the most — +the fleet is plugged in and the office is occupied for the whole of it. The "with OpenADR" +plan is pinned flat at 70 kW for the same ten hours, with the deferred energy picked up in +cheaper intervals either side of the window — at the cost of just under 2 EUR across the +whole day, for a curtailment that fully honours a 12-interval, 55.0 kWh breach. As noted in +Step 5, that clean fall to zero intervals over the limit is typical but not guaranteed: a +cloudier morning in the seeded weather can leave a residual breach right at the window's +opening edge, before rooftop PV has caught up with the office's day-level baseload. + +These numbers assume the schedule window falls on a weekday. The demo's charge points are +only 10% occupied on a weekend (vs. 85% on a weekday) and the office's heat demand drops too, +so running this step on a Saturday or Sunday will show a much weaker before-and-after +difference — there's little flexible load left for the curtailment to bite into. See +[Troubleshooting](#troubleshooting) if that's what you're seeing. + +**If the script refuses to trigger**, naming a stale or missing price window, too much time +passed since `seed_forecasts.py` last ran — it only seeds seven days from the moment it's +run, and a walkthrough left overnight or over a weekend can outlive that window. Re-seed it +and start a fresh comparison: + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_forecasts.py +``` + +then re-run both Step 4 and this step (or pass `--fresh-window` to skip straight to a new +comparison without repeating Step 4). + +--- + +## Step 11 — Inspect sensor beliefs + +After the fetch completes: + +1. Go to **Polling schedules** for `demo-ven` +2. Click **Import sensor** or **Export sensor** next to `demo-poll` +3. On the sensor page, open the chart / beliefs view + +You should see **96 beliefs** (one per 15-minute interval), sourced from **OpenADR 3 VTN**. +Each belief's `event_start` is the interval start time; values follow the shaped +import/export capacity limits set by `seed_events.py` (Step 5) — a generous 400 kW for most +of the day, dropping to a hard 70 kW over one deliberate 10-hour window pinned to +08:00-18:00 local time (the demo site's whole office day). + +To verify from the worker logs: + +```bash +docker compose logs worker | tail -20 +``` + +Look for a line like: + +``` +Fetched DR events for VEN 'demo-ven' … stored 192 beliefs. +``` + +(192 = 96 import + 96 export intervals stored across both sensors.) + +You can also open `demo-campus` in the FlexMeasures UI: its dashboard now plots the +connection capacity alongside the wired OpenADR sensors from Step 9, over the week of data +seeded in Step 3. From here, triggering a schedule on `demo-campus` will optimise the whole +hierarchy against the fetched capacity limits — which is exactly what Step 10 already did, +and its `comparison.html` is the fastest way to see the effect without reading beliefs one +sensor at a time. + +--- + +## Troubleshooting + +| Symptom | Check | +|---------|-------| +| Seed script cannot reach VTN | `docker compose ps` — is `openleadr-vtn` running? Wait ~30s after startup. | +| OAuth errors in seed script | Keycloak at `http://localhost:8080` — realm imported? | +| VEN client save fails | URLs must use Docker service names (`openleadr-vtn`, `keycloak`), not `localhost`. | +| Fetch job never runs | `docker compose ps` — both `cron-scheduler` (evaluates the schedule) and `worker` (processes the `ingestion` queue) must be up. | +| No beliefs after fetch | Re-run `seed_events.py`; event must exist on VTN before fetch. | +| HTTP blocked to VTN | `ALLOW_INSECURE_HTTP_VTN=true` is set in `docker-compose.yml`. | +| `seed_assets.py` fails with "No account named …" | The `Walkthrough Toy Account` is created by the `server` container's entrypoint on first boot; wait for `docker compose logs -f server` to show the gunicorn startup message, then re-run. | +| `seed_assets.py` reports OpenADR wiring as `not wired` | Expected until the VEN client and polling schedule exist (Steps 7-8); re-run it once they do (Step 9). | +| `seed_forecasts.py` fails with "No asset named …" | It seeds data onto the hierarchy from Step 2; run `seed_assets.py` first. | +| Scheduler fails with "Prices unknown for planning window" | The `day-ahead prices` sensor is empty over the schedule window; run `seed_forecasts.py` (Step 3), or re-run it if the seeded week has run out. | +| Scheduler fails with "No recent state-of-charge value found for sensor …" | Each storage-like device (the EVSEs and the heat pump) needs a recent belief on its `state of charge` sensor before you trigger a schedule. `seed_forecasts.py` (Step 3) records one per device at the quarter hour it runs in; if that was hours ago, re-run it, record a value manually, or pass `soc-at-start` in the scheduling request. | +| Charts are empty, or a schedule assumes zero load | Run `seed_forecasts.py` (Step 3), and re-run it once the seeded week has run out. | +| A triggered schedule (Step 4 or Step 10) stays `QUEUED` forever | `docker compose ps` — the `scheduling-worker` service must be running; check `docker compose logs scheduling-worker`. It's separate from `worker`, which only serves the `ingestion` queue. | +| `compare_schedules.py` refuses to trigger, naming a stale or missing price window | The seeded forecast/price data only covers seven days from whenever `seed_forecasts.py` last ran. Re-run it (see Step 10), or pass `--fresh-window` to start a new comparison instead of reusing Step 4's window. | +| `seed_events.py` run a second time fails on the next fetch with `OverlappingDemandResponseEventsError` | The previous event is still on the VTN; remove it first, or just don't re-run `seed_events.py` mid-walkthrough — one event is all this flow needs. | +| `seed_events.py` prints a note and the curtailment starts almost immediately instead of at 08:00 local time | Expected if you ran it while local time was already inside the pinned 08:00-18:00 window — that 10-hour office-day window no longer fits inside the 24-hour schedule horizon, so the script falls back to short notice. This is now the common case, not a rare edge case: the pinned window only fits inside the coming 24 hours when the script runs outside office hours, so a walkthrough run during the day typically takes this fallback. The comparison in Step 10 still works, it just no longer reaches into the price curve's morning and evening peaks the way the pinned window does. | +| Step 10's before-and-after comparison shows only a weak difference | Check what day the schedule window falls on: the EVSE fleet's weekend occupancy is only 10% (vs. 85% on weekdays) and heat demand is much lower too, so there's little flexible load left for the curtailment to bite into. Run the walkthrough on a weekday for the full effect. | +| `seed_assets.py` still fails with "No account named …" even after a fresh `docker compose down -v` | `fm-walkthrough-instance/.toy-account-created` is a host-side marker file that survives the volume reset, so the `server` container's entrypoint skips re-running `flexmeasures add toy-account` against the fresh database. Delete that marker file (or run `flexmeasures add toy-account --name 'Walkthrough Toy Account'` by hand in the container), then re-run `seed_assets.py`. | + +## Reference values + +All constants live in `python/settings.py`. OAuth clients are defined in +`keycloak/realm.json`: + +| Client | ID | Secret | Used for | +|--------|----|--------|----------| +| Business Logic | `test-client-id` | `my-client-secret` | `seed_events.py` | +| VEN | `test-ven-1` | `my-client-secret` | FlexMeasures UI | + +Asset names, capacities and flex-model parameters for the demo hierarchy live in +`flexmeasures/hierarchy.py`. + +## Clean up + +```bash +docker compose down -v +``` + +This removes containers and the FlexMeasures database volume. diff --git a/examples/lf-energy-demo/docker-compose.yml b/examples/lf-energy-demo/docker-compose.yml new file mode 100644 index 0000000..a51e2ce --- /dev/null +++ b/examples/lf-energy-demo/docker-compose.yml @@ -0,0 +1,256 @@ +# OpenADR 3 capacity-limits walkthrough stack. + +services: + keycloak: + image: quay.io/keycloak/keycloak:25.0.4 + command: start-dev --import-realm + environment: + KEYCLOAK_ADMIN: admin + KEYCLOAK_ADMIN_PASSWORD: admin + KC_HTTP_ENABLED: "true" + KC_HOSTNAME_STRICT: "false" + ports: + - "8080:8080" + volumes: + - ./keycloak/realm.json:/opt/keycloak/data/import/realm.json + networks: + - oadr3-walkthrough + + vtn-db: + image: postgres:18 + environment: + POSTGRES_USER: openadr + POSTGRES_PASSWORD: openadr + POSTGRES_DB: openadr + networks: + - oadr3-walkthrough + healthcheck: + test: ["CMD-SHELL", "pg_isready -U openadr -d openadr"] + interval: 5s + timeout: 5s + retries: 10 + + openleadr-vtn: + image: ghcr.io/openleadr/openleadr-rs:v0.2.6 + platform: linux/amd64 + depends_on: + vtn-db: + condition: service_healthy + keycloak: + condition: service_started + ports: + - "3000:3000" + environment: + OAUTH_TYPE: EXTERNAL + OAUTH_VALID_AUDIENCES: "https://integration.test.elaad.nl," + OAUTH_KEY_TYPE: RSA + OAUTH_JWKS_LOCATION: http://keycloak:8080/realms/integration-test-realm/protocol/openid-connect/certs + OAUTH_TOKEN_URL: http://keycloak:8080/realms/integration-test-realm/protocol/openid-connect/token + DATABASE_URL: postgres://openadr:openadr@vtn-db:5432/openadr + PG_PORT: "5432" + PG_DB: openadr + PG_USER: openadr + PG_PASSWORD: openadr + RUST_LOG: info + networks: + - oadr3-walkthrough + + fm-db: + image: postgres:18 + environment: + POSTGRES_DB: fm-walkthrough-db + POSTGRES_USER: fm-walkthrough-user + POSTGRES_PASSWORD: fm-walkthrough-pass + ports: + - "5433:5432" + volumes: + - fm-db-data:/var/lib/postgresql + networks: + - oadr3-walkthrough + healthcheck: + test: ["CMD-SHELL", "pg_isready -U fm-walkthrough-user -d fm-walkthrough-db"] + interval: 10s + timeout: 5s + retries: 5 + + queue-db: + image: redis:7 + command: redis-server --loglevel warning --requirepass fm-redis-pass + ports: + - "6380:6379" + networks: + - oadr3-walkthrough + + server: + build: + context: ../.. + dockerfile: Dockerfile + platform: linux/amd64 + ports: + - "5002:5000" + depends_on: + fm-db: + condition: service_healthy + queue-db: + condition: service_started + openleadr-vtn: + condition: service_started + keycloak: + condition: service_started + restart: on-failure + environment: + SQLALCHEMY_DATABASE_URI: postgresql://fm-walkthrough-user:fm-walkthrough-pass@fm-db:5432/fm-walkthrough-db + SECRET_KEY: notsecret + SECURITY_TOTP_SECRETS: '{"1": "something-secret"}' + FLEXMEASURES_ENV: development + FLEXMEASURES_REDIS_URL: queue-db + FLEXMEASURES_REDIS_PASSWORD: fm-redis-pass + FLEXMEASURES_PLUGINS: flexmeasures_openadr3 + ALLOW_INSECURE_HTTP_VTN: "true" + OAUTHLIB_INSECURE_TRANSPORT: "1" + OAUTHLIB_RELAX_TOKEN_SCOPE: "1" + LOGGING_LEVEL: INFO + volumes: + # flexmeasures.cfg here sets FLEXMEASURES_SECRETS_ENCRYPTION_KEYS, which + # FlexMeasures does not read from an environment variable. It must be + # identical across server, worker, and cron-scheduler, so all three + # bind-mount this same host folder. + - ./fm-walkthrough-instance:/usr/var/flexmeasures-instance:rw + - ./fm-walkthrough-instance:/app/instance:rw + - ./python:/walkthrough/python:ro + # seed_assets.py imports the shared constants from /walkthrough/python, so both + # folders must be mounted for it to run. + - ./flexmeasures:/walkthrough/flexmeasures:ro + entrypoint: ["/bin/sh", "-c"] + command: + - | + flexmeasures db upgrade + if [ ! -f /usr/var/flexmeasures-instance/.toy-account-created ]; then + flexmeasures add toy-account --name 'Walkthrough Toy Account' + touch /usr/var/flexmeasures-instance/.toy-account-created + fi + gunicorn --bind 0.0.0.0:5000 --worker-tmp-dir /dev/shm --workers 2 --threads 4 wsgi:application + networks: + - oadr3-walkthrough + + worker: + build: + context: ../.. + dockerfile: Dockerfile + platform: linux/amd64 + depends_on: + fm-db: + condition: service_healthy + queue-db: + condition: service_started + server: + condition: service_started + restart: on-failure + environment: + SQLALCHEMY_DATABASE_URI: postgresql://fm-walkthrough-user:fm-walkthrough-pass@fm-db:5432/fm-walkthrough-db + FLEXMEASURES_REDIS_URL: queue-db + FLEXMEASURES_REDIS_PASSWORD: fm-redis-pass + SECRET_KEY: notsecret + SECURITY_TOTP_SECRETS: '{"1": "something-secret"}' + FLEXMEASURES_ENV: development + FLEXMEASURES_PLUGINS: flexmeasures_openadr3 + ALLOW_INSECURE_HTTP_VTN: "true" + OAUTHLIB_INSECURE_TRANSPORT: "1" + OAUTHLIB_RELAX_TOKEN_SCOPE: "1" + LOGGING_LEVEL: INFO + volumes: + - ./fm-walkthrough-instance:/usr/var/flexmeasures-instance:rw + - ./fm-walkthrough-instance:/app/instance:rw + - ./python:/walkthrough/python:ro + - ./flexmeasures:/walkthrough/flexmeasures:ro + entrypoint: ["/bin/sh", "-c"] + command: + - flexmeasures jobs run-worker --name flexmeasures-worker --queue ingestion + networks: + - oadr3-walkthrough + + scheduling-worker: + build: + context: ../.. + dockerfile: Dockerfile + platform: linux/amd64 + depends_on: + fm-db: + condition: service_healthy + queue-db: + condition: service_started + server: + condition: service_started + restart: on-failure + environment: + SQLALCHEMY_DATABASE_URI: postgresql://fm-walkthrough-user:fm-walkthrough-pass@fm-db:5432/fm-walkthrough-db + FLEXMEASURES_REDIS_URL: queue-db + FLEXMEASURES_REDIS_PASSWORD: fm-redis-pass + SECRET_KEY: notsecret + SECURITY_TOTP_SECRETS: '{"1": "something-secret"}' + FLEXMEASURES_ENV: development + FLEXMEASURES_PLUGINS: flexmeasures_openadr3 + ALLOW_INSECURE_HTTP_VTN: "true" + OAUTHLIB_INSECURE_TRANSPORT: "1" + OAUTHLIB_RELAX_TOKEN_SCOPE: "1" + LOGGING_LEVEL: INFO + volumes: + - ./fm-walkthrough-instance:/usr/var/flexmeasures-instance:rw + - ./fm-walkthrough-instance:/app/instance:rw + entrypoint: ["/bin/sh", "-c"] + command: + # Scheduling jobs land on their own `scheduling` queue, which the `worker` service + # above does not serve, so without this service a triggered schedule stays QUEUED + # forever. It is a separate worker rather than a second queue on `worker` for two + # reasons: FlexMeasures only installs its scheduling-specific RQ exception handler + # when `--queue` is exactly `scheduling` (see flexmeasures/cli/jobs.py), which is what + # the `GET /sensors//schedules/` endpoint relies on to report a failed + # schedule; and a minutes-long MILP solve would otherwise block the OpenADR event + # ingestion that the cron-scheduler keeps queueing. + - flexmeasures jobs run-worker --name flexmeasures-scheduling-worker --queue scheduling + networks: + - oadr3-walkthrough + + cron-scheduler: + build: + context: ../.. + dockerfile: Dockerfile + platform: linux/amd64 + depends_on: + fm-db: + condition: service_healthy + queue-db: + condition: service_started + server: + condition: service_started + restart: on-failure + environment: + SQLALCHEMY_DATABASE_URI: postgresql://fm-walkthrough-user:fm-walkthrough-pass@fm-db:5432/fm-walkthrough-db + FLEXMEASURES_REDIS_URL: queue-db + FLEXMEASURES_REDIS_PASSWORD: fm-redis-pass + SECRET_KEY: notsecret + SECURITY_TOTP_SECRETS: '{"1": "something-secret"}' + FLEXMEASURES_ENV: development + FLEXMEASURES_PLUGINS: flexmeasures_openadr3 + ALLOW_INSECURE_HTTP_VTN: "true" + OAUTHLIB_INSECURE_TRANSPORT: "1" + OAUTHLIB_RELAX_TOKEN_SCOPE: "1" + LOGGING_LEVEL: INFO + volumes: + - ./fm-walkthrough-instance:/usr/var/flexmeasures-instance:rw + - ./fm-walkthrough-instance:/app/instance:rw + entrypoint: ["/bin/sh", "-c"] + command: + # Exactly one instance of this service may run: it's the sole process + # driving flexmeasures_openadr3's VEN fetch-events CronScheduler (see + # flexmeasures_openadr3/cli.py). Do not scale this service. + - flexmeasures oadr3-run-cron-scheduler + networks: + - oadr3-walkthrough + +networks: + oadr3-walkthrough: + driver: bridge + +volumes: + fm-db-data: diff --git a/examples/lf-energy-demo/flexmeasures/README.md b/examples/lf-energy-demo/flexmeasures/README.md new file mode 100644 index 0000000..2f0c116 --- /dev/null +++ b/examples/lf-energy-demo/flexmeasures/README.md @@ -0,0 +1,214 @@ +# FlexMeasures asset hierarchy + +Seeding for the [LF Energy demo walkthrough](../README.md): builds the demo site as real +`GenericAsset` and `Sensor` rows, so the OpenADR capacity limits fetched elsewhere in the +walkthrough have a site to constrain, and fills those sensors with a week of synthetic +data, so the site has something to show and something to schedule against. + +| File | Run where | Run with | Purpose | +| --- | --- | --- | --- | +| `seed_assets.py` | FlexMeasures container | `uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py` | Create the demo asset hierarchy in the `Walkthrough Toy Account` | +| `seed_forecasts.py` | FlexMeasures container | `uv run --active --no-project /walkthrough/flexmeasures/seed_forecasts.py` | Fill that hierarchy's sensors with a week of prices, forecasts, reference profiles and current states of charge | +| `hierarchy.py` | — | — | Names, capacities, flexibility limits and typical-usage shapes of the demo site | + +```bash +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py +docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_forecasts.py +``` + +Like `../python/trigger_fetch.py`, both scripts reuse the container's own uv-managed venv +(`/app/.venv`) and reach the database through the ORM, so they need no API token. + +`hierarchy.py` holds everything declarative: what the site is made of, and what it +typically does. The two scripts stay focused on how to persist that. + +## What `seed_assets.py` builds + +```text +demo-campus (building, 1 MVA connection) +├── demo-evse-hub (building, 400 kVA shared sub-connection) +│ ├── demo-evse-01 … 06 (one-way_evse, 22 kW) +│ └── demo-evse-07 … 08 (two-way_evse, 11 kW, V2G) +└── demo-office (building, 250 kVA sub-connection) + ├── demo-office-pv (solar, must-take production) + ├── demo-office-baseload (process, inflexible consumption) + └── demo-office-heat-pump (heat-storage, thermal buffer) +``` + +The campus holds the flex-context (day-ahead prices, site capacity, breach prices), which +FlexMeasures reads upwards through the tree. Each device holds a flex-model entry, which +FlexMeasures gathers downwards, so triggering a schedule on `demo-campus` covers the whole +site in one optimisation. + +The heat pump follows FlexMeasures' own `Heat Pump Template`: its coefficient of +performance is the flex-model's `charging-efficiency` (the one-way conversion from +electricity to stored heat, which is allowed to exceed 100%, unlike `roundtrip-efficiency` +which the charge points use). Its `power` sensor is therefore electrical while its +`state of charge` sensor is thermal. + +### The sensors that give flexible devices something to do + +A storage device whose flex-model only bounds its state of charge (`soc-min`/`soc-max`) has +no reason to ever consume: an empty buffer costs nothing, so the cheapest plan is to leave +it idle at 0 kW forever, regardless of price or any capacity signal. That was true of 7 of +the demo's 9 flexible devices until three extra sensors were added, each of which drives a +flex-model field that gives the scheduler an actual energy requirement to plan around: + +| Sensor | Unit | On | Drives | Purpose | +| --- | --- | --- | --- | --- | +| `power availability` | kW | each charge point | `power-capacity` | Zero while the bay is empty, the connected car's accepted power while occupied — replaces a static nameplate rating, which would let the scheduler charge an empty bay at 3am. | +| `minimum state of charge` | kWh | each charge point | `soc-minima` | The departure requirement, but recorded as nonzero in *only the single quarter-hour interval the car leaves in* — every other interval is explicitly `0.0`, not blank. This is the charge point's reason to consume at all: without it, an empty battery costs nothing and the cheapest plan is never to charge. | +| `heat demand` | kW (thermal) | `demo-office-heat-pump` | `soc-usage` | The actual drain on the thermal buffer, following the building's day: heaviest just before the first arrivals, falling back once the building is occupied and gaining incidental heat, low overnight and much lower at weekends. | + +Two details matter for `minimum state of charge`: FlexMeasures gap-fills blank intervals on +a sensor forward from the last recorded value, so leaving the non-departure intervals blank +would silently oblige the car to stay full all day instead of only at departure — they are +written as `0.0` on purpose. And the campus flex-context carries a matching +`soc-minima-breach-price: "5 EUR/kWh"`, so a stay too short for its deficit produces a +costed shortfall instead of an infeasible schedule. + +All three of these sensors are **forecasts**, written by `LF Energy demo forecaster` — they +describe the weather, building and driver behaviour the scheduler plans around, the same +category as prices and the office's inflexible load. That's different from the charge +points' and heat pump's own `power` sensors, which stay reference profiles: those describe +what the site would do unscheduled, whereas the scheduler now genuinely dispatches these +devices against the requirement. + +It is idempotent: assets and sensors are fetched by name before being created, so +re-running it only reprints the summary. + +## What `seed_forecasts.py` writes + +Seven days of quarter-hourly data, from the quarter hour the run falls in. Three kinds of +belief, from two data sources, because they do not mean the same thing: + +| Sensors | Data source | Kind | +| --- | --- | --- | +| `NL transmission zone/day-ahead prices`, `demo-office-baseload/power`, `demo-office-pv/power`, `demo-evse-01…08/power availability`, `demo-evse-01…08/minimum state of charge`, `demo-office-heat-pump/heat demand` | `LF Energy demo forecaster` (type `forecaster`) | Forecast | +| `demo-evse-01…08/power`, `demo-office-heat-pump/power` | `LF Energy demo profiles` (type `demo script`) | Reference profile | +| `demo-evse-01…08/state of charge`, `demo-office-heat-pump/state of charge` | `LF Energy demo profiles` (type `demo script`) | Measurement, one per device | + +The distinction is deliberate. The price curve is what the scheduler optimises against, and +the office's background load and its rooftop PV are the site's `inflexible-consumption` and +`inflexible-production`: the scheduler cannot move them, so it needs to know what they will +do, and a forecast is exactly the right thing to give it. Each flexible device's *energy +requirement* — a charge point's power availability and departure requirement, the heat +pump's heat demand — is a forecast for the same reason: it describes the weather, the +building and the drivers, not anything the scheduler decides, so it belongs to the +forecaster too, even though it lives on the same devices the scheduler dispatches. The +charge points' and the heat pump's own `power` sensors are what the scheduler *does* move, +so a fixed week of power values for them is not a forecast of anything — it is what the site +would do unscheduled, kept only so the charts are not empty before the first schedule runs. +The scheduler writes its own beliefs, from its own source, over the same window. + +All three energy-requirement sensors are generated from the same seeded draw of each day's +charging/heating stays as their device's reference profile, so a charge point's availability, +its departure requirement and its unscheduled power profile can never disagree about which +car showed up. The departure requirement (`minimum state of charge`) is written as an +explicit `0.0` in every quarter hour except the one the car leaves in, never left blank: +FlexMeasures gap-fills a blank interval on a sensor-referenced flex-model field forward from +the last recorded value, so a blank would inherit the departure figure and quietly oblige +the car to stay full all day. + +Prices are the only profile written outside the demo account: the `day-ahead prices` sensor +lives on FlexMeasures' public `NL transmission zone` asset, which is what the campus' +flex-context points at. `seed_forecasts.py` fetches that sensor through `seed_assets.py`, so +the two scripts cannot end up disagreeing about its unit, resolution or knowledge horizon. +It is also the one input the scheduler refuses to run without: with an empty price sensor it +stops at `Prices unknown for planning window` before it evaluates anything else. + +Both source names are specific to this walkthrough, so neither can be confused with the +plugin's own `OpenADR 3 VTN` source of genuinely fetched DR signals. + +A belief counts as a forecast when its belief time precedes the sensor's knowledge time. +These power sensors use timely-beliefs' default knowledge horizon, which puts knowledge +time at the end of the event, so stamping the whole window as believed at local midnight +of the run day makes it read as a forecast throughout. That belief time is also fixed +within a run day, which is what lets a re-run merge the same rows back over themselves — +belief time is part of a belief's key. + +The aggregate `power` sensors of `demo-campus`, `demo-evse-hub` and `demo-office` are left +empty on purpose: they are the scheduler's output, and a synthetic aggregate there would +disagree with the schedule as soon as one is triggered. + +### The shapes + +- **`day-ahead prices`** trough overnight, peak as the country wakes up, dip in the middle + of the day while the sun is on the roofs and peak again in the early evening, with the + level of the whole day drawn once and the weekend flatter. Roughly 0.02 to 0.28 EUR/kWh — + that spread is what makes the scheduler bother to move anything. +- **`demo-office-baseload`** sits on a night floor of servers, network gear and standby + losses, climbs over the hour before opening, dips shallowly over lunch and falls away + over the hour after closing. Weekends keep the floor plus a small allowance for cleaning + staff. +- **`demo-office-pv`** follows a half sine between sunrise and sunset, scaled by a cloud + cover drawn once per day, so the week holds a mix of clear and overcast days. Values are + positive, because the sensor records production as positive, and never exceed the + array's nameplate rating. +- **`demo-office-heat-pump`** pre-heats the building in the small hours and only tops the + buffer up while the building is occupied and generating its own incidental heat, with a + deep setback at the weekend. This reference profile is what an unscheduled pre-heat block + would look like; the scheduler decides the real dispatch against the `heat demand` + forecast instead (see above), which is the drain on the same thermal buffer and is what + actually obliges the scheduler to run the heat pump at all. Heaviest just before the first + arrivals, falling back once the building is occupied and gaining incidental heat, low + overnight and much lower at weekends — expressed in thermal kW, so it's the heat pump's + coefficient of performance that decides what meeting it costs electrically. +- **`demo-evse-01 … 08`** see a commuter arrive around 08:00 and leave around 17:00 on most + weekdays, with an occasional shorter stay before or after for an early bird or a late + worker, and a near-empty hub at the weekend. Arrival, departure, energy demand and the + power the car accepts are all drawn per charge point per day, so the hub's aggregate + looks like a hub rather than like eight identical rectangles. No session ever exceeds its + charge point's rating, and the two bidirectional charge points stay charge-only: + discharging is a scheduling decision, not typical usage. The same drawn stays also produce + each charge point's `power availability` and `minimum state of charge` forecasts (see + above); only commuter stays carry a departure requirement, since a short stay before or + after the working day is a top-up, not a promise of a full battery. + +**General lesson:** a FlexMeasures storage device's flex-model needs an actual energy +requirement — `soc-usage`, `soc-minima`, or `soc-targets` — not just `soc-min`/`soc-max` +bounds, or the scheduler has no reason to ever move it: an empty buffer costs nothing, so +leaving it idle is always at least as cheap as running it. That was true of the heat pump +and of 6 of the 8 charge points in an earlier version of this demo, all of which sat at +0 kW in every schedule regardless of price or capacity signal, and is why every flexible +device here carries one of these fields now. + +### Determinism + +Every profile is drawn from a generator seeded on the sensor and the local calendar day, +and each day is generated whole before being cut back to the window. A given day therefore +always gets the same numbers, whether the script runs at dawn or at noon, and a re-run +merges identical rows back over themselves rather than drifting. This is a heavier form of +idempotency than `seed_assets.py`'s fetch-or-create — the week is regenerated on every run +— but the result is the same week. + +The state-of-charge readings are the one exception: they are keyed on the quarter hour the +run falls in, so a re-run within that quarter hour rewrites them, while a later run records +a newer reading alongside the old one, which is what a fresh measurement should do. + +Pass `--days` to seed a shorter or longer horizon, and `--account-name` to target an +account other than the walkthrough's toy account. + +## Ties into the rest of the walkthrough + +Once a VEN client and polling schedule exist (walkthrough steps 7 and 8), re-run +`seed_assets.py`: it points the campus' `site-consumption-capacity` and +`site-production-capacity` at the OpenADR import and export capacity-limit sensors, turning +the fetched DR signal into a scheduling constraint. Before then it reports the wiring as +skipped. + +`../python/compare_schedules.py` triggers a schedule against this hierarchy twice — +once before this wiring exists and once after (walkthrough Steps 4 and 10) — and compares +the two plans, so you can see exactly what the capacity limits changed rather than just +that a sensor now holds new values. + +## Before triggering a schedule + +Each storage device reads its starting state of charge from its own `state of charge` +sensor, looking for a value within four time steps of the schedule start. `seed_forecasts.py` +records one per device at the quarter hour it runs in, so trigger the schedule shortly +after seeding — otherwise re-run the script, record a value manually, or pass `soc-at-start` +in the scheduling request. Without one, the scheduler stops with +`No recent state-of-charge value found for sensor `. + +See the [main walkthrough README](../README.md) for step-by-step instructions. diff --git a/examples/lf-energy-demo/flexmeasures/hierarchy.py b/examples/lf-energy-demo/flexmeasures/hierarchy.py new file mode 100644 index 0000000..9193498 --- /dev/null +++ b/examples/lf-energy-demo/flexmeasures/hierarchy.py @@ -0,0 +1,511 @@ +""" +Declarative description of the demo site that seed_assets.py builds. + +Everything describing *what* the demo site looks like lives here, so the seeding +script can stay focused on *how* to persist it idempotently. +Connection settings the rest of the walkthrough already owns are reused from +``python/settings.py`` rather than duplicated. +""" + +from __future__ import annotations + +import sys +from dataclasses import dataclass +from datetime import timedelta +from pathlib import Path + +# The walkthrough's shared constants live in the sibling `python/` helper folder. +# Appending (rather than inserting) keeps that folder from shadowing anything already importable. +_WALKTHROUGH_PYTHON_DIR = Path(__file__).resolve().parent.parent / "python" +if str(_WALKTHROUGH_PYTHON_DIR) not in sys.path: + sys.path.append(str(_WALKTHROUGH_PYTHON_DIR)) + +from settings import FLEXMEASURES_URL, POLLING_SCHEDULE_NAME, VEN_CLIENT_NAME # noqa: E402 + +# The docker-compose `server` entrypoint creates this account via `flexmeasures add toy-account`. +ACCOUNT_NAME = "Walkthrough Toy Account" + +# Amsterdam, matching the toy account's own assets so every demo asset shares one map view. +SITE_LATITUDE = 52.374 +SITE_LONGITUDE = 4.88969 +SITE_TIMEZONE = "Europe/Amsterdam" + +# Every power sensor is quarter-hourly: both the Dutch imbalance settlement period +# and the resolution the plugin uses for OpenADR capacity-limit intervals. +POWER_RESOLUTION = timedelta(minutes=15) + +# A state-of-charge sensor holds instantaneous readings, so the scheduler can align +# them with the boundaries of its own power time steps. +SOC_RESOLUTION = timedelta(0) + +# A contracted connection capacity changes at most once per contract year. +CONTRACT_RESOLUTION = "P1Y" + +# Sensor names and units are reused across assets, so every asset page reads the same, +# and both seeding scripts agree on which sensor to write to. +POWER_SENSOR_NAME = "power" +SOC_SENSOR_NAME = "state of charge" +GRID_CONNECTION_CAPACITY_SENSOR_NAME = "grid connection capacity" +INDOOR_TEMPERATURE_SENSOR_NAME = "indoor temperature" + +# What turns the flexible devices from decoration into load the scheduler has to plan for. +# A storage device whose flex-model only bounds its state of charge has no reason to ever +# consume: an empty buffer costs nothing, so the cheapest plan is to leave it idle. Each +# flexible device therefore also carries an *energy requirement*, recorded on its own sensor: +# +# - the charge points get a departure requirement (`soc-minima`) and a presence-shaped +# power ceiling (`power-capacity`), so a car has to be full when its driver leaves but +# may take its energy in whichever quarter hours of the stay are cheapest; +# - the heat pump gets the building's heat demand (`soc-usage`), so the thermal buffer it +# feeds actually drains and has to be refilled. +# +# Without these, seven of the nine flexible devices sit at exactly 0 kW in every schedule. +EVSE_AVAILABILITY_SENSOR_NAME = "power availability" +SOC_MINIMA_SENSOR_NAME = "minimum state of charge" +HEAT_DEMAND_SENSOR_NAME = "heat demand" + +POWER_UNIT = "kW" +ENERGY_UNIT = "kWh" +TEMPERATURE_UNIT = "°C" + +# Public day-ahead price sensor on the `NL transmission zone` asset, also created by `flexmeasures add toy-account`. +PRICE_SENSOR_NAME = "day-ahead prices" + +# A day-ahead market clears in hourly products. +PRICE_RESOLUTION = timedelta(hours=1) +PRICE_UNIT = "EUR/kWh" + +# --- Level 1: the site ------------------------------------------------------- +# One building with a single grid connection point, through which all demo power flows. + +CAMPUS_NAME = "demo-campus" + +# A 1 MVA connection is a realistic size for a small campus with a charging hub. +CAMPUS_POWER_CAPACITY = "1 MVA" + +# Breaching the connection limit is expensive but not physically impossible, which is +# what lets the scheduler weigh a short breach against a price spike. +CAMPUS_BREACH_PRICE = "600 EUR/kW" + +# Missing a departure requirement is priced rather than forbidden, which keeps every +# schedule solvable. A driver who finds the car half empty is far worse off than the site +# is from any energy price (the day-ahead curve peaks around 0.26 EUR/kWh), so the +# scheduler only ever leaves a car short when it physically cannot do better — a stay that +# is too short for the deficit, or a window that starts with the departure already close. +# Without this, such a case would come back as an infeasible schedule instead of a plan. +CAMPUS_SOC_MINIMA_BREACH_PRICE = "5 EUR/kWh" + +# --- Level 2a: the EVSE charging hub ----------------------------------------- +# A sub-EMS whose flex-model entry caps the aggregate power of every charger behind it. + +EVSE_HUB_NAME = "demo-evse-hub" + +# The hub sits behind a 400 kVA sub-connection, well below the summed rating of its +# chargers, so the chargers have to share it. +EVSE_HUB_POWER_CAPACITY = "400 kVA" + + +@dataclass(frozen=True, slots=True) +class EvseSpec: + """ + One charge point in the hub. + + :param name: Asset name, unique within the account. + :param asset_type: `one-way_evse` for charge-only, `two-way_evse` for V2G. + :param power_capacity: Nameplate rating, enforced as a hard limit. + :param production_capacity: Discharge limit; `0 kW` for charge-only, None to allow the full rating in both directions. + :param soc_min: Reserve the driver wants to keep in the car. + :param soc_max: Usable battery capacity of the connected car. + :param roundtrip_efficiency: Combined charger and battery losses. + """ + + name: str + asset_type: str + power_capacity: str + production_capacity: str | None + soc_min: str + soc_max: str + roundtrip_efficiency: str + + +# Six AC chargers plus two bidirectional ones: enough for the shared-capacity constraint +# to bite, few enough to stay readable on an asset page. +EVSE_SPECS: tuple[EvseSpec, ...] = ( + *( + EvseSpec( + name=f"demo-evse-{index:02d}", + asset_type="one-way_evse", + power_capacity="22 kW", + production_capacity="0 kW", + soc_min="0 kWh", + soc_max="60 kWh", + roundtrip_efficiency="92%", + ) + for index in range(1, 7) + ), + *( + EvseSpec( + name=f"demo-evse-{index:02d}", + asset_type="two-way_evse", + power_capacity="11 kW", + # No separate discharge limit: a V2G charge point may use its full rating either way. + production_capacity=None, + # A V2G driver keeps a reserve, so the car stays usable after a discharge window. + soc_min="10 kWh", + soc_max="77 kWh", + roundtrip_efficiency="88%", + ) + for index in range(7, 9) + ), +) + +# --- Level 2b: the grid-friendly office -------------------------------------- +# A second sub-EMS with rooftop PV, a heat pump and a background load. + +OFFICE_NAME = "demo-office" + +# The office's own sub-connection, capping the combined power of everything behind it: +# the PV, the background load and the heat pump all reference the office as their group. +OFFICE_POWER_CAPACITY = "250 kVA" + +OFFICE_PV_NAME = "demo-office-pv" + +# 150 kWp of rooftop PV, treated as a must-take infeed rather than a curtailable device. +OFFICE_PV_PEAK_POWER = "150 kW" + +# Lighting, servers and plug loads: the load the office cannot shift. +OFFICE_BASELOAD_NAME = "demo-office-baseload" + +# FlexMeasures ships no `heat_pump` asset type, so the heat pump plus the building's +# thermal mass is modelled as a thermal buffer the scheduler charges and discharges, +# mirroring the `Heat Pump Template` asset that FlexMeasures itself seeds. +OFFICE_HEAT_PUMP_NAME = "demo-office-heat-pump" + +# Electrical rating: what the heat pump draws from the office. +OFFICE_HEAT_PUMP_POWER_CAPACITY = "30 kW" + +# Seasonal coefficient of performance, expressed the way FlexMeasures models it: as a +# one-way `charging-efficiency` from electricity to stored heat. That field is allowed to +# exceed 100%, precisely so it can carry a COP; `roundtrip-efficiency` is capped at 100% +# and would reject this value. +OFFICE_HEAT_PUMP_COP = 3.5 +OFFICE_HEAT_PUMP_CHARGING_EFFICIENCY = f"{OFFICE_HEAT_PUMP_COP:.0%}" + +# State of charge is therefore *thermal* energy: 30 kW electrical becomes 105 kW of heat, +# so this buffer holds about four hours of full-power charging. +OFFICE_HEAT_PUMP_SOC_MIN = "0 kWh" +OFFICE_HEAT_PUMP_SOC_MAX = "420 kWh" + +# Thermal mass leaks: roughly 2% of the stored heat is lost per quarter hour. +OFFICE_HEAT_PUMP_STORAGE_EFFICIENCY = "98%" + +# --- How the demo site behaves ----------------------------------------------- +# Shape parameters for the week of synthetic data that seed_forecasts.py writes. +# They describe what the site typically *does*, the way the sections above describe what it *is*, +# so both live here and the seeding scripts stay focused on how to persist them. +# Every hour below is a local wall-clock hour in SITE_TIMEZONE, expressed as a float, +# so that 8.5 reads as 08:30. + +# A week of quarter hours: long enough to schedule a full week against, short enough to seed in seconds. +FORECAST_HORIZON = timedelta(days=7) + +# A Dutch day-ahead curve, in EUR/kWh: a night trough, a morning peak as the country wakes up, +# a dip in the middle of the day while the sun is on the roofs, and the highest hours in the early evening. +# The spread between the dip and the evening peak is what gives the scheduler something to optimise. +PRICE_NIGHT = 0.06 +PRICE_MORNING_PEAK = 0.17 +PRICE_MIDDAY_DIP = 0.02 +PRICE_EVENING_PEAK = 0.21 + +PRICE_MORNING_PEAK_HOUR = 8.0 +PRICE_MIDDAY_DIP_HOUR = 13.0 +PRICE_EVENING_PEAK_HOUR = 19.0 + +# Width of each of those three features, in hours. +PRICE_FEATURE_SPREAD_HOURS = 2.5 + +# Demand is lower at the weekend, and so is the whole curve. +PRICE_WEEKEND_FACTOR = 0.8 + +# The level of the whole day, drawn once: mostly a matter of how much wind there is. +PRICE_DAY_LEVEL_RANGE = (0.75, 1.35) + +PRICE_NOISE = 0.08 + +# The office is open on weekdays between these hours, which anchors every occupancy-driven profile below. +OFFICE_OPENING_HOUR = 8.0 +OFFICE_CLOSING_HOUR = 18.0 + +# A building does not switch between its night and day load in a single quarter hour: +# the first arrivals and the last departures spread the change over about an hour on either side. +OFFICE_RAMP_HOURS = 1.0 + +# Servers, network gear, emergency lighting and standby losses: what the office draws when nobody is in. +OFFICE_BASELOAD_NIGHT_POWER = 18.0 + +# Lighting, workstations, ventilation and the kitchen, on top of the night floor. +OFFICE_BASELOAD_DAY_POWER = 85.0 + +# Weekends keep the night floor, plus a modest allowance for cleaning staff and weekend workers. +OFFICE_BASELOAD_WEEKEND_DAY_POWER = 26.0 + +# Lunch empties the desks for a while, which shows up as a shallow dip around this hour. +OFFICE_BASELOAD_LUNCH_HOUR = 12.75 +OFFICE_BASELOAD_LUNCH_DIP = 0.12 +OFFICE_BASELOAD_LUNCH_SPREAD_HOURS = 0.9 + +# Quarter-to-quarter jitter, as a fraction of the load: enough to look measured rather than drawn. +OFFICE_BASELOAD_NOISE = 0.05 + +# A fixed daylight window rather than a real ephemeris: the demo only ever spans a week, +# and these hours are about right for Amsterdam outside midwinter. +OFFICE_PV_SUNRISE_HOUR = 6.75 +OFFICE_PV_SUNSET_HOUR = 20.5 + +# A rooftop array never reaches its nameplate rating in the field, because of angle, temperature and inverter losses. +OFFICE_PV_CLEAR_SKY_PEAK_FRACTION = 0.75 + +# Cloud cover, drawn once per day: from a thoroughly overcast day to a clear one. +OFFICE_PV_MIN_DAY_CLEARNESS = 0.25 +OFFICE_PV_MAX_DAY_CLEARNESS = 1.0 + +# Passing clouds within a day, as a fraction of that day's clear-sky output. +OFFICE_PV_CLOUD_NOISE = 0.18 + +# The heat pump pre-heats the building before the first arrivals. This is the reference +# profile only — what the site would do unscheduled; the scheduler decides the real dispatch +# against the heat demand below. +OFFICE_HEAT_PUMP_PREHEAT_START_HOUR = 4.5 +OFFICE_HEAT_PUMP_PREHEAT_POWER = 21.0 + +# During office hours the building gains incidental heat from people, lighting and equipment, +# so the heat pump only has to top the buffer up. +OFFICE_HEAT_PUMP_OCCUPIED_POWER = 7.0 + +# Outside those hours it runs at a low setback, replacing what the thermal mass leaks. +OFFICE_HEAT_PUMP_IDLE_POWER = 3.5 + +# An empty building is allowed to drift much further from its setpoint. +OFFICE_HEAT_PUMP_WEEKEND_FACTOR = 0.35 + +OFFICE_HEAT_PUMP_NOISE = 0.08 + +# Where the thermal buffer sits when the demo starts, as a fraction of its usable capacity. +OFFICE_HEAT_PUMP_INITIAL_SOC_RANGE = (0.35, 0.6) + +# --- What the heat pump actually has to deliver ------------------------------- +# The building's heat demand, in *thermal* kW, recorded on its own sensor and referenced by +# the heat pump's `soc-usage`. This is the drain on the thermal buffer, so it is what obliges +# the scheduler to run the heat pump at all — and, because the buffer holds several hours of +# it, what the scheduler is free to run early or late. Compare the electrical reference +# profile above: 30 kW electrical at COP 3.5 is 105 kW thermal, so demand stays well inside +# what the heat pump can deliver and the buffer can always be charged ahead of time. + +# An empty building at its night setback, losing heat through the envelope. +OFFICE_HEAT_PUMP_HEAT_DEMAND_NIGHT = 22.0 + +# Bringing a cooled-down building back up to temperature before the first arrivals is the +# heaviest call on the buffer of the whole day. +OFFICE_HEAT_PUMP_HEAT_DEMAND_MORNING = 62.0 + +# During office hours people, lighting and equipment contribute heat, so what the buffer +# still has to supply is well below the morning call. +OFFICE_HEAT_PUMP_HEAT_DEMAND_OCCUPIED = 34.0 + +# Nobody is coming in, so the building is allowed to drift and demand falls away. +OFFICE_HEAT_PUMP_HEAT_DEMAND_WEEKEND_FACTOR = 0.4 + +OFFICE_HEAT_PUMP_HEAT_DEMAND_NOISE = 0.06 + +# Commuters plug in around the start of the office day and leave around the end of it. +# Spread is a standard deviation; the bounds keep the tail of the distribution plausible. +EVSE_ARRIVAL_HOUR = 8.25 +EVSE_ARRIVAL_SPREAD_HOURS = 0.6 +EVSE_EARLIEST_ARRIVAL_HOUR = 7.0 +EVSE_LATEST_ARRIVAL_HOUR = 10.0 + +EVSE_DEPARTURE_HOUR = 17.0 +EVSE_DEPARTURE_SPREAD_HOURS = 0.9 +EVSE_EARLIEST_DEPARTURE_HOUR = 15.0 +EVSE_LATEST_DEPARTURE_HOUR = 19.5 + +# Nobody plugs in for less than this, so a short stay never becomes a zero-length session. +EVSE_MIN_SESSION_HOURS = 0.75 + +# How often a charge point is taken for the office day. Not every bay fills every day, +# and at the weekend the hub is nearly empty. +EVSE_WEEKDAY_OCCUPANCY = 0.85 +EVSE_WEEKEND_OCCUPANCY = 0.1 + +# How much energy a commuter needs, as a fraction of the connected car's usable battery. +# Only shapes the unscheduled reference profile; what the scheduler has to deliver is the +# departure requirement below. +EVSE_SESSION_ENERGY_FRACTION = (0.25, 0.75) + +# Most cars cannot accept the full rating of the charge point, because their onboard charger +# is the bottleneck. The floor is deliberately not much below the rating: this fraction also +# becomes the charge point's `power-capacity` while the car is plugged in, so a car that +# accepted only a fraction of the rating could not physically meet its departure requirement +# within the stay, and the scheduler would have to buy off a breach instead of planning. +EVSE_ACCEPTED_POWER_FRACTION = (0.7, 1.0) + +# Where a connected car's battery sits when the demo starts, as a fraction of its usable +# capacity. A commuter arrives on a low battery, which is precisely why the fleet is worth +# scheduling: the deficit between this and the departure requirement below is the energy the +# scheduler gets to place in the cheapest quarter hours of the stay. +EVSE_INITIAL_SOC_RANGE = (0.15, 0.35) + +# What the driver expects to find in the car at departure, as a fraction of usable battery. +# Recorded on each charge point's `minimum state of charge` sensor as a requirement that only +# applies in the quarter hour the car leaves in, and referenced by the flex-model's +# `soc-minima`. Confining it to that one quarter hour is what leaves the whole stay free: +# a requirement that also applied earlier would force charging on arrival instead. +# Only commuter stays carry one — an early bird plugged in for an hour is topping up, not +# expecting a full battery, and a short stay could not honour it anyway. +EVSE_DEPARTURE_SOC_FRACTION = (0.8, 0.95) + + +@dataclass(frozen=True, slots=True) +class ShoulderSession: + """ + A short charging session just outside the office day, for an early bird or a late worker. + + :param probability: Chance that a given charge point sees this session on a given weekday. + :param earliest_start_hour: Start of the window the session may begin in. + :param latest_start_hour: End of that window. + :param min_duration_hours: Shortest plausible stay. + :param max_duration_hours: Longest plausible stay. + :param energy_fraction: Energy taken, as a fraction of the connected car's usable battery. + """ + + probability: float + earliest_start_hour: float + latest_start_hour: float + min_duration_hours: float + max_duration_hours: float + energy_fraction: tuple[float, float] + + +# Someone who beats the traffic, charges for an hour and unplugs before the commuters arrive. +EVSE_EARLY_SESSION = ShoulderSession( + probability=0.18, + earliest_start_hour=5.75, + latest_start_hour=7.25, + min_duration_hours=0.75, + max_duration_hours=2.0, + energy_fraction=(0.08, 0.2), +) + +# Someone who stays late, or who tops up before driving home. +EVSE_LATE_SESSION = ShoulderSession( + probability=0.22, + earliest_start_hour=17.5, + latest_start_hour=19.5, + min_duration_hours=0.75, + max_duration_hours=2.5, + energy_fraction=(0.08, 0.25), +) + +__all__ = [ + "ACCOUNT_NAME", + "CAMPUS_BREACH_PRICE", + "CAMPUS_NAME", + "CAMPUS_POWER_CAPACITY", + "CAMPUS_SOC_MINIMA_BREACH_PRICE", + "CONTRACT_RESOLUTION", + "ENERGY_UNIT", + "EVSE_ACCEPTED_POWER_FRACTION", + "EVSE_ARRIVAL_HOUR", + "EVSE_ARRIVAL_SPREAD_HOURS", + "EVSE_AVAILABILITY_SENSOR_NAME", + "EVSE_DEPARTURE_HOUR", + "EVSE_DEPARTURE_SOC_FRACTION", + "EVSE_DEPARTURE_SPREAD_HOURS", + "EVSE_EARLIEST_ARRIVAL_HOUR", + "EVSE_EARLIEST_DEPARTURE_HOUR", + "EVSE_EARLY_SESSION", + "EVSE_HUB_NAME", + "EVSE_HUB_POWER_CAPACITY", + "EVSE_INITIAL_SOC_RANGE", + "EVSE_LATEST_ARRIVAL_HOUR", + "EVSE_LATEST_DEPARTURE_HOUR", + "EVSE_LATE_SESSION", + "EVSE_MIN_SESSION_HOURS", + "EVSE_SESSION_ENERGY_FRACTION", + "EVSE_SPECS", + "EVSE_WEEKDAY_OCCUPANCY", + "EVSE_WEEKEND_OCCUPANCY", + "FLEXMEASURES_URL", + "FORECAST_HORIZON", + "GRID_CONNECTION_CAPACITY_SENSOR_NAME", + "HEAT_DEMAND_SENSOR_NAME", + "INDOOR_TEMPERATURE_SENSOR_NAME", + "OFFICE_BASELOAD_DAY_POWER", + "OFFICE_BASELOAD_LUNCH_DIP", + "OFFICE_BASELOAD_LUNCH_HOUR", + "OFFICE_BASELOAD_LUNCH_SPREAD_HOURS", + "OFFICE_BASELOAD_NAME", + "OFFICE_BASELOAD_NIGHT_POWER", + "OFFICE_BASELOAD_NOISE", + "OFFICE_BASELOAD_WEEKEND_DAY_POWER", + "OFFICE_CLOSING_HOUR", + "OFFICE_HEAT_PUMP_CHARGING_EFFICIENCY", + "OFFICE_HEAT_PUMP_COP", + "OFFICE_HEAT_PUMP_HEAT_DEMAND_MORNING", + "OFFICE_HEAT_PUMP_HEAT_DEMAND_NIGHT", + "OFFICE_HEAT_PUMP_HEAT_DEMAND_NOISE", + "OFFICE_HEAT_PUMP_HEAT_DEMAND_OCCUPIED", + "OFFICE_HEAT_PUMP_HEAT_DEMAND_WEEKEND_FACTOR", + "OFFICE_HEAT_PUMP_IDLE_POWER", + "OFFICE_HEAT_PUMP_INITIAL_SOC_RANGE", + "OFFICE_HEAT_PUMP_NAME", + "OFFICE_HEAT_PUMP_NOISE", + "OFFICE_HEAT_PUMP_OCCUPIED_POWER", + "OFFICE_HEAT_PUMP_POWER_CAPACITY", + "OFFICE_HEAT_PUMP_PREHEAT_POWER", + "OFFICE_HEAT_PUMP_PREHEAT_START_HOUR", + "OFFICE_HEAT_PUMP_SOC_MAX", + "OFFICE_HEAT_PUMP_SOC_MIN", + "OFFICE_HEAT_PUMP_STORAGE_EFFICIENCY", + "OFFICE_HEAT_PUMP_WEEKEND_FACTOR", + "OFFICE_NAME", + "OFFICE_OPENING_HOUR", + "OFFICE_POWER_CAPACITY", + "OFFICE_PV_CLEAR_SKY_PEAK_FRACTION", + "OFFICE_PV_CLOUD_NOISE", + "OFFICE_PV_MAX_DAY_CLEARNESS", + "OFFICE_PV_MIN_DAY_CLEARNESS", + "OFFICE_PV_NAME", + "OFFICE_PV_PEAK_POWER", + "OFFICE_PV_SUNRISE_HOUR", + "OFFICE_PV_SUNSET_HOUR", + "OFFICE_RAMP_HOURS", + "POLLING_SCHEDULE_NAME", + "POWER_RESOLUTION", + "POWER_SENSOR_NAME", + "POWER_UNIT", + "PRICE_DAY_LEVEL_RANGE", + "PRICE_EVENING_PEAK", + "PRICE_EVENING_PEAK_HOUR", + "PRICE_FEATURE_SPREAD_HOURS", + "PRICE_MIDDAY_DIP", + "PRICE_MIDDAY_DIP_HOUR", + "PRICE_MORNING_PEAK", + "PRICE_MORNING_PEAK_HOUR", + "PRICE_NIGHT", + "PRICE_NOISE", + "PRICE_RESOLUTION", + "PRICE_SENSOR_NAME", + "PRICE_UNIT", + "PRICE_WEEKEND_FACTOR", + "SITE_LATITUDE", + "SITE_LONGITUDE", + "SITE_TIMEZONE", + "SOC_MINIMA_SENSOR_NAME", + "SOC_RESOLUTION", + "SOC_SENSOR_NAME", + "TEMPERATURE_UNIT", + "VEN_CLIENT_NAME", + "EvseSpec", + "ShoulderSession", +] diff --git a/examples/lf-energy-demo/flexmeasures/seed_assets.py b/examples/lf-energy-demo/flexmeasures/seed_assets.py new file mode 100644 index 0000000..d9ae835 --- /dev/null +++ b/examples/lf-energy-demo/flexmeasures/seed_assets.py @@ -0,0 +1,763 @@ +""" +Seed a realistic three-level asset hierarchy in the walkthrough FlexMeasures instance. + +Builds a campus site with a single grid connection point, an EVSE charging hub and a +grid-friendly office beneath it, as real GenericAssets and Sensors with flex-context +and flex-model metadata that the FlexMeasures scheduler can act on. + +Follows the pattern of trigger_fetch.py: it runs inside the FlexMeasures server +container, reusing its uv-managed venv (/app/.venv) and talking to the database +through the ORM, so no API token flow is needed: + + docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py + +The script is idempotent: every asset and sensor is fetched by name before it is +created, so re-running it changes nothing but the printed summary. +""" + +from __future__ import annotations + +import argparse +import sys +from dataclasses import dataclass, field +from typing import TYPE_CHECKING + +from flexmeasures.app import create as create_flexmeasures_app +from flexmeasures.data import db +from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType +from flexmeasures.data.models.time_series import Sensor +from flexmeasures.data.models.user import Account +from flexmeasures.data.schemas.scheduling import DBFlexContextSchema +from flexmeasures.data.schemas.scheduling.storage import DBStorageFlexModelSchema +from flexmeasures.data.scripts.data_gen import ( + add_default_asset_types, + add_transmission_zone_asset, +) +from flexmeasures.data.services.utils import get_or_create_model +from hierarchy import ( + ACCOUNT_NAME, + CAMPUS_BREACH_PRICE, + CAMPUS_NAME, + CAMPUS_POWER_CAPACITY, + CAMPUS_SOC_MINIMA_BREACH_PRICE, + CONTRACT_RESOLUTION, + ENERGY_UNIT, + EVSE_AVAILABILITY_SENSOR_NAME, + EVSE_HUB_NAME, + EVSE_HUB_POWER_CAPACITY, + EVSE_SPECS, + FLEXMEASURES_URL, + GRID_CONNECTION_CAPACITY_SENSOR_NAME, + HEAT_DEMAND_SENSOR_NAME, + INDOOR_TEMPERATURE_SENSOR_NAME, + OFFICE_BASELOAD_NAME, + OFFICE_HEAT_PUMP_CHARGING_EFFICIENCY, + OFFICE_HEAT_PUMP_COP, + OFFICE_HEAT_PUMP_NAME, + OFFICE_HEAT_PUMP_POWER_CAPACITY, + OFFICE_HEAT_PUMP_SOC_MAX, + OFFICE_HEAT_PUMP_SOC_MIN, + OFFICE_HEAT_PUMP_STORAGE_EFFICIENCY, + OFFICE_NAME, + OFFICE_POWER_CAPACITY, + OFFICE_PV_NAME, + OFFICE_PV_PEAK_POWER, + POLLING_SCHEDULE_NAME, + POWER_RESOLUTION, + POWER_SENSOR_NAME, + POWER_UNIT, + PRICE_RESOLUTION, + PRICE_SENSOR_NAME, + PRICE_UNIT, + SITE_LATITUDE, + SITE_LONGITUDE, + SITE_TIMEZONE, + SOC_MINIMA_SENSOR_NAME, + SOC_RESOLUTION, + SOC_SENSOR_NAME, + TEMPERATURE_UNIT, + VEN_CLIENT_NAME, + EvseSpec, +) +from marshmallow import ValidationError +from sqlalchemy import select +from timely_beliefs.sensors.func_store.knowledge_horizons import x_days_ago_at_y_oclock + +from flexmeasures_openadr3.utils.ven_clients import VenClientRepository + +if TYPE_CHECKING: + from datetime import timedelta + + +@dataclass +class SeedReport: + """What the run created versus what it found already in place.""" + + created_assets: list[str] = field(default_factory=list) + existing_assets: list[str] = field(default_factory=list) + created_sensors: list[str] = field(default_factory=list) + existing_sensors: list[str] = field(default_factory=list) + openadr_nesting: str | None = None + openadr_wiring: str | None = None + + def record_asset(self, asset: GenericAsset, *, was_created: bool) -> None: + """Note an asset as created or pre-existing.""" + target = self.created_assets if was_created else self.existing_assets + target.append(f"{asset.name} (id {asset.id})") + + def record_sensor(self, sensor: Sensor, *, was_created: bool) -> None: + """Note a sensor as created or pre-existing.""" + target = self.created_sensors if was_created else self.existing_sensors + target.append(f"{sensor.generic_asset.name}/{sensor.name} (id {sensor.id})") + + +def find_account(account_name: str) -> Account: + """ + Look up the account that owns the demo hierarchy. + + :param account_name: Name of an existing account, e.g. the walkthrough's toy account. + :returns: The matching account. + :raises ValueError: When no account with that name exists. + """ + account = db.session.execute(select(Account).filter_by(name=account_name)).scalar_one_or_none() + if account is None: + msg = ( + f"No account named '{account_name}' found. " + "The walkthrough's docker-compose `server` service creates it with " + f"`flexmeasures add toy-account --name '{account_name}'`; start the stack first, " + "or pass --account-name to target another account." + ) + raise ValueError(msg) + return account + + +def get_or_create_day_ahead_price_sensor() -> Sensor: + """ + Return the public day-ahead price sensor on the NL transmission zone. + + Uses the same specification as `flexmeasures add toy-account`, so this reuses that + command's sensor instead of adding a second, near-identical price sensor. + + :returns: The day-ahead price sensor. + """ + nl_zone = add_transmission_zone_asset("NL", db=db) + sensor = get_or_create_model( + Sensor, + name=PRICE_SENSOR_NAME, + generic_asset=nl_zone, + unit=PRICE_UNIT, + timezone=SITE_TIMEZONE, + event_resolution=PRICE_RESOLUTION, + knowledge_horizon=( + x_days_ago_at_y_oclock, + {"x": 1, "y": 12, "z": "Europe/Paris"}, + ), + ) + db.session.flush() + return sensor + + +class DemoSiteBuilder: + """ + Builds the demo hierarchy in one account, creating only what is missing. + + Holds the account, the default asset types and the run's report, so the individual + build steps stay free of plumbing arguments. + """ + + def __init__(self, account: Account) -> None: + self.account = account + self.asset_types: dict[str, GenericAssetType] = add_default_asset_types(db=db) + self.report = SeedReport() + # Assets this run touched, so validation stays limited to the demo hierarchy. + self.assets: list[GenericAsset] = [] + + def asset( + self, + name: str, + asset_type: str, + description: str, + parent: GenericAsset | None = None, + ) -> GenericAsset: + """ + Fetch or create one asset of the demo hierarchy. + + Assets are looked up by name within the account, which is what makes re-runs + of this script leave the hierarchy untouched. + + :param name: Asset name, unique within the account. + :param asset_type: One of the default asset type names, e.g. `building` or `one-way_evse`. + :param description: Short explanation shown on the asset page. + :param parent: Parent asset, or None for the top-level site. + :returns: The existing or freshly created asset. + """ + # Matching on the parent too keeps a name collision elsewhere in the account from + # silently adopting a foreign asset into the demo hierarchy. + has_expected_parent = GenericAsset.parent_asset_id.is_(None) if parent is None else GenericAsset.parent_asset_id == parent.id + query = select(GenericAsset).filter_by(name=name, account_id=self.account.id).filter(has_expected_parent) + existing = db.session.execute(query).scalar_one_or_none() + + if existing is not None: + self.report.record_asset(existing, was_created=False) + self.assets.append(existing) + return existing + + asset = GenericAsset( + name=name, + generic_asset_type=self.asset_types[asset_type], + owner=self.account, + description=description, + latitude=SITE_LATITUDE, + longitude=SITE_LONGITUDE, + parent_asset_id=None if parent is None else parent.id, + ) + db.session.add(asset) + db.session.flush() + self.report.record_asset(asset, was_created=True) + self.assets.append(asset) + return asset + + def sensor( + self, + asset: GenericAsset, + name: str, + unit: str, + event_resolution: timedelta | str = POWER_RESOLUTION, + attributes: dict | None = None, + ) -> Sensor: + """ + Fetch or create one sensor on an asset of the demo hierarchy. + + :param asset: Asset the sensor belongs to. + :param name: Sensor name, unique within the asset. + :param unit: Sensor unit, e.g. `kW` or `kWh`. + :param event_resolution: Resolution of the recorded events; a zero timedelta means instantaneous. + :param attributes: Extra sensor attributes, e.g. the sign convention. + :returns: The existing or freshly created sensor. + """ + existing = db.session.execute(select(Sensor).filter_by(name=name, generic_asset_id=asset.id)).scalar_one_or_none() + if existing is not None: + self.report.record_sensor(existing, was_created=False) + return existing + + sensor = Sensor( + name=name, + generic_asset=asset, + unit=unit, + timezone=SITE_TIMEZONE, + event_resolution=event_resolution, + attributes=attributes or {}, + ) + db.session.add(sensor) + db.session.flush() + self.report.record_sensor(sensor, was_created=True) + return sensor + + def power_sensor(self, asset: GenericAsset, *, consumption_is_positive: bool = True) -> Sensor: + """ + Fetch or create the standard quarter-hourly power sensor of an asset. + + :param asset: Asset the sensor belongs to. + :param consumption_is_positive: Sign convention; False for a producing asset such as PV. + :returns: The asset's power sensor. + """ + return self.sensor( + asset, + POWER_SENSOR_NAME, + POWER_UNIT, + attributes={"consumption_is_positive": consumption_is_positive}, + ) + + +def build_campus(builder: DemoSiteBuilder, price_sensor: Sensor) -> tuple[GenericAsset, Sensor]: + """ + Create the top-level site and its grid connection point. + + The campus carries the flex-context, because a flex-context is read upwards through + the asset tree: whatever the campus declares also applies to the hub and the office, + unless those override it. + + :param builder: Builder holding the target account. + :param price_sensor: Day-ahead price sensor to schedule against. + :returns: The campus asset and its power sensor. + """ + campus = builder.asset( + CAMPUS_NAME, + "building", + "Demo campus with a single grid connection point, through which all demo power flows pass.", + ) + campus_power = builder.power_sensor(campus) + builder.sensor( + campus, + GRID_CONNECTION_CAPACITY_SENSOR_NAME, + POWER_UNIT, + event_resolution=CONTRACT_RESOLUTION, + ) + + campus.flex_context = { + "consumption-price": {"sensor": price_sensor.id}, + "production-price": {"sensor": price_sensor.id}, + "site-power-capacity": CAMPUS_POWER_CAPACITY, + "site-consumption-breach-price": CAMPUS_BREACH_PRICE, + "site-production-breach-price": CAMPUS_BREACH_PRICE, + # Prices the charge points' departure requirements rather than forbidding them, so a + # stay too short for the car's deficit yields a plan with a costed shortfall instead + # of an infeasible schedule. + "soc-minima-breach-price": CAMPUS_SOC_MINIMA_BREACH_PRICE, + # The site's own power sensor doubles as the output for the scheduled aggregate. + "aggregate-consumption": {"sensor": campus_power.id}, + } + db.session.flush() + return campus, campus_power + + +def set_campus_dashboard(campus: GenericAsset, power_sensors: list[Sensor], capacity_sensors: list[Sensor]) -> None: + """ + Give the site a dashboard covering prices, power flows and connection capacity. + + Done after the subtrees are built, since it references their aggregate power sensors. + + :param campus: Top-level site asset. + :param power_sensors: Site, hub and office power sensors, in that order. + :param capacity_sensors: Any OpenADR capacity-limit sensors wired into the flex-context. + """ + connection_capacity = next(sensor for sensor in campus.sensors if sensor.name == GRID_CONNECTION_CAPACITY_SENSOR_NAME) + campus.sensors_to_show = [ + { + "title": "Day-ahead prices", + "plots": [{"asset": campus.id, "flex-context": "consumption-price"}], + }, + { + "title": "Site power flows", + "plots": [{"sensors": [sensor.id for sensor in power_sensors]}], + }, + { + "title": "Connection capacity", + "plots": [{"sensors": [connection_capacity.id, *(sensor.id for sensor in capacity_sensors)]}], + }, + ] + db.session.flush() + + +def build_evse_hub(builder: DemoSiteBuilder, campus: GenericAsset) -> tuple[GenericAsset, Sensor]: + """ + Create the EVSE charging hub and its charge points. + + The hub is a device group rather than a device: its own flex-model entry declares the + capacity of the sub-connection that all charge points behind it have to share, and each + charge point points back at it through the flex-model `group` field. + + :param builder: Builder holding the target account. + :param campus: Parent site asset. + :returns: The hub asset and its aggregate power sensor. + """ + hub = builder.asset( + EVSE_HUB_NAME, + "building", + f"EVSE charging hub behind a {EVSE_HUB_POWER_CAPACITY} sub-connection, shared by all charge points below it.", + parent=campus, + ) + hub_power = builder.power_sensor(hub) + hub.flex_model = { + "power-capacity": EVSE_HUB_POWER_CAPACITY, + "consumption": {"sensor": hub_power.id}, + } + db.session.flush() + + charge_point_power_sensors = [build_charge_point(builder, hub, spec) for spec in EVSE_SPECS] + hub.sensors_to_show = [ + { + "title": "Charging power", + "plots": [{"sensors": [hub_power.id, *(sensor.id for sensor in charge_point_power_sensors)]}], + }, + ] + db.session.flush() + return hub, hub_power + + +def build_charge_point(builder: DemoSiteBuilder, hub: GenericAsset, spec: EvseSpec) -> Sensor: + """ + Create one charge point below the hub. + + Two sensors beyond the usual power and state of charge are what make the charge point a + load the scheduler has to plan for rather than an idle battery: + + - `power availability` drives `power-capacity`, so the charge point can only move power + while a car is actually plugged in. It replaces the static nameplate rating, which + would have let the scheduler charge an empty bay at three in the morning. + - `minimum state of charge` drives `soc-minima`, the requirement that the connected car + is full when its driver leaves. This is the charge point's reason to consume at all: + with only soc-min and soc-max, an empty battery costs nothing and the cheapest plan is + never to charge — which is exactly what earlier versions of this demo produced. + + Between them the driver's deficit becomes a fixed amount of energy that has to be + delivered inside a known stay, leaving the scheduler free to choose *which* quarter + hours of that stay to use. That freedom is what the day-ahead price and the OpenADR + capacity limit then compete over. + + :param builder: Builder holding the target account. + :param hub: Parent hub asset, which also acts as the charge point's device group. + :param spec: Rating and battery limits of this charge point. + :returns: The charge point's power sensor. + """ + is_bidirectional = spec.asset_type == "two-way_evse" + charge_point = builder.asset( + spec.name, + spec.asset_type, + f"{spec.power_capacity} {'bidirectional (V2G)' if is_bidirectional else 'charge-only'} charge point.", + parent=hub, + ) + power = builder.power_sensor(charge_point) + state_of_charge = builder.sensor(charge_point, SOC_SENSOR_NAME, ENERGY_UNIT, event_resolution=SOC_RESOLUTION) + availability = builder.sensor(charge_point, EVSE_AVAILABILITY_SENSOR_NAME, POWER_UNIT) + soc_minima = builder.sensor(charge_point, SOC_MINIMA_SENSOR_NAME, ENERGY_UNIT) + + charge_point.flex_model = { + "group": {"asset": hub.id}, + # A sensor rather than spec.power_capacity: zero while the bay is empty, and the + # connected car's accepted power while it is not. + "power-capacity": {"sensor": availability.id}, + "soc-min": spec.soc_min, + "soc-max": spec.soc_max, + "soc-minima": {"sensor": soc_minima.id}, + "roundtrip-efficiency": spec.roundtrip_efficiency, + "state-of-charge": {"sensor": state_of_charge.id}, + "consumption": {"sensor": power.id}, + } + if spec.production_capacity is not None: + charge_point.flex_model["production-capacity"] = spec.production_capacity + db.session.flush() + return power + + +def build_office(builder: DemoSiteBuilder, campus: GenericAsset) -> tuple[GenericAsset, Sensor]: + """ + Create the grid-friendly office with rooftop PV, a heat pump and a background load. + + PV and background load are modelled as their own child assets carrying an inflexible + flex-model entry, rather than as flex-context fields on the office. That way the + scheduler still sees them when the campus, rather than the office, is the asset being + scheduled, because a flex-model is gathered downwards through the whole asset tree. + + Like the heat pump, both also reference the office as their `group`, which is what + makes the office's `power-capacity` a real cap on its whole sub-connection: a group's + capacity only binds the entries that name it, so leaving the two inflexible devices + out would silently give the heat pump the entire 250 kVA to itself. + + :param builder: Builder holding the target account. + :param campus: Parent site asset. + :returns: The office asset and its aggregate power sensor. + """ + office = builder.asset( + OFFICE_NAME, + "building", + f"Grid-friendly office behind a {OFFICE_POWER_CAPACITY} sub-connection, with rooftop PV, a heat pump and a background load.", + parent=campus, + ) + office_power = builder.power_sensor(office) + indoor_temperature = builder.sensor(office, INDOOR_TEMPERATURE_SENSOR_NAME, TEMPERATURE_UNIT) + office.flex_model = { + "power-capacity": OFFICE_POWER_CAPACITY, + "consumption": {"sensor": office_power.id}, + } + db.session.flush() + + pv_power = build_office_pv(builder, office) + baseload_power = build_office_baseload(builder, office) + heat_pump_power = build_office_heat_pump(builder, office) + + office.sensors_to_show = [ + { + "title": "Office power flows", + "plots": [{"sensors": [office_power.id, pv_power.id, baseload_power.id, heat_pump_power.id]}], + }, + {"title": "Indoor climate", "plots": [{"sensor": indoor_temperature.id}]}, + ] + db.session.flush() + return office, office_power + + +def build_office_pv(builder: DemoSiteBuilder, office: GenericAsset) -> Sensor: + """ + Create the office's rooftop PV as must-take production. + + :param builder: Builder holding the target account. + :param office: Parent office asset. + :returns: The PV power sensor. + """ + pv = builder.asset( + OFFICE_PV_NAME, + "solar", + f"{OFFICE_PV_PEAK_POWER} of rooftop PV, modelled as must-take production rather than a curtailable device.", + parent=office, + ) + pv_power = builder.power_sensor(pv, consumption_is_positive=False) + pv.flex_model = { + # Behind the office's connection like everything else, so its infeed counts + # against that connection's capacity even though it is not schedulable. + "group": {"asset": office.id}, + "inflexible-production": {"sensor": pv_power.id}, + } + db.session.flush() + return pv_power + + +def build_office_baseload(builder: DemoSiteBuilder, office: GenericAsset) -> Sensor: + """ + Create the office's background load as inflexible consumption. + + :param builder: Builder holding the target account. + :param office: Parent office asset. + :returns: The background load's power sensor. + """ + baseload = builder.asset( + OFFICE_BASELOAD_NAME, + "process", + "Background office load (lighting, servers, plug loads) that cannot be shifted.", + parent=office, + ) + baseload_power = builder.power_sensor(baseload) + baseload.flex_model = { + # Behind the office's connection like everything else, so it eats into the + # headroom the heat pump has left under that connection's capacity. + "group": {"asset": office.id}, + "inflexible-consumption": {"sensor": baseload_power.id}, + } + db.session.flush() + return baseload_power + + +def build_office_heat_pump(builder: DemoSiteBuilder, office: GenericAsset) -> Sensor: + """ + Create the office's heat pump as a thermal buffer. + + FlexMeasures ships no `heat_pump` asset type, so the heat pump together with the + building's thermal mass is modelled as a `heat-storage` device, mirroring the + `Heat Pump Template` asset FlexMeasures seeds itself: the scheduler charges and + discharges the buffer, and the storage efficiency stands in for heat losses. + The coefficient of performance goes in `charging-efficiency`, the one-way conversion + from electricity to stored heat, which may exceed 100% for exactly this purpose. + Power is therefore electrical while state of charge is thermal. + + The `heat demand` sensor is what gives the heat pump a reason to run. It carries the + building's heat demand in thermal kW and drives `soc-usage`, the drain on the buffer. + Without it the flex-model only bounded the buffer, so letting it sit empty cost nothing + and the scheduler left the heat pump at 0 kW in every interval of every schedule. With + it, the demand has to be met — and because the buffer holds several hours of it, the + scheduler may meet it early or late, which is what makes the heat pump a second device + that reacts to prices and to the OpenADR capacity limit. + + :param builder: Builder holding the target account. + :param office: Parent office asset, which also acts as the heat pump's device group. + :returns: The heat pump's power sensor. + """ + description = ( + f"Heat pump ({OFFICE_HEAT_PUMP_POWER_CAPACITY} electrical, COP {OFFICE_HEAT_PUMP_COP}) with the building's thermal mass as its buffer. " + "State of charge is thermal energy." + ) + heat_pump = builder.asset(OFFICE_HEAT_PUMP_NAME, "heat-storage", description, parent=office) + heat_pump_power = builder.power_sensor(heat_pump) + state_of_charge = builder.sensor(heat_pump, SOC_SENSOR_NAME, ENERGY_UNIT, event_resolution=SOC_RESOLUTION) + heat_demand = builder.sensor(heat_pump, HEAT_DEMAND_SENSOR_NAME, POWER_UNIT) + + heat_pump.flex_model = { + "group": {"asset": office.id}, + "power-capacity": OFFICE_HEAT_PUMP_POWER_CAPACITY, + # A heat pump cannot feed power back into the office. + "production-capacity": "0 kW", + "soc-min": OFFICE_HEAT_PUMP_SOC_MIN, + "soc-max": OFFICE_HEAT_PUMP_SOC_MAX, + # A list, because FlexMeasures lets several drains add up into one aggregate usage; + # the demo has just the one. Thermal kW, matching the thermal state of charge. + "soc-usage": [{"sensor": heat_demand.id}], + # Mutually exclusive with `roundtrip-efficiency`, which the charge points use instead. + "charging-efficiency": OFFICE_HEAT_PUMP_CHARGING_EFFICIENCY, + "storage-efficiency": OFFICE_HEAT_PUMP_STORAGE_EFFICIENCY, + "state-of-charge": {"sensor": state_of_charge.id}, + "consumption": {"sensor": heat_pump_power.id}, + } + db.session.flush() + return heat_pump_power + + +def find_openadr_capacity_sensors(report: SeedReport) -> dict[str, Sensor]: + """ + Look up the OpenADR capacity-limit sensors, keyed by the flex-context field they feed. + + The sensors only come into existence once a VEN client and polling schedule have been + configured (walkthrough steps 6 and 7), so finding none is a normal state to report + rather than an error. + + :param report: Run report to record why no sensors were found, if that is the case. + :returns: Flex-context field name mapped to the sensor that should drive it. + """ + ven_client = VenClientRepository().find_by_name(VEN_CLIENT_NAME) + if ven_client is None: + report.openadr_wiring = f"not wired: no VEN client '{VEN_CLIENT_NAME}' configured yet" + return {} + + sensor_config = ven_client.get_sensor_config(POLLING_SCHEDULE_NAME) + if sensor_config is None: + report.openadr_wiring = f"not wired: VEN client '{VEN_CLIENT_NAME}' has no polling schedule '{POLLING_SCHEDULE_NAME}' yet" + return {} + + found = { + context_field: sensor + for context_field, sensor in ( + ("site-consumption-capacity", sensor_config.import_sensor), + ("site-production-capacity", sensor_config.export_sensor), + ) + if sensor is not None + } + if not found: + report.openadr_wiring = f"not wired: polling schedule '{POLLING_SCHEDULE_NAME}' fetches neither import nor export capacity limits" + return found + + +def nest_ven_asset_under_campus(campus: GenericAsset, report: SeedReport) -> None: + """ + Move the VEN client asset into demo-campus's own asset tree, if one exists. + + FlexMeasures' "Edit flex-context" sensor picker (the UI form for + site-consumption-capacity/site-production-capacity) only searches an asset's + descendants, so a VEN client left as its own top-level asset can never be found + there. Nesting it here is what makes wiring those fields through the UI possible at + all, rather than only through this script. + + :param campus: Top-level site asset to nest the VEN client asset under. + :param report: Run report to record the outcome in. + """ + ven_client = VenClientRepository().find_by_name(VEN_CLIENT_NAME) + if ven_client is None: + report.openadr_nesting = f"not nested: no VEN client '{VEN_CLIENT_NAME}' configured yet" + return + if ven_client.asset.parent_asset_id == campus.id: + report.openadr_nesting = f"'{VEN_CLIENT_NAME}' already nested under '{campus.name}'" + return + ven_client.asset.parent_asset_id = campus.id + db.session.flush() + report.openadr_nesting = f"nested '{VEN_CLIENT_NAME}' under '{campus.name}'" + + +def wire_openadr_capacity_limits(campus: GenericAsset, report: SeedReport) -> list[Sensor]: + """ + Point the site's capacity limits at the OpenADR sensors, or clear them when absent. + + This is what ties the hierarchy to the rest of the walkthrough: the import and export + capacity limits fetched from the VTN become the site capacities the scheduler has to + respect. Both fields are rewritten from scratch on every run, so a hierarchy seeded + while a polling schedule existed does not keep pointing at sensors that were since + deleted along with it. + + :param campus: Top-level site asset whose flex-context is updated. + :param report: Run report to record the outcome in. + :returns: The capacity-limit sensors that were wired in, empty when there are none. + """ + sensors_by_field = find_openadr_capacity_sensors(report) + + for context_field in ("site-consumption-capacity", "site-production-capacity"): + sensor = sensors_by_field.get(context_field) + if sensor is None: + campus.flex_context.pop(context_field, None) + else: + campus.flex_context[context_field] = {"sensor": sensor.id} + + if sensors_by_field: + report.openadr_wiring = ", ".join(f"{context_field} -> {sensor.name}" for context_field, sensor in sensors_by_field.items()) + db.session.flush() + return list(sensors_by_field.values()) + + +def validate_flex_metadata(assets: list[GenericAsset]) -> None: + """ + Check the seeded flex-context and flex-model against their storage schemas. + + FlexMeasures validates these JSON columns when they are written through the API, but + not on a direct ORM write, and an invalid entry would only surface later as a failed + schedule. Validating here keeps the demo data guaranteed schedulable. + + :param assets: Assets this run built or found. + :raises ValueError: When a seeded flex-context or flex-model is invalid. + """ + for asset in assets: + for label, stored, schema in ( + ("flex-context", asset.flex_context, DBFlexContextSchema()), + ("flex-model", asset.flex_model, DBStorageFlexModelSchema()), + ): + if not stored: + continue + try: + schema.load(stored) + except ValidationError as exc: + msg = f"Seeded {label} of asset '{asset.name}' is invalid: {exc.messages}. Fix the specification in hierarchy.py." + raise ValueError(msg) from exc + + +def print_summary(report: SeedReport, campus: GenericAsset) -> None: + """ + Print what the run created, found and wired up. + + :param report: Run report gathered by the builder. + :param campus: Top-level site asset, to link to in the closing hint. + """ + print(f"Created {len(report.created_assets)} asset(s) and {len(report.created_sensors)} sensor(s).") + print(f"Found {len(report.existing_assets)} asset(s) and {len(report.existing_sensors)} sensor(s) already in place.") + for name in report.created_assets: + print(f" + asset {name}") + for name in report.created_sensors: + print(f" + sensor {name}") + for name in report.existing_assets: + print(f" = asset {name}") + print(f"OpenADR VEN asset: {report.openadr_nesting}") + print(f"OpenADR site capacity limits: {report.openadr_wiring}") + print(f"Next: open {FLEXMEASURES_URL}/assets/{campus.id} to inspect the hierarchy.") + + +def main() -> None: + """Seed the demo asset hierarchy and print a summary of the result.""" + parser = argparse.ArgumentParser(description="Seed the LF Energy demo asset hierarchy in FlexMeasures.") + parser.add_argument( + "--account-name", + default=ACCOUNT_NAME, + help=f"Account to own the demo assets (default: {ACCOUNT_NAME})", + ) + parser.add_argument( + "--skip-openadr-wiring", + action="store_true", + help=( + "Nest the VEN client asset under the campus, but leave " + "site-consumption-capacity/site-production-capacity unset -- for wiring " + "them live via demo-campus's 'Edit flex-context' UI instead." + ), + ) + args = parser.parse_args() + + app = create_flexmeasures_app() + + with app.app_context(): + account = find_account(args.account_name) + price_sensor = get_or_create_day_ahead_price_sensor() + + builder = DemoSiteBuilder(account) + campus, campus_power = build_campus(builder, price_sensor) + _, hub_power = build_evse_hub(builder, campus) + _, office_power = build_office(builder, campus) + nest_ven_asset_under_campus(campus, builder.report) + if args.skip_openadr_wiring: + capacity_sensors = [] + builder.report.openadr_wiring = "skipped (--skip-openadr-wiring); wire it via demo-campus's 'Edit flex-context' UI" + else: + capacity_sensors = wire_openadr_capacity_limits(campus, builder.report) + set_campus_dashboard(campus, [campus_power, hub_power, office_power], capacity_sensors) + + validate_flex_metadata(builder.assets) + db.session.commit() + print_summary(builder.report, campus) + + +if __name__ == "__main__": + try: + main() + except ValueError as exc: + print(f"Failed to seed the demo asset hierarchy: {exc}", file=sys.stderr) + raise SystemExit(1) from exc diff --git a/examples/lf-energy-demo/flexmeasures/seed_forecasts.py b/examples/lf-energy-demo/flexmeasures/seed_forecasts.py new file mode 100644 index 0000000..c06389b --- /dev/null +++ b/examples/lf-energy-demo/flexmeasures/seed_forecasts.py @@ -0,0 +1,1004 @@ +""" +Seed a week of synthetic beliefs onto the demo site's sensors. + +Gives the hierarchy that seed_assets.py builds something to show and something to +schedule against: a price curve to optimise against, a forecast for every load the +scheduler cannot move, a typical-usage reference profile for every device it can move, +and a current state of charge for every storage device, so a schedule can be triggered +right after this script finishes. + +Like seed_assets.py, it runs inside the FlexMeasures server container, reusing its +uv-managed venv (/app/.venv) and talking to the database through the ORM, so no API +token flow is needed: + + docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_forecasts.py + +Three kinds of belief are written, by two clearly-labelled data sources, because they do +not mean the same thing: + +- *Forecasts*, by the `LF Energy demo forecaster` source: everything the scheduler reads as + given and plans around. The day-ahead prices are what it optimises against. The office's + background load and rooftop PV are the site's `inflexible-consumption` and + `inflexible-production`. And each flexible device's *energy requirement* is a forecast + too — the building's heat demand, and every charge point's power availability and + departure requirement — because those describe the weather, the building and the drivers + rather than anything the scheduler decides. They are genuine forecasts: their belief time + precedes their knowledge time. +- *Reference profiles*, by the `LF Energy demo profiles` source, on the charge points and + the heat pump. Those are the devices the scheduler dispatches, so a fixed week of power + values is not a forecast of anything: it is what the site would do unscheduled, kept + only so the charts are not empty before the first schedule runs. The scheduler writes + its own beliefs, from its own source, onto the same sensors. +- *Measurements* of state of charge, by the same profiles source, one per storage device + at the current quarter hour, which is what the storage scheduler looks for when it + resolves `soc-at-start` from a sensor. + +The aggregate power sensors of the campus, the hub and the office are deliberately left +empty: they are the scheduler's output, and seeding a synthetic aggregate there would +make those charts disagree with the schedule. + +The values are deterministic. Every profile is generated per sensor and per local +calendar day from a seeded random generator keyed on both, so a given day always gets the +same numbers no matter when the script runs, and re-running it merges the same rows back +over themselves rather than drifting. + +This script only ever runs after seed_assets.py, so it imports that script's account and +price-sensor lookups rather than duplicating them and risking the two drifting apart. +""" + +from __future__ import annotations + +import argparse +import math +import random +import sys +from collections.abc import Callable +from dataclasses import dataclass, field +from datetime import date, timedelta +from functools import partial + +import pandas as pd +from flexmeasures.app import create as create_flexmeasures_app +from flexmeasures.data import db +from flexmeasures.data.models.data_sources import DataSource +from flexmeasures.data.models.generic_assets import GenericAsset +from flexmeasures.data.models.time_series import Sensor, TimedBelief +from flexmeasures.data.models.user import Account +from flexmeasures.data.services.data_sources import get_or_create_source +from flexmeasures.utils.unit_utils import ur +from hierarchy import ( + ACCOUNT_NAME, + CAMPUS_NAME, + ENERGY_UNIT, + EVSE_ACCEPTED_POWER_FRACTION, + EVSE_ARRIVAL_HOUR, + EVSE_ARRIVAL_SPREAD_HOURS, + EVSE_AVAILABILITY_SENSOR_NAME, + EVSE_DEPARTURE_HOUR, + EVSE_DEPARTURE_SOC_FRACTION, + EVSE_DEPARTURE_SPREAD_HOURS, + EVSE_EARLIEST_ARRIVAL_HOUR, + EVSE_EARLIEST_DEPARTURE_HOUR, + EVSE_EARLY_SESSION, + EVSE_INITIAL_SOC_RANGE, + EVSE_LATE_SESSION, + EVSE_LATEST_ARRIVAL_HOUR, + EVSE_LATEST_DEPARTURE_HOUR, + EVSE_MIN_SESSION_HOURS, + EVSE_SESSION_ENERGY_FRACTION, + EVSE_SPECS, + EVSE_WEEKDAY_OCCUPANCY, + EVSE_WEEKEND_OCCUPANCY, + FLEXMEASURES_URL, + FORECAST_HORIZON, + HEAT_DEMAND_SENSOR_NAME, + OFFICE_BASELOAD_DAY_POWER, + OFFICE_BASELOAD_LUNCH_DIP, + OFFICE_BASELOAD_LUNCH_HOUR, + OFFICE_BASELOAD_LUNCH_SPREAD_HOURS, + OFFICE_BASELOAD_NAME, + OFFICE_BASELOAD_NIGHT_POWER, + OFFICE_BASELOAD_NOISE, + OFFICE_BASELOAD_WEEKEND_DAY_POWER, + OFFICE_CLOSING_HOUR, + OFFICE_HEAT_PUMP_HEAT_DEMAND_MORNING, + OFFICE_HEAT_PUMP_HEAT_DEMAND_NIGHT, + OFFICE_HEAT_PUMP_HEAT_DEMAND_NOISE, + OFFICE_HEAT_PUMP_HEAT_DEMAND_OCCUPIED, + OFFICE_HEAT_PUMP_HEAT_DEMAND_WEEKEND_FACTOR, + OFFICE_HEAT_PUMP_IDLE_POWER, + OFFICE_HEAT_PUMP_INITIAL_SOC_RANGE, + OFFICE_HEAT_PUMP_NAME, + OFFICE_HEAT_PUMP_NOISE, + OFFICE_HEAT_PUMP_OCCUPIED_POWER, + OFFICE_HEAT_PUMP_POWER_CAPACITY, + OFFICE_HEAT_PUMP_PREHEAT_POWER, + OFFICE_HEAT_PUMP_PREHEAT_START_HOUR, + OFFICE_HEAT_PUMP_SOC_MAX, + OFFICE_HEAT_PUMP_WEEKEND_FACTOR, + OFFICE_OPENING_HOUR, + OFFICE_PV_CLEAR_SKY_PEAK_FRACTION, + OFFICE_PV_CLOUD_NOISE, + OFFICE_PV_MAX_DAY_CLEARNESS, + OFFICE_PV_MIN_DAY_CLEARNESS, + OFFICE_PV_NAME, + OFFICE_PV_PEAK_POWER, + OFFICE_PV_SUNRISE_HOUR, + OFFICE_PV_SUNSET_HOUR, + OFFICE_RAMP_HOURS, + POWER_RESOLUTION, + POWER_SENSOR_NAME, + POWER_UNIT, + PRICE_DAY_LEVEL_RANGE, + PRICE_EVENING_PEAK, + PRICE_EVENING_PEAK_HOUR, + PRICE_FEATURE_SPREAD_HOURS, + PRICE_MIDDAY_DIP, + PRICE_MIDDAY_DIP_HOUR, + PRICE_MORNING_PEAK, + PRICE_MORNING_PEAK_HOUR, + PRICE_NIGHT, + PRICE_NOISE, + PRICE_RESOLUTION, + PRICE_SENSOR_NAME, + PRICE_WEEKEND_FACTOR, + SITE_TIMEZONE, + SOC_MINIMA_SENSOR_NAME, + SOC_SENSOR_NAME, + EvseSpec, + ShoulderSession, +) +from seed_assets import find_account, get_or_create_day_ahead_price_sensor +from sqlalchemy import select + +# The two sources are named after this walkthrough and typed by what they stand for, so +# nobody mistakes either for the plugin's own `OpenADR 3 VTN` source of fetched DR signals. +FORECAST_SOURCE_NAME = "LF Energy demo forecaster" +FORECAST_SOURCE_TYPE = "forecaster" + +REFERENCE_SOURCE_NAME = "LF Energy demo profiles" +REFERENCE_SOURCE_TYPE = "demo script" + +# Monday is 0, so anything from here on is the weekend. +SATURDAY = 5 + +MINUTES_PER_HOUR = 60 +SECONDS_PER_HOUR = 3600 + +# Watts are far below the accuracy of anything synthetic; rounding also keeps re-runs writing identical rows. +BELIEF_PRECISION = 3 + +# Generates one local calendar day of values, one per event start, from a generator seeded on that day. +DayProfile = Callable[[date, pd.DatetimeIndex, random.Random], list[float]] + + +@dataclass(frozen=True, slots=True) +class ChargingSession: + """ + One car, plugged into one charge point, for one continuous stay. + + :param start: Local time the car arrives. + :param end: Local time it leaves. + :param energy: Energy the driver wants delivered during the stay, in the sensor's + energy unit. Shapes the unscheduled reference profile only. + :param power: Power the car accepts, which is at most the charge point's rating. + :param target_soc: State of charge the car has to reach by departure, in the sensor's + energy unit, or None for a stay that carries no requirement. This + is what the scheduler is actually held to. Only commuter stays have + one: an early bird plugged in for an hour is topping up, and a stay + that short could not honour a full battery anyway. + """ + + start: pd.Timestamp + end: pd.Timestamp + energy: float + power: float + target_soc: float | None + + +@dataclass +class ForecastReport: + """What the run wrote, per kind of belief.""" + + window_start: pd.Timestamp + window_end: pd.Timestamp + belief_time: pd.Timestamp + forecasts: list[str] = field(default_factory=list) + reference_profiles: list[str] = field(default_factory=list) + states_of_charge: list[str] = field(default_factory=list) + + def record_profile(self, sensor: Sensor, count: int, *, is_forecast: bool) -> None: + """ + Note a seeded power profile. + + :param sensor: Sensor the profile was written to. + :param count: Number of beliefs written. + :param is_forecast: True for a genuine forecast, False for a typical-usage reference profile. + """ + target = self.forecasts if is_forecast else self.reference_profiles + target.append(f"{sensor.generic_asset.name}/{sensor.name}: {count} beliefs") + + def record_state_of_charge(self, sensor: Sensor, value: float) -> None: + """ + Note a seeded state-of-charge reading. + + :param sensor: State-of-charge sensor the reading was written to. + :param value: The recorded value, in the sensor's own unit. + """ + self.states_of_charge.append(f"{sensor.generic_asset.name}/{sensor.name}: {value:.1f} {sensor.unit}") + + +def quantity_in(quantity: str, unit: str) -> float: + """ + Read one of hierarchy.py's quantity strings as a plain number in the given unit. + + :param quantity: A quantity with its unit, e.g. `22 kW`. + :param unit: Unit to express the result in, e.g. the unit of the sensor being written to. + :returns: The magnitude in that unit. + """ + return float(ur.Quantity(quantity).to(unit).magnitude) + + +def find_asset(account: Account, name: str) -> GenericAsset: + """ + Look up one asset of the demo hierarchy. + + :param account: Account that owns the demo hierarchy. + :param name: Asset name, as declared in hierarchy.py. + :returns: The matching asset. + :raises ValueError: When the asset does not exist, which means the hierarchy was never seeded. + """ + asset = db.session.execute(select(GenericAsset).filter_by(name=name, account_id=account.id)).scalar_one_or_none() + if asset is None: + msg = ( + f"No asset named '{name}' in account '{account.name}'. " + "Seed the hierarchy first with " + "`docker compose exec server uv run --active --no-project /walkthrough/flexmeasures/seed_assets.py`." + ) + raise ValueError(msg) + return asset + + +def find_sensor(asset: GenericAsset, name: str) -> Sensor: + """ + Look up one sensor of an asset of the demo hierarchy. + + :param asset: Asset the sensor belongs to. + :param name: Sensor name, as declared in hierarchy.py. + :returns: The matching sensor. + :raises ValueError: When the sensor does not exist, which means the hierarchy is out of date. + """ + sensor = db.session.execute(select(Sensor).filter_by(name=name, generic_asset_id=asset.id)).scalar_one_or_none() + if sensor is None: + msg = f"Asset '{asset.name}' has no sensor named '{name}'. Re-run seed_assets.py to bring the hierarchy up to date." + raise ValueError(msg) + return sensor + + +def scheduling_window(horizon: timedelta) -> tuple[pd.Timestamp, pd.Timestamp]: + """ + Return the window to seed, starting at the quarter hour the run falls in. + + Starting at the current quarter hour rather than the next one means a schedule + triggered straight after this run finds data from its very first time step. + + :param horizon: How far ahead to seed. + :returns: Window start and end, in UTC. + """ + start = pd.Timestamp.now(tz="UTC").floor(POWER_RESOLUTION) + return start, start + horizon + + +def forecast_issue_time(window_start: pd.Timestamp) -> pd.Timestamp: + """ + Return the belief time to stamp the seeded profiles with: local midnight of the run's day. + + A belief counts as a forecast when its belief time precedes the sensor's knowledge time. + These power sensors use timely-beliefs' default knowledge horizon, which places knowledge + time at the end of the event, so any belief time at or before the start of the window + makes the whole window read as a forecast rather than as a measurement. + Local midnight is used rather than the current moment because it does not move within a + run day, which is what lets a re-run merge the same rows back over themselves: belief time + is part of a belief's key. + + :param window_start: Start of the seeded window, in UTC. + :returns: The belief time, in UTC. + """ + return window_start.tz_convert(SITE_TIMEZONE).normalize().tz_convert("UTC") + + +def local_days(window: tuple[pd.Timestamp, pd.Timestamp]) -> list[date]: + """ + List every local calendar day the window touches. + + :param window: Window start and end. + :returns: The local dates, in order. + """ + first = window[0].tz_convert(SITE_TIMEZONE).date() + last = window[1].tz_convert(SITE_TIMEZONE).date() + return [first + timedelta(days=offset) for offset in range((last - first).days + 1)] + + +def local_event_starts(day: date, resolution: timedelta) -> pd.DatetimeIndex: + """ + Return every event start of one local calendar day, at the given resolution. + + Built from the day's two local midnights rather than by adding 24 hours, so the two + days a year that are 23 or 25 hours long still come out whole. + + :param day: A local calendar date. + :param resolution: Length of one event. + :returns: Localised event starts covering that day. + """ + midnight = pd.Timestamp(day, tz=SITE_TIMEZONE) + next_midnight = pd.Timestamp(day + timedelta(days=1), tz=SITE_TIMEZONE) + return pd.date_range(midnight, next_midnight, freq=resolution, inclusive="left") + + +def at_local_hour(day: date, hour: float) -> pd.Timestamp: + """ + Turn a local wall-clock hour of a day into a timestamp. + + :param day: A local calendar date. + :param hour: Hour of that day, as a float, so 8.5 means 08:30. + :returns: The localised timestamp. + """ + wall_clock = pd.Timestamp(day) + pd.Timedelta(hours=hour) + return wall_clock.tz_localize(SITE_TIMEZONE, ambiguous=True, nonexistent="shift_forward") + + +def local_hour(timestamp: pd.Timestamp) -> float: + """ + Return the local wall-clock hour of a localised timestamp, as a float. + + :param timestamp: A timestamp already localised to the site's timezone. + :returns: Hour of the day, so 08:30 becomes 8.5. + """ + return timestamp.hour + timestamp.minute / MINUTES_PER_HOUR + + +def jitter(rng: random.Random, spread: float) -> float: + """ + Return a multiplier just above or below one, to keep a drawn profile from looking drawn. + + :param rng: The profile's seeded random generator. + :param spread: Half-width of the multiplier, as a fraction. + :returns: A factor in [1 - spread, 1 + spread]. + """ + return rng.uniform(1 - spread, 1 + spread) + + +def bell(hour: float, centre: float, spread: float) -> float: + """ + Return a bell curve over the hour of the day, peaking at one. + + :param hour: Local hour to evaluate. + :param centre: Hour the curve peaks at. + :param spread: Width of the curve, in hours. + :returns: A factor in (0, 1]. + """ + return math.exp(-(((hour - centre) / spread) ** 2)) + + +def clamp(value: float, lowest: float, highest: float) -> float: + """ + Keep a drawn value inside a plausible range. + + :param value: The drawn value. + :param lowest: Lower bound. + :param highest: Upper bound. + :returns: The value, moved onto the nearest bound if it fell outside. + """ + return max(lowest, min(highest, value)) + + +def occupancy_fraction(hour: float) -> float: + """ + Return how much of the office's day-time load is switched on at a given local hour. + + Ramps up over the hour before opening and back down over the hour after closing, + because a building's first arrivals and last departures are spread out. + + :param hour: Local hour to evaluate. + :returns: A factor in [0, 1]. + """ + opening_ramp_start = OFFICE_OPENING_HOUR - OFFICE_RAMP_HOURS + closing_ramp_end = OFFICE_CLOSING_HOUR + OFFICE_RAMP_HOURS + if hour <= opening_ramp_start or hour >= closing_ramp_end: + return 0.0 + if hour < OFFICE_OPENING_HOUR: + return (hour - opening_ramp_start) / OFFICE_RAMP_HOURS + if hour <= OFFICE_CLOSING_HOUR: + return 1.0 + return (closing_ramp_end - hour) / OFFICE_RAMP_HOURS + + +def day_ahead_price_day(day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of day-ahead prices, in EUR/kWh. + + A night trough with a morning peak, a solar-driven dip in the middle of the day and the + day's highest hours in the early evening, all lifted or flattened by a level drawn once + per day. That spread is what the whole demo turns on: it is why the scheduler bothers to + move the heat pump's pre-heat and the hub's charging at all. + + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One price per timestamp. + """ + level = rng.uniform(*PRICE_DAY_LEVEL_RANGE) + if day.weekday() >= SATURDAY: + level *= PRICE_WEEKEND_FACTOR + values = [] + for timestamp in timestamps: + hour = local_hour(timestamp) + price = PRICE_NIGHT + price += (PRICE_MORNING_PEAK - PRICE_NIGHT) * bell(hour, PRICE_MORNING_PEAK_HOUR, PRICE_FEATURE_SPREAD_HOURS) + price += (PRICE_EVENING_PEAK - PRICE_NIGHT) * bell(hour, PRICE_EVENING_PEAK_HOUR, PRICE_FEATURE_SPREAD_HOURS) + price -= (PRICE_NIGHT - PRICE_MIDDAY_DIP) * bell(hour, PRICE_MIDDAY_DIP_HOUR, PRICE_FEATURE_SPREAD_HOURS) + values.append(price * level * jitter(rng, PRICE_NOISE)) + return values + + +def office_baseload_day(day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of the office's background load, in kW. + + A night floor of servers and standby losses that the working day is stacked on top of, + with a shallow dip while the desks are empty over lunch. Weekends keep the floor plus + a small allowance for cleaning staff. + + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One power value per timestamp. + """ + day_power = OFFICE_BASELOAD_DAY_POWER if day.weekday() < SATURDAY else OFFICE_BASELOAD_WEEKEND_DAY_POWER + values = [] + for timestamp in timestamps: + hour = local_hour(timestamp) + power = OFFICE_BASELOAD_NIGHT_POWER + (day_power - OFFICE_BASELOAD_NIGHT_POWER) * occupancy_fraction(hour) + power *= 1 - OFFICE_BASELOAD_LUNCH_DIP * bell(hour, OFFICE_BASELOAD_LUNCH_HOUR, OFFICE_BASELOAD_LUNCH_SPREAD_HOURS) + values.append(power * jitter(rng, OFFICE_BASELOAD_NOISE)) + return values + + +def office_pv_day(_day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of rooftop PV production, in kW, positive because the sensor records production as positive. + + A half sine between sunrise and sunset, scaled by that day's cloud cover, which is + drawn once so the week holds a mix of clear and overcast days. + + :param _day: The local calendar day; the shape only depends on the hour, so it is unused. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One power value per timestamp. + """ + nameplate = quantity_in(OFFICE_PV_PEAK_POWER, POWER_UNIT) + clear_sky_peak = nameplate * OFFICE_PV_CLEAR_SKY_PEAK_FRACTION + clearness = rng.uniform(OFFICE_PV_MIN_DAY_CLEARNESS, OFFICE_PV_MAX_DAY_CLEARNESS) + daylight_hours = OFFICE_PV_SUNSET_HOUR - OFFICE_PV_SUNRISE_HOUR + values = [] + for timestamp in timestamps: + hour = local_hour(timestamp) + if not OFFICE_PV_SUNRISE_HOUR <= hour <= OFFICE_PV_SUNSET_HOUR: + values.append(0.0) + continue + elevation = math.sin(math.pi * (hour - OFFICE_PV_SUNRISE_HOUR) / daylight_hours) + # An inverter cannot pass more than the array's rating, however bright the cloud edge. + values.append(min(nameplate, max(0.0, clear_sky_peak * clearness * elevation * jitter(rng, OFFICE_PV_CLOUD_NOISE)))) + return values + + +def office_heat_pump_day(day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of the heat pump's unscheduled electrical demand, in kW. + + Most of the work happens before the building fills up: the buffer is charged overnight + and topped up during the day, when people, lighting and equipment already contribute + heat. This is only a baseline for the charts, since the scheduler decides the real + dispatch, and moving exactly this pre-heat block towards cheap hours is the point of + the demo. + + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One power value per timestamp. + """ + setback = 1.0 if day.weekday() < SATURDAY else OFFICE_HEAT_PUMP_WEEKEND_FACTOR + capacity = quantity_in(OFFICE_HEAT_PUMP_POWER_CAPACITY, POWER_UNIT) + values = [] + for timestamp in timestamps: + hour = local_hour(timestamp) + if OFFICE_HEAT_PUMP_PREHEAT_START_HOUR <= hour < OFFICE_OPENING_HOUR: + power = OFFICE_HEAT_PUMP_PREHEAT_POWER + elif OFFICE_OPENING_HOUR <= hour < OFFICE_CLOSING_HOUR: + power = OFFICE_HEAT_PUMP_OCCUPIED_POWER + else: + power = OFFICE_HEAT_PUMP_IDLE_POWER + values.append(min(capacity, power * setback * jitter(rng, OFFICE_HEAT_PUMP_NOISE))) + return values + + +def commuter_session(spec: EvseSpec, day: date, rng: random.Random) -> ChargingSession: + """ + Draw the office-hours stay of one commuter at one charge point. + + :param spec: Rating and battery limits of the charge point. + :param day: The local calendar day. + :param rng: The profile's seeded random generator. + :returns: The stay, which always lasts at least the minimum session length. + """ + arrival_hour = clamp(rng.gauss(EVSE_ARRIVAL_HOUR, EVSE_ARRIVAL_SPREAD_HOURS), EVSE_EARLIEST_ARRIVAL_HOUR, EVSE_LATEST_ARRIVAL_HOUR) + departure_hour = clamp(rng.gauss(EVSE_DEPARTURE_HOUR, EVSE_DEPARTURE_SPREAD_HOURS), EVSE_EARLIEST_DEPARTURE_HOUR, EVSE_LATEST_DEPARTURE_HOUR) + departure_hour = max(departure_hour, arrival_hour + EVSE_MIN_SESSION_HOURS) + return ChargingSession( + start=at_local_hour(day, arrival_hour), + end=at_local_hour(day, departure_hour), + energy=quantity_in(spec.soc_max, ENERGY_UNIT) * rng.uniform(*EVSE_SESSION_ENERGY_FRACTION), + power=quantity_in(spec.power_capacity, POWER_UNIT) * rng.uniform(*EVSE_ACCEPTED_POWER_FRACTION), + target_soc=quantity_in(spec.soc_max, ENERGY_UNIT) * rng.uniform(*EVSE_DEPARTURE_SOC_FRACTION), + ) + + +def shoulder_session(session: ShoulderSession, spec: EvseSpec, day: date, rng: random.Random) -> ChargingSession: + """ + Draw the short stay of an early bird or a late worker at one charge point. + + :param session: The window and length such a stay may have. + :param spec: Rating and battery limits of the charge point. + :param day: The local calendar day. + :param rng: The profile's seeded random generator. + :returns: The stay. + """ + start_hour = rng.uniform(session.earliest_start_hour, session.latest_start_hour) + duration_hours = rng.uniform(session.min_duration_hours, session.max_duration_hours) + return ChargingSession( + start=at_local_hour(day, start_hour), + end=at_local_hour(day, start_hour + duration_hours), + energy=quantity_in(spec.soc_max, ENERGY_UNIT) * rng.uniform(*session.energy_fraction), + power=quantity_in(spec.power_capacity, POWER_UNIT) * rng.uniform(*EVSE_ACCEPTED_POWER_FRACTION), + target_soc=None, + ) + + +def without_overlaps(sessions: list[ChargingSession]) -> list[ChargingSession]: + """ + Drop any stay that would have two cars in one bay at once. + + A charge point serves one car at a time, so a drawn early or late stay that runs into + the commuter's stay simply did not happen. + + :param sessions: Candidate stays, in chronological order of their start. + :returns: The stays that fit behind one another. + """ + kept: list[ChargingSession] = [] + for session in sessions: + if kept and session.start < kept[-1].end: + continue + kept.append(session) + return kept + + +def plan_charging_sessions(spec: EvseSpec, day: date, rng: random.Random) -> list[ChargingSession]: + """ + Draw one local day's stays at one charge point. + + Every draw is made in the same order whatever its outcome, so the day stays reproducible. + + :param spec: Rating and battery limits of the charge point. + :param day: The local calendar day. + :param rng: The profile's seeded random generator. + :returns: Non-overlapping stays, in chronological order. + """ + is_weekday = day.weekday() < SATURDAY + candidates: list[ChargingSession] = [] + if is_weekday and rng.random() < EVSE_EARLY_SESSION.probability: + candidates.append(shoulder_session(EVSE_EARLY_SESSION, spec, day, rng)) + if rng.random() < (EVSE_WEEKDAY_OCCUPANCY if is_weekday else EVSE_WEEKEND_OCCUPANCY): + candidates.append(commuter_session(spec, day, rng)) + if is_weekday and rng.random() < EVSE_LATE_SESSION.probability: + candidates.append(shoulder_session(EVSE_LATE_SESSION, spec, day, rng)) + return without_overlaps(candidates) + + +def charging_power(sessions: list[ChargingSession], timestamp: pd.Timestamp, step_hours: float) -> float: + """ + Return the power drawn at one time step, given the day's stays. + + A car charges at the power it accepts until it has what the driver asked for, then sits + plugged in at zero, which is what an AC charging session actually looks like. + + :param sessions: The day's non-overlapping stays. + :param timestamp: Local event start of the time step. + :param step_hours: Length of one time step, in hours. + :returns: Power in the charge point's power unit, never above what the car accepts. + """ + for session in sessions: + if not session.start <= timestamp < session.end: + continue + delivered = session.power * (timestamp - session.start).total_seconds() / SECONDS_PER_HOUR + remaining = session.energy - delivered + if remaining <= 0: + return 0.0 + # The final step delivers only what is left, so the session's energy comes out exact. + return min(session.power, remaining / step_hours) + return 0.0 + + +def charge_point_day(spec: EvseSpec, day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of charging power at one charge point, in kW. + + The two bidirectional charge points are kept charge-only here: discharging is a + scheduling decision, not typical usage, so inventing it would misrepresent the site. + + No stay drawn here can outlast its day, so a day can be generated on its own. + + :param spec: Rating and battery limits of the charge point. + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One power value per timestamp. + """ + sessions = plan_charging_sessions(spec, day, rng) + step_hours = POWER_RESOLUTION.total_seconds() / SECONDS_PER_HOUR + return [charging_power(sessions, timestamp, step_hours) for timestamp in timestamps] + + +def charge_point_availability_day(spec: EvseSpec, day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of the power a charge point may move, in kW. + + Zero while the bay is empty and the connected car's accepted power while it is not, so + the flex-model's `power-capacity` stops the scheduler charging a bay with no car in it. + It caps both directions, which is what also keeps the two V2G charge points from + discharging a car that has already driven away. + + The day's stays are drawn exactly as charge_point_day draws them — same seeded + generator, same call, same order — so a charge point's availability, its departure + requirement and its reference profile always describe the same cars. + + :param spec: Rating and battery limits of the charge point. + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One power ceiling per timestamp. + """ + sessions = plan_charging_sessions(spec, day, rng) + return [next((session.power for session in sessions if session.start <= timestamp < session.end), 0.0) for timestamp in timestamps] + + +def charge_point_soc_minima_day(spec: EvseSpec, day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of the state of charge a charge point's car has to have reached, in kWh. + + Zero for all but one quarter hour of each commuter stay: the last quarter hour that ends + before the car leaves carries the state of charge the driver expects to find. Confining + the requirement to that single interval is deliberate — it is what leaves the scheduler + free to deliver the energy in whichever quarter hours of the stay are cheapest, which is + the flexibility the day-ahead price and the OpenADR limit then compete over. A + requirement that applied throughout the stay would instead force charging on arrival. + + Every interval gets an explicit value rather than being left empty, because FlexMeasures + fills gaps in a sensor-referenced flex-model field: a blank interval would inherit the + departure requirement and quietly oblige the car to stay full all day. + + :param spec: Rating and battery limits of the charge point. + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One required state of charge per timestamp. + """ + sessions = plan_charging_sessions(spec, day, rng) + requirements = {} + for session in sessions: + if session.target_soc is None: + continue + # The last event that ends within the stay, so the requirement is met while the car + # is still plugged in rather than a quarter hour after it has left. + within = [timestamp for timestamp in timestamps if timestamp + POWER_RESOLUTION <= session.end] + if within: + requirements[max(within)] = session.target_soc + return [requirements.get(timestamp, 0.0) for timestamp in timestamps] + + +def office_heat_demand_day(day: date, timestamps: pd.DatetimeIndex, rng: random.Random) -> list[float]: + """ + Generate one day of the building's heat demand, in thermal kW. + + This is the drain on the heat pump's thermal buffer, and therefore the reason the + scheduler runs the heat pump at all. It follows the building's day: the heaviest call is + bringing a cooled-down building back up to temperature before the first arrivals, it + falls back once people, lighting and equipment are contributing heat of their own, and + it drops to envelope losses overnight. Weekends are much lower, because nobody is coming + in and the building is allowed to drift. + + Expressed in thermal kW to match the thermal state of charge, so it is the heat pump's + coefficient of performance that decides what this costs electrically. + + :param day: The local calendar day. + :param timestamps: Every local event start of that day. + :param rng: The profile's seeded random generator. + :returns: One heat demand per timestamp. + """ + setback = 1.0 if day.weekday() < SATURDAY else OFFICE_HEAT_PUMP_HEAT_DEMAND_WEEKEND_FACTOR + values = [] + for timestamp in timestamps: + hour = local_hour(timestamp) + if OFFICE_HEAT_PUMP_PREHEAT_START_HOUR <= hour < OFFICE_OPENING_HOUR: + demand = OFFICE_HEAT_PUMP_HEAT_DEMAND_MORNING + elif OFFICE_OPENING_HOUR <= hour < OFFICE_CLOSING_HOUR: + demand = OFFICE_HEAT_PUMP_HEAT_DEMAND_OCCUPIED + else: + demand = OFFICE_HEAT_PUMP_HEAT_DEMAND_NIGHT + values.append(demand * setback * jitter(rng, OFFICE_HEAT_PUMP_HEAT_DEMAND_NOISE)) + return values + + +def generate_profile( + profile_key: str, + day_profile: DayProfile, + window: tuple[pd.Timestamp, pd.Timestamp], + resolution: timedelta = POWER_RESOLUTION, +) -> dict[pd.Timestamp, float]: + """ + Generate a profile over the whole window, one local calendar day at a time. + + Each day is generated whole, from a generator seeded on the profile and that day, and + only then cut back to the window. That is what makes a given day's values independent + of the moment the script runs: a run at noon writes the same afternoon as a run at dawn. + + :param profile_key: Stable name of the profile, e.g. the asset's name; part of the seed. + :param day_profile: Callable generating one local day of values. + :param window: Window start and end, in UTC. + :param resolution: Resolution of the target sensor's events. + :returns: Event start mapped to value, covering the window. + """ + values: dict[pd.Timestamp, float] = {} + for day in local_days(window): + timestamps = local_event_starts(day, resolution) + rng = random.Random(f"{profile_key}|{day.isoformat()}") + generated = zip(timestamps, day_profile(day, timestamps, rng), strict=True) + values.update({timestamp: value for timestamp, value in generated if window[0] <= timestamp < window[1]}) + return values + + +def store_profile(sensor: Sensor, source: DataSource, values: dict[pd.Timestamp, float], belief_time: pd.Timestamp) -> int: + """ + Write one generated profile to a sensor. + + Merging rather than adding is what makes a re-run an upsert: a belief's key is its + event start, belief time, source and cumulative probability, all of which this run + reproduces exactly. + + :param sensor: Sensor to write to. + :param source: Data source the beliefs are attributed to. + :param values: Event start mapped to value, in the sensor's own unit. + :param belief_time: When the profile is claimed to have been drawn up. + :returns: Number of beliefs written. + """ + for event_start, value in values.items(): + db.session.merge( + TimedBelief( + sensor=sensor, + source=source, + event_value=round(value, BELIEF_PRECISION), + event_start=event_start.to_pydatetime(), + belief_time=belief_time.to_pydatetime(), + ) + ) + db.session.flush() + return len(values) + + +def store_state_of_charge( + sensor: Sensor, + source: DataSource, + capacity: str, + fraction_range: tuple[float, float], + moment: pd.Timestamp, +) -> float: + """ + Record where a storage device stands right now. + + The storage scheduler resolves `soc-at-start` by looking for a belief within four time + steps of the schedule start, so without this the very first schedule stops with + "No recent state-of-charge value found". State of charge is instantaneous, so belief + time equals event start, which makes this a measurement rather than a forecast. + + Unlike the power profiles, this reading is keyed on the current quarter hour: a re-run + within the same quarter hour rewrites it, a later run records a newer one alongside it. + + :param sensor: The device's state-of-charge sensor. + :param source: Data source the reading is attributed to. + :param capacity: The device's usable capacity, as declared in hierarchy.py. + :param fraction_range: Plausible band for the reading, as a fraction of that capacity. + :param moment: The quarter hour to record the reading at. + :returns: The recorded value, in the sensor's own unit. + """ + rng = random.Random(f"{sensor.generic_asset.name}|{SOC_SENSOR_NAME}|{moment.isoformat()}") + value = round(quantity_in(capacity, sensor.unit) * rng.uniform(*fraction_range), BELIEF_PRECISION) + db.session.merge( + TimedBelief( + sensor=sensor, + source=source, + event_value=value, + event_start=moment.to_pydatetime(), + belief_time=moment.to_pydatetime(), + ) + ) + db.session.flush() + return value + + +def seed_day_ahead_prices(forecaster: DataSource, report: ForecastReport) -> None: + """ + Seed the price sensor the campus' flex-context schedules against. + + Without prices over the planning window the scheduler stops with "Prices unknown for + planning window" before it looks at anything else, so this is what makes the rest of the + seeded week schedulable. Prices are the one profile written to a public asset rather than + to the demo account: the sensor lives on the `NL transmission zone` that FlexMeasures + ships. The sensor itself is fetched through seed_assets.py, so both scripts cannot end up + disagreeing about its unit, resolution or knowledge horizon. + + :param forecaster: Data source for genuine forecasts. + :param report: Run report to record the outcome in. + """ + sensor = get_or_create_day_ahead_price_sensor() + values = generate_profile(PRICE_SENSOR_NAME, day_ahead_price_day, (report.window_start, report.window_end), PRICE_RESOLUTION) + count = store_profile(sensor, forecaster, values, report.belief_time) + report.record_profile(sensor, count, is_forecast=True) + + +def seed_office(account: Account, forecaster: DataSource, profiles: DataSource, report: ForecastReport) -> None: + """ + Seed the office's three devices. + + The background load and the PV are the site's `inflexible-consumption` and + `inflexible-production`, so what they get is a forecast the scheduler plans around. + The heat pump is dispatched by the scheduler, so what it gets is a reference profile + plus a current state of charge — and a heat demand forecast, which is not a reference + profile but an input: it is the drain on the thermal buffer that obliges the scheduler + to run the heat pump in the first place. + + :param account: Account that owns the demo hierarchy. + :param forecaster: Data source for genuine forecasts. + :param profiles: Data source for synthetic reference profiles and readings. + :param report: Run report to record the outcome in. + """ + window = (report.window_start, report.window_end) + + for asset_name, day_profile in ((OFFICE_BASELOAD_NAME, office_baseload_day), (OFFICE_PV_NAME, office_pv_day)): + sensor = find_sensor(find_asset(account, asset_name), POWER_SENSOR_NAME) + count = store_profile(sensor, forecaster, generate_profile(asset_name, day_profile, window), report.belief_time) + report.record_profile(sensor, count, is_forecast=True) + + heat_pump = find_asset(account, OFFICE_HEAT_PUMP_NAME) + power = find_sensor(heat_pump, POWER_SENSOR_NAME) + count = store_profile(power, profiles, generate_profile(OFFICE_HEAT_PUMP_NAME, office_heat_pump_day, window), report.belief_time) + report.record_profile(power, count, is_forecast=False) + + heat_demand = find_sensor(heat_pump, HEAT_DEMAND_SENSOR_NAME) + values = generate_profile(HEAT_DEMAND_SENSOR_NAME, office_heat_demand_day, window) + count = store_profile(heat_demand, forecaster, values, report.belief_time) + report.record_profile(heat_demand, count, is_forecast=True) + + state_of_charge = find_sensor(heat_pump, SOC_SENSOR_NAME) + value = store_state_of_charge(state_of_charge, profiles, OFFICE_HEAT_PUMP_SOC_MAX, OFFICE_HEAT_PUMP_INITIAL_SOC_RANGE, report.window_start) + report.record_state_of_charge(state_of_charge, value) + + +def seed_charge_points(account: Account, forecaster: DataSource, profiles: DataSource, report: ForecastReport) -> None: + """ + Seed every charge point of the hub with a typical week and a connected car's state of charge. + + Three profiles per charge point, all drawn from the same stays so they cannot disagree + about which cars turned up, but meaning three different things: + + - the *reference* power profile, what the bay would draw unscheduled, which the + scheduler overwrites; + - the *availability* forecast, the power the bay may move, which is zero whenever no car + is plugged in; + - the *departure requirement* forecast, the state of charge each commuter's car has to + have reached by the time they leave. + + The last two are forecasts of driver behaviour that the scheduler plans around, not + reference profiles it replaces, so they are attributed to the forecaster. + + :param account: Account that owns the demo hierarchy. + :param forecaster: Data source for genuine forecasts. + :param profiles: Data source for synthetic reference profiles and readings. + :param report: Run report to record the outcome in. + """ + window = (report.window_start, report.window_end) + + for spec in EVSE_SPECS: + charge_point = find_asset(account, spec.name) + + power = find_sensor(charge_point, POWER_SENSOR_NAME) + day_profile = partial(charge_point_day, spec) + count = store_profile(power, profiles, generate_profile(spec.name, day_profile, window), report.belief_time) + report.record_profile(power, count, is_forecast=False) + + # Keyed on the charge point's own name, like the reference profile above, so all + # three profiles are generated from the same seeded draw of the day's stays. + for sensor_name, generator in ( + (EVSE_AVAILABILITY_SENSOR_NAME, charge_point_availability_day), + (SOC_MINIMA_SENSOR_NAME, charge_point_soc_minima_day), + ): + sensor = find_sensor(charge_point, sensor_name) + values = generate_profile(spec.name, partial(generator, spec), window) + count = store_profile(sensor, forecaster, values, report.belief_time) + report.record_profile(sensor, count, is_forecast=True) + + state_of_charge = find_sensor(charge_point, SOC_SENSOR_NAME) + value = store_state_of_charge(state_of_charge, profiles, spec.soc_max, EVSE_INITIAL_SOC_RANGE, report.window_start) + report.record_state_of_charge(state_of_charge, value) + + +def print_summary(report: ForecastReport, campus: GenericAsset) -> None: + """ + Print what the run wrote, grouped by what the beliefs mean. + + :param report: Run report gathered while seeding. + :param campus: Top-level site asset, to link to in the closing hint. + """ + print(f"Seeded {report.window_start.isoformat()} through {report.window_end.isoformat()}, believed as of {report.belief_time.isoformat()}.") + print(f"Forecasts ({FORECAST_SOURCE_NAME}), which the scheduler plans around:") + for line in report.forecasts: + print(f" + {line}") + print(f"Reference profiles ({REFERENCE_SOURCE_NAME}), which the scheduler will override:") + for line in report.reference_profiles: + print(f" ~ {line}") + print(f"Current state of charge ({REFERENCE_SOURCE_NAME}), which the scheduler starts from:") + for line in report.states_of_charge: + print(f" = {line}") + print(f"Next: open {FLEXMEASURES_URL}/assets/{campus.id} to see the week, then trigger a schedule.") + + +def main() -> None: + """Seed the demo site's synthetic profiles and print a summary of the result.""" + parser = argparse.ArgumentParser(description="Seed synthetic forecasts and usage profiles for the LF Energy demo site.") + parser.add_argument( + "--account-name", + default=ACCOUNT_NAME, + help=f"Account owning the demo assets (default: {ACCOUNT_NAME})", + ) + parser.add_argument( + "--days", + type=int, + default=FORECAST_HORIZON.days, + help=f"How many days ahead to seed (default: {FORECAST_HORIZON.days})", + ) + args = parser.parse_args() + + app = create_flexmeasures_app() + + with app.app_context(): + account = find_account(args.account_name) + campus = find_asset(account, CAMPUS_NAME) + + window = scheduling_window(timedelta(days=args.days)) + report = ForecastReport(window_start=window[0], window_end=window[1], belief_time=forecast_issue_time(window[0])) + + forecaster = get_or_create_source(source=FORECAST_SOURCE_NAME, source_type=FORECAST_SOURCE_TYPE) + profiles = get_or_create_source(source=REFERENCE_SOURCE_NAME, source_type=REFERENCE_SOURCE_TYPE) + + seed_day_ahead_prices(forecaster, report) + seed_office(account, forecaster, profiles, report) + seed_charge_points(account, forecaster, profiles, report) + + db.session.commit() + print_summary(report, campus) + + +if __name__ == "__main__": + try: + main() + except ValueError as exc: + print(f"Failed to seed the demo site's profiles: {exc}", file=sys.stderr) + raise SystemExit(1) from exc diff --git a/examples/lf-energy-demo/fm-walkthrough-instance/flexmeasures.cfg b/examples/lf-energy-demo/fm-walkthrough-instance/flexmeasures.cfg new file mode 100644 index 0000000..1b8d9ea --- /dev/null +++ b/examples/lf-energy-demo/fm-walkthrough-instance/flexmeasures.cfg @@ -0,0 +1 @@ +FLEXMEASURES_SECRETS_ENCRYPTION_KEYS = {"1": "not-a-real-secrets-key"} diff --git a/examples/lf-energy-demo/keycloak/realm.json b/examples/lf-energy-demo/keycloak/realm.json new file mode 100644 index 0000000..3e33860 --- /dev/null +++ b/examples/lf-energy-demo/keycloak/realm.json @@ -0,0 +1,239 @@ +{ + "realm": "integration-test-realm", + "enabled": true, + "clientScopes": [ + { + "name": "test/BL", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "test/VEN", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "read_all", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "read_targets", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "read_ven_objects", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "write_programs", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "write_events", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "write_reports", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "write_subscriptions", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "write_vens", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + }, + { + "name": "write_users", + "protocol": "openid-connect", + "attributes": { + "include.in.token.scope": "true", + "display.on.consent.screen": "false" + } + } + ], + "clients": [ + { + "clientId": "test-client-id", + "enabled": true, + "clientAuthenticatorType": "client-secret", + "secret": "my-client-secret", + "protocol": "openid-connect", + "publicClient": false, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": true, + "defaultClientScopes": [ + "profile", + "email", + "test/BL", + "test/VEN", + "read_all", + "read_targets", + "read_ven_objects", + "write_programs", + "write_events", + "write_reports", + "write_subscriptions", + "write_vens", + "write_users" + ], + "protocolMappers": [ + { + "name": "audience", + "protocol": "openid-connect", + "protocolMapper": "oidc-audience-mapper", + "consentRequired": false, + "config": { + "included.client.audience": "https://integration.test.elaad.nl", + "id.token.claim": "true", + "access.token.claim": "true" + } + }, + { + "name": "nbf mapper", + "protocol": "openid-connect", + "protocolMapper": "oidc-hardcoded-claim-mapper", + "config": { + "claim.value": "1745926210", + "claim.name": "nbf", + "jsonType.label": "long", + "userinfo.token.claim": "true", + "id.token.claim": "true", + "access.token.claim": "true" + } + } + ] + }, + { + "clientId": "test-ven-1", + "enabled": true, + "clientAuthenticatorType": "client-secret", + "secret": "my-client-secret", + "protocol": "openid-connect", + "publicClient": false, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": true, + "defaultClientScopes": [ + "profile", + "email", + "test/VEN", + "read_targets", + "read_ven_objects", + "write_reports", + "write_subscriptions", + "write_vens" + ], + "protocolMappers": [ + { + "name": "audience", + "protocol": "openid-connect", + "protocolMapper": "oidc-audience-mapper", + "consentRequired": false, + "config": { + "included.client.audience": "https://integration.test.elaad.nl", + "id.token.claim": "true", + "access.token.claim": "true" + } + }, + { + "name": "nbf mapper", + "protocol": "openid-connect", + "protocolMapper": "oidc-hardcoded-claim-mapper", + "config": { + "claim.value": "1745926210", + "claim.name": "nbf", + "jsonType.label": "long", + "userinfo.token.claim": "true", + "id.token.claim": "true", + "access.token.claim": "true" + } + } + ] + }, + { + "clientId": "test-ven-2", + "enabled": true, + "clientAuthenticatorType": "client-secret", + "secret": "my-client-secret", + "protocol": "openid-connect", + "publicClient": false, + "directAccessGrantsEnabled": true, + "serviceAccountsEnabled": true, + "defaultClientScopes": [ + "profile", + "email", + "test/VEN", + "read_targets", + "read_ven_objects", + "write_reports", + "write_subscriptions", + "write_vens" + ], + "protocolMappers": [ + { + "name": "audience", + "protocol": "openid-connect", + "protocolMapper": "oidc-audience-mapper", + "consentRequired": false, + "config": { + "included.client.audience": "https://integration.test.elaad.nl", + "id.token.claim": "true", + "access.token.claim": "true" + } + }, + { + "name": "nbf mapper", + "protocol": "openid-connect", + "protocolMapper": "oidc-hardcoded-claim-mapper", + "config": { + "claim.value": "1745926210", + "claim.name": "nbf", + "jsonType.label": "long", + "userinfo.token.claim": "true", + "id.token.claim": "true", + "access.token.claim": "true" + } + } + ] + } + ] +} diff --git a/examples/lf-energy-demo/python/README.md b/examples/lf-energy-demo/python/README.md new file mode 100644 index 0000000..865b1b6 --- /dev/null +++ b/examples/lf-energy-demo/python/README.md @@ -0,0 +1,44 @@ +# Python helpers + +Scripts for the [capacity-limits walkthrough](../README.md). `seed_events.py` and +`trigger_fetch.py` are run via [uv](https://docs.astral.sh/uv/) — no manual venv or +`pip install` needed. + +| Script | Run where | Run with | Purpose | +| --- | --- | --- | --- | +| `seed_events.py` | Host machine | `uv run seed_events.py` | Create a 24h capacity-limit event on the VTN (BL OAuth client) | +| `compare_schedules.py` | Host machine | `uv run compare_schedules.py` | Trigger a `demo-campus` schedule and compare it with the other side of the OpenADR wiring | +| `trigger_fetch.py` | FlexMeasures container | `uv run --active --no-project trigger_fetch.py` | Run the fetch job immediately for `demo-ven` / `demo-poll` | +| `settings.py` | — | — | Shared URLs and form values | +| `flexmeasures_client.py` | — | — | REST client for the walkthrough's FlexMeasures instance | +| `schedule_comparison.py` | — | — | The saved-run file format and the before/after arithmetic | +| `comparison_diagram.py` | — | — | Renders a comparison as the self-contained `comparison.html` diagram | +| `event_diagram.py` | — | — | Renders a seeded event as the self-contained `event.html` diagram | + +`seed_events.py` and `compare_schedules.py` declare their dependencies inline (PEP 723) +and uv resolves them into an isolated, cached environment on first run. +`trigger_fetch.py` instead reuses the FlexMeasures container's own uv-managed venv +(`/app/.venv`), since it needs the `flexmeasures` and `flexmeasures_openadr3` packages +already installed there. + +`seed_events.py` prints the created event's JSON (as sent to the VTN) and writes +`../schedule-runs/event.html` — a self-contained page with one inline SVG panel per +payload type the event carries (`IMPORT_CAPACITY_LIMIT` and `EXPORT_CAPACITY_LIMIT`), +drawn by `event_diagram.py`. + +Run `compare_schedules.py` twice: once right after seeding forecasts and before any +OpenADR wiring exists (walkthrough Step 4), and once again after the OpenADR capacity +limits are wired into the `demo-campus` flex-context (Step 10). It works out which run is +which by reading that flex-context, saves each plan to `../schedule-runs/`, and prints a +before-and-after report plus `../schedule-runs/comparison.html` once both exist — a +self-contained page you open in a browser, with an inline SVG diagram drawn by +`comparison_diagram.py`; the page opens showing only the "Before OpenADR" plan, with +checkboxes above the diagram to bring in "With OpenADR" (or hide either one). It reuses the +first run's time window for the second +automatically, so the two plans stay comparable +interval by interval even if the manual UI steps in between (Steps 5-9) took a while; +pass `--fresh-window` to start over instead. If the seeded forecast/price data has aged out +(more than seven days since `seed_forecasts.py` last ran), it refuses to trigger and tells +you to re-seed. + +See the [main walkthrough README](../README.md) for step-by-step instructions. diff --git a/examples/lf-energy-demo/python/__init__.py b/examples/lf-energy-demo/python/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/examples/lf-energy-demo/python/compare_schedules.py b/examples/lf-energy-demo/python/compare_schedules.py new file mode 100644 index 0000000..b9f8781 --- /dev/null +++ b/examples/lf-energy-demo/python/compare_schedules.py @@ -0,0 +1,419 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [ +# "httpx>=0.27", +# ] +# /// +""" +Trigger a schedule for the demo campus and show what OpenADR changed about it. + +Run this twice in the walkthrough: + + uv run compare_schedules.py # before wiring the OpenADR capacity limits in + uv run compare_schedules.py # again, after wiring them in + +Each run triggers one schedule for `demo-campus`, waits for it, and saves the resulting +plan to `../schedule-runs/