From fe5cf7eca41b15b7dd2a0565422a91c85fb18f5f Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 18:11:16 +0000 Subject: [PATCH 01/16] Initial plan From 593e9eeaa178f6b90f1f54ceba18b521da1fd34a Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 18:37:55 +0000 Subject: [PATCH 02/16] agents/ui-specialist: introduce new agent from PR #1985 lessons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Context: - PR #1985 added the 'Create Forecast' button to the sensor page UI - No agent existed to encode Flask/Jinja2/JS UI patterns for FlexMeasures - Patterns were worked out from scratch during the session Change: - Created .github/agents/ui-specialist.md covering: - Side-panel pattern (sidepanel-container / left-sidepanel-label hierarchy) - Permission-gating idiom: always short-circuit DB calls behind permission check - JS fetch → poll loop → Toast → reload pattern with correct status-code handling - Toast/spinner setup and teardown discipline - Disabled-button vs. hidden-panel distinction - Marshmallow data_key awareness for JS POST payloads - UI test checklist (access, permissions, data availability, boundary, not-called guard) - Known architectural debt (absent CSRF, session-expiry in poll loop, type hint gap) --- .github/agents/ui-specialist.md | 207 ++++++++++++++++++ documentation/changelog.rst | 1 + documentation/features/forecasting.rst | 5 + documentation/views/sensors.rst | 9 + flexmeasures/ui/templates/sensors/index.html | 112 ++++++++++ flexmeasures/ui/tests/test_sensor_views.py | 219 +++++++++++++++++++ flexmeasures/ui/views/sensors.py | 11 + 7 files changed, 564 insertions(+) create mode 100644 .github/agents/ui-specialist.md create mode 100644 flexmeasures/ui/tests/test_sensor_views.py diff --git a/.github/agents/ui-specialist.md b/.github/agents/ui-specialist.md new file mode 100644 index 0000000000..5a020c66df --- /dev/null +++ b/.github/agents/ui-specialist.md @@ -0,0 +1,207 @@ +--- +name: ui-specialist +description: Guards UI consistency, permission patterns, JavaScript interaction patterns, and template quality in the FlexMeasures web interface +--- + +# Agent: UI Specialist + +## Role + +Owns the quality, consistency, and correctness of all FlexMeasures UI work: Flask/Jinja2 templates, Python view logic, JavaScript interaction patterns (fetch → poll → Toast → reload), CSS, and UI-focused tests. Ensures new UI features follow established side-panel patterns, permission-gate correctly, and do not introduce security regressions. Accumulated from the "Create Forecast" button PR (#1985) session. + +## Scope + +### What this agent MUST review + +- Python view files under `flexmeasures/ui/views/` +- Jinja2 templates under `flexmeasures/ui/templates/` +- JavaScript embedded in templates and under `flexmeasures/ui/static/js/` +- CSS changes under `flexmeasures/ui/static/css/` +- UI-focused tests under `flexmeasures/ui/tests/` +- Permission/data-availability guards in view code +- API call patterns from the browser (fetch, poll loops, Toast messages) + +### What this agent MUST ignore or defer to other agents + +- Core API endpoint logic (defer to API & Backward Compatibility Specialist) +- Domain model changes (defer to Architecture & Domain Specialist) +- CI/CD pipeline changes (defer to Tooling & CI Specialist) +- Documentation prose quality (defer to Documentation & DX Specialist) +- Time/timezone arithmetic (defer to Data & Time Semantics Specialist) + +## Review Checklist + +### Side Panel Pattern + +When a new side panel is added to a sensor or asset page: + +- [ ] Panel is wrapped in `
` → `
` → `
` +- [ ] Panel label text is concise (matches style of "Select dates", "Edit sensor", "Upload data") +- [ ] Panel content uses `
` with `

` heading and `` sub-label +- [ ] Action button uses classes: `btn btn-sm btn-responsive btn-success create-button` (or `btn-danger` for destructive) +- [ ] Panel is gated behind the correct Jinja2 `{% if %}` guard +- [ ] Outer permission check is placed before inner data-availability check (no unnecessary DB queries) + +### Permission Gating in Views (Python) + +- [ ] `user_can_create_children(sensor)` is used for creative actions (forecasts, uploads) +- [ ] `user_can_update(sensor)` is used for edit panels +- [ ] `user_can_delete(sensor)` is used for delete buttons +- [ ] `get_timerange` (or any other DB call) is called **only after** the permission check passes — never unconditionally +- [ ] Template variables are named consistently: `user_can__sensor`, `sensor_has__for_` +- [ ] `Sensor` objects are valid to pass to `user_can_*` helpers because `Sensor` inherits `AuthModelMixin` (same as `GenericAsset`); the `GenericAsset` type hint is advisory only + +### Fetch → Poll → Toast → Reload Pattern + +When a button triggers a background job and polls for completion: + +- [ ] Button is disabled immediately on click (`button.disabled = true`) +- [ ] Spinner is shown immediately (`spinner.classList.remove('d-none')`) +- [ ] Trigger step: POST to API endpoint, check `response.ok`, extract job ID from `data.` matching API docs +- [ ] Poll step: loop up to `maxAttempts` with `await new Promise(resolve => setTimeout(resolve, interval))` delay +- [ ] HTTP 200 from poll → job done → `showToast(..., "success")` → `window.location.reload()` +- [ ] HTTP 202 from poll → job still running → `showToast(statusData.status, "info")` +- [ ] Other HTTP status from poll or trigger → `showToast(..., "error")` and `break` +- [ ] `finally` block always restores button + hides spinner (even on error/timeout) +- [ ] Poll timeout is explicitly communicated to users via Toast message ("Forecast job timed out or failed.") +- [ ] JS block is wrapped in a Jinja2 `{% if and %}` guard to avoid registering click listeners for elements that don't exist in the DOM + +### Toast Usage + +- [ ] `showToast(message, type)` — the global function accepts `(message, type, options)` with optional third argument; do not invent a different signature +- [ ] `type` values: `"info"`, `"success"`, `"error"` +- [ ] Error messages include the API error field (e.g., `errorData.message || response.statusText`) to help users debug +- [ ] Info toasts used for progress, not success (reserve `"success"` for completion) + +### Spinner Pattern + +- [ ] Spinner element uses `id="spinner-"` and class `d-none spinner spinner-bottom-right` +- [ ] Spinner shown: `spinner.classList.remove('d-none')` +- [ ] Spinner hidden: `spinner.classList.add('d-none')` (always in `finally` or error paths) +- [ ] Spinner uses the existing Font Awesome markup: `` + +### Disabled Button Pattern + +When a feature is unavailable due to insufficient data (not insufficient permission): + +- [ ] The panel is still shown (not hidden entirely) so users understand the feature exists +- [ ] An explanatory `

...

` is shown above the disabled button +- [ ] The button has `disabled` attribute; no JS event listener is registered for it +- [ ] The enabled variant (with `id` and JS listener) and disabled variant are in separate `{% if %}` branches + +### API Field Key Awareness + +- [ ] Verify the `data_key` attribute of each Marshmallow field used in a POST body — if a field has `data_key="some-key"` the JSON must use `"some-key"`, not `"some_key"` (snake_case) +- [ ] Fields **without** a `data_key` use the Python attribute name (e.g., `duration` → `"duration"`) +- [ ] Cross-check the API spec example in the endpoint docstring against what the JS sends + +### UI Test Checklist + +- [ ] Test for basic 200 response on valid sensor ID +- [ ] Test for 404 on nonexistent sensor ID +- [ ] Test for login redirect on unauthenticated request +- [ ] Test: panel **visible** for owning-account user (permission granted) +- [ ] Test: panel **visible** for admin (admin bypasses ACL) +- [ ] Test: panel **hidden** for different-account user (no permission) +- [ ] Test: button disabled + message present when data insufficient (check `b"triggerForecastButton" not in response.data`) +- [ ] Test: button enabled + JS present when data sufficient (patch `get_timerange` to return adequate range) +- [ ] Test: boundary condition — exactly `threshold - 1s` is insufficient +- [ ] Test: verify DB-expensive call (`get_timerange` etc.) is **not called** when user has no permission (use `unittest.mock.patch` + `assert_not_called()`) +- [ ] Tests use `_get_` helper functions for DRY fixture access across multiple tests +- [ ] Tests in separate account fixture use a `scope="function"` fixture with proper `login`/`logout` teardown + +### Jinja2 Template Safety + +- [ ] Sensor/asset IDs embedded in JS use `{{ sensor.id }}` (integer, safe), not `.name` or freeform text +- [ ] User-supplied values displayed in HTML use `{{ value | e }}` or `{{ value | safe }}` (only for pre-sanitised server values like `sensor._ui_unit | safe`) +- [ ] `availableUnitsRawJSON.replace(/'/g, '"')` pattern is used for JSON embedded via template — this is the established workaround for Flask's single-quote JSON serialisation + +## Domain Knowledge + +### FlexMeasures UI Architecture + +- **View layer**: `flexmeasures/ui/views/` — Flask class-based views (`FlaskView` from `flask_classful`) +- **Templates**: `flexmeasures/ui/templates/` — Jinja2, extend `base.html`, use `{% block divs %}` +- **Static assets**: `flexmeasures/ui/static/` — `flexmeasures.js`, `flexmeasures.css`, `ui-utils.js`, `chart-data-utils.js` +- **Global JS functions**: `showToast` defined in `templates/includes/toasts.html` (attached to `window`) +- **Sensor page**: `templates/sensors/index.html` — left sidebar (col-md-2) with multiple collapsible side panels, chart area (col-sm-8), replay column (col-sm-2) + +### Permission Model + +- `user_can_create_children(entity)`: checks `"create-children"` permission; used for forecasts, uploads, child asset creation +- `user_can_update(entity)`: checks `"update"` permission; used for edit panels +- `user_can_delete(entity)`: checks `"delete"` permission; used for delete buttons +- All helpers call `check_access(entity, permission)` from `flexmeasures.auth.policy` +- `Sensor` uses `AuthModelMixin` directly (same mechanism as `GenericAsset`), so passing a `Sensor` to helpers typed as `GenericAsset` is safe at runtime +- ACL rule: every member of the account that **owns** a sensor gets `"create-children"` on it; other-account users do not + +### Side Panel Pattern (established) + +The sensor page left sidebar has three established panels: +1. **Select dates** — date-picker, always visible +2. **Edit sensor** — gated on `user_can_update_sensor` +3. **Upload data** — gated on `user_can_update_sensor` +4. **Create forecast** (new in PR #1985) — gated on `user_can_create_children_sensor` + +Pattern: `sidepanel-container` > `left-sidepanel-label` > `sidepanel left-sidepanel` > `fieldset` > content + +### Forecast Button Data-Availability Guard + +- Source: `flexmeasures.data.services.timerange.get_timerange([sensor.id])` +- Returns `(earliest_event_start, latest_event_end)` or `(now, now)` if no data +- Threshold: `(latest - earliest) >= timedelta(days=2)` +- Placed **after** permission check to avoid unnecessary DB queries for unauthorized users + +### Forecast API Interaction + +Trigger endpoint: `POST /api/v3_0/sensors//forecasts/trigger` +- Minimal payload: `{ "duration": "PT24H" }` (no `data_key` on `duration` field) +- Response: `{ "forecast": "", "status": "PROCESSED", "message": "..." }` +- JS accesses job ID via `data.forecast` + +Poll endpoint: `GET /api/v3_0/sensors//forecasts/` +- HTTP 200 → job finished, show success Toast, reload page +- HTTP 202 → job still running, response body has `{ "status": "QUEUED"|"STARTED"|"DEFERRED" }`, show info Toast +- HTTP 400 → unknown job (race condition or expired queue), show error Toast +- Default poll config: 60 attempts × 3 s = 3-minute timeout + +### `showToast` Signature + +```javascript +window.showToast(message, type, { highlightDuplicates = true, showDuplicateCount = true } = {}) +// type: "info" | "success" | "error" +// Durations: error=10s, success=2s, info=3s +``` + +## Interaction Rules + +- If a change modifies the forecast trigger/poll API contract, escalate to **API & Backward Compatibility Specialist** to verify the JS payload still matches +- If `get_timerange` or other time-arithmetic logic changes, escalate to **Data & Time Semantics Specialist** +- If test fixtures or mock strategy is complex, coordinate with **Test Specialist** +- Escalate to **Coordinator** if a new UI pattern emerges that needs to be standardised across agents + +## Self-Improvement Notes + +### Update This Agent When + +- A new panel type is added to the sensor or asset page (encode its pattern) +- The Toast API changes (e.g., new type added, signature changes) +- A new fetch→poll pattern variation is used +- A CSRF mitigation is added to the UI (currently absent — document if added) +- New permission types are used in view code +- New JS utilities are added to `ui-utils.js` or `flexmeasures.js` + +### Known Gaps / Technical Debt to Watch + +1. **CSRF protection is absent** on all browser-initiated `fetch()` POST/PATCH/DELETE calls in templates. This is an existing architectural gap (not introduced by PR #1985). If Flask-WTF CSRF tokens are added in future, the UI agent checklist should require their inclusion in all state-mutating fetch calls. +2. **Session expiry during poll loop**: A 401 response during polling is treated the same as an error, showing "Forecast job failed" rather than "Session expired — please log in". Consider adding specific handling. +3. **Hardcoded `PT24H`**: The forecast duration is not configurable via the UI. The info tooltip mentions this. If a duration picker is added later, the fetch payload and schema validation docs must be updated. +4. **Type annotation gap**: `user_can_create_children(asset: GenericAsset)` is called with `Sensor` objects. Works at runtime (both use `AuthModelMixin`), but mypy may flag it. Consider widening the type hint to `AuthModelMixin` in a future cleanup PR. + +### Session 2026-02-24 — PR #1985 Lessons + +- **Side panel pattern**: Mirror the "Upload data" panel structure exactly (outer container → label → inner div → fieldset). Consistency is important for CSS hover interactions. +- **Short-circuit the DB call**: Always gate `get_timerange` (or any DB-touching call) behind the permission check. A dedicated test (`test_get_timerange_not_called_without_permission`) should verify this. +- **Boundary test value**: Use `timedelta(days=2) - timedelta(seconds=1)` to test the boundary, not `timedelta(days=1)` — the test should be tight around the actual threshold. +- **JS guarded by Jinja2**: Wrap the event listener registration in `{% if permission_var and data_var %}` to prevent `getElementById` returning null for the disabled-button path. +- **Test fixture for cross-account user**: Create a `scope="function"` fixture that logs in a user from a different account; this makes negative-permission tests readable and reusable. diff --git a/documentation/changelog.rst b/documentation/changelog.rst index c1d4ca4148..d65796fdf8 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -13,6 +13,7 @@ New features ------------- * Improve CSV upload validation by inferring the intended base resolution even when data contains valid gaps, instead of requiring perfectly regular timestamps [see `PR #1918 `_] * New forecasting API endpoints, and all timing parameters in forecasting CLI got sensible defaults for ease of use `[POST] /sensors/(id)/forecasts/trigger `_ and `[GET] /sensors/(id)/forecasts/(uuid) `_ to forecast sensor data [see `PR #1813 `_, `PR #1823 `_, `PR #1917 `_ and `PR #1982 `_] +* Add a "Create forecast" button to the sensor page, which triggers a 24-hour regression forecast via one click. Visible to users with ``create-children`` permission on the sensor and enabled when at least two days of data are present [see `PR #1985 `_] * Support setting a resolution when triggering a schedule via the API or CLI [see `PR #1857 `_] * Support variable peak pricing and changes in commitment baselines [see `PR #1835 `_] * Support storing the aggregate power schedule [see `PR #1736 `_] diff --git a/documentation/features/forecasting.rst b/documentation/features/forecasting.rst index aced89f78b..31caf5f2a1 100644 --- a/documentation/features/forecasting.rst +++ b/documentation/features/forecasting.rst @@ -62,6 +62,11 @@ Note that: ``forecast-frequency`` together with ``max-forecast-horizon`` determine how the forecasting cycles advance through time. ``train-period``, ``from-date`` and ``to-date`` allow precise control over the training and prediction windows in each cycle. +Forecasting via the UI +----------------------- + +The quickest way to create a one-off 24-hour forecast is the **Create forecast** button on the sensor page (see :ref:`view_sensors_forecast_button`). The button is available to users with the ``create-children`` permission on the sensor, provided at least two days of historical data exist. No further configuration is needed — one click queues the job and the page shows progress messages until the forecast is ready. + Forecasting via the API ----------------------- diff --git a/documentation/views/sensors.rst b/documentation/views/sensors.rst index 494bc87d20..f6bb5d759a 100644 --- a/documentation/views/sensors.rst +++ b/documentation/views/sensors.rst @@ -34,3 +34,12 @@ And here are 4 days of (dis)-charging patterns in Seita's V2GLiberty project: Charging (blue) mostly happens in sunshine hours, discharging during high-price hours (morning & evening) So on a technical level, the daily heatmap is essentially a heatmap of the sensor's values, with dates on the y-axis and time of day on the x-axis. For individual devices, it gives an insight into the device's running times. A new button lets users switch between charts. + +.. _view_sensors_forecast_button: + +Creating a forecast +------------------- + +Users with the ``create-children`` permission on a sensor can create a 24-hour forecast directly from the sensor page by clicking the **Create forecast** button in the left side panel. The button is enabled once the sensor has at least two days of historical data. After clicking, a background job is queued and the page shows progress updates via status messages. When the job finishes, the page reloads to display the new forecast alongside the historical data. + +See :ref:`forecasting` for more details on how FlexMeasures generates forecasts. diff --git a/flexmeasures/ui/templates/sensors/index.html b/flexmeasures/ui/templates/sensors/index.html index 49e18fcbb8..1810745a41 100644 --- a/flexmeasures/ui/templates/sensors/index.html +++ b/flexmeasures/ui/templates/sensors/index.html @@ -156,6 +156,43 @@

Upload {{ sensor.name }} data

{% endif %} + {% if user_can_create_children_sensor %} +
+
Create forecast
+
+
+

+ Create forecast + +

+ Sensor: {{ sensor.name }} + {% if sensor_has_enough_data_for_forecast %} + + + {% else %} +

+ At least two days of sensor data are needed to create a forecast. +

+ + {% endif %} +
+
+
+ {% endif %} +
{% if user_can_delete_sensor %}
@@ -432,6 +469,81 @@
Statistics
deleteSensor(); } }); + + {% if user_can_create_children_sensor and sensor_has_enough_data_for_forecast %} + async function triggerForecast() { + const button = document.getElementById('triggerForecastButton'); + const spinner = document.getElementById('spinner-forecast'); + + button.disabled = true; + spinner.classList.remove('d-none'); + + showToast("Triggering forecast job...", "info"); + + try { + const response = await fetch(apiBasePath + "/api/v3_0/sensors/{{ sensor.id }}/forecasts/trigger", { + method: "POST", + headers: { + "Content-Type": "application/json", + }, + body: JSON.stringify({ duration: "PT24H" }), + }); + + if (!response.ok) { + const errorData = await response.json(); + showToast("Error triggering forecast: " + (errorData.message || response.statusText), "error"); + button.disabled = false; + spinner.classList.add('d-none'); + return; + } + + const data = await response.json(); + const jobId = data.forecast; + + showToast("Forecast job queued. Checking status...", "info"); + + let finished = false; + const maxAttempts = 60; // poll for up to 3 minutes + for (let i = 0; i < maxAttempts; i++) { + await new Promise(resolve => setTimeout(resolve, 3000)); + + let statusResponse; + try { + statusResponse = await fetch(apiBasePath + "/api/v3_0/sensors/{{ sensor.id }}/forecasts/" + jobId); + } catch (pollError) { + showToast("Error checking forecast status: " + pollError.message, "error"); + break; + } + + if (statusResponse.status === 200) { + showToast("Forecast created successfully! Reloading...", "success"); + finished = true; + await new Promise(resolve => setTimeout(resolve, 1500)); + window.location.reload(); + return; + } else if (statusResponse.status === 202) { + const statusData = await statusResponse.json(); + showToast("Forecast status: " + statusData.status + "...", "info"); + } else { + const errorData = await statusResponse.json(); + showToast("Forecast job failed: " + (errorData.message || "Unknown error"), "error"); + break; + } + } + + if (!finished) { + showToast("Forecast job timed out or failed.", "error"); + } + } catch (error) { + showToast("Error: " + error.message, "error"); + } finally { + button.disabled = false; + spinner.classList.add('d-none'); + } + } + + document.getElementById('triggerForecastButton').addEventListener('click', triggerForecast); + {% endif %} {% endblock %} diff --git a/flexmeasures/ui/tests/test_sensor_views.py b/flexmeasures/ui/tests/test_sensor_views.py new file mode 100644 index 0000000000..4525886dd1 --- /dev/null +++ b/flexmeasures/ui/tests/test_sensor_views.py @@ -0,0 +1,219 @@ +""" +Tests for the Sensor UI view (SensorUI). + +The sensor page at /sensors/ renders sensor details and optionally +a "Create forecast" side panel. These tests verify: + +- Basic access and 404 behaviour +- "Create forecast" panel visibility gated on ``create-children`` permission +- Forecast button enabled/disabled state based on available data range +- Guard that ``get_timerange`` is NOT called for users without permission +""" +from datetime import datetime, timedelta, timezone +from unittest.mock import patch + +import pytest +from flask import url_for + +from flexmeasures.data.services.users import find_user_by_email +from flexmeasures.ui.tests.utils import login, logout + + +@pytest.fixture(scope="function") +def as_supplier_user(client): + """ + Login the default test supplier user (from the *Supplier* account) and log + them out afterwards. The Supplier account does not own the Prosumer + account's sensors, so this user lacks ``create-children`` permission on + those sensors. + """ + login(client, "test_supplier_user_4@seita.nl", "testtest") + yield + logout(client) + + +def _get_prosumer_sensor(db): + """Return the first sensor that belongs to the first Prosumer asset.""" + user = find_user_by_email("test_prosumer_user@seita.nl") + sensor = user.account.generic_assets[0].sensors[0] + db.session.expunge(user) + return sensor + + +# --------------------------------------------------------------------------- +# Basic page access +# --------------------------------------------------------------------------- + + +def test_sensor_page_loads(db, client, setup_assets, as_prosumer_user1): + """Sensor page returns HTTP 200 for a logged-in owner-account user.""" + sensor = _get_prosumer_sensor(db) + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + assert response.status_code == 200 + assert sensor.name.encode() in response.data + + +def test_sensor_page_requires_login(client, setup_assets): + """Unauthenticated requests are redirected to the login page.""" + response = client.get(url_for("SensorUI:get", id=1), follow_redirects=True) + assert b"Please log in" in response.data + + +def test_sensor_page_404_for_nonexistent_sensor(db, client, as_prosumer_user1): + """Requesting a non-existent sensor ID returns a 404.""" + response = client.get( + url_for("SensorUI:get", id=999999), follow_redirects=True + ) + assert response.status_code == 404 + + +# --------------------------------------------------------------------------- +# "Create forecast" panel – visibility based on permissions +# --------------------------------------------------------------------------- + + +def test_create_forecast_panel_visible_for_account_member( + db, client, setup_assets, as_prosumer_user1 +): + """ + The "Create forecast" panel is rendered for a user who belongs to the + account that owns the sensor (Sensor ACL grants ``create-children`` to + every member of the owning account). + """ + sensor = _get_prosumer_sensor(db) + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + assert response.status_code == 200 + assert b"Create forecast" in response.data + + +def test_create_forecast_panel_visible_for_admin( + db, client, setup_assets, as_admin +): + """Admin users bypass ACL and also see the "Create forecast" panel.""" + sensor = _get_prosumer_sensor(db) + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + assert response.status_code == 200 + assert b"Create forecast" in response.data + + +def test_create_forecast_panel_hidden_for_other_account( + db, client, setup_assets, as_supplier_user +): + """ + A user from a different account (no ``create-children`` permission on the + sensor) does not see the "Create forecast" panel at all. + """ + sensor = _get_prosumer_sensor(db) + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + assert response.status_code == 200 + assert b"Create forecast" not in response.data + + +# --------------------------------------------------------------------------- +# Forecast button state – enabled vs. disabled +# --------------------------------------------------------------------------- + + +def test_forecast_button_disabled_with_insufficient_data( + db, client, setup_assets, as_prosumer_user1 +): + """ + The forecast button is disabled and an explanatory message is shown when + the sensor has fewer than two days of historical data. + + ``setup_assets`` populates each sensor with one day of beliefs + (2015-01-01 at 15-minute resolution), which is below the 2-day threshold. + """ + sensor = _get_prosumer_sensor(db) + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + assert response.status_code == 200 + assert b"Create forecast" in response.data + # The enabled button (identified by its unique id) is absent + assert b"triggerForecastButton" not in response.data + # The explanatory message is shown + assert b"At least two days of sensor data are needed" in response.data + + +def test_forecast_button_enabled_with_sufficient_data( + db, client, setup_assets, as_prosumer_user1 +): + """ + The forecast button is enabled and the JS polling code is injected when + the sensor has at least two days of historical data. + + ``get_timerange`` is patched to return a two-day span. + """ + sensor = _get_prosumer_sensor(db) + t0 = datetime(2015, 1, 1, tzinfo=timezone.utc) + t2 = t0 + timedelta(days=2) + + with patch( + "flexmeasures.ui.views.sensors.get_timerange", + return_value=(t0, t2), + ): + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + + assert response.status_code == 200 + assert b"triggerForecastButton" in response.data + assert b"At least two days of sensor data are needed" not in response.data + + +def test_forecast_boundary_just_under_two_days( + db, client, setup_assets, as_prosumer_user1 +): + """ + A data span of exactly two days minus one second does NOT pass the + ``>= timedelta(days=2)`` threshold. + """ + sensor = _get_prosumer_sensor(db) + t0 = datetime(2015, 1, 1, tzinfo=timezone.utc) + t_short = t0 + timedelta(days=2) - timedelta(seconds=1) + + with patch( + "flexmeasures.ui.views.sensors.get_timerange", + return_value=(t0, t_short), + ): + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + + assert response.status_code == 200 + assert b"triggerForecastButton" not in response.data + assert b"At least two days of sensor data are needed" in response.data + + +# --------------------------------------------------------------------------- +# Guard: get_timerange is NOT called when user has no permission +# --------------------------------------------------------------------------- + + +def test_get_timerange_not_called_without_permission( + db, client, setup_assets, as_supplier_user +): + """ + ``get_timerange`` must not be called when ``user_can_create_children`` + returns ``False`` — the view short-circuits to avoid an unnecessary DB query. + """ + sensor = _get_prosumer_sensor(db) + + with patch( + "flexmeasures.ui.views.sensors.get_timerange" + ) as mock_get_timerange: + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + + assert response.status_code == 200 + mock_get_timerange.assert_not_called() diff --git a/flexmeasures/ui/views/sensors.py b/flexmeasures/ui/views/sensors.py index 4e271bd427..60dbd564e6 100644 --- a/flexmeasures/ui/views/sensors.py +++ b/flexmeasures/ui/views/sensors.py @@ -1,3 +1,5 @@ +from datetime import timedelta + from flask import request from flask_classful import FlaskView from flask_security import login_required @@ -6,6 +8,7 @@ from flexmeasures.data import db from flexmeasures.data.schemas import StartEndTimeSchema +from flexmeasures.data.services.timerange import get_timerange from flexmeasures import Sensor from flexmeasures.ui.utils.view_utils import ( render_flexmeasures_template, @@ -13,6 +16,7 @@ ) from flexmeasures.ui.utils.breadcrumb_utils import get_breadcrumb_info from flexmeasures.ui.views.assets.utils import ( + user_can_create_children, user_can_delete, user_can_update, ) @@ -39,11 +43,18 @@ def get(self, id: int, **kwargs): sensor = db.session.get(Sensor, id) if sensor is None: raise NotFound + can_create_children = user_can_create_children(sensor) + has_enough_data = False + if can_create_children: + earliest, latest = get_timerange([sensor.id]) + has_enough_data = (latest - earliest) >= timedelta(days=2) return render_flexmeasures_template( "sensors/index.html", sensor=sensor, user_can_update_sensor=user_can_update(sensor), user_can_delete_sensor=user_can_delete(sensor), + user_can_create_children_sensor=can_create_children, + sensor_has_enough_data_for_forecast=has_enough_data, available_units=available_units(), msg="", breadcrumb_info=get_breadcrumb_info(sensor), From 2211fc314560a33a9f7611d3938ff34cec5eaaac Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 24 Feb 2026 18:38:03 +0000 Subject: [PATCH 03/16] agents/coordinator: register UI Specialist; encode UI dev patterns Context: - PR #1985 established concrete UI patterns (side panel, JS poll loop, permission-gated view logic) that the agent system had no coverage for - A new UI Specialist agent was created to own those patterns - Coordinator roster and domain knowledge should reflect this Change: - Added UI Specialist to the agent roster (item 9) - Added 'UI Development Patterns' subsection to Domain Knowledge with: - Jinja2 side-panel template snippet - Python view short-circuit data-guard pattern - JS fetch/poll/Toast/reload skeleton - Agent responsibility table for UI-touching PRs --- .github/agents/coordinator.md | 74 +++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/.github/agents/coordinator.md b/.github/agents/coordinator.md index c6c7584d82..7de7a5c53c 100644 --- a/.github/agents/coordinator.md +++ b/.github/agents/coordinator.md @@ -99,6 +99,7 @@ This agent owns the creation, structure, and evolution of all other agents. 6. **Documentation & Developer Experience Specialist** - Project understandability 7. **Tooling & CI Specialist** - Automation reliability and maintainability 8. **Review Lead** - Orchestrates agents in response to a user assignment +9. **UI Specialist** - Flask/Jinja2 templates, side-panel pattern, permission gating in views, JS fetch→poll→Toast→reload pattern, UI tests ### Standard Agent Template @@ -266,6 +267,79 @@ When reviewing PRs that change Marshmallow schemas: **Key Insight**: Tests comparing data sources are integration tests validating consistency across code paths. When they fail, investigate production code for format mismatches before changing tests. +#### UI Development Patterns + +**Context**: FlexMeasures has a growing set of interactive sensor/asset page features. Each new UI feature typically involves a Python view guard, a Jinja2 side panel, and a JS interaction pattern. Consistency across features matters for UX and maintainability. + +**Pattern: Permission-Gated Side Panels (PR #1985)** + +Structure in `sensors/index.html`: +```jinja2 +{% if user_can__sensor %} +
+
Panel label
+
+
+

Panel heading

+ Context: {{ sensor.name }} + {% if sensor_has_enough_data_for_ %} + + {% else %} + + {% endif %} +
+
+
+{% endif %} +``` + +**Pattern: View-Level Data Guard (Short-Circuit)** + +```python +can_create_children = user_can_create_children(sensor) # permission first +has_enough_data = False +if can_create_children: + earliest, latest = get_timerange([sensor.id]) # DB call only if permitted + has_enough_data = (latest - earliest) >= timedelta(days=2) +``` + +**Pattern: JS Fetch → Poll → Toast → Reload** + +```javascript +async function triggerFeature() { + button.disabled = true; + spinner.classList.remove('d-none'); + showToast("Queuing job...", "info"); + try { + const r = await fetch(url, { method: "POST", body: JSON.stringify(payload) }); + if (!r.ok) { showToast("Error: " + ..., "error"); return; } + const jobId = (await r.json()).; + for (let i = 0; i < maxAttempts; i++) { + await delay(3000); + const s = await fetch(pollUrl + jobId); + if (s.status === 200) { showToast("Done!", "success"); window.location.reload(); return; } + if (s.status === 202) { showToast((await s.json()).status, "info"); continue; } + showToast("Failed: " + ..., "error"); break; + } + if (!finished) showToast("Timed out.", "error"); + } catch (e) { + showToast("Error: " + e.message, "error"); + } finally { + button.disabled = false; + spinner.classList.add('d-none'); + } +} +``` + +**Agents responsible for UI patterns**: + +| Agent | Responsibility | +|-------|----------------| +| **UI Specialist** | Side panel, JS interaction, permission gating, Toast usage | +| **Test Specialist** | UI test coverage, mock strategy for `get_timerange` | +| **API Specialist** | Verify JS payload keys match Marshmallow `data_key` attributes | +| **Architecture Specialist** | `AuthModelMixin` usage, view layer integrity | + ## Interaction Rules ### Coordination with Other Agents From be575383b068d4b8f5f7fac4f7d5d43bc3a4de80 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 25 Feb 2026 09:03:43 +0100 Subject: [PATCH 04/16] style: black Signed-off-by: F.N. Claessen --- flexmeasures/ui/tests/test_sensor_views.py | 33 ++++++---------------- 1 file changed, 9 insertions(+), 24 deletions(-) diff --git a/flexmeasures/ui/tests/test_sensor_views.py b/flexmeasures/ui/tests/test_sensor_views.py index 4525886dd1..efaf1fc48d 100644 --- a/flexmeasures/ui/tests/test_sensor_views.py +++ b/flexmeasures/ui/tests/test_sensor_views.py @@ -9,6 +9,7 @@ - Forecast button enabled/disabled state based on available data range - Guard that ``get_timerange`` is NOT called for users without permission """ + from datetime import datetime, timedelta, timezone from unittest.mock import patch @@ -48,9 +49,7 @@ def _get_prosumer_sensor(db): def test_sensor_page_loads(db, client, setup_assets, as_prosumer_user1): """Sensor page returns HTTP 200 for a logged-in owner-account user.""" sensor = _get_prosumer_sensor(db) - response = client.get( - url_for("SensorUI:get", id=sensor.id), follow_redirects=True - ) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) assert response.status_code == 200 assert sensor.name.encode() in response.data @@ -63,9 +62,7 @@ def test_sensor_page_requires_login(client, setup_assets): def test_sensor_page_404_for_nonexistent_sensor(db, client, as_prosumer_user1): """Requesting a non-existent sensor ID returns a 404.""" - response = client.get( - url_for("SensorUI:get", id=999999), follow_redirects=True - ) + response = client.get(url_for("SensorUI:get", id=999999), follow_redirects=True) assert response.status_code == 404 @@ -83,21 +80,15 @@ def test_create_forecast_panel_visible_for_account_member( every member of the owning account). """ sensor = _get_prosumer_sensor(db) - response = client.get( - url_for("SensorUI:get", id=sensor.id), follow_redirects=True - ) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) assert response.status_code == 200 assert b"Create forecast" in response.data -def test_create_forecast_panel_visible_for_admin( - db, client, setup_assets, as_admin -): +def test_create_forecast_panel_visible_for_admin(db, client, setup_assets, as_admin): """Admin users bypass ACL and also see the "Create forecast" panel.""" sensor = _get_prosumer_sensor(db) - response = client.get( - url_for("SensorUI:get", id=sensor.id), follow_redirects=True - ) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) assert response.status_code == 200 assert b"Create forecast" in response.data @@ -110,9 +101,7 @@ def test_create_forecast_panel_hidden_for_other_account( sensor) does not see the "Create forecast" panel at all. """ sensor = _get_prosumer_sensor(db) - response = client.get( - url_for("SensorUI:get", id=sensor.id), follow_redirects=True - ) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) assert response.status_code == 200 assert b"Create forecast" not in response.data @@ -133,9 +122,7 @@ def test_forecast_button_disabled_with_insufficient_data( (2015-01-01 at 15-minute resolution), which is below the 2-day threshold. """ sensor = _get_prosumer_sensor(db) - response = client.get( - url_for("SensorUI:get", id=sensor.id), follow_redirects=True - ) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) assert response.status_code == 200 assert b"Create forecast" in response.data # The enabled button (identified by its unique id) is absent @@ -208,9 +195,7 @@ def test_get_timerange_not_called_without_permission( """ sensor = _get_prosumer_sensor(db) - with patch( - "flexmeasures.ui.views.sensors.get_timerange" - ) as mock_get_timerange: + with patch("flexmeasures.ui.views.sensors.get_timerange") as mock_get_timerange: response = client.get( url_for("SensorUI:get", id=sensor.id), follow_redirects=True ) From 611ab3ceb772faf5af9afff472d822b87237b7c0 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 25 Feb 2026 10:32:31 +0100 Subject: [PATCH 05/16] fix: set result_ttl for wrap_up_job Signed-off-by: F.N. Claessen --- .../data/models/forecasting/pipelines/train_predict.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/flexmeasures/data/models/forecasting/pipelines/train_predict.py b/flexmeasures/data/models/forecasting/pipelines/train_predict.py index 7da3a98ffd..124758599d 100644 --- a/flexmeasures/data/models/forecasting/pipelines/train_predict.py +++ b/flexmeasures/data/models/forecasting/pipelines/train_predict.py @@ -307,6 +307,11 @@ def run( "FLEXMEASURES_JOB_TTL", timedelta(-1) ).total_seconds() ), + result_ttl=int( + current_app.config.get( + "FLEXMEASURES_PLANNING_TTL", timedelta(-1) + ).total_seconds() + ), # NB job.cleanup docs says a negative number of seconds means persisting forever meta=job_metadata, ) current_app.queues[queue].enqueue_job(wrap_up_job) From 4b4f31738b8ba53fcae54b29fafc51accf85f235 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 25 Feb 2026 10:36:08 +0100 Subject: [PATCH 06/16] fix: detached data source in case of forecasting job Signed-off-by: F.N. Claessen --- flexmeasures/data/models/forecasting/utils.py | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/flexmeasures/data/models/forecasting/utils.py b/flexmeasures/data/models/forecasting/utils.py index a7819a0652..e1e4f10acd 100644 --- a/flexmeasures/data/models/forecasting/utils.py +++ b/flexmeasures/data/models/forecasting/utils.py @@ -251,11 +251,12 @@ def data_to_bdf( # source_type="forecaster", # attributes=self.data_source.attributes, # ) + source = refresh_data_source(data_source) # Convert to BeliefsDataFrame bdf = tb.BeliefsDataFrame( forecast_df.reset_index().rename(columns={"forecasts": "event_value"}), - source=data_source, + source=source, sensor=sensor_to_save, ) return bdf @@ -265,3 +266,22 @@ def floor_to_resolution(dt: datetime, resolution: timedelta) -> datetime: delta_seconds = resolution.total_seconds() floored = dt.timestamp() - (dt.timestamp() % delta_seconds) return datetime.fromtimestamp(floored, tz=dt.tzinfo) + + +def refresh_data_source(data_source: DataSource) -> DataSource: + """Refresh the potentially detached data source. + + This avoids a sqlalchemy.exc.IntegrityError / psycopg2.errors.ForeignKeyViolation + for the data source ID not being present in the data_source table. + """ + + from flexmeasures.data.services.data_sources import get_or_create_source + + source = get_or_create_source( + data_source.name, + source_type=data_source.type, + model=data_source.model, + version=data_source.version, + attributes=data_source.attributes, + ) + return source From c4486adff31bfa74b24e863828e1ed9d538037df Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 25 Feb 2026 10:44:14 +0100 Subject: [PATCH 07/16] fix: `HANDLING RQ FORECASTING EXCEPTION: : Job.__init__() missing 1 required argument: 'connection'` Signed-off-by: F.N. Claessen --- .../forecasting/pipelines/train_predict.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/flexmeasures/data/models/forecasting/pipelines/train_predict.py b/flexmeasures/data/models/forecasting/pipelines/train_predict.py index 124758599d..12caa6b0ec 100644 --- a/flexmeasures/data/models/forecasting/pipelines/train_predict.py +++ b/flexmeasures/data/models/forecasting/pipelines/train_predict.py @@ -7,6 +7,7 @@ import logging from datetime import datetime, timedelta +from redis import Redis from rq.job import Job from flask import current_app @@ -44,11 +45,12 @@ def __init__( self.delete_model = delete_model self.return_values = [] # To store forecasts and jobs - def run_wrap_up(self, cycle_job_ids: list[str]): + def run_wrap_up(self, cycle_job_ids: list[str], queue: str): """Log the status of all cycle jobs after completion.""" + connection = current_app.queues[queue].connection for index, job_id in enumerate(cycle_job_ids): logging.info( - f"forecasting job-{index}: {job_id} status: {Job.fetch(job_id).get_status()}" + f"forecasting job-{index}: {job_id} status: {Job.fetch(job_id, connection=connection).get_status()}" ) def run_cycle( @@ -199,6 +201,7 @@ def run( logging.info( f"Starting Train-Predict Pipeline to predict for {self._parameters['predict_period_in_hours']} hours." ) + connection = current_app.queues[queue].connection # How much to move forward to the next cycle one prediction period later cycle_frequency = max( self._config["retrain_frequency"], @@ -271,7 +274,7 @@ def run( self.run_cycle, # Some cycle job params override job kwargs kwargs={**job_kwargs, **cycle_params}, - connection=current_app.queues[queue].connection, + connection=connection, ttl=int( current_app.config.get( "FLEXMEASURES_JOB_TTL", timedelta(-1) @@ -299,8 +302,11 @@ def run( wrap_up_job = Job.create( self.run_wrap_up, - kwargs={"cycle_job_ids": cycle_job_ids}, # cycles jobs IDs to wait for - connection=current_app.queues[queue].connection, + kwargs={ + "cycle_job_ids": cycle_job_ids, # cycles jobs IDs to wait for + "queue": queue, + }, + connection=connection, depends_on=cycle_job_ids, # wrap-up job depends on all cycle jobs ttl=int( current_app.config.get( From 2003bf2b998e4a86d946e5e10b3f46ed228ca721 Mon Sep 17 00:00:00 2001 From: "F.N. Claessen" Date: Wed, 25 Feb 2026 10:54:07 +0100 Subject: [PATCH 08/16] fix: sidepanel subtitles Signed-off-by: F.N. Claessen --- flexmeasures/ui/templates/sensors/index.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flexmeasures/ui/templates/sensors/index.html b/flexmeasures/ui/templates/sensors/index.html index 1810745a41..9ce08f58e5 100644 --- a/flexmeasures/ui/templates/sensors/index.html +++ b/flexmeasures/ui/templates/sensors/index.html @@ -35,7 +35,7 @@

Edit {{ sensor.name }}

- Parent asset: {{ sensor.generic_asset.name }} (ID: {{ sensor.generic_asset.id }} ) + Belongs to: {{ sensor.generic_asset.name }} (ID: {{ sensor.generic_asset_id }})
@@ -62,7 +62,7 @@

Edit {{ sensor.name }}

Upload {{ sensor.name }} data

- Belongs to: {{ sensor.generic_asset.name }} (ID: {{ sensor.generic_asset_id }}) + Sensor: {{ sensor.name }} (ID: {{ sensor.id }})
@@ -169,7 +169,7 @@

style="cursor: pointer;" >

- Sensor: {{ sensor.name }} + Sensor: {{ sensor.name }} (ID: {{ sensor.id }}) {% if sensor_has_enough_data_for_forecast %} {% endif %} - {% if user_can_create_children_sensor %} + {% if user_can_create_children_sensor %}
Forecast
@@ -165,16 +165,25 @@

Create forecast

Sensor: {{ sensor.name }} (ID: {{ sensor.id }}) {% if sensor_has_enough_data_for_forecast %} - +
+ + +
+
+ + More options + + +
- {% else %}

@@ -486,7 +483,7 @@

Statistics
const durationDays = parseInt(document.getElementById('forecastDuration').value) || {{ forecast_default_duration_days }}; const durationIso = 'P' + durationDays + 'D'; - button.disabled = true; + button.classList.add('d-none'); spinner.classList.remove('d-none'); showToast("Triggering forecast job...", "info"); @@ -503,7 +500,7 @@
Statistics
if (!response.ok) { const errorData = await response.json(); showToast("Error triggering forecast: " + (errorData.message || response.statusText), "error"); - button.disabled = false; + button.classList.remove('d-none'); spinner.classList.add('d-none'); return; } @@ -548,7 +545,7 @@
Statistics
} catch (error) { showToast("Error: " + error.message, "error"); } finally { - button.disabled = false; + button.classList.remove('d-none'); spinner.classList.add('d-none'); } }