diff --git a/.github/agents/coordinator.md b/.github/agents/coordinator.md index c6c7584d82..7de7a5c53c 100644 --- a/.github/agents/coordinator.md +++ b/.github/agents/coordinator.md @@ -99,6 +99,7 @@ This agent owns the creation, structure, and evolution of all other agents. 6. **Documentation & Developer Experience Specialist** - Project understandability 7. **Tooling & CI Specialist** - Automation reliability and maintainability 8. **Review Lead** - Orchestrates agents in response to a user assignment +9. **UI Specialist** - Flask/Jinja2 templates, side-panel pattern, permission gating in views, JS fetch→poll→Toast→reload pattern, UI tests ### Standard Agent Template @@ -266,6 +267,79 @@ When reviewing PRs that change Marshmallow schemas: **Key Insight**: Tests comparing data sources are integration tests validating consistency across code paths. When they fail, investigate production code for format mismatches before changing tests. +#### UI Development Patterns + +**Context**: FlexMeasures has a growing set of interactive sensor/asset page features. Each new UI feature typically involves a Python view guard, a Jinja2 side panel, and a JS interaction pattern. Consistency across features matters for UX and maintainability. + +**Pattern: Permission-Gated Side Panels (PR #1985)** + +Structure in `sensors/index.html`: +```jinja2 +{% if user_can__sensor %} +
+
Panel label
+
+
+

Panel heading

+ Context: {{ sensor.name }} + {% if sensor_has_enough_data_for_ %} + + {% else %} + + {% endif %} +
+
+
+{% endif %} +``` + +**Pattern: View-Level Data Guard (Short-Circuit)** + +```python +can_create_children = user_can_create_children(sensor) # permission first +has_enough_data = False +if can_create_children: + earliest, latest = get_timerange([sensor.id]) # DB call only if permitted + has_enough_data = (latest - earliest) >= timedelta(days=2) +``` + +**Pattern: JS Fetch → Poll → Toast → Reload** + +```javascript +async function triggerFeature() { + button.disabled = true; + spinner.classList.remove('d-none'); + showToast("Queuing job...", "info"); + try { + const r = await fetch(url, { method: "POST", body: JSON.stringify(payload) }); + if (!r.ok) { showToast("Error: " + ..., "error"); return; } + const jobId = (await r.json()).; + for (let i = 0; i < maxAttempts; i++) { + await delay(3000); + const s = await fetch(pollUrl + jobId); + if (s.status === 200) { showToast("Done!", "success"); window.location.reload(); return; } + if (s.status === 202) { showToast((await s.json()).status, "info"); continue; } + showToast("Failed: " + ..., "error"); break; + } + if (!finished) showToast("Timed out.", "error"); + } catch (e) { + showToast("Error: " + e.message, "error"); + } finally { + button.disabled = false; + spinner.classList.add('d-none'); + } +} +``` + +**Agents responsible for UI patterns**: + +| Agent | Responsibility | +|-------|----------------| +| **UI Specialist** | Side panel, JS interaction, permission gating, Toast usage | +| **Test Specialist** | UI test coverage, mock strategy for `get_timerange` | +| **API Specialist** | Verify JS payload keys match Marshmallow `data_key` attributes | +| **Architecture Specialist** | `AuthModelMixin` usage, view layer integrity | + ## Interaction Rules ### Coordination with Other Agents diff --git a/.github/agents/ui-specialist.md b/.github/agents/ui-specialist.md new file mode 100644 index 0000000000..5a020c66df --- /dev/null +++ b/.github/agents/ui-specialist.md @@ -0,0 +1,207 @@ +--- +name: ui-specialist +description: Guards UI consistency, permission patterns, JavaScript interaction patterns, and template quality in the FlexMeasures web interface +--- + +# Agent: UI Specialist + +## Role + +Owns the quality, consistency, and correctness of all FlexMeasures UI work: Flask/Jinja2 templates, Python view logic, JavaScript interaction patterns (fetch → poll → Toast → reload), CSS, and UI-focused tests. Ensures new UI features follow established side-panel patterns, permission-gate correctly, and do not introduce security regressions. Accumulated from the "Create Forecast" button PR (#1985) session. + +## Scope + +### What this agent MUST review + +- Python view files under `flexmeasures/ui/views/` +- Jinja2 templates under `flexmeasures/ui/templates/` +- JavaScript embedded in templates and under `flexmeasures/ui/static/js/` +- CSS changes under `flexmeasures/ui/static/css/` +- UI-focused tests under `flexmeasures/ui/tests/` +- Permission/data-availability guards in view code +- API call patterns from the browser (fetch, poll loops, Toast messages) + +### What this agent MUST ignore or defer to other agents + +- Core API endpoint logic (defer to API & Backward Compatibility Specialist) +- Domain model changes (defer to Architecture & Domain Specialist) +- CI/CD pipeline changes (defer to Tooling & CI Specialist) +- Documentation prose quality (defer to Documentation & DX Specialist) +- Time/timezone arithmetic (defer to Data & Time Semantics Specialist) + +## Review Checklist + +### Side Panel Pattern + +When a new side panel is added to a sensor or asset page: + +- [ ] Panel is wrapped in `
` → `
` → `
` +- [ ] Panel label text is concise (matches style of "Select dates", "Edit sensor", "Upload data") +- [ ] Panel content uses `
` with `

` heading and `` sub-label +- [ ] Action button uses classes: `btn btn-sm btn-responsive btn-success create-button` (or `btn-danger` for destructive) +- [ ] Panel is gated behind the correct Jinja2 `{% if %}` guard +- [ ] Outer permission check is placed before inner data-availability check (no unnecessary DB queries) + +### Permission Gating in Views (Python) + +- [ ] `user_can_create_children(sensor)` is used for creative actions (forecasts, uploads) +- [ ] `user_can_update(sensor)` is used for edit panels +- [ ] `user_can_delete(sensor)` is used for delete buttons +- [ ] `get_timerange` (or any other DB call) is called **only after** the permission check passes — never unconditionally +- [ ] Template variables are named consistently: `user_can__sensor`, `sensor_has__for_` +- [ ] `Sensor` objects are valid to pass to `user_can_*` helpers because `Sensor` inherits `AuthModelMixin` (same as `GenericAsset`); the `GenericAsset` type hint is advisory only + +### Fetch → Poll → Toast → Reload Pattern + +When a button triggers a background job and polls for completion: + +- [ ] Button is disabled immediately on click (`button.disabled = true`) +- [ ] Spinner is shown immediately (`spinner.classList.remove('d-none')`) +- [ ] Trigger step: POST to API endpoint, check `response.ok`, extract job ID from `data.` matching API docs +- [ ] Poll step: loop up to `maxAttempts` with `await new Promise(resolve => setTimeout(resolve, interval))` delay +- [ ] HTTP 200 from poll → job done → `showToast(..., "success")` → `window.location.reload()` +- [ ] HTTP 202 from poll → job still running → `showToast(statusData.status, "info")` +- [ ] Other HTTP status from poll or trigger → `showToast(..., "error")` and `break` +- [ ] `finally` block always restores button + hides spinner (even on error/timeout) +- [ ] Poll timeout is explicitly communicated to users via Toast message ("Forecast job timed out or failed.") +- [ ] JS block is wrapped in a Jinja2 `{% if and %}` guard to avoid registering click listeners for elements that don't exist in the DOM + +### Toast Usage + +- [ ] `showToast(message, type)` — the global function accepts `(message, type, options)` with optional third argument; do not invent a different signature +- [ ] `type` values: `"info"`, `"success"`, `"error"` +- [ ] Error messages include the API error field (e.g., `errorData.message || response.statusText`) to help users debug +- [ ] Info toasts used for progress, not success (reserve `"success"` for completion) + +### Spinner Pattern + +- [ ] Spinner element uses `id="spinner-"` and class `d-none spinner spinner-bottom-right` +- [ ] Spinner shown: `spinner.classList.remove('d-none')` +- [ ] Spinner hidden: `spinner.classList.add('d-none')` (always in `finally` or error paths) +- [ ] Spinner uses the existing Font Awesome markup: `` + +### Disabled Button Pattern + +When a feature is unavailable due to insufficient data (not insufficient permission): + +- [ ] The panel is still shown (not hidden entirely) so users understand the feature exists +- [ ] An explanatory `

...

` is shown above the disabled button +- [ ] The button has `disabled` attribute; no JS event listener is registered for it +- [ ] The enabled variant (with `id` and JS listener) and disabled variant are in separate `{% if %}` branches + +### API Field Key Awareness + +- [ ] Verify the `data_key` attribute of each Marshmallow field used in a POST body — if a field has `data_key="some-key"` the JSON must use `"some-key"`, not `"some_key"` (snake_case) +- [ ] Fields **without** a `data_key` use the Python attribute name (e.g., `duration` → `"duration"`) +- [ ] Cross-check the API spec example in the endpoint docstring against what the JS sends + +### UI Test Checklist + +- [ ] Test for basic 200 response on valid sensor ID +- [ ] Test for 404 on nonexistent sensor ID +- [ ] Test for login redirect on unauthenticated request +- [ ] Test: panel **visible** for owning-account user (permission granted) +- [ ] Test: panel **visible** for admin (admin bypasses ACL) +- [ ] Test: panel **hidden** for different-account user (no permission) +- [ ] Test: button disabled + message present when data insufficient (check `b"triggerForecastButton" not in response.data`) +- [ ] Test: button enabled + JS present when data sufficient (patch `get_timerange` to return adequate range) +- [ ] Test: boundary condition — exactly `threshold - 1s` is insufficient +- [ ] Test: verify DB-expensive call (`get_timerange` etc.) is **not called** when user has no permission (use `unittest.mock.patch` + `assert_not_called()`) +- [ ] Tests use `_get_` helper functions for DRY fixture access across multiple tests +- [ ] Tests in separate account fixture use a `scope="function"` fixture with proper `login`/`logout` teardown + +### Jinja2 Template Safety + +- [ ] Sensor/asset IDs embedded in JS use `{{ sensor.id }}` (integer, safe), not `.name` or freeform text +- [ ] User-supplied values displayed in HTML use `{{ value | e }}` or `{{ value | safe }}` (only for pre-sanitised server values like `sensor._ui_unit | safe`) +- [ ] `availableUnitsRawJSON.replace(/'/g, '"')` pattern is used for JSON embedded via template — this is the established workaround for Flask's single-quote JSON serialisation + +## Domain Knowledge + +### FlexMeasures UI Architecture + +- **View layer**: `flexmeasures/ui/views/` — Flask class-based views (`FlaskView` from `flask_classful`) +- **Templates**: `flexmeasures/ui/templates/` — Jinja2, extend `base.html`, use `{% block divs %}` +- **Static assets**: `flexmeasures/ui/static/` — `flexmeasures.js`, `flexmeasures.css`, `ui-utils.js`, `chart-data-utils.js` +- **Global JS functions**: `showToast` defined in `templates/includes/toasts.html` (attached to `window`) +- **Sensor page**: `templates/sensors/index.html` — left sidebar (col-md-2) with multiple collapsible side panels, chart area (col-sm-8), replay column (col-sm-2) + +### Permission Model + +- `user_can_create_children(entity)`: checks `"create-children"` permission; used for forecasts, uploads, child asset creation +- `user_can_update(entity)`: checks `"update"` permission; used for edit panels +- `user_can_delete(entity)`: checks `"delete"` permission; used for delete buttons +- All helpers call `check_access(entity, permission)` from `flexmeasures.auth.policy` +- `Sensor` uses `AuthModelMixin` directly (same mechanism as `GenericAsset`), so passing a `Sensor` to helpers typed as `GenericAsset` is safe at runtime +- ACL rule: every member of the account that **owns** a sensor gets `"create-children"` on it; other-account users do not + +### Side Panel Pattern (established) + +The sensor page left sidebar has three established panels: +1. **Select dates** — date-picker, always visible +2. **Edit sensor** — gated on `user_can_update_sensor` +3. **Upload data** — gated on `user_can_update_sensor` +4. **Create forecast** (new in PR #1985) — gated on `user_can_create_children_sensor` + +Pattern: `sidepanel-container` > `left-sidepanel-label` > `sidepanel left-sidepanel` > `fieldset` > content + +### Forecast Button Data-Availability Guard + +- Source: `flexmeasures.data.services.timerange.get_timerange([sensor.id])` +- Returns `(earliest_event_start, latest_event_end)` or `(now, now)` if no data +- Threshold: `(latest - earliest) >= timedelta(days=2)` +- Placed **after** permission check to avoid unnecessary DB queries for unauthorized users + +### Forecast API Interaction + +Trigger endpoint: `POST /api/v3_0/sensors//forecasts/trigger` +- Minimal payload: `{ "duration": "PT24H" }` (no `data_key` on `duration` field) +- Response: `{ "forecast": "", "status": "PROCESSED", "message": "..." }` +- JS accesses job ID via `data.forecast` + +Poll endpoint: `GET /api/v3_0/sensors//forecasts/` +- HTTP 200 → job finished, show success Toast, reload page +- HTTP 202 → job still running, response body has `{ "status": "QUEUED"|"STARTED"|"DEFERRED" }`, show info Toast +- HTTP 400 → unknown job (race condition or expired queue), show error Toast +- Default poll config: 60 attempts × 3 s = 3-minute timeout + +### `showToast` Signature + +```javascript +window.showToast(message, type, { highlightDuplicates = true, showDuplicateCount = true } = {}) +// type: "info" | "success" | "error" +// Durations: error=10s, success=2s, info=3s +``` + +## Interaction Rules + +- If a change modifies the forecast trigger/poll API contract, escalate to **API & Backward Compatibility Specialist** to verify the JS payload still matches +- If `get_timerange` or other time-arithmetic logic changes, escalate to **Data & Time Semantics Specialist** +- If test fixtures or mock strategy is complex, coordinate with **Test Specialist** +- Escalate to **Coordinator** if a new UI pattern emerges that needs to be standardised across agents + +## Self-Improvement Notes + +### Update This Agent When + +- A new panel type is added to the sensor or asset page (encode its pattern) +- The Toast API changes (e.g., new type added, signature changes) +- A new fetch→poll pattern variation is used +- A CSRF mitigation is added to the UI (currently absent — document if added) +- New permission types are used in view code +- New JS utilities are added to `ui-utils.js` or `flexmeasures.js` + +### Known Gaps / Technical Debt to Watch + +1. **CSRF protection is absent** on all browser-initiated `fetch()` POST/PATCH/DELETE calls in templates. This is an existing architectural gap (not introduced by PR #1985). If Flask-WTF CSRF tokens are added in future, the UI agent checklist should require their inclusion in all state-mutating fetch calls. +2. **Session expiry during poll loop**: A 401 response during polling is treated the same as an error, showing "Forecast job failed" rather than "Session expired — please log in". Consider adding specific handling. +3. **Hardcoded `PT24H`**: The forecast duration is not configurable via the UI. The info tooltip mentions this. If a duration picker is added later, the fetch payload and schema validation docs must be updated. +4. **Type annotation gap**: `user_can_create_children(asset: GenericAsset)` is called with `Sensor` objects. Works at runtime (both use `AuthModelMixin`), but mypy may flag it. Consider widening the type hint to `AuthModelMixin` in a future cleanup PR. + +### Session 2026-02-24 — PR #1985 Lessons + +- **Side panel pattern**: Mirror the "Upload data" panel structure exactly (outer container → label → inner div → fieldset). Consistency is important for CSS hover interactions. +- **Short-circuit the DB call**: Always gate `get_timerange` (or any DB-touching call) behind the permission check. A dedicated test (`test_get_timerange_not_called_without_permission`) should verify this. +- **Boundary test value**: Use `timedelta(days=2) - timedelta(seconds=1)` to test the boundary, not `timedelta(days=1)` — the test should be tight around the actual threshold. +- **JS guarded by Jinja2**: Wrap the event listener registration in `{% if permission_var and data_var %}` to prevent `getElementById` returning null for the disabled-button path. +- **Test fixture for cross-account user**: Create a `scope="function"` fixture that logs in a user from a different account; this makes negative-permission tests readable and reusable. diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 720e8051bd..33142f86b5 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -13,6 +13,7 @@ New features ------------- * Improve CSV upload validation by inferring the intended base resolution even when data contains valid gaps, instead of requiring perfectly regular timestamps [see `PR #1918 `_] * New forecasting API endpoints, and all timing parameters in forecasting CLI got sensible defaults for ease of use `[POST] /sensors/(id)/forecasts/trigger `_ and `[GET] /sensors/(id)/forecasts/(uuid) `_ to forecast sensor data [see `PR #1813 `_, `PR #1823 `_, `PR #1917 `_ and `PR #1982 `_] +* Add a "Create forecast" button to the sensor page, which triggers a forecasting via one click. Visible to users with permission to record data on the sensor and enabled when at least two days of data are present [see `PR #1985 `_] * Support setting a resolution when triggering a schedule via the API or CLI [see `PR #1857 `_] * Support variable peak pricing and changes in commitment baselines [see `PR #1835 `_] * Support storing the aggregate power schedule [see `PR #1736 `_] diff --git a/documentation/features/forecasting.rst b/documentation/features/forecasting.rst index aced89f78b..8839bf34c0 100644 --- a/documentation/features/forecasting.rst +++ b/documentation/features/forecasting.rst @@ -62,6 +62,13 @@ Note that: ``forecast-frequency`` together with ``max-forecast-horizon`` determine how the forecasting cycles advance through time. ``train-period``, ``from-date`` and ``to-date`` allow precise control over the training and prediction windows in each cycle. +Forecasting via the UI +----------------------- + +The quickest way to create a one-off forecast is the **Create forecast** button on the sensor page (see :ref:`view_sensors_forecast_button`). The button is available to users with permission to record data on sensors, provided at least two days of historical data exist. The forecast duration defaults to 48 hours (configured via ``FLEXMEASURES_PLANNING_HORIZON``) but can be adjusted up to 7 days in the panel. No further configuration is needed — one click queues the job and the page shows progress messages until the forecast is ready. + +For more control over what and how to forecast, use the API. + Forecasting via the API ----------------------- diff --git a/documentation/views/sensors.rst b/documentation/views/sensors.rst index 494bc87d20..b74db9e3ae 100644 --- a/documentation/views/sensors.rst +++ b/documentation/views/sensors.rst @@ -34,3 +34,12 @@ And here are 4 days of (dis)-charging patterns in Seita's V2GLiberty project: Charging (blue) mostly happens in sunshine hours, discharging during high-price hours (morning & evening) So on a technical level, the daily heatmap is essentially a heatmap of the sensor's values, with dates on the y-axis and time of day on the x-axis. For individual devices, it gives an insight into the device's running times. A new button lets users switch between charts. + +.. _view_sensors_forecast_button: + +Creating a forecast +------------------- + +Users with permission to record data on a sensor can create a forecast directly from the sensor page by clicking the **Create forecast** button in the left side panel. The forecast duration defaults to 48 hours (configured via ``FLEXMEASURES_PLANNING_HORIZON``) but can be adjusted up to 7 days in the panel. The button is enabled once the sensor has at least two days of historical data. After clicking, a background job is queued and the page shows progress updates via status messages. When the job finishes, the chart is refreshed to display the new forecast alongside the historical data. + +See :ref:`forecasting` for more details on how FlexMeasures generates forecasts. diff --git a/flexmeasures/data/models/forecasting/pipelines/train_predict.py b/flexmeasures/data/models/forecasting/pipelines/train_predict.py index 6efb4d0fe9..8eca23ec02 100644 --- a/flexmeasures/data/models/forecasting/pipelines/train_predict.py +++ b/flexmeasures/data/models/forecasting/pipelines/train_predict.py @@ -44,11 +44,12 @@ def __init__( self.delete_model = delete_model self.return_values = [] # To store forecasts and jobs - def run_wrap_up(self, cycle_job_ids: list[str]): + def run_wrap_up(self, cycle_job_ids: list[str], queue: str): """Log the status of all cycle jobs after completion.""" + connection = current_app.queues[queue].connection for index, job_id in enumerate(cycle_job_ids): logging.info( - f"forecasting job-{index}: {job_id} status: {Job.fetch(job_id).get_status()}" + f"forecasting job-{index}: {job_id} status: {Job.fetch(job_id, connection=connection).get_status()}" ) def run_cycle( @@ -199,6 +200,7 @@ def run( logging.info( f"Starting Train-Predict Pipeline to predict for {self._parameters['predict_period_in_hours']} hours." ) + connection = current_app.queues[queue].connection # How much to move forward to the next cycle one prediction period later cycle_frequency = max( self._config["retrain_frequency"], @@ -273,7 +275,7 @@ def run( self.run_cycle, # Some cycle job params override job kwargs kwargs={**job_kwargs, **cycle_params}, - connection=current_app.queues[queue].connection, + connection=connection, ttl=int( current_app.config.get( "FLEXMEASURES_JOB_TTL", timedelta(-1) @@ -301,14 +303,22 @@ def run( wrap_up_job = Job.create( self.run_wrap_up, - kwargs={"cycle_job_ids": cycle_job_ids}, # cycles jobs IDs to wait for - connection=current_app.queues[queue].connection, + kwargs={ + "cycle_job_ids": cycle_job_ids, # cycles jobs IDs to wait for + "queue": queue, + }, + connection=connection, depends_on=cycle_job_ids, # wrap-up job depends on all cycle jobs ttl=int( current_app.config.get( "FLEXMEASURES_JOB_TTL", timedelta(-1) ).total_seconds() ), + result_ttl=int( + current_app.config.get( + "FLEXMEASURES_PLANNING_TTL", timedelta(-1) + ).total_seconds() + ), # NB job.cleanup docs says a negative number of seconds means persisting forever meta=job_metadata, ) current_app.queues[queue].enqueue_job(wrap_up_job) diff --git a/flexmeasures/data/models/forecasting/utils.py b/flexmeasures/data/models/forecasting/utils.py index a7819a0652..e1e4f10acd 100644 --- a/flexmeasures/data/models/forecasting/utils.py +++ b/flexmeasures/data/models/forecasting/utils.py @@ -251,11 +251,12 @@ def data_to_bdf( # source_type="forecaster", # attributes=self.data_source.attributes, # ) + source = refresh_data_source(data_source) # Convert to BeliefsDataFrame bdf = tb.BeliefsDataFrame( forecast_df.reset_index().rename(columns={"forecasts": "event_value"}), - source=data_source, + source=source, sensor=sensor_to_save, ) return bdf @@ -265,3 +266,22 @@ def floor_to_resolution(dt: datetime, resolution: timedelta) -> datetime: delta_seconds = resolution.total_seconds() floored = dt.timestamp() - (dt.timestamp() % delta_seconds) return datetime.fromtimestamp(floored, tz=dt.tzinfo) + + +def refresh_data_source(data_source: DataSource) -> DataSource: + """Refresh the potentially detached data source. + + This avoids a sqlalchemy.exc.IntegrityError / psycopg2.errors.ForeignKeyViolation + for the data source ID not being present in the data_source table. + """ + + from flexmeasures.data.services.data_sources import get_or_create_source + + source = get_or_create_source( + data_source.name, + source_type=data_source.type, + model=data_source.model, + version=data_source.version, + attributes=data_source.attributes, + ) + return source diff --git a/flexmeasures/data/models/generic_assets.py b/flexmeasures/data/models/generic_assets.py index e9a3e460a1..d2d64af462 100644 --- a/flexmeasures/data/models/generic_assets.py +++ b/flexmeasures/data/models/generic_assets.py @@ -637,7 +637,7 @@ def chart( ) -> dict: """Create a vega-lite chart showing sensor data. - :param chart_type: currently only "bar_chart" # todo: where can we properly list the available chart types? + :param chart_type: currently only "chart_for_multiple_sensors" # todo: where can we properly list the available chart types? :param event_starts_after: only return beliefs about events that start after this datetime (inclusive) :param event_ends_before: only return beliefs about events that end before this datetime (inclusive) :param beliefs_after: only return beliefs formed after this datetime (inclusive) diff --git a/flexmeasures/ui/templates/includes/graphs.html b/flexmeasures/ui/templates/includes/graphs.html index f99d9944f2..004625ec36 100644 --- a/flexmeasures/ui/templates/includes/graphs.html +++ b/flexmeasures/ui/templates/includes/graphs.html @@ -256,7 +256,27 @@ // Sensors To Show Updated Listener document.addEventListener('sensorsToShowUpdated', async function () { - await embedAndLoad(chartSpecsPath + 'event_starts_after=' + storeStartDate.toISOString() + '&event_ends_before=' + storeEndDate.toISOString() + '&', elementId, datasetName, previousResult, storeStartDate, storeEndDate); + $("#spinner").show(); + let newData = fetch(dataPath + '/chart_data?event_starts_after=' + '{{ event_starts_after }}' + '&event_ends_before=' + '{{ event_ends_before }}' + '&compress_json=true', { + method: "GET", + headers: { "Content-Type": "application/json" }, + signal: signal, + }) + .then(function (response) { return response.json(); }) + .then(data => decompressChartData(data)); + newData = await Promise.all([ + newData, + embedAndLoad(chartSpecsPath + 'event_starts_after=' + storeStartDate.toISOString() + '&event_ends_before=' + storeEndDate.toISOString() + '&', elementId, datasetName, previousResult, storeStartDate, storeEndDate), + ]).then(function (result) { return result[0] }).catch(console.error); + $("#spinner").hide(); + vegaView.change(datasetName, vega.changeset().remove(vega.truthy).insert(newData)).resize().run(); + var sessionStart = new Date('{{ event_starts_after }}'); + var sessionEnd = new Date('{{ event_ends_before }}'); + previousResult = { + start: sessionStart, + end: sessionEnd, + data: fetchedInitialData + }; }); {% if event_starts_after and event_ends_before %} diff --git a/flexmeasures/ui/templates/sensors/index.html b/flexmeasures/ui/templates/sensors/index.html index 49e18fcbb8..1546661a5b 100644 --- a/flexmeasures/ui/templates/sensors/index.html +++ b/flexmeasures/ui/templates/sensors/index.html @@ -35,7 +35,7 @@

Edit {{ sensor.name }}

- Parent asset: {{ sensor.generic_asset.name }} (ID: {{ sensor.generic_asset.id }} ) + Belongs to: {{ sensor.generic_asset.name }} (ID: {{ sensor.generic_asset_id }})
@@ -62,7 +62,7 @@

Edit {{ sensor.name }}

Upload {{ sensor.name }} data

- Belongs to: {{ sensor.generic_asset.name }} (ID: {{ sensor.generic_asset_id }}) + Sensor: {{ sensor.name }} (ID: {{ sensor.id }})
@@ -156,6 +156,49 @@

Upload {{ sensor.name }} data

{% endif %} + {% if user_can_create_children_sensor %} +
+
Forecast
+
+
+

+ Create forecast + +

+ Sensor: {{ sensor.name }} (ID: {{ sensor.id }}) + {% if sensor_has_enough_data_for_forecast %} +
+ + +
+
+ + More options + + + +
+ {% else %} +

+ At least two days of sensor data are needed to create a forecast. +

+ + {% endif %} +
+
+
+ {% endif %} +
{% if user_can_delete_sensor %}
@@ -432,6 +475,83 @@
Statistics
deleteSensor(); } }); + + {% if user_can_create_children_sensor and sensor_has_enough_data_for_forecast %} + async function triggerForecast() { + const button = document.getElementById('triggerForecastButton'); + const spinner = document.getElementById('spinner-forecast'); + const durationDays = parseInt(document.getElementById('forecastDuration').value) || {{ forecast_default_duration_days }}; + const durationIso = 'P' + durationDays + 'D'; + + button.classList.add('d-none'); + spinner.classList.remove('d-none'); + + showToast("Triggering forecast job...", "info"); + + try { + const response = await fetch(apiBasePath + "/api/v3_0/sensors/{{ sensor.id }}/forecasts/trigger", { + method: "POST", + headers: { + "Content-Type": "application/json", + }, + body: JSON.stringify({ duration: durationIso }), + }); + + if (!response.ok) { + const errorData = await response.json(); + showToast("Error triggering forecast: " + (errorData.message || response.statusText), "error"); + button.classList.remove('d-none'); + spinner.classList.add('d-none'); + return; + } + + const data = await response.json(); + const jobId = data.forecast; + + showToast("Forecast job queued. Checking status...", "info"); + + let finished = false; + const maxAttempts = 60; // poll for up to 3 minutes + for (let i = 0; i < maxAttempts; i++) { + await new Promise(resolve => setTimeout(resolve, 3000)); + + let statusResponse; + try { + statusResponse = await fetch(apiBasePath + "/api/v3_0/sensors/{{ sensor.id }}/forecasts/" + jobId); + } catch (pollError) { + showToast("Error checking forecast status: " + pollError.message, "error"); + break; + } + + if (statusResponse.status === 200) { + showToast("Forecast created successfully! Refreshing chart...", "success"); + finished = true; + await new Promise(resolve => setTimeout(resolve, 1500)); + document.dispatchEvent(new CustomEvent('sensorsToShowUpdated')); + break; + } else if (statusResponse.status === 202) { + const statusData = await statusResponse.json(); + showToast("Forecast status: " + statusData.status + "...", "info"); + } else { + const errorData = await statusResponse.json(); + showToast("Forecast job failed: " + (errorData.message || "Unknown error"), "error"); + break; + } + } + + if (!finished) { + showToast("Forecast job timed out or failed.", "error"); + } + } catch (error) { + showToast("Error: " + error.message, "error"); + } finally { + button.classList.remove('d-none'); + spinner.classList.add('d-none'); + } + } + + document.getElementById('triggerForecastButton').addEventListener('click', triggerForecast); + {% endif %} {% endblock %} diff --git a/flexmeasures/ui/tests/test_sensor_views.py b/flexmeasures/ui/tests/test_sensor_views.py new file mode 100644 index 0000000000..efaf1fc48d --- /dev/null +++ b/flexmeasures/ui/tests/test_sensor_views.py @@ -0,0 +1,204 @@ +""" +Tests for the Sensor UI view (SensorUI). + +The sensor page at /sensors/ renders sensor details and optionally +a "Create forecast" side panel. These tests verify: + +- Basic access and 404 behaviour +- "Create forecast" panel visibility gated on ``create-children`` permission +- Forecast button enabled/disabled state based on available data range +- Guard that ``get_timerange`` is NOT called for users without permission +""" + +from datetime import datetime, timedelta, timezone +from unittest.mock import patch + +import pytest +from flask import url_for + +from flexmeasures.data.services.users import find_user_by_email +from flexmeasures.ui.tests.utils import login, logout + + +@pytest.fixture(scope="function") +def as_supplier_user(client): + """ + Login the default test supplier user (from the *Supplier* account) and log + them out afterwards. The Supplier account does not own the Prosumer + account's sensors, so this user lacks ``create-children`` permission on + those sensors. + """ + login(client, "test_supplier_user_4@seita.nl", "testtest") + yield + logout(client) + + +def _get_prosumer_sensor(db): + """Return the first sensor that belongs to the first Prosumer asset.""" + user = find_user_by_email("test_prosumer_user@seita.nl") + sensor = user.account.generic_assets[0].sensors[0] + db.session.expunge(user) + return sensor + + +# --------------------------------------------------------------------------- +# Basic page access +# --------------------------------------------------------------------------- + + +def test_sensor_page_loads(db, client, setup_assets, as_prosumer_user1): + """Sensor page returns HTTP 200 for a logged-in owner-account user.""" + sensor = _get_prosumer_sensor(db) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) + assert response.status_code == 200 + assert sensor.name.encode() in response.data + + +def test_sensor_page_requires_login(client, setup_assets): + """Unauthenticated requests are redirected to the login page.""" + response = client.get(url_for("SensorUI:get", id=1), follow_redirects=True) + assert b"Please log in" in response.data + + +def test_sensor_page_404_for_nonexistent_sensor(db, client, as_prosumer_user1): + """Requesting a non-existent sensor ID returns a 404.""" + response = client.get(url_for("SensorUI:get", id=999999), follow_redirects=True) + assert response.status_code == 404 + + +# --------------------------------------------------------------------------- +# "Create forecast" panel – visibility based on permissions +# --------------------------------------------------------------------------- + + +def test_create_forecast_panel_visible_for_account_member( + db, client, setup_assets, as_prosumer_user1 +): + """ + The "Create forecast" panel is rendered for a user who belongs to the + account that owns the sensor (Sensor ACL grants ``create-children`` to + every member of the owning account). + """ + sensor = _get_prosumer_sensor(db) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) + assert response.status_code == 200 + assert b"Create forecast" in response.data + + +def test_create_forecast_panel_visible_for_admin(db, client, setup_assets, as_admin): + """Admin users bypass ACL and also see the "Create forecast" panel.""" + sensor = _get_prosumer_sensor(db) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) + assert response.status_code == 200 + assert b"Create forecast" in response.data + + +def test_create_forecast_panel_hidden_for_other_account( + db, client, setup_assets, as_supplier_user +): + """ + A user from a different account (no ``create-children`` permission on the + sensor) does not see the "Create forecast" panel at all. + """ + sensor = _get_prosumer_sensor(db) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) + assert response.status_code == 200 + assert b"Create forecast" not in response.data + + +# --------------------------------------------------------------------------- +# Forecast button state – enabled vs. disabled +# --------------------------------------------------------------------------- + + +def test_forecast_button_disabled_with_insufficient_data( + db, client, setup_assets, as_prosumer_user1 +): + """ + The forecast button is disabled and an explanatory message is shown when + the sensor has fewer than two days of historical data. + + ``setup_assets`` populates each sensor with one day of beliefs + (2015-01-01 at 15-minute resolution), which is below the 2-day threshold. + """ + sensor = _get_prosumer_sensor(db) + response = client.get(url_for("SensorUI:get", id=sensor.id), follow_redirects=True) + assert response.status_code == 200 + assert b"Create forecast" in response.data + # The enabled button (identified by its unique id) is absent + assert b"triggerForecastButton" not in response.data + # The explanatory message is shown + assert b"At least two days of sensor data are needed" in response.data + + +def test_forecast_button_enabled_with_sufficient_data( + db, client, setup_assets, as_prosumer_user1 +): + """ + The forecast button is enabled and the JS polling code is injected when + the sensor has at least two days of historical data. + + ``get_timerange`` is patched to return a two-day span. + """ + sensor = _get_prosumer_sensor(db) + t0 = datetime(2015, 1, 1, tzinfo=timezone.utc) + t2 = t0 + timedelta(days=2) + + with patch( + "flexmeasures.ui.views.sensors.get_timerange", + return_value=(t0, t2), + ): + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + + assert response.status_code == 200 + assert b"triggerForecastButton" in response.data + assert b"At least two days of sensor data are needed" not in response.data + + +def test_forecast_boundary_just_under_two_days( + db, client, setup_assets, as_prosumer_user1 +): + """ + A data span of exactly two days minus one second does NOT pass the + ``>= timedelta(days=2)`` threshold. + """ + sensor = _get_prosumer_sensor(db) + t0 = datetime(2015, 1, 1, tzinfo=timezone.utc) + t_short = t0 + timedelta(days=2) - timedelta(seconds=1) + + with patch( + "flexmeasures.ui.views.sensors.get_timerange", + return_value=(t0, t_short), + ): + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + + assert response.status_code == 200 + assert b"triggerForecastButton" not in response.data + assert b"At least two days of sensor data are needed" in response.data + + +# --------------------------------------------------------------------------- +# Guard: get_timerange is NOT called when user has no permission +# --------------------------------------------------------------------------- + + +def test_get_timerange_not_called_without_permission( + db, client, setup_assets, as_supplier_user +): + """ + ``get_timerange`` must not be called when ``user_can_create_children`` + returns ``False`` — the view short-circuits to avoid an unnecessary DB query. + """ + sensor = _get_prosumer_sensor(db) + + with patch("flexmeasures.ui.views.sensors.get_timerange") as mock_get_timerange: + response = client.get( + url_for("SensorUI:get", id=sensor.id), follow_redirects=True + ) + + assert response.status_code == 200 + mock_get_timerange.assert_not_called() diff --git a/flexmeasures/ui/views/sensors.py b/flexmeasures/ui/views/sensors.py index 4e271bd427..8acba20d78 100644 --- a/flexmeasures/ui/views/sensors.py +++ b/flexmeasures/ui/views/sensors.py @@ -1,11 +1,15 @@ -from flask import request +from datetime import timedelta + +from flask import current_app, request from flask_classful import FlaskView from flask_security import login_required +from humanize import naturaldelta from werkzeug.exceptions import NotFound from webargs.flaskparser import use_kwargs from flexmeasures.data import db from flexmeasures.data.schemas import StartEndTimeSchema +from flexmeasures.data.services.timerange import get_timerange from flexmeasures import Sensor from flexmeasures.ui.utils.view_utils import ( render_flexmeasures_template, @@ -13,9 +17,11 @@ ) from flexmeasures.ui.utils.breadcrumb_utils import get_breadcrumb_info from flexmeasures.ui.views.assets.utils import ( + user_can_create_children, user_can_delete, user_can_update, ) +from flexmeasures.utils.time_utils import duration_isoformat class SensorUI(FlaskView): @@ -39,11 +45,29 @@ def get(self, id: int, **kwargs): sensor = db.session.get(Sensor, id) if sensor is None: raise NotFound + can_create_children = user_can_create_children(sensor) + has_enough_data = False + planning_horizon: timedelta = current_app.config.get( + "FLEXMEASURES_PLANNING_HORIZON", timedelta(days=2) + ) + forecast_default_duration_iso = duration_isoformat(planning_horizon) + forecast_default_duration_human = naturaldelta(planning_horizon) + forecast_default_duration_days = max( + 1, min(7, int(planning_horizon.total_seconds() / 86400)) + ) + if can_create_children: + earliest, latest = get_timerange([sensor.id]) + has_enough_data = (latest - earliest) >= timedelta(days=2) return render_flexmeasures_template( "sensors/index.html", sensor=sensor, user_can_update_sensor=user_can_update(sensor), user_can_delete_sensor=user_can_delete(sensor), + user_can_create_children_sensor=can_create_children, + sensor_has_enough_data_for_forecast=has_enough_data, + forecast_default_duration_iso=forecast_default_duration_iso, + forecast_default_duration_human=forecast_default_duration_human, + forecast_default_duration_days=forecast_default_duration_days, available_units=available_units(), msg="", breadcrumb_info=get_breadcrumb_info(sensor),