From 17eac86f5a5b3c9ba6944505d6dc04ac757145d1 Mon Sep 17 00:00:00 2001 From: agrondaka Date: Mon, 6 Oct 2025 12:51:35 +0200 Subject: [PATCH 1/3] Added web app tutorial related to ADX and PBi reports --- Tutorial/docs/tutorials/web-app/.nav.yml | 9 +- .../docs/tutorials/web-app/adx-database.md | 84 ++++++++- .../web-app/adx-functions-performance.md | 176 ++++++++++++++++++ Tutorial/docs/tutorials/web-app/frontend.md | 106 ++++++++++- .../tutorials/web-app/power-bi-embedding.md | 143 ++++++++++++++ .../tutorials/web-app/power-bi-parameters.md | 109 +++++++++++ Tutorial/docs/tutorials/web-app/power-bi.md | 61 +++++- 7 files changed, 680 insertions(+), 8 deletions(-) create mode 100644 Tutorial/docs/tutorials/web-app/adx-functions-performance.md create mode 100644 Tutorial/docs/tutorials/web-app/power-bi-embedding.md create mode 100644 Tutorial/docs/tutorials/web-app/power-bi-parameters.md diff --git a/Tutorial/docs/tutorials/web-app/.nav.yml b/Tutorial/docs/tutorials/web-app/.nav.yml index 622b6c0..2c1b78d 100644 --- a/Tutorial/docs/tutorials/web-app/.nav.yml +++ b/Tutorial/docs/tutorials/web-app/.nav.yml @@ -1,5 +1,10 @@ nav: - index.md - Frontend: frontend.md - - ADX database: adx-database.md - - Power BI: power-bi.md + - ADX: + - Overview: adx-database.md + - Functions & Performance: adx-functions-performance.md + - Power BI: + - Overview: power-bi.md + - Parameters & Reuse: power-bi-parameters.md + - Embedding & Security: power-bi-embedding.md diff --git a/Tutorial/docs/tutorials/web-app/adx-database.md b/Tutorial/docs/tutorials/web-app/adx-database.md index 156f9fa..99a8bcd 100644 --- a/Tutorial/docs/tutorials/web-app/adx-database.md +++ b/Tutorial/docs/tutorials/web-app/adx-database.md @@ -1,3 +1,83 @@ -# ADX database +--- +title: ADX Database Overview +summary: Integrate and manage the Azure Data Explorer (ADX) layer for your web application. +tags: + - adx + - data + - web-app +--- -Integrate and manage the ADX database for your web application. +# ADX Database Overview + +This page introduces how the web-app consumes data from Azure Data Explorer (ADX) and how to structure your data estate for analytics & dashboards. + +## 1. Why ADX for the Solution? + +- Sub-second exploratory analytics on time-series & events +- Native integration with Power BI (DirectQuery / import) and Azure ecosystem +- Scalable ingestion & retention policies + +## 2. High-Level Flow + +``` +Sources -> Ingestion (EventHub / Blob / ADX LightIngest) -> Raw Tables -> Update Policies / Materialized Views -> Serving Functions -> Power BI / API +``` + +## 3. Core Concepts + +| Concept | Description | Example | +|---------|-------------|---------| +| Table | Physically stored records | `BrewerEvents` | +| Ingestion Mapping | Column mapping from source payload | JSON mapping for brewer events | +| Update Policy | Auto-populate derived table | Raw -> Cleaned table | +| Materialized View | Pre-computed aggregation | 15-min brew KPIs | +| Function | Reusable query logic | `GetBrewEvents()` | + +## 4. Recommended Table Design + +| Column Type | Purpose | Naming | +|-------------|---------|--------| +| Timestamp | Query time axis | `Timestamp` first column | +| Entity Keys | Filter dimensions | `Site`, `BatchId`, `Product` | +| Measures | Numeric metrics | `DurationSeconds`, `Temperature` | +| Status / Type | Categorical attributes | `EventType` | + +Partitioning automatically by ingestion time—optimize retention with policies: + +```kusto +.alter table BrewerEvents policy retention softdelete = 30d"); +``` + +## 5. Ingestion Tips + +- Batch ingestion for historical loads; streaming/EventHub for near real-time +- Validate schema with a staging table before merging +- Compress JSON payloads or prefer CSV/Parquet for large volume + +## 6. Serving Functions + +All dashboard/API queries should route via serving functions (see: [ADX Functions & Performance](./adx-functions-performance.md)). + +Benefits: +- Centralize filter logic +- Enforce parameter contracts +- Reduce duplication across Power BI & services + +## 7. Governance & Source Control + +Store Kusto artifacts (`tables.kql`, `functions/*.kql`, `policies.kql`) under version control. Deploy via CI using Azure CLI or ARM scripts. + +## 8. Observability + +Use built-in commands: +```kusto +.show table BrewerEvents details +.show queries | where StartedOn > ago(15m) +.show operations | where StartedOn > ago(1h) and Status != 'Completed' +``` + +## 9. Next + +Proceed to implementing reusable functions and performance tuning. + +> See: [Functions & Performance](./adx-functions-performance.md) diff --git a/Tutorial/docs/tutorials/web-app/adx-functions-performance.md b/Tutorial/docs/tutorials/web-app/adx-functions-performance.md new file mode 100644 index 0000000..97eeace --- /dev/null +++ b/Tutorial/docs/tutorials/web-app/adx-functions-performance.md @@ -0,0 +1,176 @@ +--- +title: ADX Functions & Performance +summary: Create reusable functions and optimize Kusto queries powering the web-app dashboards. +tags: + - adx + - performance + - web-app + - optimization +--- + +# ADX Functions & Performance + +This guide shows how to encapsulate logic into Azure Data Explorer (ADX) functions and optimize query performance for responsive dashboards. + +> Focus areas: +> 1. Reuse logic across dashboards & web-app deployments +> 2. Reduce data scanned & transferred to Power BI +> 3. Improve freshness vs cost balance +> 4. Make parameters first-class citizens + +## 1. Organize Your ADX Layer + +| Layer | Purpose | Artifacts | +|-------|---------|-----------| +| Raw ingestion | Land data as-is | Ingestion mappings, staging tables | +| Harmonized | Cleaned, typed, conformed | Update policies, materialized views | +| Serving | Query-ready, thin, parameterizable | Functions, export tables | + +Keep functions in a dedicated folder in source control (e.g. `infrastructure/adx/functions/`). Version them and deploy declaratively (ARM/Bicep/Terraform / Kusto scripts). + +## 2. Defining Functions + +Two main kinds: +- Inline (lightweight) functions +- Stored functions created once in the database + +Example: `GetBrewEvents(startTime:datetime, endTime:datetime, site:string)` + +```kusto +.create-or-alter function with (docstring = 'Brew events filtered by time & site', folder='serving/brew') +GetBrewEvents(startTime:datetime, endTime:datetime, site:string) +{ + BrewerEvents + | where Timestamp between (startTime .. endTime) + | where Site == site + | project Timestamp, Site, BatchId, EventType, DurationSeconds +} +``` + +### Best Practices +- Use PascalCase for function names, camelCase for parameters. +- Provide `docstring` and `folder` for discoverability. +- Avoid hard-coded constants; expose as parameters. + +## 3. Composing Functions + +Create small focused functions and compose: + +```kusto +// Base filter +.create-or-alter function GetBatches(startTime:datetime, endTime:datetime){ + Batches | where StartTime between (startTime .. endTime) + | project BatchId, Product, StartTime, EndTime +} + +// Duration enrichment +.create-or-alter function GetBatchDurations(startTime:datetime, endTime:datetime){ + GetBatches(startTime, endTime) + | extend DurationMinutes = datetime_diff('minute', EndTime, StartTime) +} + +// Aggregated KPI +.create-or-alter function GetOutputKPI(startTime:datetime, endTime:datetime){ + GetBatchDurations(startTime, endTime) + | summarize AvgDuration = avg(DurationMinutes), TotalBatches = count() +} +``` + +## 4. Parameter Patterns for Power BI + +When embedding, the web-app can compute a time window & site, then pass them as query parameters. + +Pattern: a single root function that accepts all Power BI-relevant parameters and outputs a narrow dataset. + +```kusto +.create-or-alter function Dashboard_Fact(startTime:datetime, endTime:datetime, site:string, product:string){ + GetBrewEvents(startTime, endTime, site) + | where product == '' or Product == product + | summarize Events=count(), TotalDuration=sum(DurationSeconds) +} +``` + +In Power BI M query (parameterized): + +```m +let + StartTime = DateTimeZone.UtcNow() - #duration(0,12,0,0), + EndTime = DateTimeZone.UtcNow(), + Site = Text.From(EnvSiteParameter), + Product = "" , + Source = Kusto.Contents("https://..kusto.windows.net", "", + "Dashboard_Fact(datetime({" & DateTimeZone.ToText(StartTime, "yyyy-MM-dd HH:mm:ss") & "}), datetime({" & DateTimeZone.ToText(EndTime, "yyyy-MM-dd HH:mm:ss") & "}), '" & Site & "', '" & Product & "')") +in + Source +``` + +(Replace the dynamic site & product with actual PBI parameters linked to slicers.) + +## 5. Performance Tactics + +| Goal | Tactic | Notes | +|------|--------|-------| +| Min scan | Use narrower projection early | Always `project` right after filters | +| Fast filter | Use ingestion time partitioning & `between` | Align queries to partition keys | +| Reuse | Materialized views | For heavy joins/expensive lookups | +| Lower transfer | Summarize before export to PBI | Return aggregated rows, not raw events | +| Adaptive freshness | Tiered functions (raw vs MV) | Switch via a boolean parameter | + +### Example: Hybrid Live + Historical + +```kusto +.create-or-alter function Dashboard_Fact_Live(startAgo:timespan, site:string){ + GetBrewEvents(now()-startAgo, now(), site) +} + +.create-or-alter function Dashboard_Fact_History(startTime:datetime, endTime:datetime, site:string){ + Materialized_BrewAgg + | where WindowStart between (startTime .. endTime) + | where Site == site +} + +.create-or-alter function Dashboard_Fact_Union(startTime:datetime, endTime:datetime, liveAgo:timespan, site:string){ + union isfuzzy=true + (Dashboard_Fact_History(startTime, endTime, site)) + (Dashboard_Fact_Live(liveAgo, site)) + | summarize Events=sum(Events), TotalDuration=sum(TotalDuration) +} +``` + +## 6. Monitoring & Diagnostics + +```kusto +.show functions +.show commands-and-queries | where StartedOn > ago(1h) +``` + +Enable Query Store & alerts for: +- Long-running queries > 15s +- High data scanned vs returned ratio + +## 7. Deployment Automation + +Store function definitions as `.kql` files and apply idempotently: + +```bash +#!/usr/bin/env bash +for f in infrastructure/adx/functions/*.kql; do + echo "Applying $f"; + az kusto script create --cluster-name $CLUSTER --database-name $DB --name $(basename $f .kql) --script-content "$(cat $f)" --force-update-tag $(date +%s); +done +``` + +## 8. Checklist + +- [ ] All dashboard queries go through a root serving function +- [ ] Parameters documented (name, type, default) +- [ ] Functions have docstrings & folder metadata +- [ ] Query completion p50 < 3s +- [ ] No function returns > 100k rows to Power BI +- [ ] Materialized views used for joins > 10M rows + +## 9. Next Steps + +Continue with Power BI parameterization to bind these functions to dynamic dashboards. + +> See: [Power BI Parameters & Reuse](./power-bi-parameters.md) diff --git a/Tutorial/docs/tutorials/web-app/frontend.md b/Tutorial/docs/tutorials/web-app/frontend.md index 57b1eec..34b1c15 100644 --- a/Tutorial/docs/tutorials/web-app/frontend.md +++ b/Tutorial/docs/tutorials/web-app/frontend.md @@ -1,3 +1,105 @@ -# Frontend +--- +title: Frontend Architecture & Integration +summary: Learn about the frontend architecture and customization of your web application. +tags: + - frontend + - web-app + - power-bi + - adx +--- -Learn about the frontend architecture and customization of your web application. +# Frontend Architecture & Integration + +The frontend orchestrates context (site, user, time window), passes parameters to embedded analytics, and optimizes perceived performance. + +## 1. Layers + +| Layer | Responsibility | Tech | +|-------|----------------|------| +| Routing | Page/screen selection | React Router / framework router | +| State | Global context (site, auth, feature flags) | Redux / Zustand / Context API | +| Services | REST / GraphQL / ADX proxy calls | Fetch / Axios | +| Embedding | Power BI iframe + SDK | `powerbi-client` | +| Components | UI widgets / filters / charts | Design system | + +## 2. Parameter Propagation Flow + +``` +User selects Site -> Global State update -> ADX function queries (server) & Embed Filters (client) -> Report visuals refresh +``` + +Core rule: single source of truth for each parameter (avoid duplicate local state). + +## 3. Environment Configuration + +Expose environment via `.env` or config endpoint: + +| Variable | Purpose | +|----------|---------| +| `PBI_AUTH_MODE` | `service-principal` / `user` | +| `PBI_BACKEND_TOKEN_URL` | Token fetch endpoint | +| `ADX_CLUSTER_URL` | Display / debugging | +| `DEFAULT_SITE` | Initial filter | + +## 4. Fetching Embed Token + +```ts +async function fetchEmbedToken(reportId: string) { + const res = await fetch(`/api/powerbi/token?reportId=${reportId}` , { credentials: 'include' }); + if (!res.ok) throw new Error('Token failed'); + return res.json(); // { token, embedUrl, datasetId } +} +``` + +## 5. Context Synchronization + +```ts +const [site, setSite] = useState(initialSite); +useEffect(() => { + reportRef.current?.setFilters([ + { $schema: 'http://powerbi.com/product/schema#basic', target: { table: 'SiteFilter', column: 'Site' }, operator: 'In', values: [site] } + ]); +}, [site]); +``` + +## 6. Performance Optimizations + +| Concern | Mitigation | +|---------|------------| +| Slow initial dashboard | Lazy-load Power BI SDK & use skeleton loaders | +| Excess re-renders | Memoize filter components & avoid prop drilling | +| Network chatter | Debounce user input before triggering queries | +| Large bundle | Code-split embedding feature | + +## 7. Error Handling Patterns + +```ts +try { + const token = await fetchEmbedToken(reportId); + embedReport(container, { ...token, site }); +} catch (e) { + logError(e); + showToast('Unable to load analytics'); +} +``` + +Surface fallback UI with actionable steps (retry / contact support). + +## 8. Observability + +- Custom performance marks (e.g. `analytics_loaded`) +- Track embed token latency +- Capture front-end errors (Sentry / App Insights) + +## 9. Security Hygiene + +- Never store secrets in frontend bundle +- Validate all query parameter values server-side +- Sanitize user-entered strings passed to functions + +## 10. Next + +Proceed to ADX or Power BI deep dives: + +- [ADX Overview](./adx-database.md) +- [Power BI Overview](./power-bi.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi-embedding.md b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md new file mode 100644 index 0000000..404b797 --- /dev/null +++ b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md @@ -0,0 +1,143 @@ +--- +title: Power BI Embedding & Security +summary: Embed interactive Power BI reports securely in the web-app with governance, caching, and performance best practices. +tags: + - power-bi + - embedding + - security + - web-app +--- + +# Power BI Embedding & Security + +This guide covers embedding models, authentication flows, Row Level Security (RLS), and performance tuning. + +## 1. Embedding Models + +| Scenario | Model | Notes | +|----------|-------|-------| +| Internal users (AAD) | User owns data | Uses user AAD token; per-user RLS naturally applied | +| External / multi-tenant | Service principal | App gets embed token; enforce tenant filter via RLS | +| Public showcase | Publish-to-web (avoid) | Not secure; never use for production | + +Prefer service principal + embed token generation backend-side for multi-tenant. + +## 2. Architecture + +``` +Browser -> Web-App (frontend) -> Backend (token endpoint) -> AAD -> Power BI Service + | + +--> ADX (DirectQuery via PBI service) +``` + +## 3. Backend: Generate Embed Token + +Pseudo-code: + +```python +from msal import ConfidentialClientApplication +import requests + +def get_embed_token(report_id, workspace_id, dataset_id): + app = ConfidentialClientApplication( + client_id=CLIENT_ID, + authority=f"https://login.microsoftonline.com/{TENANT_ID}", + client_credential=CLIENT_SECRET + ) + scope = ["https://analysis.windows.net/powerbi/api/.default"] + result = app.acquire_token_silent(scope, account=None) + if not result: + result = app.acquire_token_for_client(scopes=scope) + access_token = result['access_token'] + + body = { + "datasets": [{"id": dataset_id}], + "reports": [{"id": report_id}], + "targetWorkspaces": [{"id": workspace_id}] + } + r = requests.post( + "https://api.powerbi.com/v1.0/myorg/GenerateToken", + headers={"Authorization": f"Bearer {access_token}"}, + json=body + ) + return r.json()['token'] +``` + +## 4. Frontend: Embed + +```ts +import * as powerbi from 'powerbi-client'; + +const models = powerbi.models; + +function embedReport(container: HTMLElement, embedConfig: { id: string; embedUrl: string; token: string; site: string; }) { + const config: powerbi.IEmbedConfiguration = { + type: 'report', + id: embedConfig.id, + embedUrl: embedConfig.embedUrl, + accessToken: embedConfig.token, + tokenType: models.TokenType.Embed, + settings: { panes: { filters: { visible: false } } } + }; + const report = powerbi.embed(container, config) as powerbi.Report; + // Apply site filter from contextual data + report.on('loaded', async () => { + await report.setFilters([ + { + $schema: 'http://powerbi.com/product/schema#basic', + target: { table: 'SiteFilter', column: 'Site' }, + operator: 'In', + values: [embedConfig.site] + } + ]); + }); +} +``` + +## 5. Security Layers + +| Layer | Control | Purpose | +|-------|---------|---------| +| AAD App | App roles / scopes | Restrict who can call token endpoint | +| Backend | Tenant/site validation | Prevent arbitrary site injection | +| Power BI | RLS roles | Enforce data slice per user/tenant | +| Dataset | Parameterization | Ensure minimal data exposure | + +### Example RLS Role (Site Isolation) + +In Power BI Desktop > Modeling > Manage Roles: +``` +[Site] = USERPRINCIPALNAME()\"s site mapping OR LOOKUPVALUE(SiteMap[Site], SiteMap[UserPrincipalName], USERPRINCIPALNAME()) +``` +(Alternatively pass site via embed token identities.) + +## 6. Caching & Performance + +| Layer | Strategy | Notes | +|-------|----------|-------| +| Power BI | Aggregations | Combine DirectQuery (hot) + Import (warm) | +| Power BI | Incremental refresh | Partition large historical tables | +| ADX | Materialized views | Pre-aggregate heavy metrics | +| Web-App | Embed lifecycle | Reuse report instance across tabs, avoid reloading | + +### Measuring +Use Performance Analyzer in Power BI Desktop and `QueryTrace` in ADX. + +## 7. Observability + +- Log token requests (timestamp, workspace, dataset, site, user/tenant) +- Alert on unusual volume or failed token generation +- Track report load times (frontend performance marks) + +## 8. Checklist + +- [ ] Service principal least-privilege +- [ ] Embed token TTL minimal (e.g. 1h) +- [ ] RLS validated per tenant +- [ ] Site filter applied programmatically +- [ ] No hard-coded secrets in frontend + +## 9. Related + +- [Parameters & Reuse](./power-bi-parameters.md) +- [ADX Functions & Performance](./adx-functions-performance.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi-parameters.md b/Tutorial/docs/tutorials/web-app/power-bi-parameters.md new file mode 100644 index 0000000..01c099f --- /dev/null +++ b/Tutorial/docs/tutorials/web-app/power-bi-parameters.md @@ -0,0 +1,109 @@ +--- +title: Power BI Parameters & Reuse +summary: Design parameterized datasets and M queries to reuse dashboards across environments and web-app deployments. +tags: + - power-bi + - parameters + - reuse + - performance +--- + +# Power BI Parameters & Reuse + +Parametrization lets you deploy the same `.pbix` to multiple tenants, sites, or time scopes without cloning logic. It also reduces dataset size by pushing filters down into ADX. + +## 1. Parameter Strategy + +| Parameter | Purpose | Source | Example | +|-----------|---------|--------|---------| +| Site | Scope data to a factory/site | Web-app context / user claims | `Lyon_01` | +| TimeWindowHours | Rolling horizon | UI control (slider) | `12` | +| Product | Optional product filter | Slicer | `Pilsner` | +| RefreshMode | Switch raw vs aggregated | Hidden toggle | `"auto"` / `"historical"` | + +Keep parameters minimal; prefer enumerations (domains) when possible for caching efficiency. + +## 2. Creating Parameters in Power BI Desktop + +1. Transform Data > Manage Parameters > New +2. Name: `Site`; Type: Text; Suggested Values: List (optional) +3. Default Value: `Lyon_01`; Current Value: `Lyon_01` +4. Repeat for others. + +> Tip: Use Text/Number/TrueFalse types—avoid DateTime parameters if you can compute datetimes inside the query from `now()` plus duration. + +## 3. Using Parameters in Kusto M Query + +```m +let + Site = Text.From(Parameter_Site), + Hours = Number.ToText(Parameter_TimeWindowHours), + Product = Text.From(Parameter_Product), + Cluster = Parameter_ADXCluster, + Database = Parameter_ADXDatabase, + Query = + "Dashboard_Fact(datetime(" & DateTimeZone.ToText(DateTimeZone.UtcNow() - #duration(0, Number.From(Hours),0,0), "yyyy-MM-dd HH:mm:ss") & "), datetime(" & DateTimeZone.ToText(DateTimeZone.UtcNow(), "yyyy-MM-dd HH:mm:ss") & "), '" & Site & "', '" & Product & "')" +, + Source = Kusto.Contents(Cluster, Database, Query) +in + Source +``` + +Abstract cluster & database into parameters too for full reuse. + +## 4. Dynamic Binding from Web-App + +When embedding, you can set report-level filters or parameter values via the JavaScript SDK. + +### Option A: Embedding + Slicers +Populate hidden slicer tables with a single value coming from the web-app. + +```ts +const site = appContext.site; // e.g. from JWT +report.setFilters([ + { + $schema: "http://powerbi.com/product/schema#basic", + target: { table: "SiteFilter", column: "Site" }, + operator: "In", + values: [site] + } +]); +``` + +### Option B: Query Parameters Through DirectQuery (Preview-dependent) +If using DirectQuery, you can rewrite queries to reference a single-row parameter table that the app updates via Push API. + +## 5. Dataset Slimming + +| Technique | Effect | Notes | +|-----------|--------|-------| +| Parameter pushdown | Smaller scans | Express all filters as function parameters | +| Summarize in ADX | Less row transfer | Provide aggregated KPIs, not raw events | +| Remove unused columns | Better compression | Audit visuals & fields list | +| Incremental refresh | Faster partition processing | Combine with time parameters | + +## 6. Testing Reusability + +Create a matrix of environment x site values and script automated validation: + +```bash +for site in Lyon_01 Berlin_02; do + pbicli reports export --report "BrewDashboard" --workspace $WS --file export-$site.pbix \ + --parameter Site=$site || exit 1 + echo "Validated $site" +done +``` + +## 7. Checklist + +- [ ] All Power BI queries reference a single root ADX serving function +- [ ] Parameters documented & minimal +- [ ] No visual applies redundant filters already in function +- [ ] Dataset size < target (e.g. 100 MB) / row count limited +- [ ] Test matrix executed for each release + +## 8. Next + +Proceed to embedding details & security considerations. + +> See: [Embedding & Security](./power-bi-embedding.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi.md b/Tutorial/docs/tutorials/web-app/power-bi.md index 85241c3..531abe8 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi.md +++ b/Tutorial/docs/tutorials/web-app/power-bi.md @@ -1,3 +1,60 @@ -# Power BI +--- +title: Power BI Overview +summary: Embed and use Power BI dashboards in your web application. +tags: + - power-bi + - web-app +--- -Embed and use Power BI dashboards in your web application. +# Power BI Overview + +This section introduces how Power BI integrates with the solution for interactive analytics. + +## 1. Objectives + +- Reuse the same report across environments & sites +- Keep dashboards performant (< 5s load for key visuals) +- Enforce data security & isolation + +## 2. Architecture Snapshot + +``` +ADX (Functions) -> Power BI Dataset -> Report -> Embedded in Web-App +``` + +## 3. Key Building Blocks + +| Component | Description | Reference | +|-----------|-------------|-----------| +| Parameterized Functions | Source for DirectQuery / Import | ADX Functions & Performance | +| Power BI Parameters | Filter scoping & reuse | Parameters & Reuse | +| Embed Token Service | Secure report embedding | Embedding & Security | +| RLS Roles | Tenant / site isolation | Embedding & Security | + +## 4. Quick Start Checklist + +- [ ] Create ADX serving function `Dashboard_Fact` with parameters +- [ ] Define Power BI parameters (Site, TimeWindowHours, Product) +- [ ] Build visuals from parameterized query +- [ ] Implement embed token backend endpoint +- [ ] Add site filter application on load +- [ ] Validate RLS & parameter variations + +## 5. Subpages + +- [Parameters & Reuse](./power-bi-parameters.md) +- [Embedding & Security](./power-bi-embedding.md) + +## 6. Performance Tips (Preview) + +| Issue | Mitigation | +|-------|------------| +| Slow initial load | Pre-warm dataset via REST refresh | +| Heavy visuals | Aggregate in ADX function | +| Data bloat | Remove unused columns & tables | + +## 7. Next Steps + +Proceed to parameters for full reuse across sites. + +> See: [Parameters & Reuse](./power-bi-parameters.md) From 740713fc7453b81096075d633d3f894b1bdc96d6 Mon Sep 17 00:00:00 2001 From: agrondaka Date: Tue, 7 Oct 2025 14:03:20 +0200 Subject: [PATCH 2/3] Last changes --- Sources/onboarding-brewery-solution | 2 +- .../docs/tutorials/web-app/adx-database.md | 20 +- .../web-app/adx-functions-performance.md | 19 +- Tutorial/docs/tutorials/web-app/frontend.md | 2 + .../tutorials/web-app/power-bi-embedding.md | 194 ++++++------------ Tutorial/docs/tutorials/web-app/power-bi.md | 14 +- 6 files changed, 99 insertions(+), 152 deletions(-) diff --git a/Sources/onboarding-brewery-solution b/Sources/onboarding-brewery-solution index d66bf74..39bdad5 160000 --- a/Sources/onboarding-brewery-solution +++ b/Sources/onboarding-brewery-solution @@ -1 +1 @@ -Subproject commit d66bf7493b7b567d1c7da17256f700ace7ca5e10 +Subproject commit 39bdad5560a8181e2e7f39731743c4725860d9dd diff --git a/Tutorial/docs/tutorials/web-app/adx-database.md b/Tutorial/docs/tutorials/web-app/adx-database.md index 99a8bcd..382b5d5 100644 --- a/Tutorial/docs/tutorials/web-app/adx-database.md +++ b/Tutorial/docs/tutorials/web-app/adx-database.md @@ -54,7 +54,19 @@ Partitioning automatically by ingestion time—optimize retention with policies: - Validate schema with a staging table before merging - Compress JSON payloads or prefer CSV/Parquet for large volume -## 6. Serving Functions +## 6. Schema Provisioning via Backend + +In this project, the ADX schema (tables, mappings, policies) is defined in the backend `Create.kql` script and deployed through backend pipelines. Do not recreate tables manually from the UI. + +Workflow for changes: +1. Propose updates in `Create.kql` (new columns, policies, mappings) +2. Run local validation against a dev cluster (optional) +3. Open a PR; backend CI applies changes to non-prod +4. After approval, pipeline promotes to production + +Keep serving function contracts stable; if breaking, version functions (e.g., `Dashboard_Fact_v2`). + +## 7. Serving Functions All dashboard/API queries should route via serving functions (see: [ADX Functions & Performance](./adx-functions-performance.md)). @@ -63,11 +75,11 @@ Benefits: - Enforce parameter contracts - Reduce duplication across Power BI & services -## 7. Governance & Source Control +## 8. Governance & Source Control Store Kusto artifacts (`tables.kql`, `functions/*.kql`, `policies.kql`) under version control. Deploy via CI using Azure CLI or ARM scripts. -## 8. Observability +## 9. Observability Use built-in commands: ```kusto @@ -76,7 +88,7 @@ Use built-in commands: .show operations | where StartedOn > ago(1h) and Status != 'Completed' ``` -## 9. Next +## 10. Next Proceed to implementing reusable functions and performance tuning. diff --git a/Tutorial/docs/tutorials/web-app/adx-functions-performance.md b/Tutorial/docs/tutorials/web-app/adx-functions-performance.md index 97eeace..e9621ef 100644 --- a/Tutorial/docs/tutorials/web-app/adx-functions-performance.md +++ b/Tutorial/docs/tutorials/web-app/adx-functions-performance.md @@ -150,15 +150,16 @@ Enable Query Store & alerts for: ## 7. Deployment Automation -Store function definitions as `.kql` files and apply idempotently: - -```bash -#!/usr/bin/env bash -for f in infrastructure/adx/functions/*.kql; do - echo "Applying $f"; - az kusto script create --cluster-name $CLUSTER --database-name $DB --name $(basename $f .kql) --script-content "$(cat $f)" --force-update-tag $(date +%s); -done -``` +In this project, functions are deployed by the backend CI using the authoritative Kusto scripts. Prefer adding/altering function definitions in the backend repository and let the pipeline apply them idempotently. + +If you need local validation, run them against a dev database from your admin tools, but do not hand-deploy to shared environments. + +### CI Hints + +- Store each function in its own `.kql` file with `create-or-alter` +- Group in folders by domain (e.g., `serving/brew`) +- Add lightweight unit checks (syntax validation) in CI +- Tag releases with a migration note ## 8. Checklist diff --git a/Tutorial/docs/tutorials/web-app/frontend.md b/Tutorial/docs/tutorials/web-app/frontend.md index 34b1c15..ec5d963 100644 --- a/Tutorial/docs/tutorials/web-app/frontend.md +++ b/Tutorial/docs/tutorials/web-app/frontend.md @@ -43,6 +43,8 @@ Expose environment via `.env` or config endpoint: ## 4. Fetching Embed Token +Note: Embedding is handled in the backend. The frontend typically calls a backend endpoint to retrieve the embed configuration; SDK handling may be abstracted already by platform components. + ```ts async function fetchEmbedToken(reportId: string) { const res = await fetch(`/api/powerbi/token?reportId=${reportId}` , { credentials: 'include' }); diff --git a/Tutorial/docs/tutorials/web-app/power-bi-embedding.md b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md index 404b797..7305ffa 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi-embedding.md +++ b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md @@ -1,6 +1,6 @@ --- title: Power BI Embedding & Security -summary: Embed interactive Power BI reports securely in the web-app with governance, caching, and performance best practices. +summary: Back-end managed embedding — configure workspaces, datasets, and RLS; validate filters and performance. tags: - power-bi - embedding @@ -8,136 +8,68 @@ tags: - web-app --- -# Power BI Embedding & Security - -This guide covers embedding models, authentication flows, Row Level Security (RLS), and performance tuning. - -## 1. Embedding Models - -| Scenario | Model | Notes | -|----------|-------|-------| -| Internal users (AAD) | User owns data | Uses user AAD token; per-user RLS naturally applied | -| External / multi-tenant | Service principal | App gets embed token; enforce tenant filter via RLS | -| Public showcase | Publish-to-web (avoid) | Not secure; never use for production | - -Prefer service principal + embed token generation backend-side for multi-tenant. - -## 2. Architecture - -``` -Browser -> Web-App (frontend) -> Backend (token endpoint) -> AAD -> Power BI Service - | - +--> ADX (DirectQuery via PBI service) -``` - -## 3. Backend: Generate Embed Token - -Pseudo-code: - -```python -from msal import ConfidentialClientApplication -import requests - -def get_embed_token(report_id, workspace_id, dataset_id): - app = ConfidentialClientApplication( - client_id=CLIENT_ID, - authority=f"https://login.microsoftonline.com/{TENANT_ID}", - client_credential=CLIENT_SECRET - ) - scope = ["https://analysis.windows.net/powerbi/api/.default"] - result = app.acquire_token_silent(scope, account=None) - if not result: - result = app.acquire_token_for_client(scopes=scope) - access_token = result['access_token'] - - body = { - "datasets": [{"id": dataset_id}], - "reports": [{"id": report_id}], - "targetWorkspaces": [{"id": workspace_id}] - } - r = requests.post( - "https://api.powerbi.com/v1.0/myorg/GenerateToken", - headers={"Authorization": f"Bearer {access_token}"}, - json=body - ) - return r.json()['token'] -``` - -## 4. Frontend: Embed - -```ts -import * as powerbi from 'powerbi-client'; - -const models = powerbi.models; - -function embedReport(container: HTMLElement, embedConfig: { id: string; embedUrl: string; token: string; site: string; }) { - const config: powerbi.IEmbedConfiguration = { - type: 'report', - id: embedConfig.id, - embedUrl: embedConfig.embedUrl, - accessToken: embedConfig.token, - tokenType: models.TokenType.Embed, - settings: { panes: { filters: { visible: false } } } - }; - const report = powerbi.embed(container, config) as powerbi.Report; - // Apply site filter from contextual data - report.on('loaded', async () => { - await report.setFilters([ - { - $schema: 'http://powerbi.com/product/schema#basic', - target: { table: 'SiteFilter', column: 'Site' }, - operator: 'In', - values: [embedConfig.site] - } - ]); - }); -} -``` - -## 5. Security Layers - -| Layer | Control | Purpose | -|-------|---------|---------| -| AAD App | App roles / scopes | Restrict who can call token endpoint | -| Backend | Tenant/site validation | Prevent arbitrary site injection | -| Power BI | RLS roles | Enforce data slice per user/tenant | -| Dataset | Parameterization | Ensure minimal data exposure | - -### Example RLS Role (Site Isolation) - -In Power BI Desktop > Modeling > Manage Roles: -``` -[Site] = USERPRINCIPALNAME()\"s site mapping OR LOOKUPVALUE(SiteMap[Site], SiteMap[UserPrincipalName], USERPRINCIPALNAME()) -``` -(Alternatively pass site via embed token identities.) - -## 6. Caching & Performance - -| Layer | Strategy | Notes | -|-------|----------|-------| -| Power BI | Aggregations | Combine DirectQuery (hot) + Import (warm) | -| Power BI | Incremental refresh | Partition large historical tables | -| ADX | Materialized views | Pre-aggregate heavy metrics | -| Web-App | Embed lifecycle | Reuse report instance across tabs, avoid reloading | - -### Measuring -Use Performance Analyzer in Power BI Desktop and `QueryTrace` in ADX. - -## 7. Observability - -- Log token requests (timestamp, workspace, dataset, site, user/tenant) -- Alert on unusual volume or failed token generation -- Track report load times (frontend performance marks) - -## 8. Checklist - -- [ ] Service principal least-privilege -- [ ] Embed token TTL minimal (e.g. 1h) -- [ ] RLS validated per tenant -- [ ] Site filter applied programmatically -- [ ] No hard-coded secrets in frontend - -## 9. Related +# Power BI Embedding & Security (Backend-managed) + +In this solution, embedding is implemented in the backend. You do not write frontend embedding code. Instead, you configure the embedding context and validate security and performance. + +## 1. What’s handled by the backend + +- Acquire AAD tokens (service principal or user) +- Generate Power BI embed tokens for the selected report/dataset +- Apply identity mappings (optional) for RLS +- Expose a simplified API endpoint to the web-app + +Your frontend only consumes the embed configuration and applies high-level report filters (e.g., site) when needed. + +## 2. Required Configuration + +| Setting | Where | Example | +|--------|-------|---------| +| Workspace Id | Backend config/secret store | `00000000-0000-0000-0000-000000000000` | +| Report Id | Backend config | `11111111-1111-1111-1111-111111111111` | +| Dataset Id | Backend config | `22222222-2222-2222-2222-222222222222` | +| AAD App (Client Id/Secret) | Secret store | Azure Key Vault | +| Allowed Sites / Tenants | App settings | `LYON_01, BERLIN_02` | + +## 3. RLS and Identity Mapping + +- Define roles in the Power BI dataset (e.g., TenantIsolation, SiteIsolation) +- Backend maps the calling user/tenant/site to a role or passes identities in the embed token +- Keep the mapping source-of-truth server-side (e.g., DB or configuration) + +Checklist: +- [ ] RLS roles exist and are tested in Desktop (View as) +- [ ] Backend mapping enforces least privilege +- [ ] No ability to request arbitrary site via client-only params + +## 4. Frontend Responsibilities + +- Request the embed configuration from the backend endpoint +- Render the report with the provided embed URL/token (SDK usage is abstracted by the platform) +- Apply contextual filters (e.g., default site) via backend-approved mechanisms +- Do NOT store secrets nor generate tokens client-side + +## 5. Operational Runbook + +- Rotate client secrets regularly (Key Vault) +- Monitor token issuance failures and latency +- Track report load times and error rates + +## 6. Troubleshooting + +| Symptom | Likely cause | Action | +|--------|--------------|--------| +| Report loads but shows empty data | RLS filtering out data | Validate role mapping and site passed | +| 403 on embed token | AAD app permission or workspace access | Re-check service principal access | +| Slow visuals | Query heavy / dataset design | See performance section below | + +## 7. Performance + +- Prefer aggregated tables or ADX serving functions returning summarized data +- Validate DirectQuery vs Import mix; consider aggregations and incremental refresh +- Prewarm via scheduled refresh if suitable + +## 8. Related - [Parameters & Reuse](./power-bi-parameters.md) - [ADX Functions & Performance](./adx-functions-performance.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi.md b/Tutorial/docs/tutorials/web-app/power-bi.md index 531abe8..3376d82 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi.md +++ b/Tutorial/docs/tutorials/web-app/power-bi.md @@ -33,12 +33,12 @@ ADX (Functions) -> Power BI Dataset -> Report -> Embedded in Web-App ## 4. Quick Start Checklist -- [ ] Create ADX serving function `Dashboard_Fact` with parameters +- [ ] Configure backend with Workspace/Report/Dataset IDs and AAD app - [ ] Define Power BI parameters (Site, TimeWindowHours, Product) -- [ ] Build visuals from parameterized query -- [ ] Implement embed token backend endpoint -- [ ] Add site filter application on load -- [ ] Validate RLS & parameter variations +- [ ] Build visuals from parameterized ADX function +- [ ] Validate embed token endpoint and RLS role mapping +- [ ] Apply default site filter from frontend context +- [ ] Validate performance (Desktop Performance Analyzer + ADX metrics) ## 5. Subpages @@ -49,8 +49,8 @@ ADX (Functions) -> Power BI Dataset -> Report -> Embedded in Web-App | Issue | Mitigation | |-------|------------| -| Slow initial load | Pre-warm dataset via REST refresh | -| Heavy visuals | Aggregate in ADX function | +| Slow initial load | Pre-warm dataset via scheduled refresh | +| Heavy visuals | Aggregate in ADX function | | Data bloat | Remove unused columns & tables | ## 7. Next Steps From 067c738fc8d9fcf21b4b89899bfc2fdb4d5b873d Mon Sep 17 00:00:00 2001 From: agrondaka Date: Thu, 9 Oct 2025 11:14:37 +0200 Subject: [PATCH 3/3] ADX and PBI md files adapted to reflect build-my-app specific unboarding process --- .../docs/tutorials/web-app/adx-database.md | 144 +++++++++++++++--- .../tutorials/web-app/power-bi-embedding.md | 104 ++++++++----- .../tutorials/web-app/power-bi-parameters.md | 124 ++++++++------- Tutorial/docs/tutorials/web-app/power-bi.md | 102 +++++++++---- 4 files changed, 330 insertions(+), 144 deletions(-) diff --git a/Tutorial/docs/tutorials/web-app/adx-database.md b/Tutorial/docs/tutorials/web-app/adx-database.md index 382b5d5..31aec07 100644 --- a/Tutorial/docs/tutorials/web-app/adx-database.md +++ b/Tutorial/docs/tutorials/web-app/adx-database.md @@ -17,23 +17,39 @@ This page introduces how the web-app consumes data from Azure Data Explorer (ADX - Native integration with Power BI (DirectQuery / import) and Azure ecosystem - Scalable ingestion & retention policies -## 2. High-Level Flow +## 2. High-Level Flow (Project-specific) ``` -Sources -> Ingestion (EventHub / Blob / ADX LightIngest) -> Raw Tables -> Update Policies / Materialized Views -> Serving Functions -> Power BI / API +Sources (onboarding data / storage) + └─> ETL (Cosmo Tech runner templates) + - code/run_templates/etl_with_local_file + - code/run_templates/etl_with_azure_storage + └─> Backend pipelines (Solution_etl.yaml, Create.kql) + - Apply ADX tables/mappings/policies + - Register update policies / materialized views + - Create serving functions +→ ADX Raw / Prepared Tables +→ Serving Functions +→ Power BI (embedded via backend) ``` -## 3. Core Concepts +Notes: +- The backend is the source of truth for ADX artifacts via `Create.kql` and CI/CD. +- Onboarding data is optional and used only in non-prod to validate end-to-end. -| Concept | Description | Example | -|---------|-------------|---------| -| Table | Physically stored records | `BrewerEvents` | -| Ingestion Mapping | Column mapping from source payload | JSON mapping for brewer events | -| Update Policy | Auto-populate derived table | Raw -> Cleaned table | -| Materialized View | Pre-computed aggregation | 15-min brew KPIs | -| Function | Reusable query logic | `GetBrewEvents()` | +## 3. Core Concepts (in this project) -## 4. Recommended Table Design +| Concept | Where it’s defined | Example | +|---------|---------------------|---------| +| Tables & Mappings | Backend `Create.kql` | `BrewerEvents`, JSON/CSV mappings | +| Update Policy / MV | Backend `Create.kql` | Raw -> Cleaned / 15-min aggregates | +| Serving Function | Backend KQL scripts | `Dashboard_Fact(...)` | + +Avoid manual table/function creation in shared envs—propose changes via PR to backend scripts. + +## 4. Schema Source of Truth & Design Patterns + +`Create.kql` dictates the schema. Use these patterns when proposing changes: | Column Type | Purpose | Naming | |-------------|---------|--------| @@ -42,17 +58,21 @@ Sources -> Ingestion (EventHub / Blob / ADX LightIngest) -> Raw Tables -> Update | Measures | Numeric metrics | `DurationSeconds`, `Temperature` | | Status / Type | Categorical attributes | `EventType` | -Partitioning automatically by ingestion time—optimize retention with policies: +Retention example (applied by backend): ```kusto -.alter table BrewerEvents policy retention softdelete = 30d"); +.alter table BrewerEvents policy retention softdelete = 30d ``` -## 5. Ingestion Tips +## 5. Ingestion Tips (Project-specific) -- Batch ingestion for historical loads; streaming/EventHub for near real-time -- Validate schema with a staging table before merging -- Compress JSON payloads or prefer CSV/Parquet for large volume +- For dev/onboarding data, use the provided ETL templates to generate the source dataset: + - Local file template: `Sources/onboarding-brewery-solution/code/run_templates/etl_with_local_file/etl.py` + - Azure Storage template: `Sources/onboarding-brewery-solution/code/run_templates/etl_with_azure_storage/etl.py` + - Sample inputs live in `Sources/onboarding-brewery-solution/Simulation/Resource/scenariorun-data/` and, when zipped, under `reference/Nodes/*.csv` and `reference/Edges/*.csv`. +- ADX ingestion is executed by the backend pipeline using `Create.kql` mappings. Do not ingest directly into curated tables. If you need to validate a schema change, ingest into a staging table defined in `Create.kql` and let update policies/materialized views populate the serving tables. +- Keep file formats aligned with the repository (CSV for Nodes/ and Edges/). For larger volumes in higher environments, consider Parquet in storage and update the ingestion mapping in `Create.kql` accordingly. +- For near real-time flows, rely on backend-managed connectors and orchestration defined under `Sources/onboarding-brewery-solution/API/` (e.g., `Solution_etl.yaml`) rather than ad-hoc ingestion; coordinate with backend to register any new sources. ## 6. Schema Provisioning via Backend @@ -77,15 +97,19 @@ Benefits: ## 8. Governance & Source Control -Store Kusto artifacts (`tables.kql`, `functions/*.kql`, `policies.kql`) under version control. Deploy via CI using Azure CLI or ARM scripts. +- Store Kusto artifacts in backend repo alongside `Create.kql`. +- Use PRs and CI to apply changes to dev first, then promote. ## 9. Observability Use built-in commands: ```kusto .show table BrewerEvents details +.show functions .show queries | where StartedOn > ago(15m) .show operations | where StartedOn > ago(1h) and Status != 'Completed' +// If using managed ingestion into tables, check ingestion state +.show ingestion failures | where Database == '' and Table == 'BrewerEvents' and Timestamp > ago(1h) ``` ## 10. Next @@ -93,3 +117,87 @@ Use built-in commands: Proceed to implementing reusable functions and performance tuning. > See: [Functions & Performance](./adx-functions-performance.md) + +## 11. Junior Onboarding: ADX Setup (Step-by-step) + +This walkthrough assumes: +- You have access to the Azure subscription and target ADX cluster/database +- Backend owns deployment of schema via `Create.kql` +- You have the onboarding repo checked out at `Sources/onboarding-brewery-solution` + +### A. Prerequisites (local) + +- Azure CLI (logged in): `az login` +- ADX permissions: can view database and run read-only queries +- Coordinates from your team: + - Cluster URI: `https://..kusto.windows.net` + - Database name: `` + - Backend pipeline path for `Create.kql` (read-only) + +### B. Provision (done by Backend) + +Backend CI will apply `Create.kql` to provision tables, mappings, policies and serving functions. As a junior, you don’t run schema changes yourself. + +What you should do: verify that objects exist after deployment. + +- In Kusto Explorer or Web UI, run: +```kusto +.show tables +.show functions +``` +Confirm presence of domain tables (e.g., `BrewerEvents`) and serving functions (e.g., `Dashboard_Fact` or similar per project naming). + +### C. Load Sample Data (optional, non-prod) + +If the environment is empty and your team allows sample data loads, use the onboarding dataset. + +Location in repo: +- `Sources/onboarding-brewery-solution/Simulation/Resource/scenariorun-data/` + - `Bar.csv`, `Customer.csv`, `arc_to_Customer.csv` (example graph data) + +Typical approaches (confirm which is permitted): +- Use backend ETL runner templates under `code/run_templates/` (e.g., `etl_with_local_file`) — usually interacts with Cosmo Tech API & datasets +- Or a one-off ADX ingestion (dev only) via Web UI upload to a staging table that matches `Create.kql` schemas + +Ask your mentor which path to use in your environment. + +### D. Sanity Queries (read-only) + +Run simple queries to check data landed and functions work: +```kusto +// Replace names with your actual tables/functions +BrewerEvents | take 5 +BrewerEvents | summarize count() by Site +Dashboard_Fact(datetime(ago(12h)), datetime(now()), 'Lyon_01', '') | take 10 +``` + +Expect a few rows and reasonable counts. If empty: +- Confirm ingestion happened +- Check RLS/filters (if querying through Power BI) + +### E. Power BI Integration Check + +This project embeds Power BI via backend. Your tasks: +- Confirm the dataset’s main query references the serving function(s) +- Validate that parameter values (Site, TimeWindowHours) match what the function expects +- Open the embedded dashboard and verify visuals populate for a known site/time window + +If visuals are blank, coordinate with backend to check RLS mapping and embed token scopes. + +### F. Change Requests + +When you need schema or serving function changes: +1. Discuss with backend owner +2. Propose edits in `Create.kql` or function `.kql` files +3. Open a PR → CI applies to dev → validate → promote + +Avoid ad-hoc manual changes in shared environments. + +### G. What to Document During Onboarding + +- Cluster/Database you used +- Tables/functions you validated +- Screenshots of the sanity queries results +- Any discrepancies found & who you contacted + +This helps the next newcomer follow the same path. diff --git a/Tutorial/docs/tutorials/web-app/power-bi-embedding.md b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md index 7305ffa..b96828a 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi-embedding.md +++ b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md @@ -12,7 +12,13 @@ tags: In this solution, embedding is implemented in the backend. You do not write frontend embedding code. Instead, you configure the embedding context and validate security and performance. -## 1. What’s handled by the backend +## 0. Onboarding Flow Snapshot + +``` +Clone repos → Verify permissions (variables.yaml / gen_adx_permissions.sh) → ADX bootstrap (00-Initialization.kql via backend) → Publish Example.pbix → Backend embed endpoint test → Open web-app → Validate RLS & filters +``` + +## 1. What’s handled by the backend (Recap) - Acquire AAD tokens (service principal or user) - Generate Power BI embed tokens for the selected report/dataset @@ -21,55 +27,79 @@ In this solution, embedding is implemented in the backend. You do not write fron Your frontend only consumes the embed configuration and applies high-level report filters (e.g., site) when needed. -## 2. Required Configuration +## 2. Required Configuration (From `variables.yaml`, `workspace.yaml`, `solution.yaml`) + +| Setting / Element | Source File | YAML Path | Notes | +|-------------------|-------------|-----------|-------| +| Power BI Workspace Name | `variables.yaml` | `powerbi_workspace_name` | Target workspace for Example report | +| Report binding | `workspace.yaml` | `sidecars.azure.powerbi.workspace.reports` | Ensures Example.pbix deployed with parameters | +| Report parameters (ADX cluster/db) | `workspace.yaml` | `reports[].parameters` | Provide `ADX_Cluster`, `ADX_Database` values | +| ADX init script | `workspace.yaml` | `sidecars.azure.adx.scripts` | Executes `00-Initialization.kql` | +| Event Hub connectors | `workspace.yaml` | `sidecars.azure.eventhub.connectors` | Stream simulation outputs into ADX tables | +| Dynamic Filters | `workspace.yaml` | `webApp.options.charts.scenarioView.dynamicFilters` | Auto apply `LastSimulationRun` | +| Run Templates | `solution.yaml` | `spec.runTemplates` | Provide simulation & ETL context for data feeding report | + +If a principal is missing, update `variables.yaml` in a PR and re-run provisioning scripts. + +## 3. Permission Bootstrap + +Use (if provided) `gen_adx_permissions.sh` and backend automation to ensure ADX and Power BI roles are applied. Confirm in ADX: +```kusto +.show database principals +``` +And in Power BI workspace (Admin portal) that the service principal has Admin rights. -| Setting | Where | Example | -|--------|-------|---------| -| Workspace Id | Backend config/secret store | `00000000-0000-0000-0000-000000000000` | -| Report Id | Backend config | `11111111-1111-1111-1111-111111111111` | -| Dataset Id | Backend config | `22222222-2222-2222-2222-222222222222` | -| AAD App (Client Id/Secret) | Secret store | Azure Key Vault | -| Allowed Sites / Tenants | App settings | `LYON_01, BERLIN_02` | +## 4. RLS & Identity Mapping (Dataset) -## 3. RLS and Identity Mapping +If the Example PBIX includes RLS roles (e.g., SiteIsolation), verify them in Desktop: Modeling > Manage Roles. If not present yet, coordinate with analytics engineer to add roles BEFORE wide distribution. -- Define roles in the Power BI dataset (e.g., TenantIsolation, SiteIsolation) -- Backend maps the calling user/tenant/site to a role or passes identities in the embed token -- Keep the mapping source-of-truth server-side (e.g., DB or configuration) +Recommended role pattern: +- `TenantIsolation` (filters organization/workspace id if multi-tenant) +- `ScenarioScope` (limits to allowed scenario set) -Checklist: -- [ ] RLS roles exist and are tested in Desktop (View as) -- [ ] Backend mapping enforces least privilege -- [ ] No ability to request arbitrary site via client-only params +Backend maps user/app context → role(s) when generating embed token (or injects effective identity). -## 4. Frontend Responsibilities +## 5. Validation Steps -- Request the embed configuration from the backend endpoint -- Render the report with the provided embed URL/token (SDK usage is abstracted by the platform) -- Apply contextual filters (e.g., default site) via backend-approved mechanisms -- Do NOT store secrets nor generate tokens client-side +1. Publish `Example.pbix` to workspace (`powerbi_workspace_name`). +2. Call embed endpoint (e.g., `GET /api/analytics/embed/report/{id}`) and capture JSON (id, embedUrl, token expiry). +3. Open web-app dashboard; verify: + - Data loads without manual sign-in (service principal path) + - Only allowed scenarios/probes appear + - Switching TimeWindowHours (if parameterized) re-queries ADX +4. In ADX, list recent queries: +```kusto +.show queries | where StartedOn > ago(10m) | where Text has "GetMeasures" or Text has "GetScenarios" +``` +Confirm calls originate with expected principal. +5. Measure render time (<5s for main visuals) using browser dev tools. -## 5. Operational Runbook +## 6. Troubleshooting (Extended) -- Rotate client secrets regularly (Key Vault) -- Monitor token issuance failures and latency -- Track report load times and error rates +| Symptom | Likely Cause | Diagnostic | Fix | +|---------|--------------|-----------|-----| +| Empty visuals | RLS over-filtering | View as role in Desktop | Adjust role filter / identity mapping | +| 403 embed call | Missing workspace permission | Power BI workspace access panel | Add service principal / user | +| Slow queries | Function expanding large dynamic fields | Inspect ADX query text | Add selective projection / MV | +| Token expires too soon | Short TTL | Backend token config | Increase within security guidelines | +| Wrong dataset | Misconfigured IDs | Compare variables.yaml vs embed response | Correct backend config | -## 6. Troubleshooting +## 7. Performance (Applied) -| Symptom | Likely cause | Action | -|--------|--------------|--------| -| Report loads but shows empty data | RLS filtering out data | Validate role mapping and site passed | -| 403 on embed token | AAD app permission or workspace access | Re-check service principal access | -| Slow visuals | Query heavy / dataset design | See performance section below | +- Monitor `GetMeasures` query duration distribution (p50 < 2s target in dev) using `.show queries`. +- If dynamic expansion (bag_unpack) dominates cost, pre-flatten heavy fields in a materialized view and reference that in functions. +- Schedule dataset refresh only if Import mode; for DirectQuery ensure aggregations are defined before scaling. -## 7. Performance +## 8. Operational Runbook (Daily / Release) -- Prefer aggregated tables or ADX serving functions returning summarized data -- Validate DirectQuery vs Import mix; consider aggregations and incremental refresh -- Prewarm via scheduled refresh if suitable +| Task | Frequency | Owner | +|------|-----------|-------| +| Check failed embed calls (5xx / 4xx) | Daily | Backend Ops | +| Review ADX heavy queries (>10s) | Daily | Data Eng | +| Validate RLS after adding new users | On change | Platform Admin | +| Refresh Example PBIX (if Import) | Release / Data change | BI Dev | -## 8. Related +## 9. Related - [Parameters & Reuse](./power-bi-parameters.md) - [ADX Functions & Performance](./adx-functions-performance.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi-parameters.md b/Tutorial/docs/tutorials/web-app/power-bi-parameters.md index 01c099f..ffa4885 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi-parameters.md +++ b/Tutorial/docs/tutorials/web-app/power-bi-parameters.md @@ -12,98 +12,104 @@ tags: Parametrization lets you deploy the same `.pbix` to multiple tenants, sites, or time scopes without cloning logic. It also reduces dataset size by pushing filters down into ADX. -## 1. Parameter Strategy +## 1. Parameter Strategy (Project Context) -| Parameter | Purpose | Source | Example | -|-----------|---------|--------|---------| -| Site | Scope data to a factory/site | Web-app context / user claims | `Lyon_01` | -| TimeWindowHours | Rolling horizon | UI control (slider) | `12` | -| Product | Optional product filter | Slicer | `Pilsner` | -| RefreshMode | Switch raw vs aggregated | Hidden toggle | `"auto"` / `"historical"` | +In this training setup, parameters align with serving functions created in `00-Initialization.kql` (e.g., `GetScenarios`, `GetMeasures`). Rather than many separate filters, we focus on a small stable set: -Keep parameters minimal; prefer enumerations (domains) when possible for caching efficiency. +| Parameter | Purpose | Backing Function Usage | Source | +|-----------|---------|------------------------|--------| +| Site (or Workspace scope) | Narrow scenario / probe data | Filters Scenario/Probe tables (if multi-site) | Derived from workspace / user claim | +| TimeWindowHours | Limit recency for probe facts | Compute datetime range in Kusto (`ago(TimeWindowHours * 1h)`) | Frontend control | +| ProbeType | Focus a specific measure set | Passed to `GetMeasures(Probe)` | Report slicer | +| ScenarioState (optional) | Filter scenarios by validation state | Filter in `GetScenarios()` pipeline | Report slicer | -## 2. Creating Parameters in Power BI Desktop +Keep parameters minimal; they must map cleanly to function parameters or filters without post-processing. -1. Transform Data > Manage Parameters > New -2. Name: `Site`; Type: Text; Suggested Values: List (optional) -3. Default Value: `Lyon_01`; Current Value: `Lyon_01` -4. Repeat for others. +### Mapping to Configuration Files -> Tip: Use Text/Number/TrueFalse types—avoid DateTime parameters if you can compute datetimes inside the query from `now()` plus duration. +| Concept | Declared In | How It Appears In Report | +|---------|-------------|--------------------------| +| Run template parameters (e.g., CustomersCount) | `project/solution.yaml` (`parameters` & `parameterGroups`) | May influence simulation outputs surfaced via `GetScenarios()` | +| Dynamic filter (last run) | `project/workspace.yaml` (`dynamicFilters`) | Automatically filters report to latest simulation run (no manual parameter) | +| ADX connection parameters | `workspace.yaml` powerbi.workspace.reports.parameters (`ADX_Cluster`, `ADX_Database`) | M parameters bound during deployment | +| Probe / scenario functions | `00-Initialization.kql` | Data sources in Power BI queries | -## 3. Using Parameters in Kusto M Query +## 2. Creating Parameters in Power BI Desktop (Applied) + +Use only parameters required for query shaping. Example minimal set: `TimeWindowHours`. +If Site is handled via RLS or embedding context, skip exposing it as a manual parameter. + +Steps: +1. Transform Data > Manage Parameters > New > `TimeWindowHours` (Number, default 12) +2. (Optional) Add `ProbeType` as Text if you want a configurable default instead of a slicer. +3. If cluster/database differ per environment, add `ADXCluster` & `ADXDatabase` parameters; populate from deployment script or Power BI deployment pipeline rules. + +## 3. Using Parameters in Kusto M Query (GetMeasures) ```m let - Site = Text.From(Parameter_Site), - Hours = Number.ToText(Parameter_TimeWindowHours), - Product = Text.From(Parameter_Product), Cluster = Parameter_ADXCluster, Database = Parameter_ADXDatabase, + Hours = Number.ToText(Parameter_TimeWindowHours), + // Example: restrict to last N hours then expand probes Query = - "Dashboard_Fact(datetime(" & DateTimeZone.ToText(DateTimeZone.UtcNow() - #duration(0, Number.From(Hours),0,0), "yyyy-MM-dd HH:mm:ss") & "), datetime(" & DateTimeZone.ToText(DateTimeZone.UtcNow(), "yyyy-MM-dd HH:mm:ss") & "), '" & Site & "', '" & Product & "')" + "GetMeasures('LatencyProbe') | where ProbeDate > ago(" & Hours & "h)" , Source = Kusto.Contents(Cluster, Database, Query) in Source ``` -Abstract cluster & database into parameters too for full reuse. - -## 4. Dynamic Binding from Web-App - -When embedding, you can set report-level filters or parameter values via the JavaScript SDK. - -### Option A: Embedding + Slicers -Populate hidden slicer tables with a single value coming from the web-app. - -```ts -const site = appContext.site; // e.g. from JWT -report.setFilters([ - { - $schema: "http://powerbi.com/product/schema#basic", - target: { table: "SiteFilter", column: "Site" }, - operator: "In", - values: [site] - } -]); +For scenarios list: +```m +let + Cluster = Parameter_ADXCluster, + Database = Parameter_ADXDatabase, + Query = "GetScenarios()" +, + Source = Kusto.Contents(Cluster, Database, Query) +in + Source ``` -### Option B: Query Parameters Through DirectQuery (Preview-dependent) -If using DirectQuery, you can rewrite queries to reference a single-row parameter table that the app updates via Push API. +## 4. Dynamic Binding (Embedding Context) + +In this platform the backend embedding already injects user/workspace context. Frontend may only adjust slicers (e.g., ProbeType). Avoid duplicating Site/Workspace filters if RLS or function logic already enforces them. -## 5. Dataset Slimming +## 5. Dataset Slimming (Project-Specific) -| Technique | Effect | Notes | -|-----------|--------|-------| -| Parameter pushdown | Smaller scans | Express all filters as function parameters | -| Summarize in ADX | Less row transfer | Provide aggregated KPIs, not raw events | -| Remove unused columns | Better compression | Audit visuals & fields list | -| Incremental refresh | Faster partition processing | Combine with time parameters | +| Technique | Application Here | Notes | +|-----------|------------------|-------| +| Function parameters | Limit probe window | Use `ago()` in function or query | +| Pre-aggregation | Add MV on `ProbesMeasures` if performance degrades | Requires backend PR | +| Column pruning | Remove unused dynamic fields | Avoid expanding entire `FactsRaw` unless needed | +| Incremental refresh | If converting to Import mode for large history | Partition by `ProbeDate` | ## 6. Testing Reusability -Create a matrix of environment x site values and script automated validation: +Instead of exporting PBIX per site, validate across workspace contexts (if multi-workspace deployment) and time windows: ```bash -for site in Lyon_01 Berlin_02; do - pbicli reports export --report "BrewDashboard" --workspace $WS --file export-$site.pbix \ - --parameter Site=$site || exit 1 - echo "Validated $site" +for hours in 6 12 24; do + echo "Testing TimeWindowHours=$hours" + # Use pbicli or REST to rebind parameter (pseudo-example) + pbicli datasets update-params --dataset-id $DATASET --parameters TimeWindowHours=$hours || exit 1 + sleep 5 + # Optionally trigger a refresh if Import mode + echo "OK" done ``` -## 7. Checklist +## 7. Checklist (Adapted) -- [ ] All Power BI queries reference a single root ADX serving function -- [ ] Parameters documented & minimal -- [ ] No visual applies redundant filters already in function -- [ ] Dataset size < target (e.g. 100 MB) / row count limited -- [ ] Test matrix executed for each release +- [ ] Queries call only approved serving functions (GetMeasures/GetScenarios) +- [ ] Parameter list minimal & documented +- [ ] No duplicated filter logic in visuals and functions +- [ ] Dataset refresh / DirectQuery latency within targets +- [ ] Performance test across TimeWindowHours values ## 8. Next -Proceed to embedding details & security considerations. +Proceed to backend embedding & security validation. > See: [Embedding & Security](./power-bi-embedding.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi.md b/Tutorial/docs/tutorials/web-app/power-bi.md index 3376d82..dba577e 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi.md +++ b/Tutorial/docs/tutorials/web-app/power-bi.md @@ -22,39 +22,81 @@ This section introduces how Power BI integrates with the solution for interactiv ADX (Functions) -> Power BI Dataset -> Report -> Embedded in Web-App ``` -## 3. Key Building Blocks - -| Component | Description | Reference | -|-----------|-------------|-----------| -| Parameterized Functions | Source for DirectQuery / Import | ADX Functions & Performance | -| Power BI Parameters | Filter scoping & reuse | Parameters & Reuse | -| Embed Token Service | Secure report embedding | Embedding & Security | -| RLS Roles | Tenant / site isolation | Embedding & Security | - -## 4. Quick Start Checklist - -- [ ] Configure backend with Workspace/Report/Dataset IDs and AAD app -- [ ] Define Power BI parameters (Site, TimeWindowHours, Product) -- [ ] Build visuals from parameterized ADX function -- [ ] Validate embed token endpoint and RLS role mapping -- [ ] Apply default site filter from frontend context -- [ ] Validate performance (Desktop Performance Analyzer + ADX metrics) - -## 5. Subpages - -- [Parameters & Reuse](./power-bi-parameters.md) -- [Embedding & Security](./power-bi-embedding.md) - -## 6. Performance Tips (Preview) +## 3. Key Building Blocks (Extended) + +| Component | Source File | Purpose | +|-----------|------------|---------| +| Solution metadata | `project/solution.yaml` | Declares run templates (Example, ETL) & parameters feeding scenarios | +| Workspace definition | `project/workspace.yaml` | Defines Power BI workspace, ADX scripts, Event Hub connectors, webApp embedding options | +| ADX Init Script | `project/adx/scripts/00-Initialization.kql` | Creates tables & functions consumed by report | +| Power BI Report | `project/powerbi/Example.pbix` | Visualization layer referencing serving functions | +| Permissions | `variables.yaml` (`powerbi_permissions`, `adx_permissions`) | Grants principals access | +| Dynamic Filters | `workspace.yaml` (`webApp.options.charts.scenarioView.dynamicFilters`) | Auto-apply context filters (e.g., last simulation run) | + +## 4. Quick Start Checklist (Onboarding Flow) + +Follow these in order when onboarding: + +1. Clone repos: + - `onboarding-brewery-solution` (simulation + ETL templates) + - `build-my-app-training-workspaces` (workspace/Power BI + ADX bootstrap scripts) +2. Review `build-my-app-training-workspaces/variables.yaml` and note: + - `organization_id`, `workspace_key`, `powerbi_workspace_name` + - Power BI & ADX principal entries (`powerbi_permissions`, `adx_permissions`) +3. Ensure your user (or service principal) has been added under `powerbi_permissions` and `adx_permissions` (if not, submit a PR or request an admin update and re-run provisioning scripts). +4. Run ADX initialization (if dev environment is fresh): + - Execute `project/adx/scripts/00-Initialization.kql` via the approved backend pipeline or Kusto Script deployment job (never hand-edit prod). +5. Import / verify `Example.pbix` into the Power BI workspace named in `variables.yaml` if not already published. (Admins only; otherwise just confirm it's present.) +6. Open the report and validate required dataset parameters (Site, TimeWindowHours, etc.) exist or are represented via filters / serving functions (e.g., `GetMeasures`, `GetScenarios`). +7. Confirm backend embedding endpoint returns an embed token for the report (use internal API doc / curl if available). +8. Load the web-app; verify visuals render for default site and switch site/time context successfully. +9. Capture performance: measure first visual load (<5s target) & check ADX query durations for `GetMeasures` and `GetScenarios` functions. +10. Log findings in onboarding notes (successes, missing permissions, performance outliers). + +## 5. Key Building Blocks + +| Component | Repository / File | Purpose | +|-----------|------------------|---------| +| Workspace Variables | `build-my-app-training-workspaces/variables.yaml` | IDs, permissions, workspace naming | +| ADX Bootstrap | `project/adx/scripts/00-Initialization.kql` | Tables, mappings, baseline functions | +| Example Report | `project/powerbi/Example.pbix` | Starter visuals tied to functions | +| Serving Functions | Defined in KQL bootstrap | `GetScenarios`, `GetMeasures`, etc. | +| ETL Templates | `onboarding-brewery-solution/code/run_templates/*` | Populate dataset / scenario data | + +## 6. Performance Tips (Contextual) | Issue | Mitigation | |-------|------------| -| Slow initial load | Pre-warm dataset via scheduled refresh | -| Heavy visuals | Aggregate in ADX function | -| Data bloat | Remove unused columns & tables | +| Slow `GetMeasures` | Pre-aggregate or reduce `mv-expand` usage | +| Large dataset refresh | Limit time window in parameters (TimeWindowHours) | +| Permission errors | Re-check `variables.yaml` and re-run permission scripts | ## 7. Next Steps -Proceed to parameters for full reuse across sites. - -> See: [Parameters & Reuse](./power-bi-parameters.md) +Continue with parameter configuration: see [Parameters & Reuse](./power-bi-parameters.md) + +> After parameters and embedding validation, proceed to RLS & operational governance in [Embedding & Security](./power-bi-embedding.md) + +## 8. Unified Onboarding Checklist + +| Step | Action | File / Command | Done | +|------|--------|----------------|------| +| 1 | Clone repos | git clone both repos | [ ] | +| 2 | Inspect variables | Open `variables.yaml` | [ ] | +| 3 | Confirm PB workspace name | `powerbi_workspace_name` in `variables.yaml` | [ ] | +| 4 | Check solution run templates | `solution.yaml` runTemplates section | [ ] | +| 5 | Review workspace report binding | `workspace.yaml` powerbi.workspace.reports | [ ] | +| 6 | Verify ADX scripts referenced | `workspace.yaml` adx.scripts + `00-Initialization.kql` | [ ] | +| 7 | Ensure principals present | `powerbi_permissions` / `adx_permissions` | [ ] | +| 8 | Provision / sync backend | Run pipeline / deployment process | [ ] | +| 9 | Publish or confirm `Example.pbix` | Power BI Service | [ ] | +| 10 | Test embed endpoint | API call (embed token) | [ ] | +| 11 | Open web-app & validate visuals | Browser | [ ] | +| 12 | Run ADX sanity queries | `.show tables`, `GetScenarios()` | [ ] | +| 13 | Check dynamic filter (last run) | `workspace.yaml` dynamicFilters config | [ ] | +| 14 | Performance sanity | Measure load & ADX query p50 | [ ] | +| 15 | Document findings | Onboarding notes | [ ] | + +## 9. Next Steps + +Proceed to parameter specifics: [Parameters & Reuse](./power-bi-parameters.md) and embedding governance: [Embedding & Security](./power-bi-embedding.md)