feat(api): add findings severity timeseries endpoint #9363
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
component/api
review-django-migrations
Contributor
|
✅ All necessary |
Contributor
|
✅ Conflict Markers Resolved All conflict markers have been successfully resolved in this pull request. |
Contributor
🔒 Container Security ScanImage: 📊 Vulnerability Summary
3 package(s) affected
|
…ys of historical data
AdriiiPRodri
force-pushed
the
PROWLER-25-finding-severity-over-time-component-api
branch
from
bfda570 to
22c6a6a
Compare
Contributor
Author
SummaryThis PR implements the backend support for the "Finding Severity Over Time" chart component in the UI. The chart displays the evolution of failed findings by severity level over a configurable time range. Key Implementation Details
API Response ExampleRequest: Response: {
"data": [
{
"type": "findings-severity-over-time",
"id": "2025-12-01",
"attributes": {
"date": "2025-12-01",
"critical": 0,
"high": 0,
"medium": 0,
"low": 0,
"informational": 0,
"muted": 0
}
},
{
"type": "findings-severity-over-time",
"id": "2025-12-02",
"attributes": {
"date": "2025-12-02",
"critical": 0,
"high": 0,
"medium": 2,
"low": 25,
"informational": 0,
"muted": 0
}
}
],
"meta": {
"version": "v1"
}
}Performance EvidenceTest Data PopulationCreated test data with 100 providers and 365 days of history (36500 rows): Query Performance & Index UsageAll three query patterns were tested with proper index usage: 1. No Filters (Most Common)SELECT "daily_findings_severity"."date",
SUM("daily_findings_severity"."critical") AS "critical",
SUM("daily_findings_severity"."high") AS "high",
SUM("daily_findings_severity"."medium") AS "medium",
SUM("daily_findings_severity"."low") AS "low",
SUM("daily_findings_severity"."informational") AS "informational",
SUM("daily_findings_severity"."muted") AS "muted"
FROM "daily_findings_severity"
INNER JOIN "providers" ON ("daily_findings_severity"."provider_id" = "providers"."id")
WHERE (NOT "providers"."is_deleted"
AND "daily_findings_severity"."date" >= '2025-11-01'
AND "daily_findings_severity"."date" <= '2025-12-01'
AND "daily_findings_severity"."tenant_id" = '12646005-9067-4d2a-a098-8bb378604362')
GROUP BY "daily_findings_severity"."date"
ORDER BY "daily_findings_severity"."date" ASC
Query Plan: 2. Provider ID Filter-- filter[provider_id]=b440ccec-8865-4c3f-b079-13c9939ebbfb
Query Plan: 3. Provider Type Filter-- filter[provider_type__in]=aws,azure
Backfill Migration EvidenceTested the backfill migration with 1000 providers and 60 days of scan history: Worker Task EvidenceThe |
…ndings severity backfill task
Contributor
Author
Backfill examplefrom datetime import timedelta
from django.utils import timezone
from api.models import Scan, StateChoices, DailyFindingsSeverity
from tasks.tasks import backfill_daily_findings_severity_task
# Get scan IDs that already have backfill data (to skip them)
already_backfilled = set(
DailyFindingsSeverity.objects.using("admin")
.values_list("scan_id", flat=True)
)
print(f"Found {len(already_backfilled)} scans already backfilled")
# Get completed scans from the last 90 days, excluding already backfilled
cutoff = timezone.now() - timedelta(days=90)
scans = Scan.objects.using("admin").filter(
state=StateChoices.COMPLETED,
completed_at__gte=cutoff,
).exclude(
id__in=already_backfilled
).values_list("tenant_id", "id")
# Verify how many scans will be backfilled
print(f"Found {scans.count()} scans to backfill")
# Queue backfill tasks in batches
for tenant_id, scan_id in scans:
backfill_daily_findings_severity_task.delay(
tenant_id=str(tenant_id),
scan_id=str(scan_id),
)Query |
Contributor
Author
Queries review directly to the db |
vicferpoy
requested changes
Dec 4, 2025
AdriiiPRodri
force-pushed
the
PROWLER-25-finding-severity-over-time-component-api
branch
2 times, most recently
from
21bca44 to
f8ddcdb
Compare
AdriiiPRodri
force-pushed
the
PROWLER-25-finding-severity-over-time-component-api
branch
from
f8ddcdb to
93360c4
Compare
vicferpoy
approved these changes
Dec 5, 2025
AdriiiPRodri
deleted the
PROWLER-25-finding-severity-over-time-component-api
branch
13 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
This PR implements the backend support for the "Finding Severity Over Time" chart component in the UI. The chart will display the evolution of failed findings by severity level over a configurable time range.
The main challenge was performance: the existing
ScanSummarytable has a lot of rows in production, making it unsuitable for direct querying on every page load.Description
New aggregation table
DailyFindingsSeverityprovider_typeto avoid JOINs when filtering by cloud provider typeNew Celery task
update_daily_findings_severity_taskruns after each scan completesScanSummary(not Finding table) for efficiencyupdate_or_createso multiple scans on the same day keep only the latest datagenerate_outputs_taskto minimize scan completion timeNew API endpoint
GET /api/v1/overviews/findings_severity_over_timefilter[date_from]date_to,provider_id,provider_id__in,provider_type,provider_type__inActiveProviderManagerSteps to review
Migration: Check
0060_dailyfindingsseverity.pycreates the table with proper indexes and RLS constraintModel: Review
DailyFindingsSeverityinmodels.py:ActiveProviderManagerto exclude soft-deleted providersprovider_typefieldTask chain: In
tasks/tasks.py, verifyupdate_daily_findings_severity_taskruns in parallel withgenerate_outputs_task(inside agroup())Aggregation function: In
tasks/jobs/scan.py, checkaggregate_daily_findings_severity:ScanSummarynotFindingupdate_or_createfor idempotencyEndpoint: In
v1/views.py, reviewfindings_severity_over_time:date_fromis requiredprovider_typefor filtering (no JOIN)Checklist
UI
API
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.