Skip to content

Use OSV for ubuntu vulnerability scanning#42063

Merged
ksykulev merged 18 commits intomainfrom
40057-osv-vulns
Apr 3, 2026
Merged

Use OSV for ubuntu vulnerability scanning#42063
ksykulev merged 18 commits intomainfrom
40057-osv-vulns

Conversation

@ksykulev
Copy link
Copy Markdown
Contributor

@ksykulev ksykulev commented Mar 19, 2026

Related issue: Resolves #40057

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.

  • Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.

Testing

  • Added/updated automated tests
  • QA'd all new/changed functionality manually

Summary by CodeRabbit

  • New Features

    • OSV (Open Source Vulnerabilities) added as an optional Ubuntu vulnerability data source and enabled by default.
  • Features

    • Integrated OSV into the vulnerability scanning pipeline, artifact sync/refresh, detection, and cleanup flows.
    • Improved Ubuntu package/kernel version matching for more accurate OSV detections.
  • Chores

    • Added configuration flag and updated expected config fixtures.
  • Tests

    • Added extensive tests for OSV sync, artifact handling, analyzer logic, and cleanup behaviors.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 19, 2026

Codecov Report

❌ Patch coverage is 45.32710% with 234 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.83%. Comparing base (eed4fe1) to head (585add1).
⚠️ Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
server/vulnerabilities/osv/downloader.go 30.43% 78 Missing and 2 partials ⚠️
server/vulnerabilities/osv/analyzer.go 56.21% 64 Missing and 10 partials ⚠️
cmd/fleet/cron.go 19.11% 51 Missing and 4 partials ⚠️
server/vulnerabilities/osv/sync.go 57.62% 21 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #42063      +/-   ##
==========================================
- Coverage   66.89%   66.83%   -0.07%     
==========================================
  Files        2573     2560      -13     
  Lines      206426   206318     -108     
  Branches     9281     8983     -298     
==========================================
- Hits       138088   137890     -198     
- Misses      55784    55856      +72     
- Partials    12554    12572      +18     
Flag Coverage Δ
backend 68.61% <45.32%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ksykulev
Copy link
Copy Markdown
Contributor Author

ksykulev commented Apr 1, 2026

@coderabbitai full review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 1, 2026

✅ Actions performed

Full review triggered.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 1, 2026

Walkthrough

Adds end-to-end OSV-based Ubuntu vulnerability scanning: new config flag vulnerabilities.osv_for_vulnerabilities and corresponding VulnerabilitiesConfig field; OSV artifact sync/download (GitHub releases) and local cleanup; an OSV analyzer that loads artifacts, matches host software (including kernel normalization), computes deltas, and updates the datastore with a new UbuntuOSVSource; cron integration to run OSV checks and exclude OSV-supported platforms from OVAL processing; and related tests and test-fixture updates (including small refactors in the osv-processor).

Possibly related PRs

  • fleetdm/fleet PR 42203: Modifies cmd/osv-processor components and OSV artifact generation/processing used by the new sync/downloader.
  • fleetdm/fleet PR 41195: Touches cron vulnerability scanning and cleanup flows that intersect with the added OSV/OVAL cron logic.
  • fleetdm/fleet PR 39820: Updates vulnerability processing and cron integration, overlapping datastore/batch-insert and analysis code paths changed here.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 42.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: implementing OSV for Ubuntu vulnerability scanning, which aligns with the primary objective in the PR.
Description check ✅ Passed The PR description addresses the template requirements: links to issue #40057, confirms changes file added, validates security practices, and documents testing completion.
Linked Issues check ✅ Passed The code changes fully implement issue #40057 objectives: OSV feed integrated as new analyzer module, runs alongside OVAL feed with conditional feature flag, vulnerability data ingested via new Analyze/Refresh functions.
Out of Scope Changes check ✅ Passed All changes are scoped to OSV Ubuntu vulnerability scanning: new OSV modules, config flag, cron integration, cleanup functions, and test fixtures—no unrelated modifications detected.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 40057-osv-vulns

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@cmd/fleet/cron.go`:
- Around line 199-205: The code currently calls checkOvalVulnerabilities
unconditionally and then, when config.OSVForVulnerabilities is true, calls
checkOSVVulnerabilities and appends its results, which can cause OVAL to be
skipped for OSV-backed platforms even if OSV fails; modify the flow so that when
config.OSVForVulnerabilities is true you first call checkOSVVulnerabilities and
check its success (e.g., return value or error indicator from
checkOSVVulnerabilities), and only suppress OVAL processing for OSV-backed
platforms if checkOSVVulnerabilities succeeded; if checkOSVVulnerabilities
fails, fall back to calling checkOvalVulnerabilities for those platforms (or
keep its original results) so Ubuntu/stale platforms are processed by OVAL; use
the existing symbols checkOSVVulnerabilities, checkOvalVulnerabilities,
config.OSVForVulnerabilities, vulnAutomationEnabled and ovalVulns to implement
the gating and fallback.

In `@server/vulnerabilities/osv/analyzer.go`:
- Around line 117-129: The code in loadOSVArtifact recomputes the dated filename
with time.Now(), causing mismatch if sync used a different run time; change
loadOSVArtifact to accept the run date (e.g., a time.Time runDate) or the
already-resolved artifact filename and use that when calling osvFilename instead
of time.Now(), and ensure the caller (e.g., the Refresh/Sync orchestration that
downloads artifacts) passes the same runDate/filename so both sync and analysis
use a single consistent run timestamp; update references to loadOSVArtifact,
osvFilename, and callers accordingly.
- Around line 67-71: The current call to ds.HostIDsByOSVersion(ctx, ver, 0,
10000) only fetches the first 10k hosts and drops the rest; replace this single
call with a paginated loop that repeatedly calls ds.HostIDsByOSVersion with
increasing offsets (or using a next-token style if supported) until no more IDs
are returned, appending results into the hostIDs slice (or a new accumulator)
and returning any errors from each call; ensure you use the same ctx and ver and
handle the case where large result sets require multiple requests so that
downstream logic (the variable hostIDs) receives the complete set.
- Around line 195-205: The function extractUbuntuVersion trims suffix after
trimming spaces which fails for inputs like "24.04 LTS  "; change the order to
TrimSpace first, then TrimSuffix(" LTS"), and TrimSpace again before splitting
so inputs with trailing/leading whitespace are normalized; update
extractUbuntuVersion accordingly (keep the existing split logic returning
parts[0]+parts[1]).

In `@server/vulnerabilities/osv/downloader.go`:
- Around line 120-134: The current download block writes directly to dstPath and
risks leaving a partial file if rc/io.Copy fails; instead create a temp file in
the same directory (e.g., using os.CreateTemp or ioutil.TempFile), copy rc into
that temp file (using io.Copy), close and sync the temp file, then atomically
rename the temp to dstPath via os.Rename; update the code around rc, outFile and
io.Copy in downloader.go to create/close/sync the temp and only rename on
successful copy, and ensure temp is removed on error.

In `@server/vulnerabilities/osv/sync.go`:
- Around line 35-52: The early return when len(toDownload) == 0 skips
removeOldOSVArtifacts so stale osv-ubuntu-* files are never pruned; update the
flow in the function that calls whatToDownloadOSV/SyncOSV so
removeOldOSVArtifacts(vulnPath, now) is always invoked regardless of whether
toDownload is empty (i.e., call removeOldOSVArtifacts before the early return or
call it in both branches), and propagate its error similarly (e.g., return
toDownload, fmt.Errorf("warning: failed to clean up old OSV artifacts: %w",
err)) while keeping existing SyncOSV and error handling for the download path
intact.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b4a04710-e225-4558-b00e-a492917cccee

📥 Commits

Reviewing files that changed from the base of the PR and between d84beaa and e736933.

📒 Files selected for processing (14)
  • changes/40057-osv-vulns
  • cmd/fleet/cron.go
  • cmd/fleetctl/fleetctl/testdata/expectedGetConfigIncludeServerConfigJson.json
  • cmd/fleetctl/fleetctl/testdata/expectedGetConfigIncludeServerConfigYaml.yml
  • cmd/osv-processor/main.go
  • cmd/osv-processor/main_test.go
  • server/config/config.go
  • server/fleet/app.go
  • server/fleet/vulnerabilities.go
  • server/vulnerabilities/osv/analyzer.go
  • server/vulnerabilities/osv/analyzer_test.go
  • server/vulnerabilities/osv/downloader.go
  • server/vulnerabilities/osv/sync.go
  • server/vulnerabilities/osv/sync_test.go

Comment thread cmd/fleet/cron.go
Comment thread server/vulnerabilities/osv/analyzer.go Outdated
Comment thread server/vulnerabilities/osv/analyzer.go Outdated
Comment thread server/vulnerabilities/osv/analyzer.go
Comment thread server/vulnerabilities/osv/downloader.go
Comment thread server/vulnerabilities/osv/sync.go Outdated
@ksykulev ksykulev marked this pull request as ready for review April 2, 2026 00:15
@ksykulev ksykulev requested a review from a team as a code owner April 2, 2026 00:15
Copilot AI review requested due to automatic review settings April 2, 2026 00:15
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an OSV-backed vulnerability scanning path for Ubuntu hosts and wires it into the existing vulnerability cron workflow behind a new server configuration flag.

Changes:

  • Introduces OSV artifact sync/download, parsing, and matching logic for Ubuntu packages (including kernel-specific handling).
  • Adds vulnerabilities.osv_for_vulnerabilities config flag (default enabled) and updates the cron scanner to run OSV for supported platforms while excluding those platforms from OVAL when enabled.
  • Updates the OSV processor and CLI golden fixtures to reflect new/changed configuration/output.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
server/vulnerabilities/osv/sync.go Refreshes local OSV artifacts and cleans up old ones.
server/vulnerabilities/osv/downloader.go Fetches latest release metadata and downloads OSV artifacts.
server/vulnerabilities/osv/analyzer.go Loads OSV artifacts and analyzes Ubuntu hosts’ installed packages for CVEs.
server/vulnerabilities/osv/sync_test.go Unit tests for artifact cleanup/version selection/filename/digest logic.
server/vulnerabilities/osv/analyzer_test.go Unit tests for Ubuntu parsing and matching logic.
server/fleet/vulnerabilities.go Adds a new UbuntuOSVSource vulnerability source constant.
server/fleet/app.go Adds OSVForVulnerabilities to the vulnerabilities config struct.
server/config/config.go Adds config schema + default/loader wiring for osv_for_vulnerabilities.
cmd/fleet/cron.go Runs OSV scanning when enabled and excludes OSV-supported platforms from OVAL scanning.
cmd/osv-processor/main.go Uses map[string]struct{} for changed-file sets in delta generation.
cmd/osv-processor/main_test.go Updates tests for the changed-file set type change.
cmd/fleetctl/fleetctl/testdata/expectedGetConfigIncludeServerConfigYaml.yml Updates golden output to include osv_for_vulnerabilities.
cmd/fleetctl/fleetctl/testdata/expectedGetConfigIncludeServerConfigJson.json Updates golden output to include osv_for_vulnerabilities.
changes/40057-osv-vulns Release note entry for OSV-based Ubuntu vulnerability scanning.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread server/vulnerabilities/osv/analyzer.go Outdated
Comment thread server/vulnerabilities/osv/analyzer.go Outdated
Comment thread server/vulnerabilities/osv/downloader.go
Comment thread server/vulnerabilities/osv/downloader.go
Comment thread server/vulnerabilities/osv/sync.go Outdated
Comment thread server/vulnerabilities/osv/sync.go Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmd/fleet/cron.go Outdated
Comment thread server/vulnerabilities/osv/sync.go
Comment thread server/vulnerabilities/osv/sync.go Outdated
mostlikelee
mostlikelee previously approved these changes Apr 3, 2026
Copy link
Copy Markdown
Contributor

@mostlikelee mostlikelee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. a few nits, but nothing worth blocking

Comment thread cmd/fleet/cron_test.go Outdated
Comment thread cmd/fleet/cron.go Outdated
@ksykulev ksykulev merged commit d7b6b3c into main Apr 3, 2026
51 checks passed
@ksykulev ksykulev deleted the 40057-osv-vulns branch April 3, 2026 20:59
ksykulev added a commit that referenced this pull request Apr 6, 2026
Bug fix for
#42063
**Related issue:** Resolves #40057

# Checklist for submitter

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [ ] Confirmed that the fix is not expected to adversely impact load
test results

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ksykulev added a commit that referenced this pull request Apr 7, 2026
Bug fix for
#42063
**Related issue:** Resolves #40057

# Checklist for submitter

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [ ] Confirmed that the fix is not expected to adversely impact load
test results

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ksykulev added a commit that referenced this pull request Apr 7, 2026
Original PR: #43087
Bug fix for
#42063
**Related issue:** Resolves #40057

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@coderabbitai coderabbitai Bot mentioned this pull request Apr 9, 2026
2 tasks
ksykulev added a commit that referenced this pull request Apr 10, 2026
Unreleased bug fix for #42063
**Related issue:** Resolves #39900

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
We shouldn't need any additional load testing. This change will not have
a large impact on load.
ksykulev added a commit that referenced this pull request Apr 10, 2026
Unreleased bug fix for #42063
**Related issue:** Resolves #39900

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
We shouldn't need any additional load testing. This change will not have
a large impact on load.
ksykulev added a commit that referenced this pull request Apr 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OSV feed Ubuntu: core

4 participants