ThreatLens

ThreatLens is a Python-based, multi-source Cyber Threat Intelligence (CTI) triage pipeline. It turns public vulnerability intelligence and URLhaus IOC data into analyst-ready outputs: scored indicators, prioritized vulnerabilities, MITRE ATT&CK mappings, STIX-style bundles, Sigma hunt templates, IOC watchlists, screenshots, and a daily threat brief.

Why This Project Matters

Public CTI feeds are useful, but they are often noisy, duplicated, stale, and difficult to prioritize without organization context. ThreatLens demonstrates a real blue-team workflow:

Collect -> Normalize -> Deduplicate -> Score -> Map to ATT&CK -> Export STIX/Sigma/Watchlists -> Report

The goal is not to build another IP reputation checker. The goal is to show how raw CTI data can become prioritized, explainable, reportable, and detection-ready intelligence.

What ThreatLens Is

ThreatLens is:

• a public vulnerability and IOC collector
• an IOC and CVE normalizer
• a deduplication pipeline
• an explainable scoring engine
• a vulnerability priority engine
• a MITRE ATT&CK mapping layer
• a STIX-style export generator
• a Sigma hunt template and IOC watchlist generator
• a Markdown analyst report generator

ThreatLens is not:

• a SIEM
• an EDR
• a malware sandbox
• an attribution platform
• a production detection-rule validator

Features

• Collects public vulnerability intelligence from CISA KEV and EPSS
• Collects malware URL indicators from URLhaus
• Normalizes raw IOC and CVE records into common schemas
• Deduplicates repeated indicators
• Scores indicators with source, recency, status, and malware-tag signals
• Prioritizes vulnerabilities with KEV, batch EPSS, exposure, and criticality context
• Maps observed themes to MITRE ATT&CK techniques
• Exports supported high-priority indicator types as a STIX 2.1-style bundle
• Generates experimental Sigma hunt templates and IOC watchlists for SIEM/proxy validation
• Produces an analyst-ready daily threat brief
• Includes screenshots, self-test, and focused unit tests

Architecture

collectors/
  CISA KEV, EPSS, URLhaus
        |
normalizer/
  Indicator and Vulnerability schemas
        |
scoring/
  IOC severity and CVE priority
        |
enrichment/
  MITRE ATT&CK mapping
        |
export/ + detection/
  Markdown report, STIX bundle, Sigma rules, IOC watchlists

Data Sources

Source	Purpose	API Key
CISA KEV	Known exploited vulnerabilities	No
FIRST EPSS	Batch CVE exploit probability context	No
URLhaus	Malware distribution URLs	Yes, abuse.ch Auth-Key

Keep secrets in a local .env file. Use .env.example as the template.

Quick Start

Install dependencies:

python -m pip install -r requirements.txt

Create local secrets:

copy .env.example .env

Then set your abuse.ch URLhaus key inside .env:

URLHAUS_AUTH_KEY=replace_with_your_urlhaus_auth_key

Run the full pipeline:

python threatlens.py collect --sources kev epss urlhaus
python threatlens.py normalize
python threatlens.py score
python threatlens.py map-attack
python threatlens.py export-stix
python threatlens.py generate-sigma
python threatlens.py report
python threatlens.py self-test
python -m pytest

Commands

Command	What It Does
python threatlens.py collect --sources kev epss urlhaus	Downloads raw public CTI data
python threatlens.py normalize	Converts raw data into common IOC/CVE schemas
python threatlens.py score	Scores indicators and prioritizes vulnerabilities
python threatlens.py map-attack	Maps evidence themes to MITRE ATT&CK techniques
python threatlens.py export-stix	Exports supported high-priority indicators as a STIX 2.1-style indicator bundle
python threatlens.py generate-sigma	Generates experimental Sigma templates plus URL, domain, and IP IOC watchlists
python threatlens.py report	Builds the daily analyst threat brief
python threatlens.py self-test	Runs a lightweight smoke-test for generated output artifacts
python -m pytest	Runs focused helper tests for matching, scoring, normalization, ATT&CK mapping, EPSS, Sigma, and STIX export

Demo Results

The committed demo snapshot produced:

• 28,627 normalized indicators
• 1,587 vulnerability records
• 2,547 high-priority indicators
• 72 P1 vulnerabilities
• 5 MITRE ATT&CK mapped themes
• 100 exported STIX indicators
• 5 generated Sigma rule templates
• 3 IOC watchlists for high-priority URLs, domains, and IPs

Because URLhaus is a live feed, counts may change between runs.

Generated Vs Committed Outputs

ThreatLens keeps large live data files out of git. Files under data/raw/, data/normalized/, and data/enriched/ are generated locally after running the pipeline and are intentionally ignored except for .gitkeep placeholders.

The repository does commit small demo artifacts such as the analyst report, STIX sample bundle, Sigma templates, IOC watchlists, screenshots, and documentation.

Example Outputs

Output	Path
Analyst report sample	reports/daily_threat_brief.md
STIX-style indicator bundle sample	exports/threatlens_indicators_stix.json
Sigma hunt templates	detection/rules/*.yml
URL/domain/IP IOC watchlists	detection/watchlists/*.txt
ATT&CK mapping local output	data/enriched/attack_mapping.json
Demo HTML snapshot helper output	docs/demo_snapshot.html
Walkthrough	docs/walkthrough.md

Regenerate the committed demo HTML after a fresh pipeline run with:

python tools/build_demo_page.py

Screenshots

Pipeline Status

Scoring Distribution

STIX Export And Sigma Rules

Top Priority Vulnerabilities

MITRE ATT&CK Mapping

ThreatLens uses conservative, rule-based ATT&CK mapping. It does not perform actor attribution.

Theme	Technique
malware URL delivery	T1105 - Ingress Tool Transfer
script payload delivery	T1059 - Command and Scripting Interpreter
public-facing application exploitation	T1190 - Exploit Public-Facing Application
remote service exploitation	T1210 - Exploitation of Remote Services
RDP remote services	T1021.001 - Remote Services: Remote Desktop Protocol

Correct CTI language:

Observed evidence is consistent with ATT&CK technique X.

Not:

This activity belongs to actor Y.

Limitations

• Public CTI feeds may contain stale, noisy, incomplete, or conflicting data.
• Scores are heuristic and intended for triage.
• No actor attribution is performed.
• Generated Sigma rules and IOC watchlists are experimental and require SIEM/proxy validation.
• STIX export currently supports selected indicator types in a STIX 2.1-style indicator bundle; validated relationship modeling is future work.
• The current organization profile is a demo profile, not a real asset inventory.

Future Work

• Add AbuseIPDB IP reputation enrichment
• Add AlienVault OTX enrichment and source overlap
• Add MalwareBazaar hash enrichment
• Add Streamlit dashboard
• Add FastAPI endpoints
• Add OpenCTI or MISP import/export workflows
• Add additional unit tests around edge-case normalization and scoring

Portfolio Summary

ThreatLens shows how to transform public vulnerability intelligence and malware URL indicators into prioritized blue-team outputs: explainable IOC severity, P1 vulnerability prioritization, ATT&CK context, STIX-style sharing artifacts, Sigma hunt templates, IOC watchlists, and analyst-ready reporting.