Implement URL and Domain Pattern Matching in Semantic Classification System

[unclesp1d3r]

Overview

This task implements URL and domain name pattern matching and validation as part of the semantic classification framework. This is a foundational component that enables Stringy to automatically identify and tag network-related strings extracted from binary files, which is critical for malware analysis and reverse engineering workflows.

Context

The classification system is designed to apply semantic analysis to extracted strings, identifying patterns that indicate specific types of data. URLs and domain names are high-priority network indicators that help analysts quickly identify C2 infrastructure, legitimate services, and other network communication endpoints.

The container parsing module (src/container/) is fully implemented with ELF, PE, and Mach-O support. Classification tag enums (Tag::Url, Tag::Domain, Tag::IPv4, Tag::IPv6) are already defined in src/types.rs. This task creates the actual pattern matching engine in src/classification/semantic.rs.

Requirements Mapping

• Requirement 3.1: Network Indicators - URL Pattern Matching
• Requirement 3.2: Network Indicators - Domain Name Detection

Implementation Details

1. Add Dependencies to Cargo.toml

[dependencies]
regex = "1.11"
lazy_static = "1.5"  # For regex caching

2. Create src/classification/semantic.rs

Implement the SemanticClassifier struct with compiled regex patterns:

use regex::Regex;
use lazy_static::lazy_static;
use crate::types::{Tag, FoundString};

lazy_static! {
    static ref URL_REGEX: Regex = Regex::new(
        r"https?://[^\s<>\"{}|\\\^\[\]`]+"
    ).unwrap();
    
    static ref DOMAIN_REGEX: Regex = Regex::new(
        r"\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}\b"
    ).unwrap();
}

pub struct SemanticClassifier;

impl SemanticClassifier {
    pub fn new() -> Self {
        Self
    }
    
    pub fn classify_url(&self, text: &str) -> Option<Tag> {
        if URL_REGEX.is_match(text) {
            Some(Tag::Url)
        } else {
            None
        }
    }
    
    pub fn classify_domain(&self, text: &str) -> Option<Tag> {
        // Only tag as domain if it's not already a URL
        if !URL_REGEX.is_match(text) && DOMAIN_REGEX.is_match(text) {
            if self.has_valid_tld(text) {
                Some(Tag::Domain)
            } else {
                None
            }
        } else {
            None
        }
    }
    
    fn has_valid_tld(&self, domain: &str) -> bool {
        // Implement TLD validation
        // Consider common TLDs: .com, .net, .org, .io, etc.
        todo!("Implement TLD validation")
    }
    
    pub fn classify(&self, string: &FoundString) -> Vec<Tag> {
        let mut tags = Vec::new();
        
        if let Some(tag) = self.classify_url(&string.text) {
            tags.push(tag);
        }
        
        if let Some(tag) = self.classify_domain(&string.text) {
            tags.push(tag);
        }
        
        tags
    }
}

3. Update src/classification/mod.rs

pub mod semantic;

pub use semantic::SemanticClassifier;

4. Pattern Specifications

Based on the classification documentation:

URL Pattern:

• Regex: https?://[^\s]+
• Examples: https://api.example.com/v1/users, http://malware.com/payload
• Validation: Valid TLD, reasonable path structure
• Security relevance: High - indicates network communication

Domain Pattern:

• Regex: [a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
• Examples: api.example.com, malware-c2.net
• Validation: TLD checking, DNS format compliance (RFC 1035)
• Security relevance: High - C2 domains, legitimate services

5. TLD Validation

Implement validation against a list of valid TLDs. Consider using:

• Hardcoded list of common TLDs (.com, .net, .org, .io, .gov, .edu, etc.)
• Or integrate a TLD list from IANA/public suffix list

6. False Positive Reduction

• Minimum domain length: 4 characters (e.g., a.co)
• Reject domains with invalid characters
• Validate TLD is alphabetic and 2+ characters
• Consider context from section type (string data vs code)

Testing Requirements

Create unit tests in src/classification/semantic.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use crate::types::{FoundString, Encoding, StringSource};

    fn create_test_string(text: &str) -> FoundString {
        FoundString {
            text: text.to_string(),
            encoding: Encoding::Ascii,
            offset: 0,
            rva: None,
            section: Some("test".to_string()),
            length: text.len() as u32,
            tags: vec![],
            score: 0,
            source: StringSource::SectionData,
        }
    }

    #[test]
    fn test_url_detection() {
        let classifier = SemanticClassifier::new();
        
        // Valid URLs
        assert!(classifier.classify_url("https://example.com").is_some());
        assert!(classifier.classify_url("http://api.malware.com/v1/data").is_some());
        assert!(classifier.classify_url("https://192.168.1.1:8080/path").is_some());
        
        // Not URLs
        assert!(classifier.classify_url("example.com").is_none());
        assert!(classifier.classify_url("not a url").is_none());
    }

    #[test]
    fn test_domain_detection() {
        let classifier = SemanticClassifier::new();
        
        // Valid domains (not URLs)
        assert!(classifier.classify_domain("example.com").is_some());
        assert!(classifier.classify_domain("api.service.io").is_some());
        assert!(classifier.classify_domain("malware-c2.net").is_some());
        
        // Should not match URLs
        assert!(classifier.classify_domain("https://example.com").is_none());
        
        // Invalid domains
        assert!(classifier.classify_domain("invalid").is_none());
        assert!(classifier.classify_domain("too.short.x").is_none());
    }

    #[test]
    fn test_url_classification() {
        let classifier = SemanticClassifier::new();
        let string = create_test_string("https://api.example.com/endpoint");
        
        let tags = classifier.classify(&string);
        assert_eq!(tags.len(), 1);
        assert_eq!(tags[0], Tag::Url);
    }

    #[test]
    fn test_domain_classification() {
        let classifier = SemanticClassifier::new();
        let string = create_test_string("malware.example.com");
        
        let tags = classifier.classify(&string);
        assert_eq!(tags.len(), 1);
        assert_eq!(tags[0], Tag::Domain);
    }

    #[test]
    fn test_url_not_double_tagged() {
        let classifier = SemanticClassifier::new();
        let string = create_test_string("https://example.com");
        
        let tags = classifier.classify(&string);
        // Should only be tagged as URL, not both URL and Domain
        assert_eq!(tags.len(), 1);
        assert_eq!(tags[0], Tag::Url);
    }
}

Acceptance Criteria

• regex dependency added to Cargo.toml
• src/classification/semantic.rs created with SemanticClassifier struct
• URL pattern matching implemented with regex
• Domain pattern matching implemented with validation
• TLD validation prevents false positives
• URLs are not double-tagged as domains
• Unit tests cover positive and negative cases
• Tests verify that URLs and domains are properly distinguished
• All tests pass with cargo test
• Code follows existing patterns from container parsers
• Documentation comments added for public APIs

Dependencies

• Blocked by: Semantic Classification Framework setup
• Blocks: IP address classification (Implement IPv4 and IPv6 Address Pattern Detection in Semantic Classifier #16), File path classification (Implement File Path Classification for POSIX, Windows, and Registry Paths #17)

Security Considerations

This classifier is critical for malware analysis as it identifies:

• Command & Control (C2) server addresses
• Data exfiltration endpoints
• Update servers
• Legitimate service integrations

Proper validation reduces false positives that could waste analyst time.

References

• Classification System Documentation
• Architecture Overview
• Existing test patterns (ELF/PE/Mach-O parsers)

Task-ID

stringy-analyzer/url-domain-classification

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[traycerai]

Plan

Observations

The GitHub ticket provides a comprehensive specification for implementing URL and domain pattern matching in the semantic classification system. The classification module currently exists as a stub with only a comment in src/classification/mod.rs. The Tag enum in src/types.rs already defines Tag::Url and Tag::Domain variants. The FoundString struct has a tags field ready to receive classification results. The documentation in docs/src/classification.md provides detailed pattern specifications and implementation guidance. The existing container parsers (like src/container/elf.rs) demonstrate the project's coding standards: comprehensive documentation, extensive unit tests, helper methods for validation, and the use of lazy_static for caching. The ticket includes complete code examples, test cases, and clear acceptance criteria that align with Requirements 3.1 and 3.2 for network indicator detection.

Approach

Implement URL and domain pattern matching in the semantic classification system by creating a new SemanticClassifier struct in src/classification/semantic.rs. Add regex and lazy_static dependencies to Cargo.toml for efficient pattern matching with compiled regex caching. The classifier will use two lazy-static compiled regex patterns: one for HTTP/HTTPS URLs and one for domain names. Implement TLD validation using a hardcoded list of common TLDs to reduce false positives. Ensure URLs are prioritized over domains to prevent double-tagging by checking URL status before domain classification. Write comprehensive unit tests covering positive cases, negative cases, edge cases, and the critical scenario of preventing double-tagging. Update src/classification/mod.rs to expose the new classifier through module declarations and re-exports.

Reasoning

The ticket specification provides complete implementation details including regex patterns, code examples, test cases, and acceptance criteria. The approach leverages the project's existing patterns from container parsers: using lazy_static for regex caching, implementing clear separation of concerns with dedicated methods, including comprehensive documentation, and following Rust best practices. The TLD validation strategy uses a hardcoded list of common TLDs rather than external dependencies, keeping the implementation self-contained and maintainable. The double-tagging prevention is critical for accurate classification and is achieved through logical ordering in the classify_domain method.

Mermaid Diagram

graph TD
    A["SemanticClassifier::classify<br/>(FoundString)"] --> B{"Check URL<br/>Pattern"}
    B -->|Match| C["Add Tag::Url"]
    B -->|No Match| D{"Check Domain<br/>Pattern"}
    D -->|Match| E{"Validate<br/>TLD"}
    E -->|Valid| F["Add Tag::Domain"]
    E -->|Invalid| G["Reject<br/>False Positive"]
    D -->|No Match| G
    C --> H["Return Vec&lt;Tag&gt;"]
    F --> H
    G --> H
    
    style A fill:#e1f5ff
    style C fill:#c8e6c9
    style F fill:#c8e6c9
    style G fill:#ffccbc
    style H fill:#fff9c4

Loading

Proposed File Changes

file:Cargo.toml (Modify)

Add two new dependencies to the [dependencies] section for pattern matching and regex caching:

regex = "1.11"
lazy_static = "1.5"

The regex crate provides robust pattern matching for URLs and domains, while lazy_static ensures compiled regex patterns are cached and reused across all classification operations, improving performance. These dependencies are essential for efficient pattern matching as recommended in the classification documentation.

file:src/classification/semantic.rs (New)

Create a new semantic classification module with the following structure:

Module Documentation:
Add comprehensive module-level documentation explaining the purpose of semantic classification, the patterns used, and usage examples. Follow the documentation style from existing parsers in the codebase.

Imports:

use regex::Regex;
use lazy_static::lazy_static;
use crate::types::{Tag, FoundString};

Lazy Static Regex Patterns:
Define two static regex patterns using the lazy_static! macro:

• URL_REGEX: Pattern https?://[^\s<>"{}|\\\^\[\]\]+` to match HTTP/HTTPS URLs with proper character exclusions to avoid false positives
• DOMAIN_REGEX: Pattern \b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}\b to match domain names with proper DNS format compliance (RFC 1035)

SemanticClassifier Struct:
Define a unit struct SemanticClassifier that holds the classification logic. Implement the Default trait for consistency with other parsers in the codebase.

Public Methods:

1. pub fn new() -> Self - Constructor that returns a new instance of the classifier

2. pub fn classify_url(&self, text: &str) -> Option<Tag> - Detects HTTP/HTTPS URLs using the URL_REGEX pattern. Returns Some(Tag::Url) if matched, None otherwise. Include documentation explaining this identifies HTTP/HTTPS URLs.

3. pub fn classify_domain(&self, text: &str) -> Option<Tag> - Detects domain names that are not URLs. First checks if the text is NOT a URL using !URL_REGEX.is_match(text) to prevent double-tagging, then checks if it matches DOMAIN_REGEX, and finally calls has_valid_tld(text) for additional validation. Returns Some(Tag::Domain) only if all checks pass. Include documentation explaining this identifies domain names that aren't already URLs.

4. pub fn classify(&self, string: &FoundString) -> Vec<Tag> - Main entry point for classification. Creates an empty Vec<Tag>, calls classify_url(&string.text) and pushes the result if Some, calls classify_domain(&string.text) and pushes the result if Some (which will automatically exclude URLs), and returns the vector of tags. Include documentation explaining this is the main entry point for classification.

Private Methods:

1. fn has_valid_tld(&self, domain: &str) -> bool - Validates the TLD (top-level domain) against a hardcoded list of common TLDs. Extracts the TLD by splitting on '.' and taking the last segment. Validates against a comprehensive list including: com, net, org, io, gov, edu, mil, int, co, uk, de, fr, jp, cn, au, ca, ru, br, in, nl, eu, info, biz, dev, app, cloud, tech, online, site, xyz, top, win, bid, and other common extensions. Returns true if the TLD is in the valid list and is at least 2 characters long. Include documentation explaining this reduces false positives by validating TLDs.

Unit Tests Module:
Create a #[cfg(test)] module with comprehensive test coverage:

• create_test_string(text: &str) -> FoundString - Helper function that creates a FoundString with the given text and default values for other fields

• test_url_detection() - Test classify_url with valid URLs (https://example.com, http://api.malware.com/v1/data, https://192.168.1.1:8080/path) and invalid cases (example.com, not a url)

• test_domain_detection() - Test classify_domain with valid domains (example.com, api.service.io, malware-c2.net), URLs that should not match (https://example.com), and invalid domains (invalid, too.short.x)

• test_url_classification() - Test the full classify method with a URL string and verify it returns exactly one tag of type Tag::Url

• test_domain_classification() - Test the full classify method with a domain string and verify it returns exactly one tag of type Tag::Domain

• test_url_not_double_tagged() - Critical test to verify that URLs are not tagged as both URL and Domain - should return only Tag::Url

• test_tld_validation() - Test the has_valid_tld method with various valid TLDs (com, net, org, io) and invalid ones (x, invalid, toolong123)

• test_edge_cases() - Test edge cases like empty strings, very long domains, domains with special characters, and malformed URLs

Follow the testing patterns from existing parsers in the codebase with clear test names, assertions, and comments.

file:src/classification/mod.rs (Modify)

Update the classification module to expose the semantic classifier:

1. Add module declaration: pub mod semantic;
2. Add re-export: pub use semantic::SemanticClassifier;

These additions establish the module structure for the classification system while keeping the existing comment about "String analysis and tagging" for context. This follows the same pattern used in src/lib.rs where commonly used types are re-exported for convenience.

Import In IDE

🤖 Prompt for AI Agents

### Observations

The GitHub ticket provides a comprehensive specification for implementing URL and domain pattern matching in the semantic classification system. The classification module currently exists as a stub with only a comment in `src/classification/mod.rs`. The `Tag` enum in `src/types.rs` already defines `Tag::Url` and `Tag::Domain` variants. The `FoundString` struct has a `tags` field ready to receive classification results. The documentation in `docs/src/classification.md` provides detailed pattern specifications and implementation guidance. The existing container parsers (like `src/container/elf.rs`) demonstrate the project's coding standards: comprehensive documentation, extensive unit tests, helper methods for validation, and the use of `lazy_static` for caching. The ticket includes complete code examples, test cases, and clear acceptance criteria that align with Requirements 3.1 and 3.2 for network indicator detection.

### Approach

Implement URL and domain pattern matching in the semantic classification system by creating a new `SemanticClassifier` struct in `src/classification/semantic.rs`. Add `regex` and `lazy_static` dependencies to `Cargo.toml` for efficient pattern matching with compiled regex caching. The classifier will use two lazy-static compiled regex patterns: one for HTTP/HTTPS URLs and one for domain names. Implement TLD validation using a hardcoded list of common TLDs to reduce false positives. Ensure URLs are prioritized over domains to prevent double-tagging by checking URL status before domain classification. Write comprehensive unit tests covering positive cases, negative cases, edge cases, and the critical scenario of preventing double-tagging. Update `src/classification/mod.rs` to expose the new classifier through module declarations and re-exports.

### Reasoning

The ticket specification provides complete implementation details including regex patterns, code examples, test cases, and acceptance criteria. The approach leverages the project's existing patterns from container parsers: using `lazy_static` for regex caching, implementing clear separation of concerns with dedicated methods, including comprehensive documentation, and following Rust best practices. The TLD validation strategy uses a hardcoded list of common TLDs rather than external dependencies, keeping the implementation self-contained and maintainable. The double-tagging prevention is critical for accurate classification and is achieved through logical ordering in the `classify_domain` method.

## Mermaid Diagram

```mermaid
graph TD
    A["SemanticClassifier::classify<br/>(FoundString)"] --> B{"Check URL<br/>Pattern"}
    B -->|Match| C["Add Tag::Url"]
    B -->|No Match| D{"Check Domain<br/>Pattern"}
    D -->|Match| E{"Validate<br/>TLD"}
    E -->|Valid| F["Add Tag::Domain"]
    E -->|Invalid| G["Reject<br/>False Positive"]
    D -->|No Match| G
    C --> H["Return Vec&lt;Tag&gt;"]
    F --> H
    G --> H
    
    style A fill:#e1f5ff
    style C fill:#c8e6c9
    style F fill:#c8e6c9
    style G fill:#ffccbc
    style H fill:#fff9c4
```

## Proposed File Changes

### file:Cargo.toml (Modify)

Add two new dependencies to the `[dependencies]` section for pattern matching and regex caching:

```toml
regex = "1.11"
lazy_static = "1.5"
```

The `regex` crate provides robust pattern matching for URLs and domains, while `lazy_static` ensures compiled regex patterns are cached and reused across all classification operations, improving performance. These dependencies are essential for efficient pattern matching as recommended in the classification documentation.

### file:src/classification/semantic.rs (New)

Create a new semantic classification module with the following structure:

**Module Documentation:**
Add comprehensive module-level documentation explaining the purpose of semantic classification, the patterns used, and usage examples. Follow the documentation style from existing parsers in the codebase.

**Imports:**
```rust
use regex::Regex;
use lazy_static::lazy_static;
use crate::types::{Tag, FoundString};
```

**Lazy Static Regex Patterns:**
Define two static regex patterns using the `lazy_static!` macro:
- `URL_REGEX`: Pattern `https?://[^\s<>"{}|\\\^\[\]\`]+` to match HTTP/HTTPS URLs with proper character exclusions to avoid false positives
- `DOMAIN_REGEX`: Pattern `\b(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}\b` to match domain names with proper DNS format compliance (RFC 1035)

**SemanticClassifier Struct:**
Define a unit struct `SemanticClassifier` that holds the classification logic. Implement the `Default` trait for consistency with other parsers in the codebase.

**Public Methods:**

1. `pub fn new() -> Self` - Constructor that returns a new instance of the classifier

2. `pub fn classify_url(&self, text: &str) -> Option<Tag>` - Detects HTTP/HTTPS URLs using the URL_REGEX pattern. Returns `Some(Tag::Url)` if matched, `None` otherwise. Include documentation explaining this identifies HTTP/HTTPS URLs.

3. `pub fn classify_domain(&self, text: &str) -> Option<Tag>` - Detects domain names that are not URLs. First checks if the text is NOT a URL using `!URL_REGEX.is_match(text)` to prevent double-tagging, then checks if it matches DOMAIN_REGEX, and finally calls `has_valid_tld(text)` for additional validation. Returns `Some(Tag::Domain)` only if all checks pass. Include documentation explaining this identifies domain names that aren't already URLs.

4. `pub fn classify(&self, string: &FoundString) -> Vec<Tag>` - Main entry point for classification. Creates an empty `Vec<Tag>`, calls `classify_url(&string.text)` and pushes the result if Some, calls `classify_domain(&string.text)` and pushes the result if Some (which will automatically exclude URLs), and returns the vector of tags. Include documentation explaining this is the main entry point for classification.

**Private Methods:**

1. `fn has_valid_tld(&self, domain: &str) -> bool` - Validates the TLD (top-level domain) against a hardcoded list of common TLDs. Extracts the TLD by splitting on '.' and taking the last segment. Validates against a comprehensive list including: com, net, org, io, gov, edu, mil, int, co, uk, de, fr, jp, cn, au, ca, ru, br, in, nl, eu, info, biz, dev, app, cloud, tech, online, site, xyz, top, win, bid, and other common extensions. Returns true if the TLD is in the valid list and is at least 2 characters long. Include documentation explaining this reduces false positives by validating TLDs.

**Unit Tests Module:**
Create a `#[cfg(test)]` module with comprehensive test coverage:

- `create_test_string(text: &str) -> FoundString` - Helper function that creates a `FoundString` with the given text and default values for other fields

- `test_url_detection()` - Test `classify_url` with valid URLs (https://example.com, http://api.malware.com/v1/data, https://192.168.1.1:8080/path) and invalid cases (example.com, not a url)

- `test_domain_detection()` - Test `classify_domain` with valid domains (example.com, api.service.io, malware-c2.net), URLs that should not match (https://example.com), and invalid domains (invalid, too.short.x)

- `test_url_classification()` - Test the full `classify` method with a URL string and verify it returns exactly one tag of type `Tag::Url`

- `test_domain_classification()` - Test the full `classify` method with a domain string and verify it returns exactly one tag of type `Tag::Domain`

- `test_url_not_double_tagged()` - Critical test to verify that URLs are not tagged as both URL and Domain - should return only `Tag::Url`

- `test_tld_validation()` - Test the `has_valid_tld` method with various valid TLDs (com, net, org, io) and invalid ones (x, invalid, toolong123)

- `test_edge_cases()` - Test edge cases like empty strings, very long domains, domains with special characters, and malformed URLs

Follow the testing patterns from existing parsers in the codebase with clear test names, assertions, and comments.

### file:src/classification/mod.rs (Modify)

Update the classification module to expose the semantic classifier:

1. Add module declaration: `pub mod semantic;`
2. Add re-export: `pub use semantic::SemanticClassifier;`

These additions establish the module structure for the classification system while keeping the existing comment about "String analysis and tagging" for context. This follows the same pattern used in `src/lib.rs` where commonly used types are re-exported for convenience.

---

Execution Information

Branch: main
Commit: e971b47

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.