Implement Main String Extraction Orchestrator and Pipeline Integration

[unclesp1d3r]

Summary

Create the main extraction orchestrator that serves as the primary API for analyzing binaries and extracting meaningful strings. This orchestrator will wire together all components (format detection, parsing, extraction, classification, and ranking) into a cohesive pipeline.

Background

StringyMcStringFace aims to be a smarter alternative to the standard strings command by being data-structure aware, section-aware, and semantically intelligent. The foundation is solid with working binary format parsers (PE, ELF, Mach-O) via goblin, but the core extraction pipeline that coordinates all components needs to be implemented.

Current Status:

• ✅ Format detection (container::detect_format)
• ✅ Container parsers (ContainerParser trait with PE/ELF/Mach-O implementations)
• ✅ Type definitions (FoundString, ContainerInfo, etc.)
• ❌ String extraction engine (empty extraction/mod.rs)
• ❌ Classification/tagging system (empty classification/mod.rs)
• ❌ Ranking/scoring algorithm
• ❌ Main orchestrator API

Proposed Solution

Architecture

Create a StringAnalyzer orchestrator in src/lib.rs that provides a clean public API:

pub struct StringAnalyzer {
    min_length: usize,
    encodings: Vec<Encoding>,
    // ... configuration
}

impl StringAnalyzer {
    pub fn new() -> Self { /* ... */ }
    
    pub fn analyze(&self, data: &[u8]) -> Result<Vec<FoundString>> {
        // 1. Detect format
        // 2. Parse container metadata
        // 3. Extract strings from prioritized sections
        // 4. Classify/tag strings
        // 5. Score/rank strings
        // 6. Return sorted results
    }
}

Implementation Plan

Phase 1: Core Extraction (extraction/mod.rs)

• Implement StringExtractor trait with methods:
  • extract_ascii_utf8(data: &[u8], offset: usize) -> Vec<FoundString>
  • extract_utf16le(data: &[u8], offset: usize) -> Vec<FoundString>
  • extract_utf16be(data: &[u8], offset: usize) -> Vec<FoundString>
• Section-aware extraction that respects SectionInfo boundaries
• Configurable minimum length (default 4)
• Track source (section name, offset, RVA)

Phase 2: Classification System (classification/mod.rs)

• Implement StringClassifier with pattern matching for:
  • URLs (http://, https://, ftp://)
  • Domains (DNS patterns)
  • IP addresses (IPv4/IPv6)
  • File paths (Unix: /, Windows: C:\\, /)
  • Registry keys (HKEY_*)
  • GUIDs ({xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx})
  • User agents
  • Format strings (%s, %d, {})
  • Base64 patterns
  • Crypto constants (known algorithm identifiers)
• Return Vec<Tag> for each string

Phase 3: Ranking Algorithm

• Implement scoring system based on:
  • Section type priority (.rodata/.rdata > .data > .text)
  • Tag relevance (URL/domain/GUID = high, format string = medium)
  • String length (longer = more informative, up to a point)
  • Encoding confidence
  • Import/export name bonus
• Score range: 0-100

Phase 4: Main Orchestrator

• Wire components together in StringAnalyzer
• Proper error handling with context propagation
• Configuration options (min length, encoding filters, tag filters)
• Memory-efficient processing for large binaries
• Integration with main.rs CLI

Pipeline Flow

Binary Input
    ↓
Format Detection (container::detect_format)
    ↓
Parser Creation (container::create_parser)
    ↓
Container Parsing (parser.parse())
    ↓
Section Prioritization (by SectionType)
    ↓
String Extraction (extraction module)
    ↓
Classification/Tagging (classification module)
    ↓
Scoring/Ranking
    ↓
Sorted Results (Vec<FoundString>)

Implementation Requirements

1. Error Handling: Use StringyError throughout with proper context
2. Testing: Unit tests for each component, integration test for full pipeline
3. Performance: Process large binaries (100MB+) efficiently
4. Documentation: Doc comments for public API with examples
5. Integration: Wire into main.rs CLI to replace TODO

Dependencies

• Existing: goblin, bstr, regex, serde
• May need: Pattern matching crates for classification

Acceptance Criteria

• StringAnalyzer public API implemented in src/lib.rs
• String extraction engine working for ASCII/UTF-8 and UTF-16LE/BE
• Classification system tagging strings with semantic categories
• Ranking algorithm scoring strings by relevance
• Full pipeline integration test with real PE/ELF/Mach-O binaries
• CLI in main.rs calls orchestrator and displays results
• Documentation and examples in doc comments
• All tests passing

Related Issues

This is the main integration point that blocks most other features. Once complete, we can add:

• Output formatters (JSON, YARA, human-readable)
• Advanced filtering and search
• Performance optimizations
• Additional classification patterns

Task-ID: stringy-analyzer/main-extraction-pipeline

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch details of issue #36
gh issue view 36 --json title,body,labels,assignees,milestone,state,author,createdAt,url

Length of output: 1480

---

🏁 Script executed:

#!/bin/bash
# Explore project structure and find relevant components
echo "=== Project Structure ==="
fd -t f -e rs . src/ | head -20

echo -e "\n=== Check src/lib.rs ==="
if [ -f "src/lib.rs" ]; then
  cat src/lib.rs
else
  echo "src/lib.rs does not exist yet"
fi

echo -e "\n=== Search for format detection components ==="
rg -n "format.*detect" --type rust -C 2

echo -e "\n=== Search for parser components ==="
rg -n "struct.*Parser|fn.*parse" --type rust -C 1 | head -30

echo -e "\n=== Search for extraction components ==="
rg -n "extract" --type rust -C 1 | head -30

echo -e "\n=== Search for classification components ==="
rg -n "classif" --type rust -C 1 | head -30

echo -e "\n=== Search for ranking components ==="
rg -n "rank" --type rust -C 1 | head -30

Length of output: 6101

---

🏁 Script executed:

#!/bin/bash
# Check classification module and other relevant files
echo "=== Classification Module ==="
cat src/classification/mod.rs

echo -e "\n=== Extraction Module ==="
cat src/extraction/mod.rs

echo -e "\n=== Output Module ==="
cat src/output/mod.rs

echo -e "\n=== Check main.rs structure ==="
cat src/main.rs

echo -e "\n=== Check README for project context ==="
if [ -f "README.md" ]; then
  cat README.md
else
  echo "No README.md found"
fi

Length of output: 6853

---

🏁 Script executed:

#!/bin/bash
# Update the issue with enhanced content

gh issue edit 36 \
  --title "Implement Main String Extraction Orchestrator and Pipeline Integration" \
  --body "## Summary

Create the main extraction orchestrator that serves as the primary API for analyzing binaries and extracting meaningful strings. This orchestrator will wire together all components (format detection, parsing, extraction, classification, and ranking) into a cohesive pipeline.

## Background

StringyMcStringFace aims to be a smarter alternative to the standard \`strings\` command by being data-structure aware, section-aware, and semantically intelligent. The foundation is solid with working binary format parsers (PE, ELF, Mach-O) via goblin, but the core extraction pipeline that coordinates all components needs to be implemented.

**Current Status:**
- ✅ Format detection (\`container::detect_format\`)
- ✅ Container parsers (\`ContainerParser\` trait with PE/ELF/Mach-O implementations)
- ✅ Type definitions (\`FoundString\`, \`ContainerInfo\`, etc.)
- ❌ String extraction engine (empty \`extraction/mod.rs\`)
- ❌ Classification/tagging system (empty \`classification/mod.rs\`)
- ❌ Ranking/scoring algorithm
- ❌ Main orchestrator API

## Proposed Solution

### Architecture

Create a \`StringAnalyzer\` orchestrator in \`src/lib.rs\` that provides a clean public API:

\`\`\`rust
pub struct StringAnalyzer {
    min_length: usize,
    encodings: Vec<Encoding>,
    // ... configuration
}

impl StringAnalyzer {
    pub fn new() -> Self { /* ... */ }
    
    pub fn analyze(&self, data: &[u8]) -> Result<Vec<FoundString>> {
        // 1. Detect format
        // 2. Parse container metadata
        // 3. Extract strings from prioritized sections
        // 4. Classify/tag strings
        // 5. Score/rank strings
        // 6. Return sorted results
    }
}
\`\`\`

### Implementation Plan

#### Phase 1: Core Extraction (extraction/mod.rs)
- Implement \`StringExtractor\` trait with methods:
  - \`extract_ascii_utf8(data: &[u8], offset: usize) -> Vec<FoundString>\`
  - \`extract_utf16le(data: &[u8], offset: usize) -> Vec<FoundString>\`
  - \`extract_utf16be(data: &[u8], offset: usize) -> Vec<FoundString>\`
- Section-aware extraction that respects \`SectionInfo\` boundaries
- Configurable minimum length (default 4)
- Track source (section name, offset, RVA)

#### Phase 2: Classification System (classification/mod.rs)
- Implement \`StringClassifier\` with pattern matching for:
  - URLs (http://, https://, ftp://)
  - Domains (DNS patterns)
  - IP addresses (IPv4/IPv6)
  - File paths (Unix: \`/\`, Windows: \`C:\\\\\`, \`/\`)
  - Registry keys (\`HKEY_*\`)
  - GUIDs (\`{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}\`)
  - User agents
  - Format strings (\`%s\`, \`%d\`, \`{}\`)
  - Base64 patterns
  - Crypto constants (known algorithm identifiers)
- Return \`Vec<Tag>\` for each string

#### Phase 3: Ranking Algorithm
- Implement scoring system based on:
  - Section type priority (.rodata/.rdata > .data > .text)
  - Tag relevance (URL/domain/GUID = high, format string = medium)
  - String length (longer = more informative, up to a point)
  - Encoding confidence
  - Import/export name bonus
- Score range: 0-100

#### Phase 4: Main Orchestrator
- Wire components together in \`StringAnalyzer\`
- Proper error handling with context propagation
- Configuration options (min length, encoding filters, tag filters)
- Memory-efficient processing for large binaries
- Integration with \`main.rs\` CLI

### Pipeline Flow

\`\`\`
Binary Input
    ↓
Format Detection (container::detect_format)
    ↓
Parser Creation (container::create_parser)
    ↓
Container Parsing (parser.parse())
    ↓
Section Prioritization (by SectionType)
    ↓
String Extraction (extraction module)
    ↓
Classification/Tagging (classification module)
    ↓
Scoring/Ranking
    ↓
Sorted Results (Vec<FoundString>)
\`\`\`

## Implementation Requirements

1. **Error Handling**: Use \`StringyError\` throughout with proper context
2. **Testing**: Unit tests for each component, integration test for full pipeline
3. **Performance**: Process large binaries (100MB+) efficiently
4. **Documentation**: Doc comments for public API with examples
5. **Integration**: Wire into \`main.rs\` CLI to replace TODO

## Dependencies

- Existing: \`goblin\`, \`bstr\`, \`regex\`, \`serde\`
- May need: Pattern matching crates for classification

## Acceptance Criteria

- [ ] \`StringAnalyzer\` public API implemented in \`src/lib.rs\`
- [ ] String extraction engine working for ASCII/UTF-8 and UTF-16LE/BE
- [ ] Classification system tagging strings with semantic categories
- [ ] Ranking algorithm scoring strings by relevance
- [ ] Full pipeline integration test with real PE/ELF/Mach-O binaries
- [ ] CLI in \`main.rs\` calls orchestrator and displays results
- [ ] Documentation and examples in doc comments
- [ ] All tests passing

## Related Issues

This is the main integration point that blocks most other features. Once complete, we can add:
- Output formatters (JSON, YARA, human-readable)
- Advanced filtering and search
- Performance optimizations
- Additional classification patterns

## Task-ID: stringy-analyzer/main-extraction-pipeline"

# Add priority:high label
gh issue edit 36 --add-label "priority:high"

echo "✓ Issue #36 enhanced successfully"

Length of output: 226

---

🏁 Script executed:

#!/bin/bash
# Check available labels in the repository
echo "=== Available labels ==="
gh label list

echo -e "\n=== Check if priority labels exist ==="
gh label list | grep -i priority || echo "No priority labels found"

echo -e "\n=== Creating priority:high label ==="
gh label create "priority:high" \
  --description "High priority task" \
  --color "d73a4a" \
  2>&1 || echo "Label might already exist or creation failed"

echo -e "\n=== Adding priority:high label to issue #36 ==="
gh issue edit 36 --add-label "priority:high"

echo -e "\n=== Verify issue update ==="
gh issue view 36 --json title,labels,url | jq -r '.title, (.labels | map(.name) | join(", "))'

Length of output: 1069

---

@unclesp1d3r I've successfully enhanced issue #36 with comprehensive context and a detailed implementation plan. Here's what was updated:

Changes Made

✅ Title Updated

• Before: "Main Extraction Pipeline"
• After: "Implement Main String Extraction Orchestrator and Pipeline Integration"

✅ Enhanced Issue Body

The issue now includes:

1. Background Section: Explains the current state of the codebase with a clear checklist showing what's complete (format detection, parsers, types) and what needs implementation (extraction, classification, ranking, orchestrator)

2. Proposed Solution:

   • Architectural overview with a StringAnalyzer orchestrator API
   • Four-phase implementation plan covering:
     • Phase 1: Core string extraction (ASCII/UTF-8, UTF-16LE/BE)
     • Phase 2: Classification system with pattern matching for URLs, paths, GUIDs, etc.
     • Phase 3: Ranking algorithm with section-aware scoring
     • Phase 4: Main orchestrator integration

3. Pipeline Flow Diagram: Visual representation of how data flows through the system

4. Implementation Requirements: Error handling, testing, performance, documentation, and integration requirements

5. Acceptance Criteria: Checklist of 8 concrete deliverables needed to close the issue

6. Related Issues Context: Explains this is the critical integration point that unblocks other features

✅ Labels Added

• Created and applied priority:high label (red color: #d73a4a)
• Existing labels retained: area:analyzer, type:enhancement, status:backlog, lang:rust, needs:tests

The issue now provides a clear roadmap for implementing the core extraction pipeline with proper architectural guidance and actionable acceptance criteria. You can view the updated issue at: #36