Implement ASCII String Noise Filtering with Intelligent Heuristics

[unclesp1d3r]

Summary

Implement intelligent heuristics to distinguish legitimate ASCII strings from binary noise, padding, and non-textual data during string extraction from binary files.

Problem Context

When extracting ASCII strings from binary files, the analyzer currently captures all byte sequences that technically qualify as ASCII characters. However, this leads to several quality issues:

• Binary Noise: Random byte sequences that happen to fall within ASCII ranges but carry no semantic meaning
• Padding Data: Repetitive null bytes, spaces, or other padding characters used for alignment
• Table Data: Structured binary data (e.g., lookup tables, numerical arrays) that may contain ASCII-range values but aren't human-readable strings
• Low Entropy Content: Sequences like repeated characters or patterns that don't represent meaningful text

Without proper filtering, these false positives pollute the extraction results, making it difficult for users to identify genuinely useful strings and significantly reducing the tool's practical value.

Proposed Solution

Implement a multi-layered heuristic filtering system in Rust:

1. Character Distribution Analysis

• Calculate character frequency distributions
• Flag strings with abnormal character ratios (e.g., >80% punctuation, all same character)
• Apply entropy calculations to identify low-information content

2. Linguistic Patterns

• Check for reasonable vowel-to-consonant ratios (for English-like strings)
• Identify common word patterns and character bigrams/trigrams
• Validate against common string patterns (paths, URLs, error messages)

3. Context-Aware Filtering

• Leverage binary section information (.text, .data, .rodata, etc.)
• Apply different heuristics based on section type
• Consider string location and surrounding bytes

4. Configurable Thresholds

• Allow users to adjust sensitivity levels
• Provide options to include/exclude specific pattern types
• Support allowlist/denylist patterns

5. Scoring System

• Assign confidence scores to extracted strings
• Allow filtering by minimum confidence threshold
• Provide transparency into why strings were included/excluded

Acceptance Criteria

• Implement character distribution analysis functions
• Implement linguistic pattern detection
• Integrate section context into filtering decisions
• Add configurable threshold parameters
• Implement confidence scoring system
• Reduce false positive rate by >70% on test binaries
• Maintain >95% true positive retention
• Add comprehensive unit tests for each heuristic
• Document heuristic algorithms and tuning parameters
• Benchmark performance impact (<10% overhead target)

Technical Notes

• Consider using the entropy crate for information theory calculations
• Section context can be obtained from ELF/PE/Mach-O parsers (e.g., goblin crate)
• Performance-critical code should be profiled and optimized
• Heuristics should be modular to allow easy addition/removal

Requirements

1.4, 2.1

Dependencies

• Blocked by: Basic ASCII String Extraction (Implement ASCII String Extractor with Configurable Length Filtering #9)

Task-ID

stringy-analyzer/ascii-noise-filtering

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch the details of issue #10
gh issue view 10 --repo EvilBit-Labs/StringyMcStringFace --json title,body,labels,assignees,state,author

Length of output: 1280

---

🏁 Script executed:

#!/bin/bash
# Check available labels in the repository
gh label list --repo EvilBit-Labs/StringyMcStringFace --json name,description --limit 100

Length of output: 1182

---

🏁 Script executed:

#!/bin/bash
# Update the issue with enhanced content and metadata

# New enhanced body
cat > /tmp/issue_body.md << 'EOF'
## Summary

Implement intelligent heuristics to distinguish legitimate ASCII strings from binary noise, padding, and non-textual data during string extraction from binary files.

## Problem Context

When extracting ASCII strings from binary files, the analyzer currently captures all byte sequences that technically qualify as ASCII characters. However, this leads to several quality issues:

- **Binary Noise**: Random byte sequences that happen to fall within ASCII ranges but carry no semantic meaning
- **Padding Data**: Repetitive null bytes, spaces, or other padding characters used for alignment
- **Table Data**: Structured binary data (e.g., lookup tables, numerical arrays) that may contain ASCII-range values but aren't human-readable strings
- **Low Entropy Content**: Sequences like repeated characters or patterns that don't represent meaningful text

Without proper filtering, these false positives pollute the extraction results, making it difficult for users to identify genuinely useful strings and significantly reducing the tool's practical value.

## Proposed Solution

Implement a multi-layered heuristic filtering system in Rust:

### 1. **Character Distribution Analysis**
- Calculate character frequency distributions
- Flag strings with abnormal character ratios (e.g., >80% punctuation, all same character)
- Apply entropy calculations to identify low-information content

### 2. **Linguistic Patterns**
- Check for reasonable vowel-to-consonant ratios (for English-like strings)
- Identify common word patterns and character bigrams/trigrams
- Validate against common string patterns (paths, URLs, error messages)

### 3. **Context-Aware Filtering**
- Leverage binary section information (.text, .data, .rodata, etc.)
- Apply different heuristics based on section type
- Consider string location and surrounding bytes

### 4. **Configurable Thresholds**
- Allow users to adjust sensitivity levels
- Provide options to include/exclude specific pattern types
- Support allowlist/denylist patterns

### 5. **Scoring System**
- Assign confidence scores to extracted strings
- Allow filtering by minimum confidence threshold
- Provide transparency into why strings were included/excluded

## Acceptance Criteria

- [ ] Implement character distribution analysis functions
- [ ] Implement linguistic pattern detection
- [ ] Integrate section context into filtering decisions
- [ ] Add configurable threshold parameters
- [ ] Implement confidence scoring system
- [ ] Reduce false positive rate by >70% on test binaries
- [ ] Maintain >95% true positive retention
- [ ] Add comprehensive unit tests for each heuristic
- [ ] Document heuristic algorithms and tuning parameters
- [ ] Benchmark performance impact (<10% overhead target)

## Technical Notes

- Consider using the `entropy` crate for information theory calculations
- Section context can be obtained from ELF/PE/Mach-O parsers (e.g., `goblin` crate)
- Performance-critical code should be profiled and optimized
- Heuristics should be modular to allow easy addition/removal

## Requirements

1.4, 2.1

## Dependencies

- **Blocked by**: Basic ASCII String Extraction (#TBD)

## Task-ID

`stringy-analyzer/ascii-noise-filtering`
EOF

# Update the issue
gh issue edit 10 \
  --repo EvilBit-Labs/StringyMcStringFace \
  --title "Implement ASCII String Noise Filtering with Intelligent Heuristics" \
  --body-file /tmp/issue_body.md \
  --add-label "priority:medium" \
  --add-label "MVP"

echo "Issue #10 updated successfully!"

Length of output: 94

---

@unclesp1d3r I've successfully enhanced issue #10! Here's what I updated:

Changes Made:

📝 Title

• Before: "ASCII Noise Filtering"
• After: "Implement ASCII String Noise Filtering with Intelligent Heuristics"
• More descriptive and action-oriented

📄 Body Enhancements

Added comprehensive sections including:

1. Problem Context - Explains why noise filtering is critical, covering:

   • Binary noise issues
   • Padding data problems
   • Table data false positives
   • Low entropy content challenges

2. Proposed Solution - Multi-layered approach with:

   • Character distribution analysis
   • Linguistic pattern detection
   • Context-aware filtering (section-based)
   • Configurable thresholds
   • Confidence scoring system

3. Acceptance Criteria - 10 concrete checkboxes including:

   • Implementation requirements
   • Performance targets (>70% false positive reduction, >95% true positive retention)
   • Testing and documentation needs
   • Performance overhead target (<10%)

4. Technical Notes - Implementation guidance:

   • Suggested Rust crates (entropy, goblin)
   • Performance considerations
   • Modularity requirements

🏷️ Labels Added

• priority:medium - Important for quality but not critical
• MVP - Part of minimum viable product

You can view the updated issue here: #10

[unclesp1d3r]

@coderabbitai Enhance this issue body with additional context and a proposed solution. Improve the issue title and assign the proper labels and other metadata.