Add String Extraction Framework with Noise Filtering and Benchmarks

[unclesp1d3r]

This pull request introduces a comprehensive benchmarking suite for ASCII string extraction and noise filtering, adds the entropy crate as a dependency, and significantly expands the documentation to detail the filtering architecture, configuration, and usage. The most important changes are grouped below:

Benchmarking and Performance Evaluation

• Added a new benchmark suite in benches/ascii_extraction.rs to measure the performance of ASCII string extraction and all individual noise filters, including entropy, character distribution, linguistic, length, repetition, context-aware, and composite filtering. Also benchmarks large binary extraction scenarios.
• Registered the new benchmark target ascii_extraction in Cargo.toml for Criterion-based performance testing.

Dependency Updates

• Added the entropy crate (version 0.4) to Cargo.toml to enable Shannon entropy calculations in both filtering logic and benchmarks.

Documentation Improvements: Extraction & Filtering

• Expanded the ASCII extraction documentation in docs/src/string-extraction.md to clarify algorithm details, provide code examples, and explain configuration options for minimum/maximum length. Added a new section for UTF-8 extraction.
• Added a detailed "Noise Filtering" section describing the multi-layered filtering system, filter types, confidence scoring, configuration, and performance characteristics. Includes code samples for using and customizing filters. [1] [2]
• Updated configuration documentation to reflect new options for extraction and noise filtering, including filter weights and validation of configuration.

These changes provide robust performance testing, enable advanced filtering, and make the documentation much more actionable for users and developers.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Summary by CodeRabbit

Release Notes

• New Features

  • Added ASCII string extraction with configurable minimum and maximum length filtering
  • Implemented multi-layer noise filtering system with six independent filters (character distribution, entropy, linguistic patterns, length, repetition, context-aware)
  • Added confidence scoring (0.0–1.0) for extracted strings to prioritize results
  • Enabled section-aware extraction preserving metadata and RVA information

• Documentation

  • Added comprehensive guides with configuration examples and best practices for string extraction and noise filtering

Walkthrough

Adds a complete ASCII/UTF-8 string extraction framework with a BasicExtractor and ExtractionConfig, a multi-filter noise‑filtering system with weighted filters and per-string confidence, AsciiExtraction utilities and benchmarks, docs updates, and extensive unit/integration tests; integrates confidence into existing extraction paths and re-exports new APIs at the crate root.

Changes

Cohort / File(s)	Change Summary
Core extraction framework src/extraction/mod.rs	New trait-based extraction framework (StringExtractor, ExtractionConfig, BasicExtractor), exports and integration for ASCII/UTF-8 extraction and noise filtering; test coverage and helpers added.
ASCII extraction src/extraction/ascii.rs, tests/test_ascii_extraction.rs, tests/test_ascii_integration.rs	New ASCII scanner and section-aware extraction (AsciiExtractionConfig, is_printable_ascii, extract_ascii_strings, extract_from_section), section/RVA population, noise-filter integration, and unit/integration tests.
Noise filtering config src/extraction/config.rs	New NoiseFilterConfig and FilterWeights with Default and validate() logic and unit tests.
Noise filtering implementation src/extraction/filters.rs, tests/test_noise_filters.rs	Pluggable filters and trait (NoiseFilter, FilterContext, CharDistributionFilter, EntropyFilter, LinguisticFilter, LengthFilter, RepetitionFilter, ContextFilter) and CompositeNoiseFilter with weighted aggregation and tests.
Types & integration edits src/types.rs, src/extraction/macho_load_commands.rs, src/extraction/pe_resources.rs	FoundString.confidence added (public), helper methods (is_high_confidence, is_low_confidence), and confidence initialized (1.0) in existing extraction paths.
Crate root & exports src/lib.rs	Re-exported AsciiExtractionConfig, BasicExtractor, ExtractionConfig, StringExtractor at crate root and updated docs/examples.
Benchmarks & Cargo Cargo.toml, benches/ascii_extraction.rs	Added entropy = "0.4.2" dependency, bumped several dev deps, and new Criterion bench ascii_extraction.
Documentation docs/src/string-extraction.md	Expanded docs: ASCII extraction algorithm, multi-filter noise filtering, configuration examples, and updated public API descriptions.
Tests & CI tests/*, .github/workflows/release.yml, dist-workspace.toml	Added comprehensive test suites (unit + integration + benches), bumped GitHub Actions versions and adjusted distribution config formatting/keys.

Sequence Diagram(s)

sequenceDiagram
    participant Caller as User Code
    participant BE as BasicExtractor
    participant AS as ASCII Scanner
    participant CNF as CompositeNoiseFilter
    participant SF as Sub-filter
    participant FS as FoundString

    Caller->>BE: extract(data, ExtractionConfig)
    activate BE
    BE->>AS: scan section bytes (AsciiExtractionConfig)
    AS-->>BE: candidates (Vec<FoundString>)
    alt noise_filtering_enabled
        BE->>CNF: CompositeNoiseFilter::new(NoiseFilterConfig)
        loop each candidate
            BE->>CNF: calculate_confidence(text, FilterContext)
            loop enabled sub-filters
                CNF->>SF: calculate_confidence(text, context)
                SF-->>CNF: score
            end
            CNF-->>BE: normalized confidence
            BE->>FS: set confidence, apply threshold
        end
    else filtering_disabled
        loop each candidate
            BE->>FS: set confidence = 1.0
        end
    end
    BE-->>Caller: filtered Vec<FoundString>
    deactivate BE

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

• Cross-file review needed for interdependent modules (mod.rs, ascii.rs, config.rs, filters.rs).
• Pay attention to entropy computation → 0..1 mapping, FilterWeights validation/normalization, CompositeNoiseFilter aggregation, and Section/RVA offset math.
• Review additions to FoundString and places where confidence is initialized (PE/Mach-O extractors).
• Check tests that rely on timing/fixtures for flakiness.

Files needing extra attention:

• src/extraction/filters.rs
• src/extraction/config.rs
• src/extraction/ascii.rs and src/extraction/mod.rs
• Integration tests under tests/ that use fixtures and timing assertions

Possibly related issues

• Implement ASCII String Noise Filtering with Intelligent Heuristics #10 — Implements the multi-filter noise-filtering stack and confidence scoring described in the issue.
• Implement ASCII String Extractor with Configurable Length Filtering #9 — Adds ASCII extractor and helpers requested by the issue.
• Core String Extraction Framework: StringExtractor Trait and Configuration #8 — Implements the extraction framework (StringExtractor, ExtractionConfig, BasicExtractor) referenced by the issue.

Possibly related PRs

• Add String Extraction Framework with Noise Filtering and Benchmarks #108 — Overlapping changes: ascii extraction, noise-filter config/weights, CompositeNoiseFilter, FoundString.confidence, benches and tests.
• feat: Implement Mach-O load command extraction with section weight normalization and enhanced tagging #67 — Related Mach‑O load-command extraction and type metadata changes overlapping with confidence initialization.
• feat: Adds section-weighting and improves ELF imports #45 — Related to section weighting/context propagation used by filters.

Suggested labels

github_actions

Suggested reviewers

• Copilot

Poem

🐇
I hop through bytes and sift the night,
I chase false noise and chase it right,
Filters hum and confidence grows,
I pull the strings where meaning shows,
Hooray — the extractor gleams with light! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title describes the introduction of a core string extraction framework with StringExtractor trait and configuration, which matches the main focus of implementation changes (new extraction module, config structures, filters, and BasicExtractor). However, the PR also adds significant benchmarking, dependencies, and documentation improvements that are equally substantial but not reflected in the title.
Description check	✅ Passed	The description comprehensively covers all major changes: benchmarking suite, entropy dependency addition, and documentation expansion with filtering architecture details. It accurately reflects the changeset scope and provides context for each major component.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between baf4526 and a3060ca.

📒 Files selected for processing (17)

• .github/workflows/release.yml (11 hunks)
• Cargo.toml (2 hunks)
• benches/ascii_extraction.rs (1 hunks)
• dist-workspace.toml (2 hunks)
• docs/src/string-extraction.md (4 hunks)
• src/extraction/ascii.rs (1 hunks)
• src/extraction/config.rs (1 hunks)
• src/extraction/filters.rs (1 hunks)
• src/extraction/macho_load_commands.rs (2 hunks)
• src/extraction/mod.rs (2 hunks)
• src/extraction/pe_resources.rs (3 hunks)
• src/lib.rs (4 hunks)
• src/types.rs (1 hunks)
• tests/integration_extraction.rs (1 hunks)
• tests/test_ascii_extraction.rs (1 hunks)
• tests/test_ascii_integration.rs (1 hunks)
• tests/test_noise_filters.rs (1 hunks)

Warning

Review ran into problems

🔥 Problems

Errors were encountered while retrieving linked issues.

Errors (1)

• UTF-8: Entity not found: Issue - Could not find referenced Issue.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[dosubot]

Related Documentation

Checked 14 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[unclesp1d3r]

@coderabbitai fix the PR title

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

@unclesp1d3r I'll update the PR title to be more descriptive and properly formatted.

---

🏁 Script executed:

#!/bin/bash
# Update the PR title to be more descriptive and properly formatted
gh pr edit 108 --title "Add String Extraction Framework with Noise Filtering and Benchmarks"

# Verify the change
gh pr view 108 --json title -q '.title'

Length of output: 128

---

@unclesp1d3r Done! I've updated the PR title from:

Old: 8 core string extraction framework stringextractor trait and configuration (PR #108)

New: Add String Extraction Framework with Noise Filtering and Benchmarks

The new title is properly capitalized, removes the issue number prefix and redundant PR reference, and clearly describes the key features introduced in this PR.

[Copilot]

Pull Request Overview

This pull request introduces a comprehensive string extraction and noise filtering framework for StringyMcStringFace. The implementation provides ASCII string extraction as a foundational encoding type, along with multi-layered heuristic noise filtering to reduce false positives. The changes also include extensive benchmarking infrastructure and detailed documentation.

Key Changes:

• Added ASCII string extraction module with configurable min/max length filtering
• Implemented 6-layer noise filtering system (entropy, character distribution, linguistic patterns, length, repetition, context-aware)
• Added confidence field (0.0-1.0) to FoundString to track noise filtering results
• Introduced StringExtractor trait and BasicExtractor implementation for framework-based extraction
• Added comprehensive benchmark suite for performance validation
• Expanded documentation with filtering architecture details and usage examples

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/extraction/ascii.rs	New ASCII extraction module with byte-level scanning
src/extraction/filters.rs	Multi-layered noise filtering implementation
src/extraction/config.rs	Configuration structures for extraction and filtering
src/extraction/mod.rs	Core extraction framework with StringExtractor trait
src/types.rs	Added confidence field to FoundString
tests/test_noise_filters.rs	Comprehensive unit tests for noise filters
tests/test_ascii_extraction.rs	Unit tests for ASCII extraction
tests/test_ascii_integration.rs	Integration tests with noise filtering
tests/integration_extraction.rs	Integration tests for extraction framework
benches/ascii_extraction.rs	Benchmark suite for extraction and filtering
docs/src/string-extraction.md	Expanded documentation for extraction and filtering
Cargo.toml	Added entropy dependency and benchmark registration

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

[nitpick] The is_printable_ascii function in the ASCII module excludes whitespace control characters (tab, newline, carriage return) from the printable range, only accepting 0x20-0x7E. However, the documentation mentions that this differs from is_printable_text_byte in extraction::mod which includes these characters. This inconsistency may cause confusion when strings contain whitespace.

Consider whether ASCII extraction should include common whitespace characters like tab (\t, 0x09), newline (\n, 0x0A), and carriage return (\r, 0x0D) to better handle formatted text and multi-line strings. The current strict definition may filter out legitimate strings that contain these characters.

[Copilot]

[nitpick] The repetition filter uses a nested loop with pattern matching that could be inefficient for very long strings. While there's a max_pattern_len cap at 16 characters, the algorithm still has O(n²) complexity in the worst case when checking multiple pattern lengths. For strings approaching the max_length threshold (200 characters default), this could cause performance degradation.

Consider adding early exit optimizations or reducing the max pattern length further if benchmarks show this is a bottleneck.

[Copilot]

The ExtractionConfig struct has both encodings and enabled_encodings fields (lines 139 and 153), which appear to serve the same purpose. This duplication creates confusion about which field should be used and requires checking both in the code (see lines 467-468 and 493-494 in mod.rs where both are checked with ||).

Recommend consolidating these into a single field, likely enabled_encodings, to simplify the API and reduce confusion.