feat: Add evaluation engine with offset resolution and CI/CD automation#5
Conversation
…zation - Renumbered and reorganized tasks in `tasks.md` to improve clarity and maintain a consistent structure. - Added sub-tasks for offset resolution, type reading, operator creation, rule evaluation, and output formatting to enhance task tracking. - Updated task descriptions to reflect current implementation requirements and ensure alignment with project goals. These changes streamline the development process and provide clearer guidance for contributors on upcoming tasks and their dependencies. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Added `homepage` and `documentation` fields to `Cargo.toml` for improved project visibility and resource access. - Introduced `dist-workspace.toml` to configure workspace settings and CI parameters for building and releasing artifacts across multiple platforms. - Created a comprehensive GitHub Actions workflow in `release.yml` to automate the release process, including artifact building, uploading, and GitHub release creation. - Implemented offset resolution logic in `offset.rs`, including error handling and comprehensive unit tests to ensure robustness. These changes improve project organization, enhance automation capabilities, and ensure better handling of file offsets during evaluation. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Added `read_byte` function in `src/evaluator/types.rs` for safely reading a single byte from a buffer with bounds checking. - Introduced `TypeReadError` enum to handle buffer overrun errors during byte reading operations. - Updated `mod.rs` to include the new `types` module. - Marked task 6.1 as complete in `tasks.md`, reflecting the implementation of basic type reading for byte values. These changes enhance the evaluator's capability to interpret byte data safely, improving overall robustness and error handling in the file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Completed tasks for reading multi-byte types, including `read_short` and `read_long` functions in `src/evaluator/types.rs`, supporting both signed and unsigned interpretations with endianness handling. - Introduced `read_typed_value` function to interpret bytes according to the specified `TypeKind`, allowing for flexible type reading. - Enhanced error handling with the addition of `UnsupportedType` variant in `TypeReadError` for unsupported types. - Updated `tasks.md` to reflect the completion of tasks related to multi-byte type reading and interpretation. These changes significantly improve the evaluator's capability to handle various data types, enhancing the robustness and flexibility of the file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Completed the implementation of the basic equality operator in `src/evaluator/operators.rs` with the `apply_equal` function, allowing for type-safe comparisons between different `Value` variants. - Updated `mod.rs` to include the new `operators` module. - Marked task 7.1 as complete in `tasks.md`, reflecting the addition of value-to-value comparison functionality. These changes enhance the evaluator's capability to accurately compare values, improving the robustness of the file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…alue comparison - Completed the implementation of the inequality operator with the `apply_not_equal` function, allowing for type-safe comparisons between different `Value` variants. - Added the bitwise AND operator with the `apply_bitwise_and` function, enabling pattern matching for integer values. - Introduced the `apply_operator` function to dispatch operations based on the `Operator` enum, supporting equality, inequality, and bitwise AND operations. - Updated `tasks.md` to reflect the completion of tasks related to operator creation and application. These enhancements improve the evaluator's capability to perform complex comparisons and operations, further strengthening the file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Added the `evaluate_single_rule` function in `src/evaluator/mod.rs` to evaluate a single magic rule against a file buffer. - The function resolves the rule's offset, reads the corresponding bytes, and applies the specified operator for comparison. - Comprehensive unit tests have been included to validate the functionality of the evaluation logic, covering various scenarios including equality, inequality, and bitwise operations. - Updated `tasks.md` to reflect the completion of the task for basic rule evaluation. These enhancements significantly improve the evaluator's capability to process and validate magic rules, enhancing the overall file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Deleted the `llms-txt-updater.kiro.hook`, `rust-code-analyzer.kiro.hook`, `rust-perf-analyzer.kiro.hook`, `rust-quality-monitor.kiro.hook`, and `rust-security-hardening.kiro.hook` files as they are no longer needed. - These hooks were previously used for automated analysis and updates but have been replaced with more efficient user-triggered mechanisms to enhance control and accuracy in code quality and performance assessments. This cleanup improves the project's automation infrastructure and reduces potential confusion regarding obsolete hooks. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…state - Added the `EvaluationContext` struct in `src/evaluator/mod.rs` to maintain the state during rule evaluation, including current offset, recursion depth, and configuration settings. - Implemented methods for managing the offset and recursion depth, including incrementing and decrementing the recursion depth with error handling for exceeding limits. - Updated `lib.rs` to re-export `EvaluationContext` for convenience. - Marked task 8.2 as complete in `tasks.md`, reflecting the addition of the evaluation context structure. These enhancements improve the evaluator's ability to manage state during complex rule evaluations, contributing to a more robust file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…le processing - Completed the implementation of the `EvaluationConfig` struct in `src/lib.rs`, allowing for customizable evaluation options such as recursion limits, string length limits, and timeout settings. - Introduced the `MatchResult` struct to encapsulate results from rule evaluations, including matched messages and offsets. - Implemented the `evaluate_rules` function in `src/evaluator/mod.rs` to support hierarchical rule evaluation, enabling parent-child relationships in rules and refined matching. - Updated `tasks.md` to reflect the completion of tasks related to evaluation configuration and hierarchical processing. These enhancements improve the evaluator's flexibility and robustness, allowing for more complex and efficient file type detection processes. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…with const functions - Updated `EvaluationConfig` in `src/lib.rs` to derive `Eq` for improved comparison capabilities. - Refactored methods in `EvaluationContext` within `src/evaluator/mod.rs` to be `const fn`, allowing for compile-time evaluation and optimizations. - Adjusted method signatures for `new`, `current_offset`, `set_current_offset`, `recursion_depth`, `config`, `should_stop_at_first_match`, `max_string_length`, `enable_mime_types`, `timeout_ms`, and `reset` to reflect the new const nature. - Updated error handling in `src/io/mod.rs` to simplify path handling in error types. These changes improve the efficiency and usability of the evaluation configuration and context, enhancing the overall robustness of the file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…hooks - Modified the `rust-code-analyzer.kiro.hook` to analyze only the changed files in the current branch, enhancing its accuracy in identifying code quality issues. - Updated the `rust-perf-optimizer.kiro.hook` to reflect the same change, ensuring performance analysis is based on the current diff. - These updates improve the hooks' functionality and maintainability, aligning them with the project's focus on code quality and performance. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…or enhanced clarity and functionality - Updated the `MatchResult` struct in `src/output/mod.rs` to include additional fields such as `length`, `rule_path`, `confidence`, and `mime_type`, providing more comprehensive match information. - Introduced the `EvaluationResult` struct to encapsulate the results of file evaluations, including metadata about the evaluation process and a vector of successful matches. - Enhanced documentation with examples for both structs, improving usability and understanding of the evaluation results. - Adjusted the `Cargo.toml` to modify lint settings, changing specific lints from "allow" to "warn" for better code quality enforcement. These changes improve the output formatting module's capability to represent evaluation results, contributing to a more robust file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Created a new module `text.rs` in the `src/output` directory to handle text formatting of match results, providing a human-readable output style compatible with the GNU `file` command. - Implemented functions for formatting single and multiple match results, as well as complete evaluation results, ensuring clear and structured output. - Updated `tasks.md` to reflect the completion of tasks related to text output formatting, including unit tests for various scenarios. These enhancements improve the output capabilities of the library, facilitating better user interaction and understanding of file type detection results. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…dling - Changed `set_current_offset` and `reset` methods in `src/evaluator/mod.rs` from `const fn` to regular functions to allow for mutable state changes. - Improved error handling in `evaluate_rules` to provide more context for rule evaluation failures, including rule message and offset in error messages. - Enhanced `read_byte` function in `src/evaluator/types.rs` to use `.copied()` for better clarity and safety. - Added TODOs across various modules for comprehensive error handling, validation, and performance monitoring improvements, including file metadata validation and match result checks. These changes enhance the robustness and clarity of the evaluation logic, paving the way for improved error management and validation in the file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Added detailed documentation for `load_from_file` and `evaluate_file` methods in `src/lib.rs`, including arguments, error handling, and usage examples to improve clarity and usability. - Introduced a constant `SMALL_FILE_THRESHOLD` in `src/io/mod.rs` to indicate the threshold for small file handling, enhancing code readability and maintainability. These updates improve the overall documentation quality, making it easier for users to understand and utilize the library's functionality effectively. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Added `nix` crate as a development dependency in `Cargo.toml` for enhanced file type handling capabilities. - Updated GitHub Actions in `dist-workspace.toml` to use newer versions for `attest-build-provenance` and `upload-artifact`, ensuring compatibility and improved functionality. - Refactored the `release.yml` workflow to utilize the updated action version for build provenance. These changes enhance the development environment and CI/CD pipeline, contributing to a more robust and maintainable project structure. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
… configuration - Reformatted the `allowed_elements` array in `.markdownlint.json` to a single line for improved readability and consistency. - This change enhances the clarity of the configuration file without altering its functionality. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…n configuration - Added comprehensive security documentation in `src/lib.rs`, outlining memory safety, bounds checking, resource limits, input validation, error handling, and timeout protection. - Improved the `validate` method in `EvaluationConfig` to include checks for stack overflow, memory exhaustion, denial of service, and integer overflow vulnerabilities. - Introduced tests for potential integer overflow vulnerabilities in offset calculations in `src/evaluator/offset.rs`. - Enhanced the `read_byte` function in `src/evaluator/types.rs` with strict bounds checking to prevent buffer overruns and ensure secure byte reading. - Updated `create_memory_mapping` in `src/io/mod.rs` to include security considerations for memory mapping. These changes significantly bolster the security posture of the library, ensuring safer file type detection and evaluation processes. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
- Modified the `nix` dependency in `Cargo.toml` to enable the `fs` feature, enhancing file system capabilities for the project. - Added an ignore attribute to the FIFO rejection test in `src/io/mod.rs` to prevent hanging issues in CI environments. - Updated the text output formatting test in `src/output/text.rs` to clarify handling of Windows paths on Unix systems. These changes improve the development environment and ensure more robust testing practices, contributing to a more reliable file type detection process. Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…tting - Update filename extraction logic in `format_evaluation_result` function - Add platform-specific handling for Windows and Unix paths - Modify test case to support different path extraction behaviors - Ensure consistent filename display across different operating systems - Add Windows-specific assertion for filename extraction Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
|
Caution Review failedThe pull request is closed. Summary by CodeRabbit
WalkthroughAdds a GitHub Actions release workflow and a dist workspace; introduces a public evaluator API (offset resolution, operators, typed readers, EvaluationContext), structured output types and text formatters, enhanced I/O validation and symlink helpers, EvaluationConfig and timeout error, many unit tests, dependency and pre-commit updates, and removal/adjustment of several .kiro hooks. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Dev as Developer (push tag)
participant GH as GitHub Actions
participant Plan as Plan Job
participant BuildLocal as Build-local-artifacts (matrix)
participant BuildGlobal as Build-global-artifacts
participant Host as Host
participant Release as Create Release (gh)
Dev->>GH: push version tag
GH->>Plan: run plan (dist plan/host)
Plan-->>GH: manifest outputs
GH->>BuildLocal: spawn matrix per target
BuildLocal-->>GH: upload per-target artifacts & manifests
GH->>BuildGlobal: aggregate artifacts, run global builds, generate SBOMs
BuildGlobal-->>GH: upload global artifacts (archives, SBOM)
GH->>Host: publish artifacts to host path (dist host)
Host-->>GH: host manifest for release notes
GH->>Release: generate body & run gh CLI to create release + attach artifacts
Release-->>Dev: release created
sequenceDiagram
autonumber
participant Caller as Evaluation API caller
participant EvalCtx as EvaluationContext
participant Offset as offset::resolve_offset
participant Reader as types::read_typed_value
participant Operator as operators::apply_operator
participant Results as output::EvaluationResult
Caller->>EvalCtx: create or with_config
Caller->>EvalCtx: evaluate_rules_with_config(rules, buffer)
EvalCtx->>Offset: resolve_offset(spec, buffer)
Offset-->>EvalCtx: resolved offset / OffsetError
EvalCtx->>Reader: read_typed_value(buffer, offset, type)
Reader-->>EvalCtx: Value or TypeReadError
EvalCtx->>Operator: apply_operator(op, left, right)
Operator-->>EvalCtx: bool (match)
EvalCtx-->>Results: collect MatchResult(s)
Results-->>Caller: EvaluationResult (metadata / error)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Pre-merge checks and finishing touches and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro Disabled knowledge base sources:
📒 Files selected for processing (22)
Comment |
…atting - Add cfg-if dependency to Cargo.toml for conditional compilation - Update text output formatting test to handle different path extraction on Windows and Unix - Implement cfg_if! macro to conditionally assert filename extraction based on platform - Enhance test coverage for filename extraction in evaluation result formatting Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
|
@CodeRabbit improve the PR title and update the metadata, such as labels |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 1657 🏁 Script executed: Length of output: 17 🏁 Script executed: Length of output: 953 🏁 Script executed: Length of output: 50 @unclesp1d3r: I've successfully updated the PR metadata: Title updated:
Labels added:
The new title better captures the full scope of this PR, which includes not just offset resolution but the complete evaluation engine implementation, CI/CD automation via cargo-dist, and significant enhancements across multiple modules. |
- Modify Linux platform matrix to use ubuntu-22.04 instead of generic 'arm' - Ensure consistent and reliable Linux platform testing configuration - Improve cross-platform compatibility for CI/CD pipeline Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
This pull request introduces significant improvements to the project's CI/CD pipeline and updates automation hooks for code quality and documentation. The main changes include adding an autogenerated release workflow for building and publishing artifacts, removing an outdated documentation updater hook, and refining the Rust code analyzer hook to trigger on user request and improve its file detection logic.
CI/CD Pipeline Enhancements:
.github/workflows/release.ymlfile that automates the release process usingcargo-dist. This workflow builds platform-specific and global artifacts, uploads them, and creates GitHub Releases with autogenerated changelogs. It supports both regular and prerelease versions, and includes artifact attestation and SBOM generation for security and compliance.Automation Hooks Updates:
.kiro/hooks/llms-txt-updater.kiro.hookfile, discontinuing automatic updates to thellms.txtfile when documentation or repository structure changes. This means future updates tollms.txtwill need to be handled manually..kiro/hooks/rust-code-analyzer.kiro.hookto trigger analysis via user action rather than on every file edit, reducing unnecessary runs. The hook now usesgit diffagainstorigin/mainto identify changed files, ensuring more accurate and relevant code quality checks.