feat: add QualityScorer module for automatic data quality assessment (#34)

[AliiiBenn]

Summary

• Implements the QualityScorer module for automatic data quality assessment of pandas DataFrames
• Provides comprehensive quality reports with scores, grades, and actionable recommendations
• Detects common data quality issues: null values, duplicates, empty columns, outliers
• Includes 29 comprehensive tests with 99% code coverage

Implementation Details

QualityScorer Class (excel_to_sql/auto_pilot/quality.py)

Core Features:

• Quality Score (0-100): Starts at 100, deducts points for:

  • Null values above threshold (default 10%)
  • Duplicates in potential primary key columns (2 points each)
  • Empty columns (5 points each)
  • Statistical outliers (0.1 points each, capped at 10 total)

• Letter Grade Assignment:

  • A: 90-100 (excellent quality)
  • B: 75-89 (good quality)
  • C: 60-74 (acceptable quality)
  • D: 1-59 (poor quality)
  • F: 0 (unusable)

• Issue Detection:

  • Null value percentage per column
  • Duplicate values in potential PK columns
  • Empty columns (100% null)
  • Type mismatches (numeric data stored as object)
  • Statistical outliers using 3-sigma rule

• Per-Column Statistics:

  • Data type, null count/percentage
  • Unique count/percentage
  • Sample values (top 5)
  • Primary key potential (≥95% unique)
  • Empty column flag

API Usage:

from excel_to_sql.auto_pilot.quality import QualityScorer

scorer = QualityScorer()
report = scorer.generate_quality_report(df, "products")

print(f"Score: {report['score']}/100")  # e.g., 85
print(f"Grade: {report['grade']}")      # e.g., 'B'
print(f"Issues: {report['issues']}")    # List of detected issues

Configuration:

scorer = QualityScorer(
    null_threshold=15,    # Deduct for nulls above 15%
    grade_a_min=85,       # A grade at 85+
    grade_b_min=70,       # B grade at 70+
    grade_c_min=55        # C grade at 55+
)

Testing (tests/test_quality_scorer.py)

29 comprehensive tests covering:

• Quality report generation (basic, high quality, with issues)
• Edge cases (empty DataFrame, insufficient data)
• Duplicate detection in potential PKs
• Empty column detection
• Outlier detection using 3-sigma rule
• Letter grade assignment
• Column statistics accuracy
• Score calculation logic
• Configuration (default/custom thresholds)
• Type hints and docstrings
• Integration tests with realistic datasets

Test Results: ✅ All 29 tests passing
Code Coverage: 99% for quality module

Technical Notes

Outlier Detection

• Uses 3-sigma rule: values outside mean ± 3×std are outliers
• Requires at least 10 data points for meaningful detection
• Capped at 10 points deduction total to avoid excessive penalties

Primary Key Detection

• Columns with ≥95% unique values are flagged as potential PKs
• Duplicate checking only applies to potential PK columns
• Helps identify unique identifiers and reference data

Type Hints

• Full type annotation support using from __future__ import annotations
• All public methods have comprehensive docstrings with examples

Resolves

#34 - Implement missing QualityScorer module for Auto-Pilot mode

Checklist

• Implementation complete
• All tests passing (29/29)
• Code coverage: 99%
• Docstrings and examples added
• Atomic commits created
• Ready for review

🤖 Generated with Claude Code