Investigate validation failures - maintain 83.8% pass rate#46
Merged
Conversation
**Investigation Summary:** Analyzed remaining 16.2% validation failures to identify root causes and potential fixes. Created comprehensive investigation notes documenting findings and recommendations. **Failure Patterns Identified:** 1. High-frequency (25.90 MHz): 17 failures - SNR deviations 12-47 dB 2. Low-frequency (6.10, 7.20 MHz): 13 failures - reliability under-predicted ~20-25% 3. Mid-frequency spot failures: 5 isolated incidents **Key Findings:** - MUFday probability calculation shows 100x discrepancy at high frequencies (DVOACAP: 0.0002, VOACAP: 0.02 for hour 06, 25.90 MHz) - Sigma values vary by 2x between different calculation points - F2D deviation formula verified correct against Pascal source - Attempted fix (*2 scaling) failed, reducing pass rate to 62% **Conclusion:** Current 83.8% pass rate represents good agreement for most scenarios. Remaining failures concentrated in edge cases (near/over MUF, nighttime low freqs). Further improvements require VOACAP community consultation or real-world calibration. **Artifacts:** - INVESTIGATION_NOTES.md - Detailed findings and recommendations - Debug scripts for MUFday, F2D tables, and failure analysis - Debug hooks in prediction_engine.py and fourier_maps.py (disabled) **No functionality changes** - All debug code is disabled by default. Validation pass rate remains at 83.8% (181/216 tests passing).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Investigation Summary:
Analyzed remaining 16.2% validation failures to identify root causes and potential fixes. Created comprehensive investigation notes documenting findings and recommendations.
Failure Patterns Identified:
Key Findings:
Conclusion:
Current 83.8% pass rate represents good agreement for most scenarios. Remaining failures concentrated in edge cases (near/over MUF, nighttime low freqs). Further improvements require VOACAP community consultation or real-world calibration.
Artifacts:
No functionality changes - All debug code is disabled by default. Validation pass rate remains at 83.8% (181/216 tests passing).