Skip to content

Add paper-aligned trap metrics, FFT/burden variants, and trap verification/remove workflow#359

Open
charlesmartin14 wants to merge 1 commit intomasterfrom
codex/add-paper-aligned-metrics-to-analyze_traps-48pssg
Open

Add paper-aligned trap metrics, FFT/burden variants, and trap verification/remove workflow#359
charlesmartin14 wants to merge 1 commit intomasterfrom
codex/add-paper-aligned-metrics-to-analyze_traps-48pssg

Conversation

@charlesmartin14
Copy link
Copy Markdown
Member

Motivation

  • Align trap analysis with the NeurIPS paper by producing Porter-Thomas-centered localization metrics and a clear scalar "variance burden" while preserving legacy diagnostics for compatibility.
  • Provide richer experiment support (burden variant sweeps, Fourier/FFT diagnostics) to let users compare localization/burden definitions and frequency-domain behavior.
  • Improve reproducibility and safety of trap removal by recording permutation identity, signing artifacts, and adding verification logic to remove_traps.

Description

  • Introduces new modules: trap_burden_variants.py, trap_fourier.py, and trap_identity.py implementing burden-variant math, Fourier/FFT diagnostics, and permutation/trap identity & verification helpers.
  • Extends trap analysis to return paper-facing metrics (trap_delta, trap_ipr, trap_q, trap_diffuseness, trap_top_sector_overlap, trap_variance_burden, layer_trap_variance_burden) and optional burden variants and FFT-derived columns, plus options burden_variants, return_burden_components, return_burden_raw, trap_fft and trap_fft_config.
  • Adds deterministic permutation metadata (perm_signature, permutation_n, trap_seed, trap_identity_key) and verification tooling used by remove_traps; remove_traps gained parameters verify_traps, return_analyze, traps, rtol, atol, and min_vector_cosine, and can return a verification dataframe when requested.
  • Refactors and consolidates trap helper functions into trap_analysis.py and hooks them into weightwatcher.WeightWatcher (new helper methods: compute_trap_delta, compute_trap_ipr_q, compute_trap_ipr_q_uniform, compute_top_sector_overlap, compute_trap_variance_burden, and compute_original_basis_for_traps), and preserves legacy heuristics for backward compatibility.
  • Documentation and README updated to describe the new metrics, defaults, and recommended FFT config fields.

Testing

  • Ran the trap unit tests including tests/test_analyze_traps.py (expanded to cover new paper metrics, top-sector behavior, and index conventions), and they passed.
  • Added and ran tests/test_trap_burden_variants.py covering burden math and API behavior, and tests/test_trap_fft.py covering Fourier helpers and integration hooks, and these tests passed.
  • Updated tests/test_remove_traps.py with verification/return_analyze tests exercising the new remove_traps behavior, and those tests passed (torch-dependent tests are skipped when PyTorch is unavailable).

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant