Skip to content

HyperBB: add binary calibration file support (.hbb_cal / .hbb_tcal)#13

Draft
Copilot wants to merge 4 commits intodevfrom
copilot/add-support-for-new-data-formats
Draft

HyperBB: add binary calibration file support (.hbb_cal / .hbb_tcal)#13
Copilot wants to merge 4 commits intodevfrom
copilot/add-support-for-new-data-formats

Conversation

Copy link

Copilot AI commented Mar 13, 2026

scipy.interpolate.interp2d was removed in scipy 1.14.0, making HyperBB completely non-functional. This PR fixes that and adds support for the 2025 Sequoia calibration file format update, while keeping backward compatibility with existing users.

Critical fix: interp2d → RegularGridInterpolator

All three dark-offset interpolators and the temperature correction interpolation now use RegularGridInterpolator. The temperature grid is pre-computed once in __init__ over 0–50°C (resolves an existing TODO optimize comment):

# Before (broken with scipy ≥ 1.14)
self.f_dark_cal_scat_1 = interp2d(gain_vals, wl_vals, scat1, kind='linear')
# ...called as: scat1 - self.f_dark_cal_scat_1(raw[:, idx_PmtGain], wl)

# After
self.f_dark_cal_scat_1 = RegularGridInterpolator(
    (wl_vals, gain_vals), scat1, method='linear', bounds_error=False, fill_value=None)
# ...called as: scat1 - self.f_dark_cal_scat_1(np.column_stack([wl, raw[:, idx_PmtGain]]))

New binary calibration file format

Sequoia now ships binary calibration files (.hbb_cal for plaque, .hbb_tcal for temperature) produced by Hbb_ConvertCalibrations.m. The calibration format is selected explicitly in the Setup dialog:

  • Legacy format (cal_format='legacy'): two separate .mat files — a plaque file (containing a cal struct) and a temperature file (containing a cal_temp struct). Behaviour unchanged for existing users.
  • Current format (cal_format='current'): binary .hbb_cal plaque file + binary .hbb_tcal temperature file. Both files are required for live data temperature correction.

Two new static methods implement the binary readers following the exact format defined in Hbb_ReadBinaryCalFile.m and Hbb_ReadBinaryTempCalFile.m (Sequoia Scientific, 2024):

  • _read_binary_plaque_cal() — reads the fixed 72-byte header then variable-length wavelength, gain, and dark-offset arrays. 2-D dark offset arrays are reshaped with Fortran (column-major) order to match MATLAB's fwrite/reshape behaviour. Handles multiple appended records and returns the last (consistent with MATLAB's cal = cal(end)). Returns a dict with the same field names as the legacy .mat cal struct so no downstream changes are required.
  • _read_binary_temp_cal() — reads wavelengths and polynomial coefficient matrix (column-major). Returns {'wl': ..., 'coeff': ...} compatible with the legacy .mat cal_temp struct.

Data output formats

Data output formats are unchanged: legacy, advanced, and light are the only supported formats.

Setup UI changes

  • New Cal. Format dropdown (legacy / current) in the Calibration Files group.
  • Temperature File is required for both formats.
  • Browse dialogs accept both .mat (legacy) and .hbb_cal/.hbb_tcal (current) files.

📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

Copilot AI and others added 2 commits March 13, 2026 21:08
… formats

Co-authored-by: doizuc <6456289+doizuc@users.noreply.github.com>
…ing, add doc comment

Co-authored-by: doizuc <6456289+doizuc@users.noreply.github.com>
Copilot AI changed the title [WIP] Add support for updated data and calibration file formats HyperBB: fix broken scipy interp2d removal, add new calibration and data formats Mar 13, 2026
Copilot AI requested a review from doizuc March 13, 2026 21:11
@doizuc doizuc changed the title HyperBB: fix broken scipy interp2d removal, add new calibration and data formats HyperBB: add new calibration format Mar 15, 2026
@doizuc
Copy link
Member

doizuc commented Mar 16, 2026

@copilot The instrument output the same data formats as before, please remove the new 'standard' format and keep the current light, advanced, and legacy output format.
The only thing that change is the calibration files. Adjust the setup menu to either select 'legacy' or 'current' calibration files. If legacy provide the current two calibration files, if current only use the single calibration format provided in the new format. Please make sure to use the Hbb_ReadBinaryCalFile.m file provided by Sequoia (in the matlab demo code available here) to read the data.

…dd explicit cal_format UI

Co-authored-by: doizuc <6456289+doizuc@users.noreply.github.com>
Copy link
Author

Copilot AI commented Mar 16, 2026

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Copilot AI changed the title HyperBB: add new calibration format HyperBB: add binary calibration file support (.hbb_cal / .hbb_tcal) Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants