Use ProbeTable to generate NP probes information #349

chrishalcrow · 2025-05-15T13:39:19Z

PR to use ProbeTable (https://github.com/billkarsh/ProbeTable/) to generate NeuroPixels probe information, instead of hardcoding this ourselves.

Summary

Import differences in src/probeinterface/neuropixels_tools.py

Does a few things:

Replaces the information in the npx_descriptions hardcoded dict, with information we can read from ProbeTable.
We now extract a mux_table_array from the probe_features.json, file, which tells us the multiplexing groupings. This can then be used in spikeinterface for the inter_sample_shift. PR incoming. This is saved as a probe annotation.
Removes probe_type from metadata creation, and from the read_openephys and read_spikeglx functions. Still needed for the read/write imro functions, where probe_type is stored in the format. E.g. tests/data/imro/test_single_shak_2.0.imro starts “(21,384)(…” and the “21” is the probe_type of the probe.
Adds a get_probe_contour_vertices function which computes the probe contour vertices from metadata.

We now additionally support the following probes: 'NP2005', 'NP3011', 'NP1033', 'NP1122', 'NP3010', 'NP1120', 'NP1210', 'NP1014', 'NP3021', 'NP2020', 'NP3020', 'NP1011', 'NP3000', 'NP1013', 'NP1123', 'NP2021', 'NP1200', 'NP2006', 'NP1012'
We LOSE support for NP1110

Implementation questions and annoying edge cases:

We had a model_name for each probe, which ProbeTable doesn’t contain. @alejoe91 suggested using the description, which changes all our probe names. Ok??
I’ve saved the json file at src/probeinterface/resources/probe_features.json. I think it has to be in source to work for a normal pip installation? Any better places to store?
We want to check backwards compatibility. This PR mostly changes the npx_description, so it's not too radical. So we only really need check that the old hardcoded dict matches the new function. To do this, I’ve moved the old hardcoded dict to a test file. Any other ideas for how to check this?

Backwards compatibility

There are some mismatches between the ProbeTable and old ProbeInterface implementation we found by comparing the npx_descriptions dict with the make_npx_descriptions function (which now hasn't survived during the evolution of the PR). These are:

Probe type	key	ProbeTable	ProbeInterface
1020	x_pitch	87	91
	stagger	16	12
1021	x_pitch	87	91
	stagger	16	12
1030	x_pitch	87	56
	stagger	16	12
1031	x_pitch	87	91
	stagger	16	12
1121	x_pitch	0	6
	contact_width	2.5	2
1110	fields_in_imro_table	(“group”, “bankA”, “bankB”)	(“channel_ids”, “banks”, “references”, “ap_gains”, “lf_gains”, “ap_hp_filters”

We also tested the neuropixels directory from the probeinterface_library and the new library generated by running generate_neuropixels_library.py. We checked their differences using the following code. First, we checks which probes were generated from each library:

import json
from pathlib import Path

new_library_path = Path('/Users/christopherhalcrow/Work/fromgit/probeinterface/resources/neuropixels_library_generated/')
new_generated_paths = list(new_library_path.glob('*'))
probes_in_new_paths = set([path.name for path in new_generated_paths])

old_library_path = Path("/Users/christopherhalcrow/Work/fromgit/probeinterface_library/neuropixels")
old_generated_paths = list(old_library_path.glob('*'))
probes_in_old_paths = set([path.name for path in old_generated_paths])

new_not_old = probes_in_new_paths.difference(probes_in_old_paths)
old_not_new = probes_in_old_paths.difference(probes_in_new_paths)

print(new_not_old)
print(old_not_new)

>>> {'NP2005', 'NP3011', 'NP1033', 'NP1122', 'NP3010', 'NP1120', 'NP1210', 'NP1014', 'NP3021', 'NP2020', 'NP3020', 'NP1011', 'NP3000', 'NP1013', 'NP1123', 'NP2021', 'NP1200', 'NP2006', 'NP1012'}
>>> {'README.txt', 'NP1110'}

The new_not_old contains all the new probes which are in ProbeTable but not in our previous table. Nice! The old_not_new contained NP1110 which we currently don't support (@alejoe91 @samuelgarcia I think we're not supporting it, right??)

We can then check that all the generated probes are equal by comparing their json files. Here's some code that does that. We do assertions, then I've added continues when we want to skip a property (if we know it's different):

for probe_name in old_generated_paths[:2]:
    if probe_name.name not in ['NP1110', 'README.txt']:

        old_json_filename = probe_name / (probe_name.name + '.json')
        new_json_filename = new_library_path / probe_name.name / (probe_name.name + '.json')
        
        with open(old_json_filename) as f:
            old_probe_info = json.load(f)
        with open(new_json_filename) as g:
            new_probe_info = json.load(g)

        for a, old_probe in enumerate(old_probe_info['probes']):
            new_probe = new_probe_info['probes'][a]
            for key, value in old_probe.items():
                if key == "annotations":
                    for annotations_key, annotations_value in value.items():
                        new_annotations_value = new_probe['annotations'][annotations_key]
                        if annotations_key in ["model_name", "shank_tips"]:
                            continue
                        assert annotations_value == new_probe["annotations"][annotations_key], f"{annotations_key} doesn't match.\nOLD: {annotations_value}.\nNEW: {new_annotations_value}"
                elif key in ["probe_planar_contour"]:
                    continue
                elif probe_name.name == "NP1121" and key == "contact_shape_params":
                    continue
                elif probe_name.name in ["NP1031", "NP1030", "NP1020", "NP1021"] and key == "contact_positions":
                    continue
                else:
                    assert value == new_probe[key], f"{probe_name}. {key} doesn't match.\nOLD: {value}.\nNEW: {new_probe[key]}"

The skipped properties exactly match those in the table above.

for more information, see https://pre-commit.ci

codecov · 2025-05-15T13:40:39Z

Codecov Report

❌ Patch coverage is 94.48819% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.91%. Comparing base (43ccfcb) to head (9bf8f15).
⚠️ Report is 56 commits behind head on main.

Files with missing lines	Patch %	Lines
src/probeinterface/neuropixels_tools.py	94.48%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #349      +/-   ##
==========================================
+ Coverage   89.75%   89.91%   +0.15%     
==========================================
  Files          12       12              
  Lines        1972     2012      +40     
==========================================
+ Hits         1770     1809      +39     
- Misses        202      203       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

for more information, see https://pre-commit.ci

h-mayorquin · 2025-05-26T17:55:49Z

Thanks for taking on this, @chrishalcrow

I’ve saved the json file at src/probeinterface/resources/probe_features.json. I think it has to be in source to work for a normal pip installation? Any better places to store?

Yes, you will need to modify the pyproject.toml to ensure that this is packaged on the wheel/stds. An alternative is to store it as a python dict which will make package simpler (that way you don't need to modify the gitignore).

All that said, json should be fine as well, any reason you prefer it? Whatever we choose, we should add a cron test that runs weekly or monthy to ensure that what we have matches:

https://github.com/billkarsh/ProbeTable/blob/main/Tables/probe_features.json

I can help with that if you want.

src/probeinterface/neuropixels_tools.py

h-mayorquin · 2025-05-26T19:13:31Z

src/probeinterface/neuropixels_tools.py

+
+    is_phase3a = False
+    # These are all prototype NP1.0 probes, not contained in ProbeTable
+    if probe_part_number in ["PRB_1_4_0480_1", "PRB_1_4_0480_1_C", "PRB_1_2_0480_2", None]:


PRB_1_4_0480_1 and PRB_1_4_0480_1_C are on the table, why are we fetching NP1010 instead for those?

Oops - tests were failing for these and we used to group them together so I thought that was why. Now realise there's a typo in PRB_1_2_0480_2 in the ProbeTable. Will raise an issue over there.

h-mayorquin · 2025-05-26T19:20:43Z

We had a model_name for each probe, which ProbeTable doesn’t contain. @alejoe91 suggested using the description, which changes all our probe names. Ok??

I think this is a good idea.I am +1 on it.

src/probeinterface/neuropixels_tools.py

for more information, see https://pre-commit.ci

chrishalcrow · 2025-05-28T10:22:03Z

src/probeinterface/neuropixels_tools.py

-    imro_table_header_str, *imro_table_values_list, _ = imro_str.strip().split(")")
-    imro_table_header = tuple(map(int, imro_table_header_str[1:].split(",")))

-    if imDatPrb_pn is None:


Note: we push this logic to the read_imro function.

for more information, see https://pre-commit.ci

h-mayorquin · 2025-05-28T19:02:43Z

The logic seems fine and the tests are passing. We can improve later if we need to so this is fine to me. To points left are that:

The packaging works. We should see if the json artifact is packed on the sdist and wheels for distribution.
See that if things differ with the current https://github.com/SpikeInterface/probeinterface_library/tree/main/neuropixels they should differ for a good reason.

chrishalcrow · 2025-06-01T11:22:57Z

Great! I've...

added more backwards compatibility information in the PR description.
added a github action (99% written by Claude) which checks the probe_features.json every week. It works fine on chrishalcrow/probeinterface but I'm unsure how to test it here.

Not sure how to check the packaging stuff - we can discuss offline.

chrishalcrow · 2025-06-03T08:43:39Z

Me and @h-mayorquin discussed the GitHub action and checked if the probe_features.json file would be included in the PyPi build - it is! I'm now happy with this PR.

for more information, see https://pre-commit.ci

…ce into use-mux-tables

chrishalcrow · 2025-06-26T09:53:26Z

src/probeinterface/neuropixels_tools.py


+    # annotate with MUX table
+    if mux_table is not None:
+        print("Adding MUX table to probe")


Suggested change

print("Adding MUX table to probe")

chrishalcrow · 2025-06-26T09:59:29Z

tests/test_neuropixels_backwards_compat.py

+
+        if probe_part_number not in ["3000", "1200"]:
+
+            probe_info = make_npx_description(probe_part_number)


Now that we've removed the make_npx_description function, it's pretty difficult to test backwards compatibility inside the library, so we'll just get rid of this test?

We could externally make a repo that builds the probes from two ProbeInterface versions and compares them.

Could we copy the old make_npx_description here in this tests and keep it for a while ?

finally no too complex

chrishalcrow · 2025-06-26T10:01:03Z

src/probeinterface/neuropixels_tools.py

+    probe = _make_npx_probe_from_description(pt_metadata, elec_ids, shank_ids, mux_table)

-    # this is scalar annotations
+    mux_table_format_type = pt_metadata["mux_table_format_type"]


Should delete these three lines, and the line mux_table_array=mux_table_array, below

you do not want to annotate the mux ?

samuelgarcia · 2025-07-02T08:28:41Z

tests/test_neuropixels_backwards_compat.py

@@ -0,0 +1,495 @@
+from probeinterface.neuropixels_tools import make_npx_description, probe_part_number_to_probe_type
+


Suggested change

# this code is here only for testing the refactoring of neuropixel probe handling done by Chris in 2025 and based on .....

# this test could be removed in 202X ....

filenally remove this file

samuelgarcia · 2025-07-02T08:30:56Z

src/probeinterface/resources/probe_features.json

@@ -0,0 +1,2086 @@
+{


I would rename this file to neuropixels_probe_features.json no ?

load json only once on demand remove backwards compativility file because it is not possible anymore to have it.

for more information, see https://pre-commit.ci

…ce into use-mux-tables

samuelgarcia · 2025-07-02T09:38:40Z

This is OK for me for merging.

@alejoe91 @chrishalcrow I guess that the mux array annotations will be used in get_neuropixels_sample_shifts on spikeinterface side.
Should we make the PR before merging it ?

samuelgarcia · 2025-07-02T12:36:17Z

ET voilà!

h-mayorquin · 2025-07-02T17:39:29Z

Hey guys, thanks for taking this to the finishing line and proving me wrong when I said this would take a while : )

chrishalcrow and others added 2 commits May 15, 2025 14:27

initial commit

9112e43

[pre-commit.ci] auto fixes from pre-commit.com hooks

247f538

for more information, see https://pre-commit.ci

chrishalcrow and others added 2 commits May 19, 2025 18:13

update docstrings

bb8237e

[pre-commit.ci] auto fixes from pre-commit.com hooks

11db7ec

for more information, see https://pre-commit.ci

h-mayorquin reviewed May 26, 2025

View reviewed changes

src/probeinterface/neuropixels_tools.py Show resolved Hide resolved

h-mayorquin reviewed May 26, 2025

View reviewed changes

src/probeinterface/neuropixels_tools.py Show resolved Hide resolved

h-mayorquin reviewed May 26, 2025

View reviewed changes

src/probeinterface/neuropixels_tools.py Outdated Show resolved Hide resolved

h-mayorquin reviewed May 26, 2025

View reviewed changes

src/probeinterface/neuropixels_tools.py Outdated Show resolved Hide resolved

h-mayorquin reviewed May 26, 2025

View reviewed changes

src/probeinterface/neuropixels_tools.py Show resolved Hide resolved

h-mayorquin reviewed May 26, 2025

View reviewed changes

src/probeinterface/neuropixels_tools.py Outdated Show resolved Hide resolved

This was referenced May 26, 2025

SpikeGLX: Adding NP1123 support NeuralEnsemble/python-neo#1716

Merged

Discussion: deprecate read_imro #351

Open

chrishalcrow and others added 2 commits May 28, 2025 11:19

respond to Heberto

8d49927

[pre-commit.ci] auto fixes from pre-commit.com hooks

833b14f

for more information, see https://pre-commit.ci

chrishalcrow commented May 28, 2025

View reviewed changes

chrishalcrow and others added 2 commits May 28, 2025 11:33

Bits and pieces

0c42f58

[pre-commit.ci] auto fixes from pre-commit.com hooks

dd4d632

for more information, see https://pre-commit.ci

chrishalcrow mentioned this pull request May 28, 2025

Typo in PRB_1_2_0480_2 name billkarsh/ProbeTable#2

Closed

add github action to copy probe features

c9973ce

chrishalcrow added 2 commits June 1, 2025 12:25

try to fix action whitespace

f00695a

defo fix whitespace

851a696

chrishalcrow and others added 2 commits June 20, 2025 10:56

Remove make_npx_description from read_spikeglx

3cb1768

[pre-commit.ci] auto fixes from pre-commit.com hooks

783dba9

for more information, see https://pre-commit.ci

chrishalcrow and others added 3 commits June 26, 2025 09:59

add mux_table_array to read_spikeglx

280f103

Refactor Open Ephys and add mux table contact annotation

43c57c7

Merge branch 'use-mux-tables' of github.com:chrishalcrow/probeinterfa…

b3299ee

…ce into use-mux-tables

chrishalcrow commented Jun 26, 2025

View reviewed changes

samuelgarcia reviewed Jul 2, 2025

View reviewed changes

samuelgarcia and others added 5 commits July 2, 2025 11:16

rename neuropixels ressource file

2650ee7

load json only once on demand remove backwards compativility file because it is not possible anymore to have it.

rename neuropixels ressource file

02c3540

load json only once on demand remove backwards compativility file because it is not possible anymore to have it.

[pre-commit.ci] auto fixes from pre-commit.com hooks

607519b

for more information, see https://pre-commit.ci

clean mux_table_arraymux_channels

9567cfd

Merge branch 'use-mux-tables' of github.com:chrishalcrow/probeinterfa…

9bf8f15

…ce into use-mux-tables

samuelgarcia approved these changes Jul 2, 2025

View reviewed changes

samuelgarcia merged commit b3f3542 into SpikeInterface:main Jul 2, 2025
5 checks passed

This was referenced Jul 3, 2025

New neuropixel library generated from ProbeTable SpikeInterface/probeinterface_library#22

Closed

New neuropixel library generated from ProbeTable SpikeInterface/probeinterface_library#23

Merged

chrishalcrow mentioned this pull request Jul 24, 2025

Refactor Neuropixels Info to use ProbeTable #341

Closed

alejoe91 mentioned this pull request Aug 1, 2025

Proposal: Add ProbeInterface Support for ONIX Source Plugin #358

Open


		if probe_part_number not in ["3000", "1200"]:

		probe_info = make_npx_description(probe_part_number)

		@@ -0,0 +1,495 @@
		from probeinterface.neuropixels_tools import make_npx_description, probe_part_number_to_probe_type


	# this code is here only for testing the refactoring of neuropixel probe handling done by Chris in 2025 and based on .....
	# this test could be removed in 202X ....

Use ProbeTable to generate NP probes information #349

Use ProbeTable to generate NP probes information #349

Uh oh!

Conversation

chrishalcrow commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Backwards compatibility

Uh oh!

codecov bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

h-mayorquin commented May 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

h-mayorquin commented May 26, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

h-mayorquin commented May 28, 2025

Uh oh!

chrishalcrow commented Jun 1, 2025

Uh oh!

chrishalcrow commented Jun 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samuelgarcia commented Jul 2, 2025

Uh oh!

samuelgarcia commented Jul 2, 2025

Uh oh!

Uh oh!

h-mayorquin commented Jul 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chrishalcrow commented May 15, 2025 •

edited

Loading

codecov bot commented May 15, 2025 •

edited

Loading