Commit2 regularisation - Error float division

[arnaudbore]

Dear experts,

I'm updating scilpy with your new release (2.4.1).

I've got some issues related to the regularisation (code).

After setting the regularisation when I run the fit I end up with this error:

Code

            # set the regularisation
            mit.set_regularisation(
                regularisers=('group_lasso', None, None),
                is_nonnegative=(True, True, True),
                lambdas=(perc_lambda, None, None),
                params=(params_IC, None, None))

            mit.fit(tol_fun=1e-3, max_iter=args.nbr_iter)

Error

INFO:root:Adding commit weights to /tmp/tmpa2_a7n_2/Results_StickZeppelinBall/decompose_commit.h5.
/home/local/USHERBROOKE/bora2502/p/scil/s/scilpy/src/scilpy/cli/scil_tractogram_commit.py:501: RuntimeWarning: invalid value encountered in scalar divide
  mit.set_regularisation(
Traceback (most recent call last):
  File "/home/local/USHERBROOKE/bora2502/.virtualenvs/scilpy312/bin/scil_tractogram_commit", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/local/USHERBROOKE/bora2502/p/scil/s/scilpy/src/scilpy/cli/scil_tractogram_commit.py", line 507, in main
    mit.fit(tol_fun=1e-3, max_iter=args.nbr_iter)
  File "commit/core.pyx", line 1455, in commit.core.Evaluation.fit
  File "commit/core.pyx", line 1456, in commit.core.Evaluation.fit
  File "/home/local/USHERBROOKE/bora2502/.virtualenvs/scilpy312/lib/python3.12/site-packages/commit/solvers.py", line 241, in solve
    return fista(y, A, At, omega, prox, confidence_array, tol_fun, tol_x, max_iter, verbose, x0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/local/USHERBROOKE/bora2502/.virtualenvs/scilpy312/lib/python3.12/site-packages/commit/solvers.py", line 272, in fista
    prox( xhat, 1.0 )
  File "/home/local/USHERBROOKE/bora2502/.virtualenvs/scilpy312/lib/python3.12/site-packages/commit/solvers.py", line 201, in <lambda>
    prox = lambda x, scaling: proxIC(proxEC(proxISO(x, scaling), scaling), scaling)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/local/USHERBROOKE/bora2502/.virtualenvs/scilpy312/lib/python3.12/site-packages/commit/solvers.py", line 97, in <lambda>
    proxIC = lambda x, scaling: non_negativity(prox_group_lasso(x,groupIdxIC,groupSizeIC,dictIC_params['group_weights'],scaling*lambda_group_IC),startIC,sizeIC)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "commit/proximals.pyx", line 103, in commit.proximals.prox_group_lasso
  File "commit/proximals.pyx", line 130, in commit.proximals.prox_group_lasso
ZeroDivisionError: float division

Is there any new thing in the API I didn't catch ?

Here are some data

This is the command:

scil_tractogram_commit sub-01__decomposed.h5 sub-01_desc-preproc_dwi.nii.gz sub-01_desc-preproc_dwi.bval sub-01_desc-preproc_dwi.bvec "sub-01__results_bzs/" --processes 1 --para_diff 1.62E-3 --iso_diff 2.92E-3 --commit2 --lambda_commit_2 0.001 --nbr_dir 500 --in_peaks sub-01_desc-peaks.nii.gz -v DEBUG -f

I also have a second error when I set the number of threads at 2, I end up with this error:

Distributing workload to different threads                                 
INFO:core:Distributing workload to different threads
   * Number of threads: 2
Level 21:core:   * Number of threads: 2
   * A operator   Level 21:core:   * A operator 
\ ERROR  Too many threads for the EC compartments to evaluate; try          
decreasing the number                                                        <module:scil_tractogram_commit, line:477>
ERROR:core:Too many threads for the EC compartments to evaluate; try decreasing the number

Thank you very much in advance for your help
Arnaud

[daducci]

Hi Arnaud,

We didn't change much since 2.4.0, so this is strange. Did you test the 2.4.1 version on the same data you used before?

[arnaudbore]

Hey Alessandro,

Thank you for taking some time to look at it.

Actually we are moving from commit 2.3.1 -> 2.4.1
I'm doing some testing so I can give you more information.

Arnaud

[arnaudbore]

It's not specific to 2.4.1 I also have the same error with commit 2.3.1.

[daducci]

I've also added @fullbat to the loop, as many modifications occurred between those two versions.

[fullbat]

Hi Arnaud,
Could you please share the tractogram in .tck format? We are not familiar with .h5 files. Is there an easy way to convert it if needed?

Thank you,
Matteo

[arnaudbore]

Hey Matteo,

Thank you for your help. Here is the new data.

Arnaud

[ilariagabusi]

Hi Arnaud,
Thank you for the converted tractogram!

I was trying to reproduce your error using the standard COMMIT code, but I'm missing the bundle organization required for COMMIT2. Can you share the group definition you used? Either the actual array of streamline indices or the GM parcellation, together with the command you used for the assignment, should work fine. Also, are you passing a WM mask when generating the dictionaries (i.e., as filename_mask in trk2dictionary)?

Regarding the error about the threads, I assume there is a problem in the peaks file, as it contains only 84 non-empty voxels (see the attached screenshot). Can you check if this is the case?

Thank you,
Ilaria

[gagnonanthony]

Hi @fullbat and @ilariagabusi,

Sorry for the delay in our response, we were making sure the problem did not come from our data. You can find the new data here.

I've added the groups as data per streamline, I've also exported them in a separate file. Let me know if that is sufficient for you!

Maybe this could also help, but we tried an old version of COMMIT (1.6.5) and it seems to work seamlessly.

Thanks again!

[ilariagabusi]

Hi @gagnonanthony and @arnaudbore,
Thank you for sharing the new data!

I've identified the cause of the error you encountered when running the fitting with group lasso regularisation.
In the group_ids.npy file, some values are skipped (e.g., id=3). This results in empty groups in the array provided as params_IC['group_idx'], which caused the division by zero indicated in the error. This happens because, by default, the weighting of the groups takes into account their cardinality (as detailed in eq. 6 of the COMMIT2 paper).

This error never occurred to us as we always use the procedure reported in the wiki, which considers only the non-zero entries of the connectome when creating the group structure. Thanks to your feedback, we are making COMMIT even more flexible 😄
We added a check for this case, which will automatically remove empty groups and provide a warning message. The changes are available in the branch release/v2.4.2 and we plan to add them to the master soon. Could you please confirm if this resolves the issue on your end as well?

Thanks,
Ilaria

P.S.: Using the new file for the peaks, I see that the multithreaded computation works without problems.

[gagnonanthony]

Thank you @ilariagabusi !

I've tested it using the release/v2.4.2 branch and can confirm it solves our issue!

Thanks again,
Anthony