Skip to content

[WIP] GUI-less training example script #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
C-Achard merged 8 commits into from
Nov 10, 2023
Merged

[WIP] GUI-less training example script #52

C-Achard merged 8 commits into from
Nov 10, 2023

Conversation

@C-Achard
Copy link
Collaborator

@C-Achard C-Achard commented Oct 17, 2023

Adds :

  • A Dockerfile draft for CellSeg3D
  • A script for training without napari

@C-Achard C-Achard added enhancement New feature or request ML Related to machine learning : MONAI, Torch... labels Oct 17, 2023
@C-Achard C-Achard requested a review from CYHSM October 17, 2023 07:26
@C-Achard C-Achard self-assigned this Oct 17, 2023
CYHSM
CYHSM approved these changes Oct 17, 2023
View reviewed changes
Copy link
Contributor

@CYHSM CYHSM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@C-Achard
Copy link
Collaborator Author

C-Achard commented Oct 17, 2023

LGTM!

Thanks a lot @CYHSM ! I will merge once the Dockerfile is approved as well.

@C-Achard C-Achard marked this pull request as ready for review November 10, 2023 14:47
@C-Achard
Copy link
Collaborator Author

C-Achard commented Nov 10, 2023

No news on Dockerfile so it will be omitted for now

@C-Achard C-Achard merged commit de813e8 into cy/jupyter-books-docs Nov 10, 2023
@C-Achard C-Achard mentioned this pull request Nov 10, 2023
Merged
43 tasks
MMathisLab added a commit that referenced this pull request Nov 15, 2023
@C-Achard @Timokleia @MMathisLab
Docs update & jupyter-books port (#50)
8ae9a14 
* Moved docs to source folder

* Updated to jupyter-books

* Update GH Action to jupyter-book

* Updated guides

* Update reqs for jupyter-books

- Fixed typo

* Fix typo and file duplicate

* Added auto API docs

* Update welcome.rst

* Update interface docstrings

* Update ruff to enforce docstring style

* Updated docstrings in interface/config

* Add additional docstring rule exceptions

* Update utils docstring

* Update plugins.py

* Update dev_scripts

* Updated code_plugin folder docstrings

* Update code_models docstrings

* Update tests docstrings

* Update docstring rules to enforce docs

* Fix warnings when building docs

* Restore docs images

* Remove old API docs

* Added module docstring for all public modules

* Update pyproject.toml

* Update folder desc

* Update inference and training guides

* Update inference_module_guide.rst

* Guides updates

* Change indexing, guides updates

* Fix unindent warning

* Update _toc.yml

* Fixed model list inconsistencies

* Update model list

* Trying to fully use autosummary (see #51)

* Docs update

* Fix missing refactor in CRF

* Update cropping_module_guide.rst

* Updated docs ref for code

* Figs, refs, utils guide update

* Update images to figures where relevant
* Updated refs to source code
* Improved utils guide layout and contents

* Review update + crop/utils

* Review update

* Small inference tweaks for pipeline

* Config changes

* Update worker_inference.py

* Small instance seg refactor

* Added properties for instance seg params

* Utils cleanup

* Small refactor of unclear train/val split terminology

* Fix log parameters incorrectly printing def dir

* Enable sliding window for all instance seg methods

* Update instance_segmentation.py

Refactor sliding window to not shadow builtin "function"

* Fixed mistake in auto-check for WNet results

* Fix for restarting train worker after force stop

* Docs & images update

* mv welcome image

* Improved inference docs

* Update inference_module_guide.rst

* Attempt to fix M1 yaml

* Config docstring update

* Welcome page update

* Docs update

* Add WandB for supervised models

* Added auto contrast reset in cropping

* Added checks for instance labels in training

* Check for custom weights before locking window inference

* Improve review plots makers

* Documentation update (Timokleia) (#53)

* Update cropping_module_guide.rst

changes in the cropping module guide

* Update cropping_module_guide.rst

add ``

* Update inference_module_guide.rst

additions to inference module

* Update metrics_module_guide.rst

adjusting the metrics module

* Update review_module_guide.rst

review module updates

* Update training_module_guide.rst

changes in the training module

* Update training_wnet.rst

additions in the wnet module

* Update utils_module_guide.rst

utilities updates

* review parameters image

* Update cropping_module_guide.rst

emoji

* Update metrics_module_guide.rst

emoji

* Update training_module_guide.rst

emoji

* Update utils_module_guide.rst

emoji

* Update cropping_module_guide.rst

mild style changes

* Update review_module_guide.rst

adjusting image code

* Update welcome.rst

changes to welcome

* Add files via upload

* Review and training tweaks

- Fixed a few links, indents, typos
- Replaced some emojis and added new ones for other pages

* Welcome and inference tweaks

- Fixed minor formatting, tables, broken links issues, etc

* RM unused model wrappers attributes

* Inference edits

* Update WNet and inference

* Images update

* Update plots_train.png

* Restore walkthrough

* Update detailed_walkthrough.rst

* Update detailed_walkthrough.rst

editions

* Minor edits

---------

Co-authored-by: C-Achard <cyril.achard@epfl.ch>

* Added floats in small remove

* Try to fix issue with float/long

* Add except handling for layer removal in train

* Add first version of dedicated install page

- Add install page
- Refacto/clean up welcome page
- Add more useful plugins
- Update TODO

* Improve training guide

* Move M1 install

* Added label stats utils

* Added label stats doc

* Added check for dtype in stats labels

* Update utils_module_guide.rst

* Experimental classifier to reject bad preds on empty

* Experimental classifier to reject bad preds on empty

* Update classifier_test.ipynb

* WIP Trying to fix some QoL issues

- Better exception handling in inference
- Swapaxes at mode output
- Remove small in Voronoi-Otsu

* Fix exception handling

* Locating csv issue

* Enable ruff again

* Handle empty labels in csv stats

* Fix call to get_dict when stats is None

* Ensure consistent rotation

* Fix rotation checks + auto-channel select

* Fix test for review

* Update installation_guide.rst (#54)

refining

* Minor edits to install guide

* [WIP] GUI-less training example script (#52)

* Add remote train scripts

* mv dockerfile

* Paths for remote train

* Change device

* Update remote_training.py

* Change results path to include more info

* Update colab_training.py

* Delete Dockerfile.cellseg3d

* Make all wandb inits point to same project

* Remove deprecated call to AddChannel

* Fix tests issue with MONAI 1.3.0

* Create whole_brain_utils.py

Add segmentation correction scripts

* Update whole_brain_utils.py

* Update whole_brain_utils.py

* Update whole_brain_utils.py

* Update whole_brain_utils.py

* Update _config.yml

---------

Co-authored-by: Timokleia <86098649+Timokleia@users.noreply.github.com>
Co-authored-by: Mackenzie Mathis <mathis@rowland.harvard.edu>
C-Achard added a commit that referenced this pull request Nov 15, 2023
@C-Achard @Timokleia @MMathisLab
WandB/Colab + docs (#48)
eef7eac 
* WANDB integration for WNet

* Colab training script

* Update colab_training.py

* Update colab_training.py

* Update colab_training.py

* Update colab_training.py

* Update colab_training.py

* Added minimal Colab notebook

* ZeroCostDL4Mic-style notebook

* Added WANDB to Colab notebook

* Added softmax param

* Fix reset of paths in training UI

* WNet visualization

* Disable intensity transforms in WNet

* Fixed supervised loss saving

* Update view_wnet.ipynb

* Update view_wnet.ipynb

* Fixed filepaths

* Added more checks for csv + fix eval batch size

* Utils+training fixes

* Colab training improvements (#49)

* Update colab_wnet_training.ipynb

* Update colab_wnet_training.ipynb

---------

Co-authored-by: Cyril Achard <94955160+C-Achard@users.noreply.github.com>

* Reverted include_background=True in Dice

* Reintroduced best Dice channel seeking + refacto

* Improve filepath messages

* Fix unsup image loading when not validating

* Fixed deletion of Qt imports in interface

* Added auto-check for which channel has foreground

Preventing background to be shown first when using WNet inference, for better UX

* Fixed eval_batch_size assignment

* Fix training tests

* Experimental auto-discard

* Improved auto-discard utility

* Create experimental sliding window Voronoi-Otsu for large volumes

* Disable resize error handler

* Docs update & jupyter-books port (#50)

* Moved docs to source folder

* Updated to jupyter-books

* Update GH Action to jupyter-book

* Updated guides

* Update reqs for jupyter-books

- Fixed typo

* Fix typo and file duplicate

* Added auto API docs

* Update welcome.rst

* Update interface docstrings

* Update ruff to enforce docstring style

* Updated docstrings in interface/config

* Add additional docstring rule exceptions

* Update utils docstring

* Update plugins.py

* Update dev_scripts

* Updated code_plugin folder docstrings

* Update code_models docstrings

* Update tests docstrings

* Update docstring rules to enforce docs

* Fix warnings when building docs

* Restore docs images

* Remove old API docs

* Added module docstring for all public modules

* Update pyproject.toml

* Update folder desc

* Update inference and training guides

* Update inference_module_guide.rst

* Guides updates

* Change indexing, guides updates

* Fix unindent warning

* Update _toc.yml

* Fixed model list inconsistencies

* Update model list

* Trying to fully use autosummary (see #51)

* Docs update

* Fix missing refactor in CRF

* Update cropping_module_guide.rst

* Updated docs ref for code

* Figs, refs, utils guide update

* Update images to figures where relevant
* Updated refs to source code
* Improved utils guide layout and contents

* Review update + crop/utils

* Review update

* Small inference tweaks for pipeline

* Config changes

* Update worker_inference.py

* Small instance seg refactor

* Added properties for instance seg params

* Utils cleanup

* Small refactor of unclear train/val split terminology

* Fix log parameters incorrectly printing def dir

* Enable sliding window for all instance seg methods

* Update instance_segmentation.py

Refactor sliding window to not shadow builtin "function"

* Fixed mistake in auto-check for WNet results

* Fix for restarting train worker after force stop

* Docs & images update

* mv welcome image

* Improved inference docs

* Update inference_module_guide.rst

* Attempt to fix M1 yaml

* Config docstring update

* Welcome page update

* Docs update

* Add WandB for supervised models

* Added auto contrast reset in cropping

* Added checks for instance labels in training

* Check for custom weights before locking window inference

* Improve review plots makers

* Documentation update (Timokleia) (#53)

* Update cropping_module_guide.rst

changes in the cropping module guide

* Update cropping_module_guide.rst

add ``

* Update inference_module_guide.rst

additions to inference module

* Update metrics_module_guide.rst

adjusting the metrics module

* Update review_module_guide.rst

review module updates

* Update training_module_guide.rst

changes in the training module

* Update training_wnet.rst

additions in the wnet module

* Update utils_module_guide.rst

utilities updates

* review parameters image

* Update cropping_module_guide.rst

emoji

* Update metrics_module_guide.rst

emoji

* Update training_module_guide.rst

emoji

* Update utils_module_guide.rst

emoji

* Update cropping_module_guide.rst

mild style changes

* Update review_module_guide.rst

adjusting image code

* Update welcome.rst

changes to welcome

* Add files via upload

* Review and training tweaks

- Fixed a few links, indents, typos
- Replaced some emojis and added new ones for other pages

* Welcome and inference tweaks

- Fixed minor formatting, tables, broken links issues, etc

* RM unused model wrappers attributes

* Inference edits

* Update WNet and inference

* Images update

* Update plots_train.png

* Restore walkthrough

* Update detailed_walkthrough.rst

* Update detailed_walkthrough.rst

editions

* Minor edits

---------

Co-authored-by: C-Achard <cyril.achard@epfl.ch>

* Added floats in small remove

* Try to fix issue with float/long

* Add except handling for layer removal in train

* Add first version of dedicated install page

- Add install page
- Refacto/clean up welcome page
- Add more useful plugins
- Update TODO

* Improve training guide

* Move M1 install

* Added label stats utils

* Added label stats doc

* Added check for dtype in stats labels

* Update utils_module_guide.rst

* Experimental classifier to reject bad preds on empty

* Experimental classifier to reject bad preds on empty

* Update classifier_test.ipynb

* WIP Trying to fix some QoL issues

- Better exception handling in inference
- Swapaxes at mode output
- Remove small in Voronoi-Otsu

* Fix exception handling

* Locating csv issue

* Enable ruff again

* Handle empty labels in csv stats

* Fix call to get_dict when stats is None

* Ensure consistent rotation

* Fix rotation checks + auto-channel select

* Fix test for review

* Update installation_guide.rst (#54)

refining

* Minor edits to install guide

* [WIP] GUI-less training example script (#52)

* Add remote train scripts

* mv dockerfile

* Paths for remote train

* Change device

* Update remote_training.py

* Change results path to include more info

* Update colab_training.py

* Delete Dockerfile.cellseg3d

* Make all wandb inits point to same project

* Remove deprecated call to AddChannel

* Fix tests issue with MONAI 1.3.0

* Create whole_brain_utils.py

Add segmentation correction scripts

* Update whole_brain_utils.py

* Update whole_brain_utils.py

* Update whole_brain_utils.py

* Update whole_brain_utils.py

* Update _config.yml

---------

Co-authored-by: Timokleia <86098649+Timokleia@users.noreply.github.com>
Co-authored-by: Mackenzie Mathis <mathis@rowland.harvard.edu>

---------

Co-authored-by: Timokleia <86098649+Timokleia@users.noreply.github.com>
Co-authored-by: Mackenzie Mathis <mathis@rowland.harvard.edu>
@C-Achard C-Achard deleted the cy/remote-train branch December 5, 2023 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@CYHSM CYHSM CYHSM approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@C-Achard C-Achard

Labels

enhancement New feature or request ML Related to machine learning : MONAI, Torch...

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@C-Achard @CYHSM