Skip to content

general structure updates#315

Merged
FerriolCalvet merged 11 commits intomainfrom
dev
Jul 20, 2025
Merged

general structure updates#315
FerriolCalvet merged 11 commits intomainfrom
dev

Conversation

@FerriolCalvet
Copy link
Collaborator

@FerriolCalvet FerriolCalvet commented Jul 8, 2025

New version of the pipeline, containing:

  • More extensive documentation (still not fully complete)
  • Automated validation via nf-core plugin
  • Passing all nextflow lint, but modules.config
  • Update mutation rate to mutation density
  • bug fixing

AI summary

This pull request introduces updates across several areas of the deepCSA project, including configuration changes, documentation enhancements, and code improvements. Key changes involve replacing "mutationrate" with "mutationdensity" in multiple configuration files, improving the README.md to provide clearer instructions and additional context, and refining the modules.config file to simplify and standardize prefixes for process outputs.

Configuration Updates:

  • Replaced mutationrate with mutationdensity in multiple configuration files (conf/bladder.config, conf/chip.config, conf/chip_dwnsyndr.config, conf/chip_pancancer.config, conf/kidney.config, conf/kidney_bulk.config, conf/lung.config, conf/mice.config, conf/mice_mm10.config) to reflect a more accurate parameter naming convention. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Documentation Enhancements:

  • Updated README.md to include a new workflow overview image, clearer usage instructions, and a warning about parameter configuration. It also now includes a credits section listing contributors and a link to the documentation. [1] [2] [3] [4]
  • Added a .markdownlint.json file to configure markdown linting rules, disabling MD013 and customizing MD024.

Code Improvements:

  • Enhanced deepcsa_maf2samplevcfs.py script with detailed comments specifying mandatory columns for input mutations, improving usability and clarity.

Process Configuration Refinements:

  • Simplified and standardized output prefixes in conf/modules.config by removing ${meta.id} references and using consistent naming conventions (e.g., .subset_depths, .all, .exons). These changes improve maintainability and readability. [1] [2] [3] [4] [5]
  • Removed the ENSEMBLVEP_DOWNLOAD process configuration, indicating a cleanup of unused or redundant processes.

Other Changes:

  • Fixed minor issues in conf/modules.config related to publish directory patterns and process configurations, ensuring consistency and correctness. [1] [2] [3] [4]

FerriolCalvet and others added 6 commits June 8, 2025 12:57
* First changes in README starting to document

* remove VEP prepare cache step

- clean commented lines

* update mutabilities to relative mutabilities

* update minimum nxf version to 25.04.2

* refactor: update ext.prefix definitions to use meta parameter for consistency

* Refactor and enhance plotting and annotation scripts

- Added a placeholder for VAF distribution plot in plot_maf.py.
- Filtered gene list in plot_selection_omega.py and plot_selectionsideplots.py to include only canonical symbols.
- Cleaned up vcf2maf.py by removing commented-out plotting code and added a TODO for molecule activation.
- Removed unused args definitions in various Nextflow processes to streamline code.
- Updated main.nf and other workflow scripts to improve clarity and maintainability, including removing unnecessary comments and consolidating options.
- Enhanced the handling of parameters in the pipeline initialization workflow.

* fix linting of timestamp

* update prefix definition and clean config

* remove lib and all references to it

- partial fix since MULTIQC is not complete

* finalize cleaning of VEP_PREPARE_CACHE

* minor removal

* reactivate multiqc sections

* add outputing of relativemutability and rename

* Apply suggestions from code review

* update logo function

* fix functions input definition

* update signature extraction name handling

* update signature subgroups explanation

* update matrix concatenation namings

* fix review comments
* fix relative mutabilities output

* explicitly define postprocessveppanel outdir

* force outputting postprocesspanel

* update storing fixes
* rename mutrate to mut density

- reimplement with click
- partial renaming
- simplification of sample_name logic

* full update of mutation rate to mutation density

* define other_sample_SNP based on all VAF

* update mutation density functions

- clean code
- add explanation on mutation density

* apply review changes
reimplement it with click solved the problem
- only_canonical boolean working
- not tested
* first doc update

* update in usage documentation

- list params
- list structural parameters and files

* backbone of output docs

* update usage description with custom sets of mutations

* fix headers

* docs: Update usage with vep information

* update output description

* update order of usage information

* update distribution of information in the docs

* fix typo in docs/output.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix markdown linting

* remove unnecessary validation params from config

- REQUIRES TESTING

* remove remaining references to download VEP cache

* update dag format to mmd

* update in usage

* update documentation of file formatting and some params

* add examples in file formatting docs

* apply review comments

* remove Nextflow parameters section

* minor fix in nextflow.config

---------

Co-authored-by: Miquel L. Grau <miguel.grau@irbbarcelona.org>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@FerriolCalvet FerriolCalvet requested a review from Copilot July 8, 2025 23:17
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the deepCSA pipeline to replace the “mutation rate” modules and parameters with “mutation density,” simplifies various process prefixes, removes the VEP cache download subworkflow, and enhances documentation/configuration for mutation density.

  • Rename all mutationrate modules, channels, CLI flags, and configs to mutationdensity
  • Remove the PREPARE_CACHE subworkflow and related VEP‐cache parameters
  • Simplify module prefix logic and clean up unused imports

Reviewed Changes

Copilot reviewed 106 out of 122 changed files in this pull request and generated no comments.

Show a summary per file
File Description
workflows/deepcsa.nf Swapped mutation rate modules for mutation density
nextflow_schema.json Renamed mutationratemutationdensity
nextflow.config Added mutationdensity param; removed mutationrate
subworkflows/local/mutationdensity/main.nf Introduced new mutation density subworkflow
modules/local/select_mutdensity/main.nf Replaced SELECT_MUTRATES with SELECT_MUTDENSITIES
subworkflows/local/utils_nfcore_deepcsa/main.nf Updated logo/citation helpers; removed purple blocks
conf/tools/mutdensity.config Configured filters for mutation density processes
Comments suppressed due to low confidence (4)

subworkflows/local/mutationdensity/main.nf:1

  • A new MUTATION_DENSITY subworkflow has been added; please add or update tests to cover its functionality and prevent regressions.
include { TABIX_BGZIPTABIX_QUERY    as SUBSETMUTATIONS      } from '../../../modules/nf-core/tabix/bgziptabixquery/main'

modules/local/select_mutdensity/main.nf:1

  • The SELECT_MUTDENSITIES process replaces SELECT_MUTRATES; ensure you add corresponding tests to validate the new process logic.
process SELECT_MUTDENSITIES {

workflows/deepcsa.nf:7

  • The imports for validateParameters, samplesheetToList, paramsHelp, and paramsSummaryLog were removed. Ensure any calls to those functions have also been removed or replaced to prevent missing-function errors at runtime.
include { paramsSummaryMap          } from 'plugin/nf-schema'

nextflow.config:64

  • The signatures parameter is defined more than once in this block; remove the duplicate to avoid confusion or unintended overrides.
    signatures                  = false

FerriolCalvet and others added 3 commits July 10, 2025 08:25
* update consensus building to filter genes

- add consensus compliance param
- add list of genes param
- NOT tested

* tested gene filter implementation

- consensus panels implemented in polars
- allowing subset for specific genes
@FerriolCalvet
Copy link
Collaborator Author

I am not sure if the review are required since you already accepted each of the branches that were merged here.
Let me know if you want to review or I just merge
@migrau @FedericaBrando

Copy link
Member

@FedericaBrando FedericaBrando left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@migrau
Copy link
Member

migrau commented Jul 15, 2025

go ahead!

@FerriolCalvet FerriolCalvet merged commit 7b0fd9b into main Jul 20, 2025
@FerriolCalvet FerriolCalvet deleted the dev branch July 24, 2025 12:18
FerriolCalvet added a commit that referenced this pull request Jul 28, 2025
commit 10c12aa
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 24 12:35:40 2025 +0200

    update gnomAD threshold to 0.001

    - ignore errors in omega plot

commit 31e741e
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Tue Jul 22 18:34:11 2025 +0200

    update description and fix broken link

commit 7b0fd9b
Merge: 72dc4d9 cb18adc
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Sun Jul 20 22:34:46 2025 +0200

    Merge pull request #315 from bbglab/dev

commit cb18adc
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Sat Jul 19 12:40:39 2025 +0200

    fix bug in mut density & update omega container

commit 1b4a52d
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Tue Jul 15 15:52:07 2025 +0200

    fix broken path for test_real

commit db6d640
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Fri Jul 11 15:50:45 2025 +0200

    Allow gene selection in consensus (#316)

    * update consensus building to filter genes

    - add consensus compliance param
    - add list of genes param
    - NOT tested

    * tested gene filter implementation

    - consensus panels implemented in polars
    - allowing subset for specific genes

commit b92c2b9
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 10 23:56:47 2025 +0200

    add metro map

commit 11fb1c5
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 10 08:25:31 2025 +0200

    update description in main README

commit 628f282
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Mon Jul 7 19:14:21 2025 +0200

    Add more complete docs (#306)

    * first doc update

    * update in usage documentation

    - list params
    - list structural parameters and files

    * backbone of output docs

    * update usage description with custom sets of mutations

    * fix headers

    * docs: Update usage with vep information

    * update output description

    * update order of usage information

    * update distribution of information in the docs

    * fix typo in docs/output.md

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

    * fix markdown linting

    * remove unnecessary validation params from config

    - REQUIRES TESTING

    * remove remaining references to download VEP cache

    * update dag format to mmd

    * update in usage

    * update documentation of file formatting and some params

    * add examples in file formatting docs

    * apply review comments

    * remove Nextflow parameters section

    * minor fix in nextflow.config

    ---------

    Co-authored-by: Miquel L. Grau <miguel.grau@irbbarcelona.org>
    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 72dc4d9
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Mon Jul 7 15:53:47 2025 +0200

    temporary LICENSE definition

commit 14c5246
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Fri Jun 27 11:19:45 2025 +0200

    fix bug in panel_annotation (#313)

    reimplement it with click solved the problem
    - only_canonical boolean working
    - not tested

commit 764782a
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Thu Jun 26 23:23:05 2025 +0200

    Update mutation rate to mutation density (#307)

    * rename mutrate to mut density

    - reimplement with click
    - partial renaming
    - simplification of sample_name logic

    * full update of mutation rate to mutation density

    * define other_sample_SNP based on all VAF

    * update mutation density functions

    - clean code
    - add explanation on mutation density

    * apply review changes

commit aae8f0b
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Thu Jun 26 10:49:00 2025 +0200

    Ensure POSTPROCESSVEPPANEL in output (#311)

    * fix relative mutabilities output

    * explicitly define postprocessveppanel outdir

    * force outputting postprocesspanel

    * update storing fixes

commit dd25b9b
Merge: 05c80ee 26d5d9b
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Wed Jun 4 23:09:29 2025 +0200

    Merge pull request #303 from bbglab/dev

    First pre-release merge

commit 05c80ee
Merge: ea9a301 e06218a
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Tue Apr 29 10:30:23 2025 +0200

    Merge pull request #289 from bbglab/tmp-dev

    First release

commit e06218a
Merge: 7559d7f ea9a301
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Tue Apr 29 10:11:05 2025 +0200

    Merge branch 'main' into tmp-dev

commit ea9a301
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 25 16:05:10 2024 +0200

    update schema
@FerriolCalvet FerriolCalvet self-assigned this Jul 28, 2025
FerriolCalvet added a commit that referenced this pull request Aug 1, 2025
* plotting wishlist

- subworkflow definition
- shortlisting plots to add

missing:
- nf scripts for the modules
- python scripts for the plots

* add raw version of supplementary figure plotting

* update omega plotting

* plotting update: omega & needles & stacked

* update plotting cohort plots working

* clean code
list inputs

* saturation data loading working

* gene saturation all tracks working with TP53

* tested additional complementary plots

missing:
- handle sample information input files
- handle reference datasets
- handle multiple genes

* update gene saturation inputs from pipeline

- not tested
- pending to decide creation of unique_splice_sites

* Squashed commit of the following:

commit 10c12aa
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 24 12:35:40 2025 +0200

    update gnomAD threshold to 0.001

    - ignore errors in omega plot

commit 31e741e
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Tue Jul 22 18:34:11 2025 +0200

    update description and fix broken link

commit 7b0fd9b
Merge: 72dc4d9 cb18adc
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Sun Jul 20 22:34:46 2025 +0200

    Merge pull request #315 from bbglab/dev

commit cb18adc
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Sat Jul 19 12:40:39 2025 +0200

    fix bug in mut density & update omega container

commit 1b4a52d
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Tue Jul 15 15:52:07 2025 +0200

    fix broken path for test_real

commit db6d640
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Fri Jul 11 15:50:45 2025 +0200

    Allow gene selection in consensus (#316)

    * update consensus building to filter genes

    - add consensus compliance param
    - add list of genes param
    - NOT tested

    * tested gene filter implementation

    - consensus panels implemented in polars
    - allowing subset for specific genes

commit b92c2b9
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 10 23:56:47 2025 +0200

    add metro map

commit 11fb1c5
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 10 08:25:31 2025 +0200

    update description in main README

commit 628f282
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Mon Jul 7 19:14:21 2025 +0200

    Add more complete docs (#306)

    * first doc update

    * update in usage documentation

    - list params
    - list structural parameters and files

    * backbone of output docs

    * update usage description with custom sets of mutations

    * fix headers

    * docs: Update usage with vep information

    * update output description

    * update order of usage information

    * update distribution of information in the docs

    * fix typo in docs/output.md
    * fix markdown linting

    * remove unnecessary validation params from config

    - REQUIRES TESTING

    * remove remaining references to download VEP cache

    * update dag format to mmd

    * update in usage

    * update documentation of file formatting and some params

    * add examples in file formatting docs

    * apply review comments

    * remove Nextflow parameters section

    * minor fix in nextflow.config

    ---------

    Co-authored-by: Miquel L. Grau <miguel.grau@irbbarcelona.org>

commit 72dc4d9
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Mon Jul 7 15:53:47 2025 +0200

    temporary LICENSE definition

commit 14c5246
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Fri Jun 27 11:19:45 2025 +0200

    fix bug in panel_annotation (#313)

    reimplement it with click solved the problem
    - only_canonical boolean working
    - not tested

commit 764782a
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Thu Jun 26 23:23:05 2025 +0200

    Update mutation rate to mutation density (#307)

    * rename mutrate to mut density

    - reimplement with click
    - partial renaming
    - simplification of sample_name logic

    * full update of mutation rate to mutation density

    * define other_sample_SNP based on all VAF

    * update mutation density functions

    - clean code
    - add explanation on mutation density

    * apply review changes

commit aae8f0b
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Thu Jun 26 10:49:00 2025 +0200

    Ensure POSTPROCESSVEPPANEL in output (#311)

    * fix relative mutabilities output

    * explicitly define postprocessveppanel outdir

    * force outputting postprocesspanel

    * update storing fixes

commit dd25b9b
Merge: 05c80ee 26d5d9b
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Wed Jun 4 23:09:29 2025 +0200

    Merge pull request #303 from bbglab/dev

    First pre-release merge

commit 05c80ee
Merge: ea9a301 e06218a
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Tue Apr 29 10:30:23 2025 +0200

    Merge pull request #289 from bbglab/tmp-dev

    First release

commit e06218a
Merge: 7559d7f ea9a301
Author: Ferriol Calvet <38539786+FerriolCalvet@users.noreply.github.com>
Date:   Tue Apr 29 10:11:05 2025 +0200

    Merge branch 'main' into tmp-dev

commit ea9a301
Author: FerriolCalvet <ferriolcalvet@gmail.com>
Date:   Thu Jul 25 16:05:10 2024 +0200

    update schema

* update stacked plot needles

* minor fixes after merge

- NOT WORKING

* update linewidth and size

- add mutation types

* fix plot saturation within pipeline
- collect sitecomparisons
- remove reference to ddg. requires external
- fix input unique to keep header and minimal information
- tested and works

* fix o3d logs output

* plot multiple genes, not only TP53

* apply review suggestions

- temporary solution to domain information loading

* update domain definition and plotting

- subset domains to in_panel ones
- update domain name definition
- plotting modules works
- missing autoexons plot
- update signatures output

* - separate generation of depth per exon
- add general error handling to all plotting modules
- generate exons bedfile within panel

* batch update of exon definitions

- use correct exons definition
- update custom bedfile name
- not tested

* fix exons panel generation

* add error handling to omega plot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants