Skip to content

Project: update Python to 3.9, update all dependencies#214

Merged
taimontgomery merged 17 commits intomasterfrom
issue-187
Aug 4, 2022
Merged

Project: update Python to 3.9, update all dependencies#214
taimontgomery merged 17 commits intomasterfrom
issue-187

Conversation

@AlexTate
Copy link
Member

@AlexTate AlexTate commented Jul 20, 2022

The update to Python 3.9 from 3.7 has passed testing. This allows us to use the latest versions of bowtie (thanks to my bugfix for the conda recipe), fastp, HTSeq, matplotlib, numpy, pandas, and cwltool. All other dependencies have been updated to the highest compatible version.

In this PR, r-base and DESeq2 are now installed and managed by conda, which at the very least has the benefit of removing the 20 minute compiler wait time if DESeq2 installation is required.

New warnings and bugs resulting from the update have been addressed in the codebase.

Closes #187

AlexTate added 10 commits July 20, 2022 10:28
…esents a forward jump of 2-4 years for many of these dependencies
… py3.8 switched to spawn by default on macOS, which means that workers did not inherit global variables as expected
…ges in HTSeq. Also removed string interning for attribute keys/values since tiny-count's design has evolved to handle these values minimally and only during the GFF parsing step, so it no longer has much of an affect on our runtimes.
…he "is" operator (even though the values I was comparing to are guaranteed to be interned for current and prior Python versions)
…st used in the Wildcard class, for consistency
…s are possible on linux but not osx). Also reorganized the ordering of environment.yml.
… is now managed by conda (which also means we can skip the 20 minute wait while it compiles). Bioconductor-related variables have been removed and the Python version has been bumped.
…es have been updated to reflect changes in environment.yml
@AlexTate AlexTate requested a review from taimontgomery July 20, 2022 19:20
@AlexTate
Copy link
Member Author

I am waiting for a Cloudflare --> Anaconda issue to be resolved before I can resume testing on Linux. Currently certain packages return HTTP 403 when fetched

AlexTate added 5 commits July 26, 2022 07:16
… include an environment variable which should help with installation stability on Linux. Lock files will not convey environment variables so setup.sh must do so.

If a user had previously installed any component in the dependency tree via pip --user, these components end up in ~/.local/lib/python3.x/site-packages and Python actually gives this directory higher priority in sys.path than the environment's site-packages. Importing a package that can be found in the .local directory will result in that package being preferred even if it is the wrong version. Wild. This strikes me as a bug in conda because it breaks conda's isolation model. Others have thought so as well, but each time this subject has been raised it is ultimately dismissed as intended functionality.
@AlexTate AlexTate marked this pull request as ready for review July 26, 2022 15:05
@AlexTate
Copy link
Member Author

A note on PYTHONNOUSERSITE, new in conda/environment.yml and setup.sh:
This addresses a curious issue I encountered on Linux. If a user has previously installed any component in the dependency tree via pip --user, these components end up in ~/.local/lib/python3.x/site-packages and Python actually gives this directory higher priority in sys.path than the environment's own site-packages. Importing a package that can be found in the ~/.local directory will result in that package taking precedence over the environment's package, even if it is the wrong version. Wild. This strikes me as a bug in conda because it breaks conda's isolation model. Others have thought so as well, but each time this subject has been raised it is ultimately dismissed as intended functionality. Setting PYTHONNOUSERSITE=1 makes Python ignore the ~/.local directory. I've taken care to ensure that this variable is set only within the tinyRNA environment.

AlexTate added 2 commits July 27, 2022 09:41
…ting the index via positional argument produces a future deprecation warning. This commit also contains a small correction to the specification and doc string for the `reads` parameter which is more inline with its type
… user preferences for run_parallel. This has been corrected so that tinyRNA respects the run_parallel setting when run_native is true.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Project: update Python and dependencies

2 participants