Improve conda-forge automation #5
Description
The ESIP community relies on the conda-forge channel for installing the packages then need to enable their workflows, as these packages are often not found in the defaults conda channel. Maintaining 5000+ packages is a lot of work, and there are some straightforward improvements that could be made with modest funding.
Conda-forge has always relied on heavy automation to reduce maintenance burdens and keep the stack up to date and stable. There are some remaining problems however.
Existing Problems
-
Due to a constantly shifting landscape of ABI incompatibilities conda-forge periodically needs to change the pinnings on critical packages to guarantee interoperability of the entire stack. When this happens portions of the stack must be rebuilt so that they use the new binaries. Currently setting up these migrations is a manual process.
-
At the heart of conda-forge is a Directed Graph which describes the dependency relationships for all the packages in the distribution. This graph is used for properly installing packages into environments and for migrating packages to new pinnings and compilers. However, it is possible to get these dependencies wrong in the specification of the package recipe.
Proposed work:
-
Automate the process so that when pinnings change our stack can seamlessly be rebuilt to take advantage of the newer binaries. This will require inspection of the dependency graph and extraction of the portions touched by the newly pinned package, and automation of determining if a new pinning has been issued.
-
Re-generate Python (and maybe R) recipes using the package metadata to get the most accurate dependency list possible. That can be achieved via some small improvements in conda-skeleton recipe generator.
-
Extend the work of Eric Dill on https://github.com/ericdill/depfinder to automatically find dependencies for packages in the conda-forge ecosystem. If unreported dependencies are found they can be PRed into the recipes, helping to keep their recipes up to correct. This approach would then be extended beyond pure python packages with depfinder analogues for other languages.