Feature Request: Create an importable Zip bundle #429
Description
🚀 feature request
Relevant Rules
Similar to py_library and py_bazel
Description
I'm looking to have a rule which bundles a python module(s) and its dependencies in to a container (i.e. a zip) which can then be executed.
For example, let's assume there's an graphresults module that has dependencies on matplotlib and boto3, like in the following directory structure:
- WORKSPACE (has pip_install for matplotlib and boto3)
- requirements.txt
- BUILD
- graphresults
| - BUILD
| - src
| - __init__.py
| - graphresults.py
would generate a zip file that has a structure of:
! OPTION A !
- graphresults.py
- matplotlib
| - __init__.py
| - [matplotlib stuff]
- boto3
| - __init__.py
| - [boto stuff]
or
! OPTION B !
- graphresults
| - __init__.py
| - graphresults.py
- matplotlib
| - __init__.py
| - [matplotlib stuff]
- boto3
| - __init__.py
| - [boto stuff]
The main focus I have is to deploy to AWS Lambda, either as the codebase for a function or to create a lambda layer.
Describe the solution you'd like
A rule like this:
py_bundle(
name = "...",
srcs = [ a list of labels to package up ],
module_name = "...", # if OPTION B is picked, this can be the containing folder for srcs
deps = [ a list of labels to depend on ] # e.g. boto3, matplotlib
container = "zip|tar", # defaults to zip
)
The rule would iterate through the dependencies and wrap them explicitly. When the dependency is a dependency generated by pip_import we need to drop down one levels to pick the dependency, as well as analyse its BUILD file to find its dependencies.
For example, boto3 depends on botocore, so if we specify boto3 in deps we need to look at pypi__boto3: to find its dependencies (in this case botocore) and so on; botocore depends on urllib3 etc.
Describe alternatives you've considered
I've tried using zipper which is bundled within Bazel, derived from a Stack Overflow post:
genrule(
name = "gen_zip",
srcs = [
requirement("boto3")
],
tools = ["@bazel_tools//tools/zip:zipper"],
outs = ["files.zip"],
cmd = "$(location @bazel_tools//tools/zip:zipper) c $@ $(SRCS) ",
)
However this results in a directory structure like:
- pypi__boto
| - boto3
| - __init__.py
Another alternative is to write a genrule which goes through each requirement in srcs and uses a heuristic to determine if it's a python dependency, then strips the containing pypi__[lib_name]. However this is brittle and there's not a simple way to specify source and then dependencies, and it depends on rules_python not changing how it labels pip_import dependencies.