small lambdas by default (zappa on diet)

[kapilt]

i happened to have a big virtualenv when i installed zappa, it seems to pull everything in the venv by default which resulted in a big zip, ideally before going to upload that it could just barf and say its not supported. even better would be just pulling deps from requirements.txt file or dependency.

[chromakey]

Your situation sounds different than mine, but I use Zappa for a lot of Flask deployments where the static folder gets sucked into the zip and uploaded with the rest of the materials. This frequently pushes my zip well over 50MB. It doesn't seem to impact me at all and if Zappa puked at a 51MB zip, I'd have a lot of trouble with my existing workflows. Maybe we can create some flags around size limits or dependencies?

[Miserlou]

Hey @kapilt ! I think we met after your great talk at Serverless London..

You may be very interested in this commit that will hopefully be merged soon: #548

[kapilt]

yeah its actually on the basis of extracted size that it barfed not zip size. @Miserlou indeed we did, thanks, playing around with apigw approaches at the moment.

[kapilt]

@Miserlou i had a look at #548, afaics that's the wrong solution, because i think the goal is actually just making zappa zipped lambdas smaller by default with the correct dependency stack.

ie. is the goal about keeping the size small ? or supporting large projects with over 250mb of runtime dependencies?

for keeping the size small which i think is the more universal win, conceptually, the solution in that PR feels like the wrong solution and just adds complexity and cold start latency, the right solution imo, would focus on keeping the zappa handler size minimal and getting a better default exclude. the fixed overhead on zappa handler should only be 1mb uncompressed (werkzeug and requestlogger) ~ +310k on the zip

afaics keeping the lambda size small needs a few things

• better logical separation of runtime vs cli deps on zappa
• use a better default exclude that takes into account the lambda env (no dist-info, or dups of lambda env modules) get rid of duplicates and extraneous libraries #581
• zappa handler deps should be minimal, the callback for cli.certify seems to be the worse offender
• always serialize config files as json in the lambda (add to lambda, to avoid carrying toml, yaml deps forward)
• use explicit dependencies from a requirements.txt file instead of copying the whole virtualenv

some of this is easy and should dramatically cut down the zip file size, other parts require code refactoring, like the certify stuff. in general it feels like some refactoring would be good, the kitchensink class approach with cross call chains complicates separation of the cli env and the runtime env.

[cjerdonek]

@Miserlou Do you have a sense of how challenging / how much work it would be to make Zappa operate on a "target" / different virtualenv that doesn't itself contain Zappa? I feel like this would go a long way towards addressing both this issue, as well as issue #525 and related issues.

In the approach I'm describing, the "target" virtualenv would have only those dependencies needed on the Lambda server, and none of the dependencies needed to upload and deploy, etc. This probably requires in some sense "introspecting" the target virtualenv without actually installing Zappa inside. My sense is that there would be lots of possibilities for this, but you're probably most familiar with what's involved.

[lestephane]

Removing zappa would make my 'zappa update' upload take about 8 secs instead of the current 1 minute 6 secs (on my current network connectivity). The time spent waiting for uploads adds up over time. I'm looking at serverless-wsgi + sls as a workaround (in case others need an alternative too). Still, any update on this? I can't assist as i'm a first time python user.