[AN-503] Update to Dataproc 2.2.X

[LizBaldo]

Jira ticket: https://broadworkbench.atlassian.net/browse/AN-503

Summary of changes

What

• Updates our dataproc image from 2.1.x, to 2.2.x, which required:
• • Updating docker compose to a supported version
• • Installing a new log agent for the jupyter, welder, and proxy logs
• • Forcing the GCE cluster to assign an external IP address to each VM node as the behavior has changed between 2.1.x and 2.2.x (https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/network#create-a-dataproc-cluster-with-internal-IP-addresses-only)
• Caches the terra-docker images (hail and AOU) with the latest hail version (0.3.134) see: [AN-503] Update hail to 0.2.134 terra-docker#504
• Also caches the AOU docker image that is currently in production (2.2.13), and the corresponding dataproc 2.1.x image. Leonardo will deploy aou 2.2.13 on the dataproc 2.1.x image, while any other image will be deployed on dataproc 2.2.x

Why

• We need to keep up with the latest releases from the Hail team to enable our users to continue do their analyses on a tool that remains supported
• Being able to cache both dataproc 2.1.x and 2.2.x will help the AOU team do testing and transition to their new 2.2.16 image on their own timeline

Testing these changes

I pointed my BEE to this PR and was able to successfully launch a Hail and AOU image with both a spark single node, and a spark cluster with 2 nodes.
When opening a jupyter notebook, I can import and initialize the new version of hail:

I also was able to launch the AOU image that is currently I prod using the legacy Dataproc 2.1 image. So we should be safe to merge this as it won't impact RWB and they can move over to Dataproc 2.2 when they want:

• This change is covered by automated tests
  • NB: Rerun automation tests on this PR by commenting jenkins retest or jenkins multi-test.
• I validated this change
• Primary reviewer validated this change
• I validated this change in the dev environment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.67%. Comparing base (f58279b) to head (7c9abd7).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4839      +/-   ##
===========================================
- Coverage    74.67%   74.67%   -0.01%     
===========================================
  Files          166      166              
  Lines        14623    14622       -1     
  Branches      1156     1143      -13     
===========================================
- Hits         10920    10919       -1     
  Misses        3703     3703              

Files with missing lines	Coverage Δ
...bench/leonardo/config/ClusterResourcesConfig.scala	100.00% <ø> (ø)
...titute/dsde/workbench/leonardo/config/Config.scala	97.78% <100.00%> (+<0.01%)	⬆️
...te/dsde/workbench/leonardo/util/BucketHelper.scala	60.00% <100.00%> (+0.42%)	⬆️
.../workbench/leonardo/util/DataprocInterpreter.scala	63.15% <100.00%> (-0.46%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f58279b...7c9abd7. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[LizBaldo]

This is the main change in addition to updating docker compose, and that sneaky change in external IP assignment behavior.

It is annoying that the new log agent does not come pre-built into the dataproc image itself, but the install and setup was not too too bad in the end

[aednichols]

I appreciate the detailed comments.

[LizBaldo]

@Qi77Qi I modified the PR to make sure that Leonardo can support both the deployment of the AOU 2.2.13 image on Dataproc 2.1.x (aka what you currently have in production), and AOU 2.2.16 image on Dataproc 2.2.x. I will do some testing on my BEE, but this should let us release the new hail/dataproc version on terra without impacting RWB (you can switch your pre prod / prod environments on your own timeline).
Note that the dataproc 2.1.x image will expire in 60 days, so please let the team know if you have not migrated to using the new AOU / dataproc image by then