AN-497 Ubuntu 20 -> 22 for all images #503

jgainerdewar · 2025-04-09T18:44:15Z

Rebuild all terra-jupyter-* images on gcr.io/deeplearning-platform-release/tf2-cu123.2-17.py310. This upgrades us from ubuntu 20 to ubuntu 22.

Leo changes in DataBiosphere/leonardo#4843

jgainerdewar · 2025-04-24T19:28:55Z

terra-jupyter-aou/Dockerfile

+# of jupyter and they're google-managed packages we don't have access to older versions of.
+# Retain two plugins: google.cloud.bigquery and sql
+RUN sed -i "s/^\( *'beatrix_jupyterlab',\)/#\1/g" /etc/ipython/ipython_kernel_config.py
+RUN sed -i "s/^\( *'bq_stats',\)/#\1/g" /etc/ipython/ipython_kernel_config.py


This worries me a little because I'm not sure whether anyone is using these extensions. They are google-managed Python packages and we can't install versions that are compatible with our older notebook version, so I'm turning them off. Same deal in terra-jupyter-python.

I am comfortable making that decision. Once we update to a newer version of notebooks we can add them back if we think we need it.

jgainerdewar · 2025-04-24T19:30:35Z

terra-jupyter-base/Dockerfile


 RUN chown -R $USER:users $JUPYTER_HOME \
-    # Disable nb_conda for now. Consider re-enable in the future
-    && jupyter nbextension disable nb_conda --py --sys-prefix \


This line no longer works because the extension can't be found. No need to disable if it isn't there!

jgainerdewar · 2025-04-24T19:32:04Z

terra-jupyter-python/tests/smoke_test.ipynb

   "outputs": [],
   "source": [
-    "output = !pip3 show pendulum\n",
+    "output = !pip3 show emoji\n",


This test checks whether we can install a new package - in the new image, pendulum is already installed, so it makes the test fail. Switch to a package that is not already installed.

jgainerdewar · 2025-04-24T19:32:55Z

terra-jupyter-python/tests/smoke_test.ipynb

+    "output = !pip3 show emoji\n",
    "print(output)  # Should show not yet installed.\n",
-    "assert(0 == output.count('Name: pendulum'))"
+    "assert(0 == len([i for i in output if 'Name: emoji' in i]))"


Needed to adjust this because the output strings now contain some color-coding junk characters. We need to check whether the strings contain the expected message rather than being equal to it.

aednichols · 2025-04-24T20:02:30Z

terra-jupyter-python/Dockerfile

+# Turn off beatrix-jupyterlab and bq_stats plugins, they don't activate cleanly in our version
+# of jupyter and they're google-managed packages we don't have access to older versions of.
+# Retain two plugins: google.cloud.bigquery and sql
+RUN sed -i "s/^\( *'beatrix_jupyterlab',\)/#\1/g" /etc/ipython/ipython_kernel_config.py
+RUN sed -i "s/^\( *'bq_stats',\)/#\1/g" /etc/ipython/ipython_kernel_config.py


A reasonable compromise, that speaks to the need to delegate package selection to users.

LizBaldo

This is amazing, I can feel the pain in every git diff 🙈. I am approving to unblock but I would wait until all images are cached and can be launched on your BEE in leo before merging and triggering the GHA that will generate the version file in the documentation bucket

LizBaldo · 2025-04-24T20:56:49Z

README.md

-It is not advised to run build.sh locally, as this will push to the remote docker repo and delete the image locally upon completion. 
+Apple Silicon chips later then M1 (ex. M3) need:
+
+`FROM --platform=linux/amd64/v2 <image name>`


LizBaldo · 2025-04-24T20:58:36Z

config/legacy_static_images.json

-    "label": "Legacy GATK (GATK 4.2.4.0, Python 3.7.12, R 4.3.0)",
-    "version": "2.2.14",
+    "updated": "2025-04-24",
+    "image": "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-gatk:2.3.9",


I think you mean 2.3.8 for GATK legacy instead

Yes, yes I do, good catch!

LizBaldo · 2025-04-24T21:01:03Z

terra-jupyter-aou/Dockerfile

+# of jupyter and they're google-managed packages we don't have access to older versions of.
+# Retain two plugins: google.cloud.bigquery and sql
+RUN sed -i "s/^\( *'beatrix_jupyterlab',\)/#\1/g" /etc/ipython/ipython_kernel_config.py
+RUN sed -i "s/^\( *'bq_stats',\)/#\1/g" /etc/ipython/ipython_kernel_config.py


I am comfortable making that decision. Once we update to a newer version of notebooks we can add them back if we think we need it.

LizBaldo · 2025-04-24T21:05:16Z

terra-jupyter-python/requirements.txt

 scikit-learn-intelex
 xgboost
 #older version of nbconvert fail to convert notebooks to html
 nbconvert>=7.7.3


Interesting that this did not need to change in order to fix the notebook download issue on Terra 🤔

hmm that is odd, I poked around a bit and there's an nbconvert_support section of jupyter_contrib_nbextensions, maybe the old version of that was incompatible for some reason

This is saying we want anything equal to or later than 7.7.3, so I assume there was a new package version published since we built the last images, which fixed the issue.

lucymcnatt · 2025-04-24T21:41:26Z

terra-jupyter-python/tests/smoke_test.ipynb

    "import numpy as np\n",
    "import pandas as pd\n",
-    "from pandas_profiling import ProfileReport\n",
+    "from ydata_profiling import ProfileReport\n",


did the name of this package change?

jgainerdewar added 30 commits April 2, 2025 14:34

Adjust apt packages for jammy

5f235be

ENV formatting

935c8bf

Get Jupyter running

cfc549c

Changelog

b598caf

Update versions

1c195b2

Update immediately-downstream images to use new base

9733c6e

Update smoke test to install new package

a18adbc

Pin some requirements

7fd8b71

Remove incompatible ipython plugins

cc2b8fa

Don't build the base image if we pulled it successfully

75e3f35

Bogus commit to trigger CI

f576623

Revert incorrect fix

dcd0498

More pinning

8959c78

Test fixes

a8d01f1

Fix quotes

dc2be8a

Uuuuggghhh

95df5a6

Consistent soft tabs

8add199

Disable dependency tracking for protobuf build

ef0acee

Successful apt package install

40986ef

Use new base image for terra-jupyter-hail

078a43a

Fix env syntax

d8d9ed9

Get R image building

c29c1be

Rev bioconductor image

03e89c1

Update Hail in config

ce2c29e

Update bioconductor image in config

0061a89

Set gatk image to use new base images

e998527

New terra-jupyter-aou image based on new GATK image

497d2a1

Update package names/versions for new Ubuntu release

a9110c4

Upgrade ICU

e554e86

Upgrade pangenie due to build error

7e57517

jgainerdewar added 5 commits April 23, 2025 10:27

Update doc for post-M1 machines

e3a5785

Turn off unused ipython plugins

7b3dd7c

Cleanup

20e1299

Clean up changelogs

4adf198

Update static legacy image conf

c644331

jgainerdewar commented Apr 24, 2025

View reviewed changes

jgainerdewar marked this pull request as ready for review April 24, 2025 19:36

lucymcnatt self-requested a review April 24, 2025 19:43

aednichols approved these changes Apr 24, 2025

View reviewed changes

LizBaldo approved these changes Apr 24, 2025

View reviewed changes

lucymcnatt reviewed Apr 24, 2025

View reviewed changes

lucymcnatt approved these changes Apr 24, 2025

View reviewed changes

jgainerdewar mentioned this pull request Apr 25, 2025

AN-497 Update Leo to point at new ubuntu 22 images DataBiosphere/leonardo#4843

Merged

4 tasks

jgainerdewar merged commit 7d4b885 into master Apr 30, 2025
9 checks passed

jgainerdewar deleted the jd_AN-497_ubuntu22 branch April 30, 2025 17:07

AN-497 Ubuntu 20 -> 22 for all images #503

AN-497 Ubuntu 20 -> 22 for all images #503

Uh oh!

Conversation

jgainerdewar commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LizBaldo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgainerdewar Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jgainerdewar commented Apr 9, 2025 •

edited

Loading

jgainerdewar Apr 25, 2025 •

edited

Loading