Update Fence to Python 3.13 + Run as gen3 user #1312

nss10 · 2025-12-12T21:24:30Z

Link to JIRA ticket if there is one: PD-77

New Features

Breaking Changes

Bug Fixes

Improvements

Replace raw strings in session.execute() with sqlalchemy.text
Update deprecated datetime.utcnow() and datetime.utcfromtimestamp() with 3.13 compatible equivalents
Replace deprecated request attributes with request.payload
Replace deprecated SqlAlchemy's query.get() with Session.get()
Eliminate all boto usage in favor of boto3
Add a local implementation of strtobool since distutils is no longer available since Python 3.12
Add NoAsyncMagicMock since MagicMock returns AsyncMocks even if the source object is partially async
Replace assert <mocked_method>.not_called with <mocked_method>.assert_not_called

Dependency updates

Update Fence to Python 3.13

Deployment changes

Change Docker image to run with gen3 user instead of root

…ocal_settings.py`

…dependency_updates

…update_fence_3_13

coveralls · 2025-12-12T21:29:30Z

Pull Request Test Coverage Report for Build 20471262045

Details

0 of 0 changed or added relevant lines in 0 files are covered.
423 unchanged lines in 17 files lost coverage.
Overall coverage increased (+0.02%) to 75.002%

Files with Coverage Reduction	New Missed Lines	%
blueprints/misc.py	1	88.24%
ea7e1b843f82_optional_client_redirect_uri.py	1	81.82%
resources/audit/client.py	1	77.31%
scripting/google_monitor.py	1	40.17%
a04a70296688_non_unique_client_name.py	2	89.47%
blueprints/oauth2.py	2	76.03%
blueprints/login/ras.py	6	75.47%
blueprints/data/blueprint.py	13	86.9%
resources/ga4gh/passports.py	15	89.13%
9b3a5a7145d7_authlib_update_1_2_1.py	18	60.22%

Totals
Change from base Build 20076693184:	0.02%
Covered Lines:	8440
Relevant Lines:	11253

💛 - Coveralls

github-actions · 2025-12-12T22:46:29Z

Failed to Prepare CI environment

Please find the Github Action logs here

github-actions · 2025-12-12T23:43:29Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	2	2	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_user_token.py	5	0	0	5
tests/test_register_user.py	2	0	0	2
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	1	1	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	65	3	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-15T15:30:14Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	2	2	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_user_token.py	5	0	0	5
tests/test_register_user.py	2	0	0	2
tests/test_oidc_client.py	2	0	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	66	2	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

nss10 · 2025-12-18T17:08:02Z

tests/test_app_config.py

        patchers.append(patcher)

+    # create a fresh local app
+    from flask import Flask


This did not throw an error in older versions of fence due to the order in which tests ran.

nss10 · 2025-12-18T17:40:53Z

tests/data/test_data.py

        headers = {"Authorization": "Bearer " + encoded_creds_jwt.jwt}
        response = client.delete("/data/{}".format(did), headers=headers)
        assert response.status_code == 204
-        assert mock_boto_delete.called_once()


Since IndexedFile.delete_files is patched with a MagicMock, we never really hit the core boto delete functionality in the past, since we were patching IndexedFile.delete_files before we reached the core functionality we wanted to test

The tests were passing in the past, because assert .called_once() returned a new MagicMock which is non-falsy in nature, making the assert to pass

nss10 · 2025-12-18T18:01:26Z

tests/data/test_indexed_file.py

            """
            raise Exception("url not available")

-    with patch("fence.blueprints.data.indexd.flask.current_app", return_value=app):


Since flask is coming from a fixture, flask.current_app is already set to app.
Setting this patch, was making everything internally to MagicMock. Removing this patch altogether tests the logic better

nss10 · 2025-12-18T18:29:57Z

tests/ras/test_ras.py

    assert username_to_log_in == username
    iss_sub_pair_to_user_records = db_session.query(IssSubPairToUser).all()
    assert len(iss_sub_pair_to_user_records) == 1
-    iss_sub_pair_to_user = db_session.query(IssSubPairToUser).get((iss, sub))


# LegacyAPIWarning: The Query.get() method is considered legacy as of the 1.x series of SQLAlchemy and becomes a legacy construct in 2.0. The method is now available as Session.get() (deprecated since: 2.0) (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

github-actions · 2025-12-18T18:59:55Z

Failed to Prepare CI environment

Please find the Github Action logs here

github-actions · 2025-12-19T03:21:33Z

filepath	passed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	15
tests/test_centralized_auth.py	16	0	16
tests/test_audit_service.py	3	3	6
tests/test_data_upload.py	8	1	9
tests/test_presigned_url.py	7	0	7
tests/test_user_token.py	5	0	5
tests/test_dbgap.py	4	1	5
tests/test_drs_endpoint.py	4	0	4
tests/test_oidc_client.py	2	0	2
tests/test_client_credentials.py	1	0	1
tests/test_google_data_access.py	1	0	1
tests/test_register_user.py	2	0	2
tests/test_ras_authn.py	0	3	3
TOTAL	68	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

paulineribeyre

just minor comments!

fence/resources/openid/idp_oauth2.py

migrations/versions/9b3a5a7145d7_authlib_update_1_2_1.py

tests/dbgap_sync/test_user_sync.py

tests/job/test_access_token_updater.py

tests/ras/test_ras.py

tests/conftest.py

tests/test_app_config.py

Dockerfile

paulineribeyre · 2025-12-23T17:32:12Z

pyproject.toml

+  "gunicorn",  # not imported but used in Dockerfile
+  "psycopg2-binary",   # not imported but used by sqlalchemy
+  "urllib3", # Needed to be pinned to a version >=2.6.0
+  "sniffio", # required by gen3authz, but not installed as a transitive dependency during poetry install


i added it here, if you upgrade gen3authz to the latest version you should be able to remove sniffio here

gen3authz 3.0.0 requires backoff>=2.2.1, but fence has backoff pinned to <2 several years ago. I'm trying to remove that restriction and run unit tests to see if anything breaks.

Dockerfile

github-actions · 2025-12-23T17:52:05Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	15	1	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	4	0	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_register_user.py	2	0	0	2
tests/test_user_token.py	5	0	0	5
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_ras_authn.py	0	0	3	3
TOTAL	67	1	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

nss10 · 2025-12-23T18:21:42Z

tests/job/test_access_token_updater.py

    run_async(updater_queue.put(mock_users[0]))

    # Mock the client to return a valid update process
-    mock_oidc_clients[0].update_user_authorization = AsyncMock()


The method update_user_authorization (https://github.com/uc-cdis/fence/blob/e288114ca81d31c2abd28dbd20d2d64e18ed87e0/fence/resources/openid/idp_oauth2.py#L446C20-L446C20) is not Async to use AsyncMock

paulineribeyre

Looks good, just need to update gen3authz and remove the sniffio dependency if you can

github-actions · 2025-12-23T20:01:24Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	16	0	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	4	0	1	5
tests/test_user_token.py	4	1	0	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_register_user.py	2	0	0	2
tests/test_google_data_access.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_client_credentials.py	1	0	0	1
tests/test_ras_authn.py	0	0	3	3
TOTAL	67	1	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-23T21:35:29Z

filepath	passed	failed	skipped	SUBTOTAL
tests/test_oauth2.py	15	0	0	15
tests/test_centralized_auth.py	15	1	0	16
tests/test_audit_service.py	3	0	3	6
tests/test_data_upload.py	8	0	1	9
tests/test_presigned_url.py	7	0	0	7
tests/test_dbgap.py	4	0	1	5
tests/test_drs_endpoint.py	4	0	0	4
tests/test_register_user.py	2	0	0	2
tests/test_user_token.py	5	0	0	5
tests/test_google_data_access.py	1	0	0	1
tests/test_client_credentials.py	1	0	0	1
tests/test_oidc_client.py	2	0	0	2
tests/test_ras_authn.py	0	0	3	3
TOTAL	67	1	8	76

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2025-12-24T19:22:23Z

filepath	passed	SUBTOTAL
tests/test_centralized_auth.py	16	16
TOTAL	16	16

Please find the detailed integration test report here

Please find the Github Action logs here

nss10 added 26 commits October 10, 2025 14:54

Update dependencies in fence

4feccd7

Remove Deprecated local_settings.py

44a6eea

Delete references of local_settings and files that are linked to `l…

71102ee

…ocal_settings.py`

Merge branch 'master' of https://github.com/uc-cdis/fence into chore/…

c04e664

…dependency_updates

Merge branch 'master' of https://github.com/uc-cdis/fence into chore/…

ce9c522

…dependency_updates

Update poetry lock

62909d0

Update poetry lock to resolve snyk errors

bb89c64

Add a forced dependency on idna, to overcome a snyk dependency

fc45c35

Initial commit with 3.13 -- failing unit tests due to boto

9b61c79

Update boto to boto3 in cleversafe. Failing dbgap unit tests

e288114

Fix unit tests to work with Python 3.13

ca80e1b

Fix python version in ci

0b326af

Update deprecated distutils

58134c7

Custom implementation of strtobool., instead of using setuptools

674bbbf

Update raw sql statement from str to sq.text

08db93c

Update Dockerfile with 3.13 image

093d288

Revert cov report to include coveralls

1b49239

Merge branch 'master' of https://github.com/uc-cdis/fence into chore/…

c8e62ca

…update_fence_3_13

Update Arborist MagicMock to NoAsyncMagicMock

a59830d

Make changes to remove warnings

7664b57

Run dockerfile with root user

c4ab7b9

Set the final user for fence as gen3 instead of root

9a76456

Add empty line for clarity

01147ee

Set the user back to root

1d8e3a8

parallel image with gen3 user

6c7ea5d

Use custom helm chart to work with gen3 user

61f80d2

nss10 commented Dec 18, 2025

View reviewed changes

Final updates before making PR ready for review

55c2e3b

Run with HELM master branch

0317dc7

nss10 requested review from Avantol13 and k-burt-uch and removed request for k-burt-uch December 19, 2025 03:24

Update pyproject and docstring comments

7534748

nss10 requested a review from paulineribeyre December 23, 2025 17:05

paulineribeyre reviewed Dec 23, 2025

View reviewed changes

nss10 commented Dec 23, 2025

View reviewed changes

nss10 added 2 commits December 23, 2025 13:01

Respond to PR comments

48c0c0a

Update comment

506a2c8

nss10 requested a review from paulineribeyre December 23, 2025 19:11

paulineribeyre reviewed Dec 23, 2025

View reviewed changes

Eliminate sniffio and update gen3authz

da5b213

nss10 added the TestCentralizedAuth label Dec 24, 2025

nss10 requested a review from paulineribeyre December 29, 2025 18:03

paulineribeyre approved these changes Jan 2, 2026

View reviewed changes

nss10 merged commit 12eb038 into master Jan 2, 2026
11 of 12 checks passed

nss10 deleted the temp/python_313_gen3_user branch January 2, 2026 19:19

Update Fence to Python 3.13 + Run as gen3 user #1312

Update Fence to Python 3.13 + Run as gen3 user #1312

Uh oh!

Conversation

nss10 commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

Uh oh!

coveralls commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 20471262045

Details

💛 - Coveralls

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

nss10 Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

paulineribeyre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulineribeyre Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

nss10 Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 23, 2025

Uh oh!

nss10 Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulineribeyre left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 23, 2025

Uh oh!

github-actions bot commented Dec 23, 2025

Uh oh!

github-actions bot commented Dec 24, 2025

Uh oh!

Uh oh!

nss10 commented Dec 12, 2025 •

edited

Loading

coveralls commented Dec 12, 2025 •

edited

Loading

nss10 Dec 18, 2025 •

edited

Loading

nss10 Dec 18, 2025 •

edited

Loading

nss10 Dec 23, 2025 •

edited

Loading