Feat/py3.12 by brosenberg42 · Pull Request #396 · openmpf/openmpf-components

brosenberg42 · 2025-03-11T14:22:26Z

Issues:

Upgrade to Python 3.12 openmpf#1884

Related PRs:

This change is

jrobble

Reviewed 12 of 12 files at r1, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @brosenberg42)

a discussion (no related file):
Is there an assoc. PR for openmpf-python-component-sdk? I didn't see one.

Please update openmpf-python-component-sdk/README.md : If not already installed, [build and install Python 3.8.]( ...

Please update openmpf-python-component-sdk/detection/nlp_text_splitter/README.md: apt-get install g++ python3.8-dev

python/NlpTextCorrection/nlp_correction_component/nlp_correction_component.py line 125 at r1 (raw file):

        self._hunspell = Hunspell('en_US')
        self._hunspell.clear_cache()

Is this no longer supported, or just not required?

python/ArgosTranslation/Dockerfile line 80 at r1 (raw file):

<<eot
    if [ "${RUN_TESTS,,}" == true ]; then
        echo test

Can this echo be removed?

jrobble · 2025-08-08T02:48:58Z

I was able to run PythonTestComponent locally without issues, but PythonOcvComponent failed after registering with:

2025-08-05 10:57:51,079 INFO  [__init__.py:1539] - Connecting to ActiveMQ broker at: failover:(tcp://localhost:61616)?maxReconnectAttempts=1
2025-08-05 10:57:51,091 INFO  [__init__.py:1539] - Creating ActiveMQ consumer for queue: MPF.DETECTION_PYTHONOCVTEST_REQUEST
2025-08-05 10:57:51,152 FATAL [__init__.py:1592] - A fatal error occurred: ModuleNotFoundError: No module named 'pkg_resources'

At:
  /home/mpf/openmpf-projects/openmpf/trunk/install/plugins/PythonOcvComponent/venv/lib/python3.12/site-packages/ocv_component/ocv_component.py(28): <module>
  <frozen importlib._bootstrap>(488): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(999): exec_module
  <frozen importlib._bootstrap>(950): _load_unlocked
  <frozen importlib._bootstrap>(1334): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1360): _find_and_load
  /home/mpf/openmpf-projects/openmpf/trunk/install/plugins/PythonOcvComponent/venv/lib/python3.12/site-packages/ocv_component/__init__.py(27): <module>
  <frozen importlib._bootstrap>(488): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(999): exec_module
  <frozen importlib._bootstrap>(950): _load_unlocked
  <frozen importlib._bootstrap>(1333): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1360): _find_and_load
  <frozen importlib._bootstrap>(1387): _gcd_import
  <frozen importlib._bootstrap>(488): _call_with_frames_removed
  <frozen importlib._bootstrap>(1310): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1360): _find_and_load
  <frozen importlib._bootstrap>(1387): _gcd_import
  /usr/lib/python3.12/importlib/__init__.py(90): import_module
  /usr/lib/python3.12/importlib/metadata/__init__.py(205): load

To fix this I modified openmpf-python-component-sdk/detection/examples/PythonOcvComponent/setup.cfg by adding setuptools to the bottom of this section:

[options]
packages = ocv_component
install_requires =
    mpf_component_api>=9.0
    mpf_component_util>=9.0
    setuptools

In the base Python Docker image we do pip install --upgrade setuptools when creating the venv so it's unnecessary to add setuptools to the components' setup.cfg.

Note that others have had this issue with Python 3.12: mu-editor/mu#2485 (comment)

Python 3.12 has removed pkg_resources from the standard library (moved to setuptools)

The pkg_resources approach is deprecated in favor of importlib.*: https://setuptools.pypa.io/en/latest/pkg_resources.html

I also got this to work:

import logging
# import pkg_resources
from importlib import resources  # NEW
import os
from typing import Iterable

<snip>

path = resources.files(__name__).joinpath("models")  # NEW
logger.info(f'HELLO: {path}')  # NEW

# (mpf_util.ModelsIniParser(pkg_resources.resource_filename(__name__, 'models'))
ModelSettings = (mpf_util.ModelsIniParser(path)  # NEW
                 .register_path_field('network')
                 .register_path_field('names')
                 .register_int_field('num_classes')
                 .build_class())

Minimally, we should update the example Python component to use importlib.* since that is what devs should be using moving forward.

I think we should make the same change in all of the components that are currently using pkg_resources as well.

brosenberg42

Reviewable status: 10 of 16 files reviewed, 4 unresolved discussions (waiting on @brosenberg42 and @jrobble)

a discussion (no related file):

Previously, jrobble (Jeff Robble) wrote…

Is there an assoc. PR for openmpf-python-component-sdk? I didn't see one.

Please update openmpf-python-component-sdk/README.md : If not already installed, [build and install Python 3.8.]( ...

Please update openmpf-python-component-sdk/detection/nlp_text_splitter/README.md: apt-get install g++ python3.8-dev

Done.

a discussion (no related file):

Previously, jrobble (Jeff Robble) wrote…

I was able to run PythonTestComponent locally without issues, but PythonOcvComponent failed after registering with:

2025-08-05 10:57:51,079 INFO  [__init__.py:1539] - Connecting to ActiveMQ broker at: failover:(tcp://localhost:61616)?maxReconnectAttempts=1
2025-08-05 10:57:51,091 INFO  [__init__.py:1539] - Creating ActiveMQ consumer for queue: MPF.DETECTION_PYTHONOCVTEST_REQUEST
2025-08-05 10:57:51,152 FATAL [__init__.py:1592] - A fatal error occurred: ModuleNotFoundError: No module named 'pkg_resources'

At:
  /home/mpf/openmpf-projects/openmpf/trunk/install/plugins/PythonOcvComponent/venv/lib/python3.12/site-packages/ocv_component/ocv_component.py(28): <module>
  <frozen importlib._bootstrap>(488): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(999): exec_module
  <frozen importlib._bootstrap>(950): _load_unlocked
  <frozen importlib._bootstrap>(1334): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1360): _find_and_load
  /home/mpf/openmpf-projects/openmpf/trunk/install/plugins/PythonOcvComponent/venv/lib/python3.12/site-packages/ocv_component/__init__.py(27): <module>
  <frozen importlib._bootstrap>(488): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(999): exec_module
  <frozen importlib._bootstrap>(950): _load_unlocked
  <frozen importlib._bootstrap>(1333): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1360): _find_and_load
  <frozen importlib._bootstrap>(1387): _gcd_import
  <frozen importlib._bootstrap>(488): _call_with_frames_removed
  <frozen importlib._bootstrap>(1310): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1360): _find_and_load
  <frozen importlib._bootstrap>(1387): _gcd_import
  /usr/lib/python3.12/importlib/__init__.py(90): import_module
  /usr/lib/python3.12/importlib/metadata/__init__.py(205): load

To fix this I modified openmpf-python-component-sdk/detection/examples/PythonOcvComponent/setup.cfg by adding setuptools to the bottom of this section:

[options]
packages = ocv_component
install_requires =
    mpf_component_api>=9.0
    mpf_component_util>=9.0
    setuptools

In the base Python Docker image we do pip install --upgrade setuptools when creating the venv so it's unnecessary to add setuptools to the components' setup.cfg.

Note that others have had this issue with Python 3.12: mu-editor/mu#2485 (comment)

Python 3.12 has removed pkg_resources from the standard library (moved to setuptools)

The pkg_resources approach is deprecated in favor of importlib.*: https://setuptools.pypa.io/en/latest/pkg_resources.html

I also got this to work:

import logging
# import pkg_resources
from importlib import resources  # NEW
import os
from typing import Iterable

<snip>

path = resources.files(__name__).joinpath("models")  # NEW
logger.info(f'HELLO: {path}')  # NEW

# (mpf_util.ModelsIniParser(pkg_resources.resource_filename(__name__, 'models'))
ModelSettings = (mpf_util.ModelsIniParser(path)  # NEW
                 .register_path_field('network')
                 .register_path_field('names')
                 .register_int_field('num_classes')
                 .build_class())

Minimally, we should update the example Python component to use importlib.* since that is what devs should be using moving forward.

I think we should make the same change in all of the components that are currently using pkg_resources as well.

Done.

python/ArgosTranslation/Dockerfile line 80 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Can this echo be removed?

Done.

python/NlpTextCorrection/nlp_correction_component/nlp_correction_component.py line 125 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Is this no longer supported, or just not required?

I had to switch the hunspell library because the old one only supports up to Python 3.9.

jrobble

@jrobble reviewed 6 of 6 files at r2, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @brosenberg42)

jrobble

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @brosenberg42)

a discussion (no related file):
I saw this in the Azure Translation log:

2025-09-08 20:47:08,865 INFO  [acs_translation_component.py:442] - [Job 7:dracula-es-sentence.txt] Sending POST https://somehost/translator/text/v3.0//detect?api-version=3.0
2025-09-08 20:47:08,866 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "Text": "Temía ir muy lejos de la estación, ya que habíamos llegado tarde y comenzaría lo más cerca posible de la hora correcta."
    }
]
2025-09-08 20:47:08,944 INFO  [acs_translation_component.py:446] - [Job 7:dracula-es-sentence.txt] Received response from https://somehost/translator/text/v3.0//detect?api-version=3.0.
2025-09-08 20:47:08,945 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "isTranslationSupported": true,
        "isTransliterationSupported": false,
        "language": "es",
        "score": 1.0
    }
]
2025-09-08 20:47:08,945 INFO  [acs_translation_component.py:361] - [Job 7:dracula-es-sentence.txt] Detected the primary language of the text was es with score 1.0
2025-09-08 20:47:08,945 INFO  [acs_translation_component.py:337] - [Job 7:dracula-es-sentence.txt] Sending POST to https://somehost/translator/text/v3.0//translate?api-version=3.0&to=en&from=es
2025-09-08 20:47:08,945 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "Text": "Temía ir muy lejos de la estación, ya que habíamos llegado tarde y comenzaría lo más cerca posible de la hora correcta."
    }
]
2025-09-08 20:47:09,073 INFO  [acs_translation_component.py:343] - [Job 7:dracula-es-sentence.txt] Received response from https://somehost/translator/text/v3.0//translate?api-version=3.0&to=en&from=es.
2025-09-08 20:47:09,073 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "translations": [
            {
                "text": "I was afraid to go too far from the station as we had arrived late and would start as close to the right time as possible.",
                "to": "en"
            }
        ]
    }
]
2025-09-08 20:47:09,073 INFO  [acs_translation_component.py:246] - [Job 7:dracula-es-sentence.txt] Successfully translated the "TEXT" property.

Please make the lines that output the raw text content and translated text content debug lines in order to prevent bloating the logs. More generally, consider making all of the lines that log the request / response debug lines.

brosenberg42

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @jrobble)

a discussion (no related file):

Previously, jrobble (Jeff Robble) wrote…

I saw this in the Azure Translation log:

2025-09-08 20:47:08,865 INFO  [acs_translation_component.py:442] - [Job 7:dracula-es-sentence.txt] Sending POST https://somehost/translator/text/v3.0//detect?api-version=3.0
2025-09-08 20:47:08,866 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "Text": "Temía ir muy lejos de la estación, ya que habíamos llegado tarde y comenzaría lo más cerca posible de la hora correcta."
    }
]
2025-09-08 20:47:08,944 INFO  [acs_translation_component.py:446] - [Job 7:dracula-es-sentence.txt] Received response from https://somehost/translator/text/v3.0//detect?api-version=3.0.
2025-09-08 20:47:08,945 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "isTranslationSupported": true,
        "isTransliterationSupported": false,
        "language": "es",
        "score": 1.0
    }
]
2025-09-08 20:47:08,945 INFO  [acs_translation_component.py:361] - [Job 7:dracula-es-sentence.txt] Detected the primary language of the text was es with score 1.0
2025-09-08 20:47:08,945 INFO  [acs_translation_component.py:337] - [Job 7:dracula-es-sentence.txt] Sending POST to https://somehost/translator/text/v3.0//translate?api-version=3.0&to=en&from=es
2025-09-08 20:47:08,945 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "Text": "Temía ir muy lejos de la estación, ya que habíamos llegado tarde y comenzaría lo más cerca posible de la hora correcta."
    }
]
2025-09-08 20:47:09,073 INFO  [acs_translation_component.py:343] - [Job 7:dracula-es-sentence.txt] Received response from https://somehost/translator/text/v3.0//translate?api-version=3.0&to=en&from=es.
2025-09-08 20:47:09,073 DEBUG [acs_translation_component.py:762] - [Job 7:dracula-es-sentence.txt] [
    {
        "translations": [
            {
                "text": "I was afraid to go too far from the station as we had arrived late and would start as close to the right time as possible.",
                "to": "en"
            }
        ]
    }
]
2025-09-08 20:47:09,073 INFO  [acs_translation_component.py:246] - [Job 7:dracula-es-sentence.txt] Successfully translated the "TEXT" property.

Please make the lines that output the raw text content and translated text content debug lines in order to prevent bloating the logs. More generally, consider making all of the lines that log the request / response debug lines.

The lines with the request content are already debug messages.

jrobble

Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @brosenberg42)

brosenberg42 added 4 commits March 6, 2025 08:24

Changes for Python upgrade

8124907

Fix nlp text correction

18252a0

Upgrade whisper

86f2ebe

Fix nlp text correction

b82ed12

brosenberg42 self-assigned this Mar 11, 2025

This was referenced Mar 11, 2025

Feat/py3.12 openmpf/openmpf#1885

Merged

Feat/py3.12 openmpf/openmpf-docker#237

Merged

brosenberg42 requested a review from jrobble March 11, 2025 14:29

Fix east

67ffb5a

brosenberg42 removed the request for review from jrobble March 12, 2025 17:32

brosenberg42 mentioned this pull request Mar 17, 2025

Feat/py3.12 openmpf/openmpf-build-tools#49

Merged

brosenberg42 requested a review from jrobble March 17, 2025 12:32

brosenberg42 mentioned this pull request Mar 17, 2025

Add Python3.12 install instructions openmpf/openmpf.github.io#195

Merged

Merge branch 'develop' into feat/py3.12

d653480

jrobble requested changes Aug 4, 2025

View reviewed changes

brosenberg42 added 5 commits August 14, 2025 12:04

Merge branch 'develop' into feat/py3.12

ccb2ff4

Fix spacy version in AzureTranslation

d3d2d1b

Change assertEquals to assertEqual

91cfee6

Remove pkg_resources

be94abc

Remove debug line

d495cf7

brosenberg42 mentioned this pull request Aug 19, 2025

Feat/py3.12 openmpf/openmpf-python-component-sdk#92

Merged

brosenberg42 commented Aug 19, 2025

View reviewed changes

jrobble reviewed Aug 26, 2025

View reviewed changes

jrobble requested changes Sep 8, 2025

View reviewed changes

brosenberg42 commented Sep 9, 2025

View reviewed changes

jrobble approved these changes Sep 9, 2025

View reviewed changes

brosenberg42 merged commit 9adca03 into develop Sep 10, 2025
2 checks passed

brosenberg42 deleted the feat/py3.12 branch September 10, 2025 17:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/py3.12#396

Feat/py3.12#396
brosenberg42 merged 11 commits into
developfrom
feat/py3.12

brosenberg42 commented Mar 11, 2025 •

edited

Loading

Uh oh!

jrobble left a comment

Uh oh!

jrobble commented Aug 8, 2025

Uh oh!

brosenberg42 left a comment

Uh oh!

jrobble left a comment

Uh oh!

jrobble left a comment

Uh oh!

brosenberg42 left a comment

Uh oh!

jrobble left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

brosenberg42 commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble commented Aug 8, 2025

Uh oh!

brosenberg42 left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

brosenberg42 left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

brosenberg42 commented Mar 11, 2025 •

edited

Loading