Skip to content

workflows: update marker from require_non_self_hosted to require_docker_in_docker in Pposargs #35598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
damccorm merged 1 commit into from
Jul 15, 2025
Merged

workflows: update marker from require_non_self_hosted to require_docker_in_docker in Pposargs #35598

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/beam_PostCommit_Python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,8 @@ jobs:
-PuseWheelDistribution \
-Pposargs="${{
contains(matrix.os, 'self-hosted') &&
'-m ''not require_non_self_hosted''' ||
'-m ''require_non_self_hosted'''
'-m ''not require_docker_in_docker''' ||
'-m ''require_docker_in_docker'''
}}" \
-PpythonVersion=${{ matrix.python_version }} \
env:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/beam_PreCommit_Python_ML.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ jobs:
arguments: |
-Pposargs="${{
contains(matrix.os, 'self-hosted') &&
'apache_beam/ml/ -m ''not require_non_self_hosted''' ||
'apache_beam/ml/ -m ''require_non_self_hosted'''
'apache_beam/ml/ -m ''not require_docker_in_docker''' ||
'apache_beam/ml/ -m ''require_docker_in_docker'''
Copy link
Contributor

@damccorm damccorm Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, it looks like this will currently not work because of

beam/sdks/python/scripts/run_pytest.sh

Line 34 in 8930c8d

echo "$0 cannot be called with -m as it interferes with 'no_xdist' logic, see BEAM-12985."

Copy link
Contributor

@damccorm damccorm Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/beam/actions/runs/16300739721/job/46034393155

Copy link
Contributor

@damccorm damccorm Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the context - https://issues.apache.org/jira/browse/BEAM-12985

Copy link
Contributor

@damccorm damccorm Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a simple fix would be to replace

beam/sdks/python/scripts/run_pytest.sh

Line 34 in 8930c8d

echo "$0 cannot be called with -m as it interferes with 'no_xdist' logic, see BEAM-12985."

and instead of failing, collect the extra -m arguments and add the appropriate not no_xdist and no_xdist flags in

beam/sdks/python/scripts/run_pytest.sh

Lines 44 to 48 in 8930c8d

pytest -v -rs -o junit_suite_name=${envname} \
--junitxml=pytest_${envname}.xml -m 'not no_xdist' -n 6 --import-mode=importlib ${pytest_args} --pyargs ${posargs}
status1=$?
pytest -v -rs -o junit_suite_name=${envname}_no_xdist \
--junitxml=pytest_${envname}_no_xdist.xml -m 'no_xdist' --import-mode=importlib ${pytest_args} --pyargs ${posargs}

Copy link
Contributor Author

@mohamedawnallah mohamedawnallah Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just seen this notification comments...

Copy link
Contributor Author

@mohamedawnallah mohamedawnallah Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a simple fix would be to replace

beam/sdks/python/scripts/run_pytest.sh

Line 34 in 8930c8d

echo "$0 cannot be called with -m as it interferes with 'no_xdist' logic, see BEAM-12985."

and instead of failing, collect the extra -m arguments and add the appropriate not no_xdist and no_xdist flags in

beam/sdks/python/scripts/run_pytest.sh

Lines 44 to 48 in 8930c8d

pytest -v -rs -o junit_suite_name=${envname} \
--junitxml=pytest_${envname}.xml -m 'not no_xdist' -n 6 --import-mode=importlib ${pytest_args} --pyargs ${posargs}
status1=$?
pytest -v -rs -o junit_suite_name=${envname}_no_xdist \
--junitxml=pytest_${envname}_no_xdist.xml -m 'no_xdist' --import-mode=importlib ${pytest_args} --pyargs ${posargs}

This approach looks good 👍

Copy link
Contributor Author

@mohamedawnallah mohamedawnallah Jul 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've also another approach of just using -k instead -m for keyword matching but I think your approach is more reliable in the future

(.venv) dev@dev:~/beam_dev/beam/sdks/python/apache_beam/ml/rag/enrichment$ pytest -v -k "require_docker_in_docker"  --test-pipeline-options="--runner=DirectRunner"
====================================================== test session starts =======================================================
platform linux -- Python 3.9.22, pytest-7.4.4, pluggy-1.6.0 -- /home/dev/beam_dev/beam/sdks/python/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /home/dev/beam_dev/beam/sdks/python
configfile: pytest.ini
plugins: xdist-3.8.0, requests-mock-1.12.1, timeout-2.4.0, anyio-4.9.0, hypothesis-6.135.31
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 36 items / 27 deselected / 9 selected

milvus_search_it_test.py::TestMilvusSearchEnrichment::test_empty_input_chunks PASSED                                       [ 11%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_filtered_search_with_bm25_full_text_and_batching PASSED         [ 22%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_filtered_search_with_cosine_similarity_and_batching PASSED      [ 33%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_hybrid_search PASSED                                            [ 44%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_invalid_query_on_non_existent_collection PASSED                 [ 55%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_invalid_query_on_non_existent_field PASSED                      [ 66%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_keyword_search_with_inner_product_sparse_embedding PASSED       [ 77%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_vector_search_with_euclidean_distance PASSED                    [ 88%]
milvus_search_it_test.py::TestMilvusSearchEnrichment::test_vector_search_with_inner_product_similarity PASSED              [100%]

=============================================== 9 passed, 27 deselected in 30.26s ================================================

}}" \
-PpythonVersion=${{ matrix.python_version }}
- name: Archive Python Test Results
Expand Down
Loading