Skip to content

GH-40079: [CI][Packaging] Enable Azure in more tests and builds #40080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kou merged 11 commits into from
Feb 22, 2024
Merged

GH-40079: [CI][Packaging] Enable Azure in more tests and builds #40080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
11cfcb6
Add missing export that prevented any pyarrow tests from running
Tom-Newton Feb 14, 2024
8064499
Enable pyarrow tests on sdist_test
Tom-Newton Feb 14, 2024
1f455b3
Enable Azure on python macos builds
Tom-Newton Feb 14, 2024
cf8ce6a
Enable azure in conda builds and install dependencies
Tom-Newton Feb 14, 2024
5f1e50b
Install node from conda instead of apt
Tom-Newton Feb 14, 2024
79b1c5d
Update comment
Tom-Newton Feb 15, 2024
2e3e51c
Pin specific versions to avoid segfault
Tom-Newton Feb 17, 2024
cbe5c46
Less restrictive pin and add a comment about the issue
Tom-Newton Feb 18, 2024
7c27978
Correct comment typo
Tom-Newton Feb 18, 2024
5e3f846
Add retry mitigation suggested for GH-40121
Tom-Newton Feb 19, 2024
53b9d2c
Remove conda version restriction now that latest release works
Tom-Newton Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ jobs:
timeout-minutes: 60
env:
ARROW_HOME: /usr/local
ARROW_AZURE: ON
ARROW_DATASET: ON
ARROW_FLIGHT: ON
ARROW_GANDIVA: ON
Expand Down
6 changes: 6 additions & 0 deletions ci/conda_env_cpp.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@
# under the License.

aws-sdk-cpp=1.11.68
azure-core-cpp>=1.10.3
azure-identity-cpp>=1.6.0
azure-storage-blobs-cpp>=12.10.0
azure-storage-common-cpp>=12.5.0
azure-storage-files-datalake-cpp>=12.9.0
benchmark>=1.6.0
boost-cpp>=1.68.0
brotli
Expand All @@ -34,6 +39,7 @@ libutf8proc
lz4-c
make
ninja
nodejs
orc
pkg-config
python
Expand Down
8 changes: 8 additions & 0 deletions ci/docker/conda-cpp.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ RUN mamba install -q -y \
valgrind && \
mamba clean --all

# Ensure npm, node and azurite are on path. npm and node are required to install azurite, which will then need to
# be on the path for the tests to run.
ENV PATH=/opt/conda/envs/arrow/bin:$PATH

COPY ci/scripts/install_azurite.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_azurite.sh

# We want to install the GCS testbench using the same Python binary that the Conda code will use.
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
RUN /arrow/ci/scripts/install_gcs_testbench.sh default
Expand All @@ -50,6 +57,7 @@ COPY ci/scripts/install_sccache.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_sccache.sh unknown-linux-musl /usr/local/bin

ENV ARROW_ACERO=ON \
ARROW_AZURE=ON \
ARROW_BUILD_TESTS=ON \
ARROW_DATASET=ON \
ARROW_DEPENDENCY_SOURCE=CONDA \
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/cpp_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ ctest \
--label-regex unittest \
--output-on-failure \
--parallel ${n_jobs} \
--repeat until-pass:3 \
--timeout ${ARROW_CTEST_TIMEOUT:-300} \
"${ctest_options[@]}" \
"$@"
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/python_sdist_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ export PARQUET_TEST_DATA=${arrow_dir}/cpp/submodules/parquet-testing/data
export PYARROW_CMAKE_GENERATOR=${CMAKE_GENERATOR:-Ninja}
export PYARROW_BUILD_TYPE=${CMAKE_BUILD_TYPE:-debug}
export PYARROW_WITH_ACERO=${ARROW_ACERO:-ON}
export PYARROW_WITH_AZURE=${ARROW_AZURE:-OFF}
export PYARROW_WITH_S3=${ARROW_S3:-OFF}
export PYARROW_WITH_ORC=${ARROW_ORC:-OFF}
export PYARROW_WITH_CUDA=${ARROW_CUDA:-OFF}
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/python_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ fi
: ${PYARROW_TEST_S3:=${ARROW_S3:-ON}}

export PYARROW_TEST_ACERO
export PYARROW_TEST_AZURE
export PYARROW_TEST_CUDA
export PYARROW_TEST_DATASET
export PYARROW_TEST_FLIGHT
Expand Down