Fix non existing test pyspark master #15
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raulcd
pushed a commit
that referenced
this pull request
Apr 12, 2024
…ache#41152) ### Rationale for this change An error is received installing R duckdb: ``` #15 18.13 > remotes::install_github('duckdb/duckdb-r', build = FALSE) #15 18.27 Error: Failed to install 'unknown package' from **GitHub:** #15 18.27 Line starting 'Roxyg ...' is malformed! ``` Some searching seems to suggest that this is because R cannot process UTF-8 characters in DESCRIPTION files if the `LANG` is set to `C`. ### What changes are included in this PR? The `LANG` is set to `C.UTF-8` in the dockerfile for this CI job ### Are these changes tested? The change only affects a test ### Are there any user-facing changes? No * GitHub Issue: apache#41145 Authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
raulcd
pushed a commit
that referenced
this pull request
Dec 18, 2024
…n timezone (apache#45051) ### Rationale for this change If the timezone database is present on the system, but does not contain a timezone referenced in a ORC file, the ORC reader will crash with an uncaught C++ exception. This can happen for example on Ubuntu 24.04 where some timezone aliases have been removed from the main `tzdata` package to a `tzdata-legacy` package. If `tzdata-legacy` is not installed, trying to read a ORC file that references e.g. the "US/Pacific" timezone would crash. Here is a backtrace excerpt: ``` #12 0x00007f1a3ce23a55 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #13 0x00007f1a3ce39391 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6 #14 0x00007f1a3f4accc4 in orc::loadTZDB(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #15 0x00007f1a3f4ad392 in std::call_once<orc::LazyTimezone::getImpl() const::{lambda()#1}>(std::once_flag&, orc::LazyTimezone::getImpl() const::{lambda()#1}&&)::{lambda()#2}::_FUN() () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #16 0x00007f1a4298bec3 in __pthread_once_slow (once_control=0xa5ca7c8, init_routine=0x7f1a3ce69420 <__once_proxy>) at ./nptl/pthread_once.c:116 #17 0x00007f1a3f4a9ad0 in orc::LazyTimezone::getEpoch() const () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #18 0x00007f1a3f4e76b1 in orc::TimestampColumnReader::TimestampColumnReader(orc::Type const&, orc::StripeStreams&, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #19 0x00007f1a3f4e84ad in orc::buildReader(orc::Type const&, orc::StripeStreams&, bool, bool, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #20 0x00007f1a3f4e8dd7 in orc::StructColumnReader::StructColumnReader(orc::Type const&, orc::StripeStreams&, bool, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #21 0x00007f1a3f4e8532 in orc::buildReader(orc::Type const&, orc::StripeStreams&, bool, bool, bool) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #22 0x00007f1a3f4925e9 in orc::RowReaderImpl::startNextStripe() () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #23 0x00007f1a3f492c9d in orc::RowReaderImpl::next(orc::ColumnVectorBatch&) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 #24 0x00007f1a3e6b251f in arrow::adapters::orc::ORCFileReader::Impl::ReadBatch(orc::RowReaderOptions const&, std::shared_ptr<arrow::Schema> const&, long) () from /tmp/arrow-HEAD.ArqTs/venv-wheel-3.12-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.12/site-packages/pyarrow/libarrow.so.1900 ``` ### What changes are included in this PR? Catch C++ exceptions when iterating ORC batches instead of letting them slip through. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#40633 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
raulcd
added a commit
that referenced
this pull request
Jan 21, 2026
…ption instead of final Azure::Storage::StorageException and set minimum nodejs on conda env to 16 for Azurite to work (apache#48895) ### Rationale for this change nodejs 12 is currently being installed on conda. CI jobs are failing and or segfaulting due to azurite failing with old versions. ``` 2026-01-13T18:32:39.6961900Z #15 [ 9/11] RUN /arrow/ci/scripts/install_azurite.sh 2026-01-13T18:32:39.9624124Z #15 0.417 Node.js version = v12.4.0 2026-01-13T18:32:42.2087322Z #15 2.663 npm WARN deprecated rimraf@ 3.0.2: Rimraf versions prior to v4 are no longer supported 2026-01-13T18:32:42.3917601Z #15 2.846 npm WARN deprecated uuid@ 3.4.0: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. 2026-01-13T18:32:51.4870197Z #15 11.94 npm WARN deprecated glob@ 7.2.3: Glob versions prior to v9 are no longer supported 2026-01-13T18:32:51.7035681Z #15 12.01 npm WARN deprecated inflight@ 1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful. 2026-01-13T18:33:02.1406491Z #15 22.59 /opt/conda/envs/arrow/bin/azurite -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/azurite.js 2026-01-13T18:33:02.3841290Z #15 22.60 /opt/conda/envs/arrow/bin/azurite-queue -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/queue/main.js 2026-01-13T18:33:02.3842792Z #15 22.60 /opt/conda/envs/arrow/bin/azurite-blob -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/blob/main.js 2026-01-13T18:33:02.3844216Z #15 22.60 /opt/conda/envs/arrow/bin/azurite-table -> /opt/conda/envs/arrow/lib/node_modules/azurite/dist/src/table/main.js 2026-01-13T18:33:02.3846002Z #15 22.66 npm WARN applicationinsights@ 2.9.8 requires a peer of applicationinsights-native-metrics@* but none is installed. You must install peer dependencies yourself. 2026-01-13T18:33:02.3847278Z #15 22.66 2026-01-13T18:33:02.3847564Z #15 22.66 + azurite@ 3.35.0 2026-01-13T18:33:02.3848038Z #15 22.66 added 376 packages from 296 contributors in 20.644s 2026-01-13T18:33:02.3848830Z #15 22.69 /opt/conda/envs/arrow/bin/azurite 2026-01-13T18:33:02.8929329Z #15 23.35 /opt/conda/envs/arrow/lib/node_modules/azurite/node_modules/fs-extra/lib/util/async.js:14 2026-01-13T18:33:02.8930231Z #15 23.35 (err) => err ?? new Error('unknown error') 2026-01-13T18:33:02.8930740Z #15 23.35 ^ ``` The job on PyArrow was segfaulting due to an Exception being thrown but not catch. In general we were using `Azure::Storage::StorageException` but `Azure::Core::Http::TransportException` could also be thrown on some cases. Bot are final but inherit from `Azure::Core::RequestFailedException`. ### What changes are included in this PR? - Pin minimum nodejs version to 16 so the failure doesn't happen again. - Update catching `Azure::Storage::StorageException` to `Azure::Core::RequestFailedException` so `Azure::Core::Http::TransportException` is also catch. ### Are these changes tested? Yes on CI. ### Are there any user-facing changes? No * GitHub Issue: apache#48894 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.