Skip to content

ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integration test #9889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alamb wants to merge 3 commits into from
Closed

ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integration test #9889

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a1a5460
Print out the size of docker container
alamb Apr 3, 2021
3a6564a
add comments to trigger integration test
alamb Apr 4, 2021
6317315
ARROW-12204: [Rust][CI] Reduce size of Rust build artifacts in integr…
alamb Apr 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,3 +415,46 @@ jobs:
export RUSTFLAGS="-C debuginfo=0"
cd rust/arrow
cargo build --target wasm32-unknown-unknown

# test the projects can build without default features
default-build:
name: AMD64 Debian 10 Rust ${{ matrix.rust }} test no defaults
runs-on: ubuntu-latest
strategy:
matrix:
arch: [amd64]
rust: [nightly-2020-11-24]
Copy link
Member

@jorgecarleitao jorgecarleitao Apr 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we change this to stable(or have both), since this is the target version?

Copy link
Contributor Author

@alamb alamb Apr 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that is a good idea. I will make the change

Copy link
Contributor Author

@alamb alamb Apr 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in #9904

container:
image: ${{ matrix.arch }}/rust
env:
# Disable full debug symbol generation to speed up CI build
# "1" means line tables only, which is useful for panic tracebacks.
RUSTFLAGS: "-C debuginfo=1"
ARROW_TEST_DATA: /__w/arrow/arrow/testing/data
PARQUET_TEST_DATA: /__w/arrow/arrow/cpp/submodules/parquet-testing/data
steps:
- uses: actions/checkout@v2
with:
submodules: true
- name: Cache Cargo
uses: actions/cache@v2
with:
path: /github/home/.cargo
# this key equals the ones on `linux-build-lib` for re-use
key: cargo-cache-
- name: Cache Rust dependencies
uses: actions/cache@v2
with:
path: /github/home/target
key: ${{ runner.os }}-${{ matrix.arch }}-target-wasm32-cache-${{ matrix.rust }}
- name: Setup Rust toolchain
run: |
rustup toolchain install ${{ matrix.rust }}
rustup override set ${{ matrix.rust }}
rustup component add rustfmt
- name: Build arrow crate
run: |
export CARGO_HOME="/github/home/.cargo"
export CARGO_TARGET_DIR="/github/home/target"
cd rust/arrow
cargo check --all-targets --no-default-features
14 changes: 8 additions & 6 deletions ci/scripts/rust_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ set -ex

source_dir=${1}/rust

# This file is used to build the rust binaries needed for the
Copy link
Contributor Author

@alamb alamb Apr 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing a full rust build as part of the integration test appears to be unecessary

# archery integration tests. Testing of the rust implementation
# in normal CI is handled by github workflows

# Disable full debug symbol generation to speed up CI build / reduce memory required
export RUSTFLAGS="-C debuginfo=1"

Expand All @@ -32,12 +36,10 @@ rustup show

pushd ${source_dir}

# build entire project
cargo build --all-targets
# build only the integration testing binaries
cargo build -p arrow-integration-testing

# make sure we can build Arrow sub-crate without default features
pushd arrow
cargo check --all-targets --no-default-features
popd
# Remove incremental build artifacts to save space
rm -rf target/debug/deps/ target/debug/build/

popd
3 changes: 3 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1179,6 +1179,9 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/go_build.sh /arrow &&
/arrow/ci/scripts/java_build.sh /arrow /build &&
df -h &&
du -hx --threshold=100M / &&
du -hx --threshold=500M /arrow/ &&
/arrow/ci/scripts/js_build.sh /arrow /build &&
/arrow/ci/scripts/integration_arrow.sh /arrow /build"]

Expand Down
2 changes: 1 addition & 1 deletion rust/datafusion/src/execution/dataframe_impl.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

//! Implementation of DataFrame API
//! Implementation of DataFrame API.

use std::sync::{Arc, Mutex};

Expand Down