Skip to content

Add TravisCI job that builds and tests on ARM64 CPU architecture#10562

Merged
nishantmonu51 merged 1 commit intoapache:masterfrom
martin-g:feature/build-on-arm64
Nov 16, 2020
Merged

Add TravisCI job that builds and tests on ARM64 CPU architecture#10562
nishantmonu51 merged 1 commit intoapache:masterfrom
martin-g:feature/build-on-arm64

Conversation

@martin-g
Copy link
Copy Markdown
Member

@martin-g martin-g commented Nov 6, 2020

More and more software deployment is being done on ARM64 CPU architecture.
It would be good if Apache Druid is being regularly tested on ARM64.

This PR adds an additional TravisCI job that runs the build and tests on ARM64 node.
Apache Maven is not pre-installed on ARM64 so it has to be added as an addon.

Copy link
Copy Markdown
Member

@nishantmonu51 nishantmonu51 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@nishantmonu51 nishantmonu51 merged commit 9e69786 into apache:master Nov 16, 2020
@nishantmonu51
Copy link
Copy Markdown
Member

Thanks for the contribution @martin-g

@martin-g
Copy link
Copy Markdown
Member Author

Thanks for merging, @nishantmonu51 !

@himanshug
Copy link
Copy Markdown
Contributor

@martin-g curious about what makes ARM64 CPU architecture special for a pure java project ?

@martin-g martin-g deleted the feature/build-on-arm64 branch November 19, 2020 09:09
@martin-g
Copy link
Copy Markdown
Member Author

@himanshug You are right that JVM as a multi-platform should just run on any of the supported platforms.
The testing on ARM64 architecture adds some confidence that the software does actually run on that platform.
Druid itself could be pure Java project but it may depend on a third party library that uses native libraries. For example Apache Pig failed to run on aarch64 because it was using old version of Snappy library (compression) that didn't provide aarch64 native library (.so).
Another possible problem could arise if your Java project uses some JDK APIs which hit some corner case that is not covered for aarch64. The JDK has a lot of tests but no project has 100% coverage! As any other maintained project the JDK project constantly evolves and regressions happen.

@himanshug
Copy link
Copy Markdown
Contributor

yeah, that is technically true and more testing is always better :)

however, I think trade-off here is the build time which is probably ok considering the new build runs in parallel to others hopefully , however travis might put some limits on build resources used overall/concurrently and that would actually increase it.

@martin-g
Copy link
Copy Markdown
Member Author

I've asked Apache Infra in Slack team for a comment whether the new pricing model of TravisCI affects us anyhow but so far I have no answer.
In any case if this proves to be a problem the extra job could be removed!

abhishekagarwal87 pushed a commit to abhishekagarwal87/druid that referenced this pull request Dec 14, 2020
* scaffolding

* readme

* adjust

* more better, janky heap metadata store, primitive job queue that can submit to overlord, it works - sort of

* test scaffolding

* move InputFormat into IngestSchema

* imply-5135 Create & list ingest tables

* Addressed PR comments

* Removed bean IngestTable

* job processing + sql metadata job table (#78)

* Add indexed-table-loader (#65)

* Add indexed-table-loader

* Fix checkstyle

* Fix intelliJ inspections

* Fix analyze dependencies

* fix license check job

* Add imply-druid-security (#66)

* Add imply-druid-security

* fix checkstyle

* Fix analyze dependencies

* Fix license check job

* Update license header for all imply extensions

* fix intelliJ inspections

* code review

* modify access to protected SQLMetadataConnector methods to allow extensions to create SQL metadata tables using implementation specific constructs (payload type, serial type, etc) (apache#10573)

* Correct getRandomBalancerSegmentHolderTest (apache#10569)

* Add missing docs for timeout exceptions (apache#10554)

* Add missing docs for timeout exceptions

* Add info on auth failures

* Fix ingestion failure of pretty-formatted JSON message (apache#10383)

* support multi-line text

* add test cases

* split json text into lines case by case

* improve exception handle

* fix CI

* use IntermediateRowParsingReader as base of JsonReader

* update doc

* ignore the non-immutable field in test case

* add more test cases

* mark `lineSplittable` as final

* fix testcases

* fix doc

* add a test case for SqlReader

* return all raw columns when exception occurs

* fix CI

* fix test cases

* resolve review comments

* handle ParseException returned by index.add

* apply Iterables.getOnlyElement

* fix CI

* fix test cases

* improve code in more graceful way

* fix test cases

* fix test cases

* add a test case to check multiple json string in one text block

* fix inspection check

* Add TravisCI job that builds and tests on ARM64 CPU architecture (apache#10562)

* Ensure Krb auth before killing YARN apps in graceful shutdown (apache#9785)

* job processing + sql metadata

* Web console: fix data loader schema table column ordering bug and other polish (apache#10588)

* remove unused fields

* keep tables live

* advanced

* fix schema view

* better indication

* tests pass

* Show more instead of show advanced

* fix tests

* extract dynamic configs

* update snapshots

* fix issues

* update snapshot

* reword without >

* some javadoc

* modify druid.historical.cache.maxEntrySize property in Unified format (apache#10590)

Co-authored-by: yuezhang <yuezhang@freewheel.tv>

* Fix license header for imply extensions (#76)

* Fix license header for imply extensions

* arm64 packaging should use jdk8

* maybe this time

* jobs and states and status and whatever

* use indexing client and coordinator client instead of leader client

* always running

* simplify

* fix readme

* Add zero period support to TIMESTAMPADD (apache#10550)

* Allow zero period for TIMESTAMPADD

* update test cases

* add empty zone test case

* add unit test cases for TimestampShiftMacro

* add -Pimply-saas distribution profile, table exists check

* update readme

Co-authored-by: Suneet Saldanha <suneet.saldanha@imply.io>
Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Atul Mohan <atulmohan.mec@gmail.com>
Co-authored-by: frank chen <frank.chen021@outlook.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
Co-authored-by: zhangyue19921010 <69956021+zhangyue19921010@users.noreply.github.com>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>

* fix style and headers

* fix fails

* fix auth

Co-authored-by: Agustin Gonzalez <agustin.gonzalez@imply.io>
Co-authored-by: Suneet Saldanha <suneet.saldanha@imply.io>
Co-authored-by: Lucas Capistrant <capistrant@users.noreply.github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Atul Mohan <atulmohan.mec@gmail.com>
Co-authored-by: frank chen <frank.chen021@outlook.com>
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>
Co-authored-by: zhangyue19921010 <69956021+zhangyue19921010@users.noreply.github.com>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
@jihoonson jihoonson added this to the 0.21.0 milestone Jan 4, 2021
JulianJaffePinterest pushed a commit to JulianJaffePinterest/druid that referenced this pull request Jan 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants