Skip to content

build(bindings/java): Allow building on linux-aarch_64#3527

Merged
Xuanwo merged 2 commits intoapache:mainfrom
questdb:build_py_compat
Nov 8, 2023
Merged

build(bindings/java): Allow building on linux-aarch_64#3527
Xuanwo merged 2 commits intoapache:mainfrom
questdb:build_py_compat

Conversation

@amunra
Copy link
Copy Markdown
Contributor

@amunra amunra commented Nov 8, 2023

Allow building the Java bindings on Linux ARM64.

Additionally allows the script to run with older Python versions that require strings in subprocess calls.

@amunra amunra requested a review from tisonkun as a code owner November 8, 2023 09:48
@github-actions github-actions Bot added the releases-note/build The PR modifies build related content or has a title that begins with "build" label Nov 8, 2023
@Xuanwo Xuanwo requested a review from G-XD November 8, 2023 09:50
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks a lot!

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Nov 8, 2023

Unfortunately, we lack the CI resources to accommodate this change, but it's likely to function.

@Xuanwo Xuanwo merged commit 9546db7 into apache:main Nov 8, 2023
@amunra
Copy link
Copy Markdown
Contributor Author

amunra commented Nov 8, 2023

This obviously brings in the whole question of actually cutting the binaries for this platform.
For now, I've been adding incubator-opendal as a git submodule and pulling it by adding it as a module in pom.xml.

Our CI (for our own code which build opendal as part of it) ran this so I have some degree of confidence it's correct, though I haven't run the opendal tests: I've just used the lib as a dependency.

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Nov 8, 2023

This obviously brings in the whole question of actually cutting the binaries for this platform. For now, I've been adding incubator-opendal as a git submodule and pulling it by adding it as a module in pom.xml.

https://github.com/apache/incubator-opendal/blob/9546db7d9736b4fcfd42713731942c443fb54dc4/.github/workflows/release_java.yml#L67-L75

We are performing cross compile for macos aarch64 support, maybe we can do it for linux aarch64 too.

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Nov 8, 2023

Our CI (for our own code which build opendal as part of it) ran this so I have some degree of confidence it's correct, though I haven't run the opendal tests: I've just used the lib as a dependency.

That's impressive! I wasn't aware that opendal-java had a user. Is your project open source, perhaps questdb? May I include you as a listed user in the README.md file?

@amunra
Copy link
Copy Markdown
Contributor Author

amunra commented Nov 8, 2023

Our CI (for our own code which build opendal as part of it) ran this so I have some degree of confidence it's correct, though I haven't run the opendal tests: I've just used the lib as a dependency.

That's impressive! I wasn't aware that opendal-java had a user. Is your project open source, perhaps questdb? May I include you as a listed user in the README.md file?

How we use OpenDAL:

  • We have a closed-source edition of QuestDB called "Enterprise".
  • This release has replication support which works by storing WALs in an object store.
  • Our original implementation supported S3 only, last week we swapped out our implementation from S3 to opendal in our Rust code (our replication feature is written in Rust because we want to avoid garbage collection) in the interest of supporting Azure Blob.
  • I now have an incomplete PR that swaps out our Java tests from using the AWS Java SDK to using opendal-java. This is where the set of bugfixes are coming from.
  • Our DB builds on a number of platforms, including Linux/ARM64 which is where this patch comes in.
  • It would be awesome to get Linux/ARM64 binaries for opendal-java.
  • We intend to support additional object stores over time (webhdfs is probably next up), but this will be on a by-need basis dependent on what our customers are after.

Thanks for building this lib! It's coming together nicely!

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Nov 8, 2023

Bravo!

@tisonkun
Copy link
Copy Markdown
Member

tisonkun commented Nov 8, 2023

maybe we can do it for linux aarch64 too.

The last time I try it, cargo build returns error. So at least some code doesn't support cross compile to aarch64 on x86_64 on linux.

@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented Nov 8, 2023

The last time I try it, cargo build returns error. So at least some code doesn't support cross compile to aarch64 on x86_64.

Got it, let's track this in a new issue instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/build The PR modifies build related content or has a title that begins with "build"

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants