This issue is a sibling of #12809.
Progress
TL;DR:
Pinot uses LArray, which is unsafe, platform dependent and deprecated. It also doesn't work with Java >= 15, so we had to create our own buffer library to be able to run Pinot with Java >= 15.
This issue proposes to remove LArray dependency and always use our own library instead
Context
As we may know Java buffers length has been limited to 2GBs. In Apache Pinot we decided to skip that limit by using LArray, a third party buffer library that uses JNI to create larger buffers. That library is not maintained anymore (see xerial/larray#75 (comment) or xerial/larray#80
This library has several problems, including:
- It may produce segmentation faults or buffer overflows (there is no index check when using it).
- It doesn't work in Mac using arm architecture.
- It doesn't support Java >= 15.
In order to solve these problems, we introduced a new buffer implementation called unsafe. The name comes from the fact that uses sun.misc.Unsafe, but AFAIK it is safer than LArray. It works and that is what we use by default if at runtime we detect we are using Java >= 15.
We should remove LArray and just use our library instead. In the future a version built on top of the official Java Foreign Memory API should be used, but it can only be used when the runtime is Java 22 (or maybe 21 with enabled preview). See #12809 to know more about that.
This issue is a sibling of #12809.
Progress
TL;DR:
Pinot uses LArray, which is unsafe, platform dependent and deprecated. It also doesn't work with Java >= 15, so we had to create our own buffer library to be able to run Pinot with Java >= 15.
This issue proposes to remove LArray dependency and always use our own library instead
Context
As we may know Java buffers length has been limited to 2GBs. In Apache Pinot we decided to skip that limit by using LArray, a third party buffer library that uses JNI to create larger buffers. That library is not maintained anymore (see xerial/larray#75 (comment) or xerial/larray#80
This library has several problems, including:
In order to solve these problems, we introduced a new buffer implementation called unsafe. The name comes from the fact that uses sun.misc.Unsafe, but AFAIK it is safer than LArray. It works and that is what we use by default if at runtime we detect we are using Java >= 15.
We should remove LArray and just use our library instead. In the future a version built on top of the official Java Foreign Memory API should be used, but it can only be used when the runtime is Java 22 (or maybe 21 with enabled preview). See #12809 to know more about that.