RoaringBitmap support

[zhicwu]

Object	Read	Write
RoaringBitmap	Y	Y
ImmutableRoaringBitmap	as RoaringBitmap	Y
MutableRoaringBitmap	as RoaringBitmap	Y
Roaring64Bitmap	as Roaring64NavigableMap and only when cardinality <= 32	as Roaring64NavigableMap
Roaring64NavigableMap	only when cardinality <= 32	Y

Usage

// use JDBC interface - NOT recommended before 0.3.1
try (PreparedStatement statement = connection.prepareStatement("insert into my_bitmap_table values(..., ?, ...)")) {
    ...
    // RoaringBitmap bitmap = RoaringBitmap.bitmapOf(1,2,3,...);
    s.setObject(index++, ClickHouseBitmap.wrap(bitmap, ClickHouseDataType.UInt32));
    ...
    // the actual SQL in 0.3.0 will be something like below, which is also why batch insertion does not work...
    // insert into my_bitmap_table values(..., bitmapBuild([toUInt32(1),toUInt32(3),toUInt32(3),...]) ...)
    s.execute();
}

// use extended API - recommended in 0.3.0
try (ClickHouseStatement statement = connection.createStatement()) {
    statement.sendRowBinaryStream("insert into my_bitmap_table", new ClickHouseStreamCallback() {
        public void writeTo(ClickHouseRowBinaryStream stream) throws IOException {
            ...
            // RoaringBitmap bitmap = RoaringBitmap.bitmapOf(1,2,3,...);
            // In addition to RoaringBitmap, you can pass:
            // ImmutableRoaringBitmap, MutableRoaringBitmap and even Roaring64NavigableMap
            stream.writeBitmap(ClickHouseBitmap.wrap(bitmap, ClickHouseDataType.UInt32));
            ...
        }
    });
}

[github-actions]

Benchmark                           (client)  (statement)   Mode  Cnt       Score       Error  Units
Basic.insertOneRandomNumber  clickhouse-jdbc       normal  thrpt   20  533147.555 ± 33574.299  ops/s
Basic.insertOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20     256.789 ±    26.478  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc       normal  thrpt   20    1142.817 ±   149.810  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20    1179.416 ±   158.935  ops/s

[lzzuo2]

您好 请问下用哪种方式去从CK里面读取RoaringBitmap比较好?直接读的话 貌似会有内存溢出的问题

[github-actions]

Benchmark                           (client)  (statement)   Mode  Cnt       Score      Error  Units
Basic.insertOneRandomNumber  clickhouse-jdbc       normal  thrpt   20  560076.510 ± 8225.068  ops/s
Basic.insertOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20     238.036 ±   31.159  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc       normal  thrpt   20    1455.879 ±  224.413  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20    1330.789 ±  179.090  ops/s

[zhicwu]

您好 请问下用哪种方式去从CK里面读取RoaringBitmap比较好?直接读的话 貌似会有内存溢出的问题

目前使用什么方式读取导致内存溢出？当前版本还是建议用extended API，例如：

try (ClickHouseRowBinaryInputStream in = statement.executeQueryClickhouseRowBinaryStream("SELECT * FROM my_bitmap_table")) {
    ...
    ClickHouseBitmap bitmap = in.readBitmap(ClickHouseDataType.UInt32);
    RoaringBitmap rb = (RoaringBitmap)bitmap.unwrap();
    ...
}

[github-actions]

Benchmark                           (client)  (statement)   Mode  Cnt       Score       Error  Units
Basic.insertOneRandomNumber  clickhouse-jdbc       normal  thrpt   20  526151.979 ± 10659.838  ops/s
Basic.insertOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20     251.023 ±    27.312  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc       normal  thrpt   20    1181.098 ±   193.725  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20    1165.106 ±   148.765  ops/s

[github-actions]

Benchmark                           (client)  (statement)   Mode  Cnt       Score       Error  Units
Basic.insertOneRandomNumber  clickhouse-jdbc       normal  thrpt   20  521913.493 ± 13423.092  ops/s
Basic.insertOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20     245.400 ±    25.875  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc       normal  thrpt   20    1324.137 ±   174.162  ops/s
Basic.selectOneRandomNumber  clickhouse-jdbc     prepared  thrpt   20    1186.195 ±   221.061  ops/s

[LeeMouRen]

It's so convenient

[wntp]

用上面的方法单条提交没有问题，如果批量提交的话，就报 void DB::ParalleParsingBlockInputStream::onBackgroundException(): Code 33, e.displayText() = DB::Exception Cannot read all data.Bytes expected: 105. Stack trace....
完全参考https://github.com/ClickHouse/clickhouse-jdbc/issues/532的写法，只是excute 换成了batchexcute。

[sloan-zhang]

This method will generate a large size sql script, isn't it written directly insert into a bitmap object?
If so, how should I write a bitmap object with a large cardinality...

[zhicwu]

This method will generate a large size sql script, isn't it written directly insert into a bitmap object? If so, how should I write a bitmap object with a large cardinality...

I guess you're using JDBC API which wii generate a large SQL as you said - you should use extended API. Behind the scene, in most cases, it's just about how to use input function and/or external/temp table.

Starting 0.3.2, the situation has changed: 1) extended API is replaced by new Java client; and 2) you can load bitmap or use it as a query parameter using standard JDBC API. Please take a look at examples at here.