Add Primitive time/timestamp_nanos/timestampntz_nanos/uuid test file by klion26 · Pull Request #92 · apache/parquet-testing

klion26 · 2025-08-12T12:09:02Z

This wants to add binary artifacts for
primitive_time.[metadata/value], primitive_timestamp_nanos.[metadata/value], primitive_timestampntz_nanos.[metadata/value] and primitive_uuid.[metadata/data]. They were generated by the modified iceberg code, as currently Spark does not support these primitives

modified code to generate the binary artifacts

  private String writeVariantFile(int rowId, Variant variant) throws IOException {
    String variantFile = String.format("case-%03d_row-%d.variant.bin", caseNumber, rowId);

    try (OutputStream out =
            IO.newOutputFile(CASE_LOCATION + "/" + caseNumber + ".metadata").createOrOverwrite()) {
      ByteBuffer bufferMeta =
              ByteBuffer.allocate(variant.metadata().sizeInBytes())
                      .order(ByteOrder.LITTLE_ENDIAN);
      variant.metadata().writeTo(bufferMeta, 0);
      out.write(bufferMeta.array());
    }
    try (OutputStream out =
                 IO.newOutputFile(CASE_LOCATION + "/" + caseNumber + ".value").createOrOverwrite()) {
      ByteBuffer bufferValue =
              ByteBuffer.allocate(variant.value().sizeInBytes())
                      .order(ByteOrder.LITTLE_ENDIAN);
      variant.value().writeTo(bufferValue, 0);
      out.write(bufferValue.array());
    }
    ByteBuffer buffer = ParquetVariantUtil.toByteBuffer(variant.metadata(), variant.value());
    try (OutputStream out =
        IO.newOutputFile(CASE_LOCATION + "/" + variantFile).createOrOverwrite()) {
      out.write(buffer.array());
    }

    return variantFile;
  }

binary artifacts content

~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_time.metadata                                                                                        1 ↵ qiucongxian@bogon
00000000: 0100 00                                  ...
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_time.value                                                                                               qiucongxian@bogon
00000000: 44c0 f229 880a 0000 00                   D..).....
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_uuid.metadata                                                                                             qiucongxian@bogon
00000000: 0100 00                                  ...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_uuid.value                                                                                                qiucongxian@bogon
00000000: 50f2 4f9b 6481 fa49 d1b7 4e8c 09a6 e31c  P.O.d..I..N.....
00000010: 56                                       V
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestamp_nanos.metadata                                                                                  qiucongxian@bogon
00000000: 0100 00                                  ...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestamp_nanos.value                                                                                     qiucongxian@bogon
00000000: 4815 413a 6cb7 af05 18                   H.A:l....
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestampntz_nanos.metadata                                                                               qiucongxian@bogon
00000000: 0100 00                                  ...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestampntz_nanos.value                                                                                  qiucongxian@bogon
00000000: 4c15 413a 6cb7 af05 18                   L.A:l....

klion26 · 2025-08-13T01:00:49Z

This wants to add the binary artifacts for Variant::TimeNTZ, so that we can use them to do cross-language testing

cc @mapleFU @alamb and @aihuaxu

klion26 · 2025-08-13T01:01:43Z

-    "short_string": "Less than 64 bytes (\u2764\ufe0f with utf8)"
-}
+    "short_string": "Less than 64 bytes (\u2764\ufe0f with utf8)",
+    "primitive_time": "12:33:54:123456"


The value copied from iceberg test

mapleFU · 2025-08-13T01:17:23Z

@@ -78,6 +78,7 @@
 -- https://github.com/apache/parquet-testing/issues/79
 -- is not clear how to create the following types using Spark SQL
 -- TODO TimeNTZ                    (Type ID 17)


Remove this line?

ok, will do it

mapleFU · 2025-08-13T01:17:36Z

 -- TODO 'timestamp with timezone (NANOS)'  (Type ID 18)
 -- TODO 'timestamp with time zone (NANOS)' (Type ID 19)
 -- TODO 'UUID'                     (Type ID 20)


Would you mind generate other types?

Ok, will update the pr

klion26 · 2025-08-13T03:13:53Z

@mapleFU I've updated the pr, please have another look when you're free, thanks.

~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_uuid.metadata                                                                                             qiucongxian@bogon
00000000: 0100 00                                  ...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_uuid.value                                                                                                qiucongxian@bogon
00000000: 50f2 4f9b 6481 fa49 d1b7 4e8c 09a6 e31c  P.O.d..I..N.....
00000010: 56                                       V
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestamp_nanos.metadata                                                                                  qiucongxian@bogon
00000000: 0100 00                                  ...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestamp_nanos.value                                                                                     qiucongxian@bogon
00000000: 4815 413a 6cb7 af05 18                   H.A:l....
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestampntz_nanos.metadata                                                                               qiucongxian@bogon
00000000: 0100 00                                  ...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/arrow-rs/parquet-testing (primitive_time) » xxd variant/primitive_timestampntz_nanos.value                                                                                  qiucongxian@bogon
00000000: 4c15 413a 6cb7 af05 18                   L.A:l....

mapleFU · 2025-08-13T04:08:38Z

I would checking these files after work about 8PM in UTC-8

alamb · 2025-08-13T12:50:17Z

Thanks @mapleFU and @klion26

mapleFU · 2025-08-13T12:52:32Z

-- TODO 'timestamp with timezone (NANOS)'  (Type ID 18)
-- TODO 'timestamp with time zone (NANOS)' (Type ID 19)
-- TODO 'UUID'                     (Type ID 20)
+-- binary artifacts of 'TimeNTZ'/'timestamp with timezone (NANOS)'/'timestamp with time zone (NANOS)'/'UUID' was generated by the iceberg test code, please ref to https://github.com/apache/parquet-testing/pull/92 for more detail


timestamp with timezone is duplicated?

Ah, there is one for without, fixed it.

mapleFU · 2025-08-13T13:28:48Z

-}
+    "primitive_timestamp_nanos": "2024-11-07T12:33:54.123456789+00:00",
+    "primitive_timestampntz_nanos": "2024-11-07T12:33:54.123456789",
+    "short_string": "Less than 64 bytes (\u2764\ufe0f with utf8)",


the ordering here is a bit weird, can we order this by alphabet?

Thanks for pointing this out, fixed it.

mapleFU · 2025-08-13T13:41:11Z

Other verified and looks ok to me

mapleFU · 2025-08-14T01:10:54Z

Before merging, can you update the readme with latest info?

mapleFU · 2025-08-14T05:22:27Z

(Sorry, I mean pr description, my bad...)

klion26 · 2025-08-14T05:22:33Z

@mapleFU I've updated the readme, please take another look when you're free, thanks.

klion26 · 2025-08-14T05:28:46Z

@mapleFU, thanks for the reply. I've updated the PR description. Do I need to revert the latest commit?

klion26 · 2025-08-14T06:30:39Z

@mapleFU Thanks for the review and merging!

add primitive_time

d2c8f8c

klion26 mentioned this pull request Aug 12, 2025

[Variant] Add Variant::Time primitive and cast logic apache/arrow-rs#8114

Merged

update data_dictionary and regen script

04db763

klion26 commented Aug 13, 2025

View reviewed changes

mapleFU reviewed Aug 13, 2025

View reviewed changes

add timestamp_nanos(tz&ntz) and uuid binary artifacts

d33d507

klion26 requested a review from mapleFU August 13, 2025 03:13

klion26 changed the title ~~Add Primitive time test file~~ Add Primitive time/timestamp_nanos/timestampntz_nanos/uudi test file Aug 13, 2025

mapleFU changed the title ~~Add Primitive time/timestamp_nanos/timestampntz_nanos/uudi test file~~ Add Primitive time/timestamp_nanos/timestampntz_nanos/uuid test file Aug 13, 2025

mapleFU reviewed Aug 13, 2025

View reviewed changes

fix typo

0123a86

mapleFU reviewed Aug 13, 2025

View reviewed changes

update the primitive data order in data_dictionary.json

9867649

mapleFU approved these changes Aug 13, 2025

View reviewed changes

update readme

c3bfa1a

mapleFU approved these changes Aug 14, 2025

View reviewed changes

mapleFU merged commit 5cbfc43 into apache:master Aug 14, 2025

klion26 deleted the primitive_time branch August 14, 2025 06:30

klion26 mentioned this pull request Aug 14, 2025

Add example Variant values for primitive types TimeNTZ (Type ID 17) 'timestamp with timezone' (Type ID 18+19) and UUID (Type ID 20) #79

Closed

Conversation

klion26 commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klion26 commented Aug 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klion26 commented Aug 13, 2025

Uh oh!

mapleFU commented Aug 13, 2025

Uh oh!

alamb commented Aug 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mapleFU commented Aug 13, 2025

Uh oh!

mapleFU commented Aug 14, 2025

Uh oh!

mapleFU commented Aug 14, 2025

Uh oh!

klion26 commented Aug 14, 2025

Uh oh!

klion26 commented Aug 14, 2025

Uh oh!

klion26 commented Aug 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

klion26 commented Aug 12, 2025 •

edited

Loading