feat(schema): Migrate HoodieRecord methods to use HoodieSchema instead of Avro.Schema #17772

KiteSoar · 2026-01-03T03:07:18Z

Describe the issue this Pull Request addresses

This PR addresses the migration of HoodieRecord API from using org.apache.avro.Schema to org.apache.hudi.common.schema.HoodieSchema for all record-related methods. This change reduces coupling with Avro-specific implementations.
Closes #17689

Summary and Changelog

Users can now use HoodieSchema consistently across all HoodieRecord methods. This provides a unified schema abstraction layer.

Impact

All HoodieRecord methods that previously accepted org.apache.avro.Schema now accept org.apache.hudi.common.schema.HoodieSchema

Risk Level

Low

Documentation Update

N/A

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

… Avro Schema.

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java

hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java

hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieAvroDataBlock.java

hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java

.../hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java

...ent/hudi-client-common/src/main/java/org/apache/hudi/execution/CopyOnWriteInsertHandler.java

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java

hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieHiveRecord.java

voonhous · 2026-01-03T16:14:53Z

Sorry, went through the changes again and found that there are more things that can be improved. I believe this should be all.

Let's wait for @the-other-tim-brown's review too after you've made the changes i suggested.

Thank you so much for the contribution, we really appreciate it!

.../hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java

hudi-hadoop-common/src/test/java/org/apache/hudi/metadata/TestHoodieTableMetadataUtil.java

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieHiveRecord.java

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java

…files.

hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java

...spark-client/src/test/java/org/apache/hudi/execution/TestBoundedInMemoryExecutorInSpark.java

...ient/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestBoundedInMemoryQueue.java

...ent/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java

voonhous

LGTM

hudi-bot · 2026-01-05T15:20:29Z

CI report:

5c8cf22 UNKNOWN
691a8ae UNKNOWN
57bec67 UNKNOWN
5887e38 UNKNOWN
b515c76 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

yihua · 2026-01-08T04:58:15Z

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

      return OrderingValues.getDefault();
    } else {
      return OrderingValues.create(orderingFields, field -> {
        if (recordSchema.getField(field) == null) {


HoodieSchema#getField never returns null. It returns Option.empty() if the field is not found. So all invocations of HoodieSchema#getField should be revisited and revised so that they have parity as before (i.e., use recordSchema.getField(field).isEmpty() or recordSchema.getField(field).isPresent() instead of null check).

Four places to check in HoodieFlinkRecord

doGetOrderingValue(HoodieSchema, Properties, String[]) if (recordSchema.getField(field) == null) { getOrderingValueAsJava(HoodieSchema, Properties, String[]) if (recordSchema.getField(field) == null) { getRecordKey(HoodieSchema, Option<BaseKeyGenerator>) ValidationUtils.checkArgument(recordSchema.getField(RECORD_KEY_METADATA_FIELD) != null, updateMetaField(HoodieSchema, int, String) boolean withOperation = recordSchema.getField(OPERATION_METADATA_FIELD) != null;

Also could you add unit tests to cover this method?

yihua

Thanks for your patience on the review

yihua · 2026-01-08T05:02:23Z

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

      return OrderingValues.getDefault();
    } else {
      return OrderingValues.create(orderingFields, field -> {
        if (recordSchema.getField(field) == null) {


Four places to check in HoodieFlinkRecord

doGetOrderingValue(HoodieSchema, Properties, String[]) if (recordSchema.getField(field) == null) { getOrderingValueAsJava(HoodieSchema, Properties, String[]) if (recordSchema.getField(field) == null) { getRecordKey(HoodieSchema, Option<BaseKeyGenerator>) ValidationUtils.checkArgument(recordSchema.getField(RECORD_KEY_METADATA_FIELD) != null, updateMetaField(HoodieSchema, int, String) boolean withOperation = recordSchema.getField(OPERATION_METADATA_FIELD) != null;

yihua · 2026-01-08T05:03:44Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java


  @Override
-  public Comparable<?> getOrderingValueAsJava(Schema recordSchema, Properties props, String[] orderingFields) {
+  public Comparable<?> getOrderingValueAsJava(HoodieSchema recordSchema, Properties props, String[] orderingFields) {


Same on if (recordSchema.getField(field) == null) { to avoid null check

yihua · 2026-01-08T05:08:01Z

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

+  public String getRecordKey(HoodieSchema recordSchema, String keyFieldName) {
    if (key == null) {
-      String recordKey = Objects.toString(RowDataAvroQueryContexts.fromAvroSchema(recordSchema).getFieldQueryContext(keyFieldName).getFieldGetter().getFieldOrNull(data));
+      String recordKey = Objects.toString(RowDataAvroQueryContexts.fromAvroSchema(recordSchema.toAvroSchema()).getFieldQueryContext(keyFieldName).getFieldGetter().getFieldOrNull(data));


Is RowDataAvroQueryContexts going to be migrated away from Avro schema separately?

The reason I ask is that there's still back and forth conversion to Avro Schema for RowDataAvroQueryContexts to use.

We'll take this on in #17739

yihua · 2026-01-08T05:09:23Z

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

      return OrderingValues.getDefault();
    } else {
      return OrderingValues.create(orderingFields, field -> {
        if (recordSchema.getField(field) == null) {


Also could you add unit tests to cover this method?

yihua · 2026-01-08T05:35:12Z

...ent/hudi-client-common/src/test/java/org/apache/hudi/testutils/HoodieWriteableTestTable.java

        int seqId = 1;
        for (HoodieRecord record : records) {
-          GenericRecord avroRecord = (GenericRecord) record.rewriteRecordWithNewSchema(schema, CollectionUtils.emptyProps(), schema).getData();
+          GenericRecord avroRecord = (GenericRecord) record.rewriteRecordWithNewSchema(HoodieSchema.fromAvroSchema(schema),


nit: The Avro schema can be converted beforehand once, instead of conversion per record.

yihua · 2026-01-08T05:35:45Z

...ent/hudi-client-common/src/test/java/org/apache/hudi/testutils/HoodieWriteableTestTable.java

        int seqId = 1;
        for (HoodieRecord record : records) {
-          GenericRecord avroRecord = (GenericRecord) record.toIndexedRecord(schema, CollectionUtils.emptyProps()).get().getData();
+          GenericRecord avroRecord = (GenericRecord) record.toIndexedRecord(HoodieSchema.fromAvroSchema(schema), CollectionUtils.emptyProps()).get().getData();


nit: similar

yihua · 2026-01-08T05:40:41Z

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java

+  public String getRecordKey(HoodieSchema recordSchema, String keyFieldName) {
    if (key == null) {
-      String recordKey = Objects.toString(RowDataAvroQueryContexts.fromAvroSchema(recordSchema).getFieldQueryContext(keyFieldName).getFieldGetter().getFieldOrNull(data));
+      String recordKey = Objects.toString(RowDataAvroQueryContexts.fromAvroSchema(recordSchema.toAvroSchema()).getFieldQueryContext(keyFieldName).getFieldGetter().getFieldOrNull(data));


The reason I ask is that there's still back and forth conversion to Avro Schema for RowDataAvroQueryContexts to use.

yihua · 2026-01-08T06:12:19Z

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java

    while (recordItr.hasNext()) {
      HoodieRecord record = recordItr.next();
-      String recordKey = record.getRecordKey(readerSchema, keyFieldName);
+      String recordKey = record.getRecordKey(HoodieSchema.fromAvroSchema(readerSchema), keyFieldName);


HoodieSchema should be generated outside the while loop

yihua · 2026-01-08T06:15:28Z

hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/TestHoodieHiveRecord.java

  void testConvertColumnValueForLogicalTypeWithNullValue() {
-    Schema dateSchema = Schema.create(Schema.Type.INT);
-    LogicalTypes.date().addToSchema(dateSchema);
+    HoodieSchema dateSchema = HoodieSchema.create(HoodieSchemaType.INT);


Should this be HoodieSchema.createDate();?

KiteSoar force-pushed the hoodieRecord-migration branch 2 times, most recently from 5c8cf22 to 691a8ae Compare January 3, 2026 03:34

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Jan 3, 2026

refactor: Migrate HoodieRecord methods to use HoodieSchema instead of…

8cc8fa2

… Avro Schema.

KiteSoar force-pushed the hoodieRecord-migration branch from 691a8ae to 8cc8fa2 Compare January 3, 2026 03:40

KiteSoar changed the title ~~refactor: Migrate HoodieRecord methods to use HoodieSchema instead of Avro Schema.~~ feat(scheme): Migrate HoodieRecord methods to use HoodieSchema instead of Avro Schema. Jan 3, 2026

KiteSoar changed the title ~~feat(scheme): Migrate HoodieRecord methods to use HoodieSchema instead of Avro Schema.~~ feat(schema): Migrate HoodieRecord methods to use HoodieSchema instead of Avro Schema. Jan 3, 2026

voonhous reviewed Jan 3, 2026

View reviewed changes

voonhous changed the title ~~feat(schema): Migrate HoodieRecord methods to use HoodieSchema instead of Avro Schema.~~ feat(schema): Migrate HoodieRecord methods to use HoodieSchema instead of Avro.Schema Jan 3, 2026

refactor(schema): Optimize schema usage.

b892713

KiteSoar force-pushed the hoodieRecord-migration branch from 88e0a16 to b892713 Compare January 3, 2026 08:39

voonhous reviewed Jan 3, 2026

View reviewed changes

the-other-tim-brown reviewed Jan 3, 2026

View reviewed changes

.../hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieMergeHelper.java Outdated Show resolved Hide resolved

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java Outdated Show resolved Hide resolved

the-other-tim-brown reviewed Jan 3, 2026

View reviewed changes

refactor(schema): Replace Avro Schema with HoodieSchema.

d40ef89

KiteSoar force-pushed the hoodieRecord-migration branch from 57bec67 to d40ef89 Compare January 4, 2026 15:36

the-other-tim-brown reviewed Jan 5, 2026

View reviewed changes

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/HoodieFlinkRecord.java Outdated Show resolved Hide resolved

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java Show resolved Hide resolved

refactor(schema): Fix Avro Schema to HoodieSchema conversion in test …

d59fa57

…files.

voonhous reviewed Jan 5, 2026

View reviewed changes

hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java Outdated Show resolved Hide resolved

refactor(schema): Fix Avro Schema to HoodieSchema in test files.

4608f20

KiteSoar force-pushed the hoodieRecord-migration branch from 5887e38 to 4608f20 Compare January 5, 2026 04:48

KiteSoar requested a review from the-other-tim-brown January 5, 2026 08:16

voonhous reviewed Jan 5, 2026

View reviewed changes

refactor(schema): Fix Avro Schema to HoodieSchema in test files.

b515c76

voonhous approved these changes Jan 5, 2026

View reviewed changes

the-other-tim-brown approved these changes Jan 5, 2026

View reviewed changes

the-other-tim-brown assigned vinothchandar Jan 7, 2026

the-other-tim-brown mentioned this pull request Jan 7, 2026

feat(schema): Remove direct usage of Avro schema in Flink-client path #17739

Open

3 tasks

yihua reviewed Jan 8, 2026

View reviewed changes

feat(schema): Migrate HoodieRecord methods to use HoodieSchema instead of Avro.Schema #17772

Are you sure you want to change the base?

feat(schema): Migrate HoodieRecord methods to use HoodieSchema instead of Avro.Schema #17772

Conversation

KiteSoar commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

voonhous commented Jan 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

voonhous left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Jan 5, 2026

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

KiteSoar commented Jan 3, 2026 •

edited

Loading