Skip to content

Support Avro GenericRecord as a valid format for StorageWrite API on BigQueryIO #24274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Abacn merged 11 commits into from
Feb 17, 2023
Merged

Support Avro GenericRecord as a valid format for StorageWrite API on BigQueryIO #24274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
685fc06
initial impl
prodriguezdefino Nov 18, 2022
4be8f9d
force unit tests
prodriguezdefino Nov 21, 2022
7772e82
Merge branch 'master' into avro_support_bq_storagewrite
prodriguezdefino Dec 1, 2022
4dcf9a4
addressing comments
prodriguezdefino Dec 1, 2022
ea52993
spotless and checkstyle checks fixes
prodriguezdefino Dec 1, 2022
962cf51
spotless and checkstyle checks fixes
prodriguezdefino Dec 1, 2022
e3ff8cb
addressing more comments
prodriguezdefino Dec 6, 2022
60db300
Merge branch 'apache:master' into avro_support_bq_storagewrite
prodriguezdefino Jan 18, 2023
5f571e5
Merge branch 'master' into avro_support_bq_storagewrite
prodriguezdefino Jan 31, 2023
33c7974
merge from master, fix tests and validations
prodriguezdefino Jan 31, 2023
fd50138
copying the changes in AvroUtils located in core module to the avro e…
prodriguezdefino Feb 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,14 @@ public class AvroUtils {
GenericData.get().addLogicalTypeConversion(new JodaTimestampConversion());
}

// Unwrap an AVRO schema into the base type an whether it is nullable.
static class TypeWithNullability {
public final org.apache.avro.Schema type;
public final boolean nullable;
/** Unwrap an AVRO schema into the base type an whether it is nullable. */
public static class TypeWithNullability {
final org.apache.avro.Schema type;
final boolean nullable;

public static TypeWithNullability create(org.apache.avro.Schema avroSchema) {
return new TypeWithNullability(avroSchema);
}

TypeWithNullability(org.apache.avro.Schema avroSchema) {
if (avroSchema.getType() == org.apache.avro.Schema.Type.UNION) {
Expand Down Expand Up @@ -182,6 +186,14 @@ static class TypeWithNullability {
nullable = false;
}
}

public Boolean isNullable() {
return nullable;
}

public org.apache.avro.Schema getType() {
return type;
}
}

/** Wrapper for fixed byte fields. */
Expand Down
14 changes: 13 additions & 1 deletion ...sions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,10 +152,14 @@ public class AvroUtils {
}

// Unwrap an AVRO schema into the base type an whether it is nullable.
static class TypeWithNullability {
public static class TypeWithNullability {
public final org.apache.avro.Schema type;
public final boolean nullable;

public static TypeWithNullability create(org.apache.avro.Schema avroSchema) {
return new TypeWithNullability(avroSchema);
}

TypeWithNullability(org.apache.avro.Schema avroSchema) {
if (avroSchema.getType() == Type.UNION) {
List<org.apache.avro.Schema> types = avroSchema.getTypes();
Expand Down Expand Up @@ -184,6 +188,14 @@ static class TypeWithNullability {
nullable = false;
}
}

public Boolean isNullable() {
return nullable;
}

public org.apache.avro.Schema getType() {
return type;
}
}

/** Wrapper for fixed byte fields. */
Expand Down
Loading