Skip to content

Protobuf Schema support AutoConsumeSchema and Presto table  #7642

@hnail

Description

@hnail

Is your enhancement request related to a problem? Please describe.

  1. Now , ProtobufSchema create and persistence dependency Java implementation class of GeneratedMessageV3 , create action code trace is ProtobufSchema.of(Class pojo) ->org.apache.avro.protobuf.ProtobufData.getSchema(Class c), but avro-protobuf lib i haven't find method to reflect class by Descriptor,so ProtobufSchema.loadReader() is don't support , and AutoConsumeSchema.generateSchema() haven't support PROTOBUF。
  2. I am working for Presto pulsar table , but pulsar current protobuf comsumer need Java implementation class of GeneratedMessageV3 , Presto is a Long-Running server, so top server then update class is unfriendly。
  3. Regist ProtobufSchema with Java implementation class of GeneratedMessageV3 is unfriendly for other languages instead of JAVA User : SimpleTypedProducerConsumerTest

Describe the solution you'd like

AutoConsumeSchema support Protobuf , Presto table work with Protobuf schema register

  1. ProtobufSchema regist use protoc descriptor_set_out file, persistence this single file content to Schema Registry,.desc conten is language-independent and readable .
    e.g. protoc --include_imports --descriptor_set_out=<filename>.desc <filename>.proto <filename>.proto <filename>.proto...
  2. consumer support Protobuf AutoConsumeSchema by DynamicMessage ,e.g.
BufferedInputStream dscFile = new BufferedInputStream(new FileInputStream(new File ("desc file path")).getPath())
DescriptorProtos.FileDescriptorSet fileDescriptorSet = DescriptorProtos.FileDescriptorSet.parseFrom(dscFile);
  • single_message example:

DynamicMessage.parseFrom(fileDescriptor.getMessageTypes().get(0), outputStream.toByteArray());

  • multiple_message example :
    DynamicMessage.parseFrom(fileDescriptor.findMessageTypeByName(${MessageName}), outputStream.toByteArray());
    MessageName persistence with SchemaInfo.properties when create。
  1. Then Presto ProtobufSchemaHandler can extract Field(etc. Value\List\Map ) from DynamicMessage

Additional context
If this proposal is approved,I glad to make a more detailed description or PR 。

tag : Client \Schema Registry with Protobuffer , Presto

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/enhancementThe enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions