-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Closed
Labels
type/enhancementThe enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messagesThe enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Description
Is your enhancement request related to a problem? Please describe.
- Now ,
ProtobufSchemacreate and persistence dependency Java implementation class of GeneratedMessageV3 , create action code trace is ProtobufSchema.of(Class pojo) ->org.apache.avro.protobuf.ProtobufData.getSchema(Class c), but avro-protobuf lib i haven't find method to reflect class by Descriptor,so ProtobufSchema.loadReader() is don't support , and AutoConsumeSchema.generateSchema() haven't support PROTOBUF。 - I am working for Presto pulsar table , but pulsar current protobuf comsumer need Java implementation class of GeneratedMessageV3 , Presto is a Long-Running server, so top server then update class is unfriendly。
- Regist ProtobufSchema with Java implementation class of GeneratedMessageV3 is unfriendly for other languages instead of JAVA User :
SimpleTypedProducerConsumerTest
Describe the solution you'd like
AutoConsumeSchema support Protobuf , Presto table work with Protobuf schema register
- ProtobufSchema regist use protoc descriptor_set_out file, persistence this single file content to Schema Registry,.desc conten is language-independent and readable .
e.g.protoc --include_imports --descriptor_set_out=<filename>.desc <filename>.proto <filename>.proto <filename>.proto... - consumer support Protobuf AutoConsumeSchema by DynamicMessage ,e.g.
BufferedInputStream dscFile = new BufferedInputStream(new FileInputStream(new File ("desc file path")).getPath())
DescriptorProtos.FileDescriptorSet fileDescriptorSet = DescriptorProtos.FileDescriptorSet.parseFrom(dscFile);
- single_message example:
DynamicMessage.parseFrom(fileDescriptor.getMessageTypes().get(0), outputStream.toByteArray());
- multiple_message example :
DynamicMessage.parseFrom(fileDescriptor.findMessageTypeByName(${MessageName}), outputStream.toByteArray());
MessageName persistence with SchemaInfo.properties when create。
- Then Presto ProtobufSchemaHandler can extract Field(etc. Value\List\Map ) from DynamicMessage
Additional context
If this proposal is approved,I glad to make a more detailed description or PR 。
tag : Client \Schema Registry with Protobuffer , Presto
Metadata
Metadata
Assignees
Labels
type/enhancementThe enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messagesThe enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages