add avro + kafka + schema registry integration test#10929
add avro + kafka + schema registry integration test#10929suneet-s merged 9 commits intoapache:masterfrom
Conversation
clintropolis
changed the title
suneet-s
left a comment
suneet-s
left a comment
There was a problem hiding this comment.
Overall LGTM, I think some comments in the java code, explaining where the hard coded user names, ports, etc. come from would help. What do you think?
| bb.rewind(); | ||
| // When | ||
| GenericRecord record = new SchemaRegistryBasedAvroBytesDecoder(registry).parse(bb); | ||
| Assert.assertNull(record); |
There was a problem hiding this comment.
| Assert.assertNull(record); | |
| Assert.fail("ParseException should have been thrown"); |
There was a problem hiding this comment.
Or just delete this line? Similar comment in testParseCorruptedPartial
There was a problem hiding this comment.
deleted this and other similar lines that would never be reached
| public String getSchemaRegistryInternalHost() | ||
| { | ||
| return "schema-registry:8085"; | ||
| } |
There was a problem hiding this comment.
It would be good to add comments here pointing devs to the source of these definitions. I believe it's integration-tests/docker/docker-compose.schema-registry.yml
There was a problem hiding this comment.
I believe it's docker-compose.base.yml rather than integration-tests/docker/docker-compose.schema-registry.yml that would be relevant here.
There was a problem hiding this comment.
added javadoc to DockerConfigProvider indicating that the values should be kept in sync with the docker-compose files values.
| "basic.auth.credentials.source", "USER_INFO", | ||
| "basic.auth.user.info", "druid:diurd" |
There was a problem hiding this comment.
Is this meant to always be in sync with integration-tests/src/test/resources/stream/data/avro_schema_registry/parser/input_row_parser.json?
There was a problem hiding this comment.
once avro schema registry supports InputFormat this would be used for that as well, but yeah AvroEventSerializer and AvroSchemaRegistryEventSerializer are a bit more tied to the wikipedia schema used by those tests than some of the other event serializers
| # the 'high availability' test cluster with multiple coordinators and overlords | ||
| echo "-f ${DOCKERDIR}/docker-compose.high-availability.yml" | ||
| elif [ "$DRUID_INTEGRATION_TEST_GROUP" = "kafka-data-format" ] | ||
| then |
There was a problem hiding this comment.
Maybe add a comment above so it doesn't look like this was an accidental empty branch
There was a problem hiding this comment.
this is just matching the style of the other comment blocks in this script so going to leave like it is to be consistent
3 participants
Description
Builds on top of #10314 to add an integration test using a schema-registry docker container to test Apache Kafka + Apache Avro + Confluent Schema Registry secured with basic auth. This container only runs when the
kafka-data-formatintegration test group is run.I also adjusted the parse method to distinguish failures to get the Avro schema from the registry from actual parse exceptions, since it seems like failure to get the schema would cause no messages to ever be parsable.
This PR has: