topicctl load metadata from MSK cluster failing, suddenly (works on other cluster) #62
Description
For some reasons our dev kafka cluster is not accessible from topicctl anymore, I can NC in from my machine:
nc -vz b-1.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com 9098
Connection to b-1.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com port 9098 [tcp/*] succeeded!
I can run kafka-topic.sh:
aws-vault exec dev -- /Users/david/Downloads/kafka_2.13-3.1.0/bin/kafka-topics.sh \
--bootstrap-server=b-1.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com:9098 \
--command-config client-config.properties \
--list
__amazon_msk_canary
__consumer_offsets
# ... complete list of topics
However, running topicctl fails on trying to read the metadata, we use IAM and the config below:
"meta":
"description": "config for connecting topicctl to dev with IAM"
"environment": "dev"
"name": "thecompany-eventbus"
"region": "us-east-1"
"spec":
"bootstrapAddrs":
- "b-1.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com:9098"
- "b-2.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com:9098"
- "b-3.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com:9098"
"sasl":
"enabled": true
"mechanism": "AWS-MSK-IAM"
"tls":
"enabled": true
The debug output does not give much info, more than potentially it being a timeout of some sort:
[2022-04-13 13:52:10] DEBUG Connecting to cluster on address b-1.platform-us-east-1-de.xxxxxx.xxx.kafka.us-east-1.amazonaws.com:9098 with TLS enabled=true, SASL enabled=true
[2022-04-13 13:52:10] DEBUG Getting supported API versions
[2022-04-13 13:52:18] DEBUG Supported API versions: &{Error:<nil> ApiKeys:[{ApiKey:0 ApiName:Produce MinVersion:0 MaxVersion:8} {ApiKey:1 ApiName:Fetch MinVersion:0 MaxVersion:11} {ApiKey:2 ApiName:ListOffsets MinVersion:0 MaxVersion:5} {ApiKey:3 ApiName:Metadata MinVersion:0 MaxVersion:9} {ApiKey:4 ApiName:LeaderAndIsr MinVersion:0 MaxVersion:4} {ApiKey:5 ApiName:StopReplica MinVersion:0 MaxVersion:3} {ApiKey:6 ApiName:UpdateMetadata MinVersion:0 MaxVersion:6} {ApiKey:7 ApiName:ControlledShutdown MinVersion:0 MaxVersion:3} {ApiKey:8 ApiName:OffsetCommit MinVersion:0 MaxVersion:8} {ApiKey:9 ApiName:OffsetFetch MinVersion:0 MaxVersion:7} {ApiKey:10 ApiName:FindCoordinator MinVersion:0 MaxVersion:3} {ApiKey:11 ApiName:JoinGroup MinVersion:0 MaxVersion:7} {ApiKey:12 ApiName:Heartbeat MinVersion:0 MaxVersion:4} {ApiKey:13 ApiName:LeaveGroup MinVersion:0 MaxVersion:4} {ApiKey:14 ApiName:SyncGroup MinVersion:0 MaxVersion:5} {ApiKey:15 ApiName:DescribeGroups MinVersion:0 MaxVersion:5} {ApiKey:16 ApiName:ListGroups MinVersion:0 MaxVersion:4} {ApiKey:17 ApiName:SaslHandshake MinVersion:0 MaxVersion:1} {ApiKey:18 ApiName:ApiVersions MinVersion:0 MaxVersion:3} {ApiKey:19 ApiName:CreateTopics MinVersion:0 MaxVersion:5} {ApiKey:20 ApiName:DeleteTopics MinVersion:0 MaxVersion:4} {ApiKey:21 ApiName:DeleteRecords MinVersion:0 MaxVersion:2} {ApiKey:22 ApiName:InitProducerId MinVersion:0 MaxVersion:3} {ApiKey:23 ApiName:OffsetForLeaderEpoch MinVersion:0 MaxVersion:3} {ApiKey:24 ApiName:AddPartitionsToTxn MinVersion:0 MaxVersion:1} {ApiKey:25 ApiName:AddOffsetsToTxn MinVersion:0 MaxVersion:1} {ApiKey:26 ApiName:EndTxn MinVersion:0 MaxVersion:1} {ApiKey:27 ApiName:WriteTxnMarkers MinVersion:0 MaxVersion:0} {ApiKey:28 ApiName:TxnOffsetCommit MinVersion:0 MaxVersion:3} {ApiKey:29 ApiName:DescribeAcls MinVersion:0 MaxVersion:2} {ApiKey:30 ApiName:CreateAcls MinVersion:0 MaxVersion:2} {ApiKey:31 ApiName:DeleteAcls MinVersion:0 MaxVersion:2} {ApiKey:32 ApiName:DescribeConfigs MinVersion:0 MaxVersion:3} {ApiKey:33 ApiName:AlterConfigs MinVersion:0 MaxVersion:1} {ApiKey:34 ApiName:AlterReplicaLogDirs MinVersion:0 MaxVersion:1} {ApiKey:35 ApiName:DescribeLogDirs MinVersion:0 MaxVersion:2} {ApiKey:36 ApiName:SaslAuthenticate MinVersion:0 MaxVersion:2} {ApiKey:37 ApiName:CreatePartitions MinVersion:0 MaxVersion:2} {ApiKey:38 ApiName:CreateDelegationToken MinVersion:0 MaxVersion:2} {ApiKey:39 ApiName:RenewDelegationToken MinVersion:0 MaxVersion:2} {ApiKey:40 ApiName:ExpireDelegationToken MinVersion:0 MaxVersion:2} {ApiKey:41 ApiName:DescribeDelegationToken MinVersion:0 MaxVersion:2} {ApiKey:42 ApiName:DeleteGroups MinVersion:0 MaxVersion:2} {ApiKey:43 ApiName:ElectLeaders MinVersion:0 MaxVersion:2} {ApiKey:44 ApiName:IncrementalAlterConfigs MinVersion:0 MaxVersion:1} {ApiKey:45 ApiName:AlterPartitionReassignments MinVersion:0 MaxVersion:0} {ApiKey:46 ApiName:ListPartitionReassignments MinVersion:0 MaxVersion:0} {ApiKey:47 ApiName:OffsetDelete MinVersion:0 MaxVersion:0} {ApiKey:48 ApiName:DescribeClientQuotas MinVersion:0 MaxVersion:0} {ApiKey:49 ApiName:AlterClientQuotas MinVersion:0 MaxVersion:0}]}
[2022-04-13 13:52:18] DEBUG Supported features: {Reads:true Applies:true Locks:false DynamicBrokerConfigs:true}
[2022-04-13 13:52:18] DEBUG Metadata request: {Addr:<nil> Topics:[]}
[2022-04-13 13:52:18] DEBUG Metadata response: <nil> (kafka.(*Client).Metadata: context deadline exceeded)
[2022-04-13 13:52:18] ERROR kafka.(*Client).Metadata: context deadline exceeded
We have a similar cluster in another environment, which just seem to work with the above command still.
- topicctl: 1.3.1
MSK:
- brokers(3x): kafka.m5.large
- kafka: 2.6.2
- auth: IAM auth enabled
- IAM role executing: full admin rights
Any ideas on how to debug further?