Intermittent errors caused by Kafka Flush Timeout

[oto-macenauer-absa]

Describe the bug

The message publish to Kafka fails intermittently with this error:

Caused by org.apache.http.client.HttpResponseException: status code: 500, reason phrase: POST request to https://event-gate-api-prod.pdintdebdtools.aws.dsarena.com/topics/public.cps.za.runs was unsuccessful. Status code: 500, response body: Some({"success": false, "statusCode": 500, "errors": [{"type": "kafka", "message": "Kafka flush timeout after 5.0s: 1 message(s) still pending"}]})

To Reproduce

Happens intermittently

Expected behavior

The connection to Kafka should handle the error. Either retry or produce only warning if rest of the ways to produce messages fail.

Screenshots