Fix Fargate logging for AWS system tests

[vincbeck]

Both system tests example_eks_with_fargate_in_one_step and example_eks_with_fargate_profile are failing for the same reason. When the operator EksPodOperator is used with get_logs=True, the operator tries to get log once the POD started. When doing so, 90% of the time it fails because of:

HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \\"https://10.0.5.84:10250/containerLogs/default/run-pod-6rjxgqrs/base?follow=true\\u0026timestamps=true\\": remote error: tls: internal error","code":500}\n'

After investigation, it turns out there is delay between when the pod starts and when the CSR is available, signed and approved. If you try to get logs with the command kubectl logs <pod-name> -n <namespace> when the CSR is not available, signed and approved, you'll get the exact same error. If you wait until the CSR is there, you'll get the logs.

Therefore, in order to fix it, I decided to just retry on ApiException which the is the exception we get in such scenario.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vincbeck]

@ferruzzi who made most of the work here

[ferruzzi]

You've reverted everything except adding the exception. Was that intentional?

[dstandish]

@vincbeck can you share an example traceback that caused you to add tenacity to consume_logs?

I am just digging into KPO logging a bit again, and finding that the whole thing has become extremely messy and complicated, and I'm trying to chip away at some of the mess.

And as part of that, the behavior of consume_logs is such that it is already called in a loop if there's an error. So it's a bit odd to also wrap it with tenacity, a kind of mixing of two different retry strategies that makes it a bit confusing.

Now what can be done.... Well please observe that consume_logs calls read_pod_logs, which I believe is where the APIException that you catch is actually encountered, and which already retries using tenacity. So my thought is we can just update that function to handle your case, and thus move the retry logic closer to where it is needed, and reduce the tenacity wrappers from 2 to 1. But to do this I need more information about specifically where that exception is raised. My suspicion is that it would be raised at logs = self._client.read_namespaced_pod_log in read_pod_logs. But I recognize it's also possible that it is not raised until the logs consumer is iterated in at for raw_line in logs: in consume_logs. Thanks for the help.

[vincbeck]

Hey @dstandish . I removed the tenacity wrapper around consume_logs locally and ran the system test to get the stack trace. The system test ran successfully twice. So I guess, it should be safe to remove it?

[dstandish]

Hey, thanks @vincbeck , kind of you to check that.

Ok, i'll make a PR to remove it for now. And then if it comes back, we can revisit.

Thanks