Skip to content

Conversation

@vincbeck
Copy link
Contributor

Resolves #40875.

As mentioned in #40875, in some cases EcsRunTaskOperator have missing logs in the log group specified through awslogs_group. When awslogs_group is provided, the operator pull logs from awslogs_group (where the ECS task logs are) and log them so that the user can see them in the UI. The issue is, somehow (and I really cannot explain why), when multiple logs from awslogs_group have the same timestamp, only one of them (the first one) is logged. An example to better understand:

log_events = self._get_log_events(continuation_token)
for log_event in log_events:
    self.logger.info(self.event_to_str(log_event))

If log_events has 2 events with the same timestamp, self.logger.info(self.event_to_str(log_event)) will run twice but will log once.

Adding a delay between the logging operators "fix" the issue.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Aug 15, 2024
@vincbeck vincbeck marked this pull request as draft August 15, 2024 21:37
@vincbeck vincbeck marked this pull request as ready for review August 15, 2024 21:59
@eladkal eladkal merged commit e92cf81 into apache:main Aug 17, 2024
Artuz37 pushed a commit to Artuz37/airflow that referenced this pull request Aug 19, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Aug 20, 2024
@vincbeck vincbeck deleted the vincbeck/ecs branch September 20, 2024 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EcsRunTaskOperator log output in UI does not contain all events

2 participants