Skip to content

Conversation

@dstandish
Copy link
Contributor

demo

cc @RNHTTR

@boring-cyborg boring-cyborg bot added provider:cncf-kubernetes Kubernetes (k8s) provider related issues area:logging area:providers area:Scheduler including HA (high availability) scheduler provider:amazon AWS/Amazon - related issues provider:microsoft-azure Azure-related issues provider:google Google (including GCP) related issues labels May 12, 2023
@collinmcnulty
Copy link
Contributor

This looks awesome.

@dstandish dstandish changed the title Enable shipping task log messages from anywhere DEMO: Enable shipping task log messages from anywhere May 24, 2023
@pankajkoti pankajkoti force-pushed the add-arbitrary-log-shipping branch from 2fff1f5 to fa599eb Compare June 23, 2023 12:03
@pankajkoti pankajkoti force-pushed the add-arbitrary-log-shipping branch from fa599eb to 04d9489 Compare July 2, 2023 13:24
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably we can / should avoid creating the session if not using USE_PER_RUN_LOG_ID

Comment on lines 249 to 251
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar to above, I suspect that there is an optimization here, if USE_PER_RUN_LOG_ID is false, then we should not need to go to the database i think. this is probably not a huge issue on individual task logging i.e. on the worker but from something like scheduler you would probably want to avoid queries. though, not sure if it's really possible given the templating etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

USE_PER_RUN_LOG_ID is not used here, and the case that USE_PER_RUN_LOG_ID == False is when users use airflow < 2.3.3. As I think we don't want to roll back the code to 2.3.3, I'm confused about what you mean by USE_PER_RUN_LOG_ID is false here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah as you mentioned on slack, i was just making quick conclusions based on what i saw in the code but it appears that this is not a global feature, just a backcompat flag in ES handler. but maybe we should make it a config and then we could use it here.

@pankajkoti pankajkoti force-pushed the add-arbitrary-log-shipping branch from 0f12304 to 69f01f9 Compare July 12, 2023 10:00
@pankajkoti
Copy link
Member

Following up on this PR with some refactoring here: #32646

@github-actions
Copy link

github-actions bot commented Sep 1, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 1, 2023
@github-actions github-actions bot closed this Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:logging area:providers area:Scheduler including HA (high availability) scheduler provider:amazon AWS/Amazon - related issues provider:cncf-kubernetes Kubernetes (k8s) provider related issues provider:google Google (including GCP) related issues provider:microsoft-azure Azure-related issues stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants