-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Fix hardcoded container name in remote logging option for Azure Blob Storage #32779
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@eladkal since it touches both airflow core and provider, do we need to separate into two PRs? |
potiuk
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! I like the doc improvement too :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I rethought it a bit.
I know we still have provider-specific code in "logging" section, but with the recent move of configuration to providers, we should rather aim to eventually move all of those to provider-specific configuration.
Can you please do it this way @Adaverse (even if it is not consistent with the other remote logging, this will be the "target" - so we can start with this wasb change). This will also setup convention on how to name such "remote logging" section for other providers.
Few things to change:
- add config section in provider.yaml of the azure provider ("logging-azure-wasb" ?)
- read it from there
- add configuration page in azure provider
- update examples
|
And "version_added" in this case should be the next minor version of the azure provider, not 2.7.0 of airflow. |
|
BTW. This should make the change "azure-provider-only". As it should be. |
So in order to remove provider specific config from loading from core Airflow, below are the steps I would follow -
So we would compare the value in If we can dynamically update the config doc to only contain the possible values of the supported remote loggers, then i guess we are good to go. Let me know if above sounds fine or any changes you suggest? |
That approach would be far in the future I think. We should approach it gradually - maybe in Airlfow 2.8 or 2.9 we might want to do a bigger change. For now I just propose to create the new configuration option via the |
Got it @potiuk! Let me make the change. |
Moved the config to the provider but having the below issue where the config won't load in the doc. Am I missing anything here? |
The config is hidden when I put the version (for this case) anything greater than 2.7.0 For the above case, had put the next release of Azure provider i.e 6.3.0 |
Let's fix the above before merging this PR. Let me give it a try. |

Currently, the container name for remote logging in Azure Blob Storage is hard coded in
airflow_local_settings.py. Hence, in this PR -remote_wasb_log_container^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.