-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Allow variables to be printed to STDOUT with airflow variables export. #31854
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wondering if it makes sense to use AirflowConsole, it would allow print in multiple formats.
|
If by console, you mean the UI, that makes sense, however this allows the cli to do so as well, similar to connections. In the past i have had to switch between the cli and the UI for connections and variables. |
In stdout only. I was thinking if we could print in different formats like json/yaml etc But I see we do print in different formats for connection export like |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we please use more meaningful variable names instead of qry, var, val?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair point. I just refactored the helper function into this. Will update the patch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| except Exception: | |
| except json.decoder.JSONDecodeError: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's always a good idea to handle specific exceptions as close to the source as possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method doesn't really add much value, I would directly use fileio.name == "<stdout>" in line 96. Saves one lookup to a different method. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, this was just to maintain some consistency with connection_command.py's export. Will update the patch
|
Thanks for the reviews @utkarsharma2. Updated the patch. Please take a look when you have some time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| query = session.query(Variable).all() | |
| data = json.JSONDecoder() | |
| for variable in query: | |
| try: | |
| value = data.decode(variable.val) | |
| except json.decoder.JSONDecodeError: | |
| value = variable.val | |
| var_dict[variable.key] = value | |
| rows = session.query(Variable).all() | |
| data = json.JSONDecoder() | |
| for row in rows: | |
| try: | |
| value = data.decode(row.val) | |
| except json.decoder.JSONDecodeError: | |
| value = row.val | |
| var_dict[row.key] = value |
Maybe something like this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can drop the all() as well since it’s only iterated once. Also why is the decoder called the data?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the delay, had been unwell lately.
Thanks for the inputs. Ive updated the patch.
Similar to connections, this change allows the cli to export the variables to the stdout using 'airflow variables export -' Fixes apache#31851
- handle JSONDecodeError instead of Exception
| var_dict[row.key] = value | ||
|
|
||
| with args.file as f: | ||
| f.write(json.dumps(var_dict, sort_keys=True, indent=4)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| f.write(json.dumps(var_dict, sort_keys=True, indent=4)) | |
| json.dump(f, var_dict, sort_keys=True, indent=4) |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
This change allows airflow cli to export variables to stdout similar to connections using
airflow variables export -.Fixes #31851
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.