Hide sensitive data in UI

[n4rk0o]

Description

I'm using Airflow for 2 years now and I have a plugin that get password for a specific account in a Vault and then push it through a XCOM to reuse it on another tasks.

The fact is that if the value is sensitive like a password, I can't hide it in the UI except for XCOM if I add an underscore in the prefix name of the key value.

Eg: kwargs['ti'].xcom_push('key':'_password', 'value':'my_value')

But for rendered template UI page, I didn't find anything similar, so if I try to pull a XCOM, it will show the value in the UI and I want to avoid it.

Maybe is it possible to add a condition in https://github.com/apache/airflow/blob/master/airflow/www/views.py after line 635

elif template_field.startswith('_'):
    html_dict[template_field] = ("<pre><code>sensitive data will not be exposed here</pre></code>")

Use case / motivation

I know that I can use connections but in my case, and due to security politic in my company, we have to store it in a dedicated Vault.

Related Issues

N/A

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kaxil]

Airflow 1.10.10 allows getting connections from a Vault: https://airflow.apache.org/blog/airflow-1.10.10/#allow-retrieving-airflow-connections-variables-from-various-secrets-backend

Does that help your use-case?

[n4rk0o]

Airflow 1.10.10 allows getting connections from a Vault: https://airflow.apache.org/blog/airflow-1.10.10/#allow-retrieving-airflow-connections-variables-from-various-secrets-backend

Does that help your use-case?

So basically, with Airflow 1.10.10, if I configure the airflow.cfg to use Hashicorp Vault, I can use connections and variables as usual but instead of getting data from Airflow database, it will got it from Vault?

[kaxil]

Airflow 1.10.10 allows getting connections from a Vault: https://airflow.apache.org/blog/airflow-1.10.10/#allow-retrieving-airflow-connections-variables-from-various-secrets-backend
Does that help your use-case?

So basically, with Airflow 1.10.10, if I configure the airflow.cfg to use Hashicorp Vault, I can use connections and variables as usual but instead of getting data from Airflow database, it will got it from Vault?

Exactly. Here is one of the guide: https://www.astronomer.io/guides/airflow-and-hashicorp-vault/ to test it out locally and following docs:

• https://airflow.apache.org/docs/1.10.10/concepts.html#storing-variables-in-environment-variables
• https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#configuration

[marcusianlevine]

@kaxil I'm trying to implement a very similar workflow using the new VaultBackend for Variables storage, but I noticed that sensitive Variables fetched from Vault can be leaked out from the UI via Rendered Templates if you include a Variable value in a field that is templated

Could we take a similar approach to AIRFLOW-45 and mask any Variables that match a set of pattern names in the Rendered Templates view?

[kaxil]

aah I see, apologies @n4rk0o I should have read your description more carefully. Yes the Rendered UI Field currently exposes everything. We should have a way of hiding this.

I see two options here:

1. Same as what @marcusianlevine described
2. Have a flag in airflow.cfg to hide rendered templated_fields globally. Sometime though users would love to check if the field was rendered correctly and hence it acts as a good debugging tool.

[marcusianlevine]

I do like the idea of having this be an optional behavior

I think it would be really great if we could selectively obscure Variables too - some are sensitive while others are not, which is why I like the approach in #1530 of using a list of patterns

Also, I noticed that Variables can be leaked via the Task Instance Details page too, even if the fields aren't rendered templates, so we should be sure to mask sensitive variables everywhere we can in the UI (obviously if they are logged by a task it's still possible to deliberately leak them through the logs...)

[kaxil]

100% agree. Would you like to work on a proposal or a PR for that one, it would be definitely good to get this in for 2.0

[marcusianlevine]

Hmm it's been awhile since I've worked on the UI

I'll do some digging in the code and see if I can come up with a reasonable design...

Do you happen to know off-hand if there is any way to tell at run-time whether DAG code is being compiled by the webserver?

Because we need the Variable's key to do the conditional masking, we really need to mask the Variable values at the time that Variable.get is called, otherwise we'd have to search for every possible Variable value in every field's contents...

I'm also not sure how this would work with serialized DAGs: once they're serialized they will have the unmasked values, so if the webserver just reads the serialized DAG directly we'd have no way to know whether to mask certain Variables since they would already be just plain values

Maybe we need a more dynamic way of injecting Variables rather than reading them at DAG compile time and writing their values into the DB? I'm thinking some kind of Variable pointer syntax like <Var key=my-variable> that gets rendered differently based on whether the code is being run on the webserver or not

[kaxil]

This should happen before the templates are rendered I think for hiding sensitive info in Rendered Fields

[ausiddiqui]

I have a similar use case, but passing the password from a connection object I created in the UI to the environment variable in the KubernetesPodOperator and it is appearing in plain text in the Rendered Template part of the UI for the task. There should be a way to avoid this being printed and visible.

[marcusianlevine]

I should have some time to work on a proposal for this in the next few weeks, but I'm still not sure the best way to approach the design

The key question I'm stuck on is at what point the templates get filled in... obviously the executors need to have access to the unmasked values, but if they get injected at DAG compile time then they will be visible in the UI, so I'm guessing the unmasked injection needs to happen at task execution time