Sync DAG specific permissions when parsing

[jedcunningham]

This POC allows the DAG specific permissions to be created/updated during DAG parsing, instead of during webserver start or cli sync-perm.

With a large number of DAGs, walking through them all to do DAG specific permissions isn't exactly fast and they can only change during the scheduler parsing anyways. Overall more efficient as we don't need to check every DAG as well, we only need to check a given DAG when it changes.

This also fixed a bug where the default webserver DAG specific syncing didn't handle access_control.

Closes #8609

[jedcunningham]

@jhtimmins, curious is you know a better way or trick to using the security manager somewhere where we don't want/need the whole flask app?

[jhtimmins]

@jedcunningham Oof I need to think about this, because generally speaking we really don't want to extend the webserver-level controls into Airflow core.

[jhtimmins]

Ok, after thinking more about this, I don't think we should be extending the security manager into the /airflow/models directory. I'd much rather create a sync-permissions API endpoint if one doesn't exist, and hitting that from the CLI via a separate HTTP request.

Stale review

[chodankarcc]

which airflow version has this fixed change?

[jedcunningham]

2.1.0:

airflow/CHANGELOG.txt

Line 100 in e3f5eb9

- Sync DAG specific permissions when parsing (#15311)

[chodankarcc]

2.1.0:

airflow/CHANGELOG.txt

Line 100 in e3f5eb9

- Sync DAG specific permissions when parsing (#15311)

Thanks for quick reply. I am using composer-1.16.7-airflow-1.10.15 (Google Composer), and unfortunately composer don't have this airflow version available yet to upgrade to. So is there any alternative other than admin clicking on refresh to update permissions as I want to automate solution,

[jedcunningham]

I believe running airflow sync-perm should do it as well.

[chodankarcc]

No sync_perm is not working as expected. Its not updating roles permission as per DAG access control.

[jedcunningham]

Interesting, the code looks like it should do it 🤷‍♂️. Sorry, I'm not sure.

airflow/airflow/bin/cli.py

Lines 2075 to 2080 in 5786dcd

	print('Updating permission on all DAG views')
	dags = DagBag(store_serialized_dags=settings.STORE_SERIALIZED_DAGS).dags.values()
	for dag in dags:
	appbuilder.sm.sync_perm_for_dag(
	dag.dag_id,
	dag.access_control)

[chodankarcc]

store_serialized_dags

I was able to solve issue by updating store_serialized_dags = False in airflow config. Thanks for your pointer

[ashb]

If something is not working in Composer that is fixed in open source Airflow then you should raise that issue with Composer support.

[ali-hafidz]

@chodankarcc I also facing the same issue. then already updating store_serialized_dags = False . but I have new issue, the dag that I set running sequentially running not in order, the dag running from middle dag I think its bug ui, when I updating store_serialized_dags = True its working normally . Have you facing the same problem ? I also using composer.