-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Closed
Description
How does one add new dags to the system without restarting the webserver? Running "python my_dag.py" is not importing the dag in to the db either. I am running on EC2 with a Postgresql database. I am able to see the data gets imported only on restart of the web app.
Here is the code:
This code will become the EP Data Pipeline.
The flow may eventually look like :
* 1. SQS Message Written Detect
* 2.a. Email Send Flow Started
* 2.b. Database Row Written Detect
* 3. SQS Queue Empty Detect
* 4. Email Send Flow Complete
"""
from airflow import DAG
from airflow.operators import EmailOperator
from datetime import datetime
print ' got here 1'
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2015, 1, 1),
'email': ['sanand@agari.com'],
'email_on_failure': True,
'email_on_retry': True,
}
print ' got here 1'
dag = DAG('ep_demo_1', default_args=default_args)
print ' got here 1'
t1 = EmailOperator(
task_id='email_pipeline_start',
to='sanand@agari.com',
subject='EP Demo Pipeline Started (TAD)',
html_content='',
dag=dag)
After restarting the webapp, I still don't see any entries in the dag table in the sqlite db for my dags. I see the dags in the UI -- there seems to be a difference between loading and importing a dag.
2015-06-18 17:49:21,175 - root - INFO - Loaded DAG <DAG: ep_demo_1>
2015-06-18 17:49:21,177 - root - INFO - Importing /usr/local/lib/python2.7/site-packages/airflow/example_dags/example_bash_operator.py
2015-06-18 17:49:21,178 - root - INFO - Loaded DAG <DAG: example_bash_operator>
2015-06-18 17:49:21,179 - root - INFO - Importing /usr/local/lib/python2.7/site-packages/airflow/example_dags/example_python_operator.py
2015-06-18 17:49:21,180 - root - INFO - Loaded DAG <DAG: example_python_operator>
2015-06-18 17:49:21,181 - root - INFO - Importing /usr/local/lib/python2.7/site-packages/airflow/example_dags/tutorial.py
2015-06-18 17:49:21,182 - root - INFO - Loaded DAG <DAG: tutorial>
Metadata
Metadata
Assignees
Labels
No labels