Add @latest, @now, @start_date, @end_date to the `airflow test` CLI

[underyx]

It's often quite a drag to have to calculate and type a valid datestring for airflow test when I just want to see whether my code runs. It'd be really nice if I could just run airflow test dag task latest and have airflow think about the execution date instead of me.

[r39132]

Good idea. PR coming.. @underyx What are the expectations of
airflow test test_dag_garthcn op_1 @latest? Do you want it to run the current time of something that makes sense for your schedule_interval?

[underyx]

@r39132 I would expect it to run with the same execution time that a run_only_latest=True task would get scheduled for at that moment.

[r39132]

@underyx Here you go!
cc @mistercrunch @criccomini @bolkedebruin @jlowin

I've added 4 short-cuts: @now, @latest, @start_date, and @end_date to airflow test

I will be adding documentation as well, but here is how this works:

@now
This passes datettime.datetime.now() as the task execution date

airflow test simple_dummy_once_dag_v1 run_this @now
[2016-04-19 23:19:13,543] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-04-19 23:19:13.512938

@start_date
This raises an exception if the dag has no start_date. Otherwise, the dag's start date will be used. For example, the dag simple_dummy_dag_v2 has a start_date of 2016-03-26 00:00:00

airflow test simple_dummy_dag_v2 run_this @start_date
[2016-04-19 23:25:02,732] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-03-26 00:00:00

@end_date
This raises an exception if the dag has no end_date. Otherwise, the dag's end date will be used. For example, the dag simple_dummy_dag_v3 has a start_date of 2016-03-26 00:00:00 and an end_date of 2016-03-29 00:00:00. Dag simple_dummy_dag_v4 has a start_date of 2016-03-26 00:00:00 and an end_date of 2016-05-29 00:00:00. Note: the code does not forbid future end_dates

Non-future end_date

airflow test simple_dummy_dag_v3 run_this @end_date
[2016-04-19 23:36:02,611] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-03-29 00:00:00

Future end_date

airflow test simple_dummy_dag_v4 run_this @end_date
[2016-04-19 23:36:56,767] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-05-29 00:00:00

@latest
This passes the latest day while honoring the start and end dates of a DAG as the task execution date. If the DAG has a schedule interval of None or @once, datetime.datetime.now() is used. If the DAG has a different schedule interval, an exception is raised if the dag has no start_date. For example, the dag simple_dummy_dag_v2 has a start_date of 2016-03-26 00:00:00 and no end_date (and a schedule interval of @daily. The dag simple_dummy_dag_v3 has a start_date of 2016-03-26 00:00:00 and an end_date of 2016-03-29 00:00:00. Dag simple_dummy_dag_v4 has a start_date of 2016-03-26 00:00:00 and an end_date of 2016-05-29 00:00:00. Note: the code does not forbid future end_dates as shown below

Non-future end_date

airflow test simple_dummy_dag_v3 run_this @latest
[2016-04-19 23:36:02,611] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-03-29 00:00:00

Future end_date

airflow test simple_dummy_dag_v4 run_this @latest
[2016-04-19 23:36:56,767] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-05-29 00:00:00

No end_date

airflow test simple_dummy_dag_v2 run_this @latest
[2016-04-21 00:41:43,126] {models.py:1191} INFO - Executing <Task(DummyOperator): run_this> on 2016-04-20 00:00:00

[underyx]

Wow, that's just lovely, awesome job, @r39132!

If you're okay with nitpicking, I'd mention that I'm not sure if @start / @end wouldn't already be just as clear as @start_date / @end_date, with shorter names (which is certainly an advantage when the feature is directed at the lazy 👯 ).

[r39132]

@underyx That was my first approach but felt some standard naming trumped the needs of the lazy.

[underyx]

@r39132 fair enough, you've got me convinced :D

[underyx]

Moved to https://issues.apache.org/jira/browse/AIRFLOW-28