Issue 23 Explanation

[afarbos]

I am re-reading the last sentence of your answer of this issue #23.

"If you're planning on doing some large distributed computation that uses lots of memory, Airflow might not be the right platform. Instead, you may want to use a distributed system, or write some sort of service, and invoke it from Airflow."

I am not sure to really understand. Could you give me more information about it?

[mistercrunch]

Airflow dispatches work to other systems. Hadoop, Hive, MySQL, Redshift, ... Airflow can take on some workload, but its role is not to crunch huge amount of data on its own. The unit of work (task instance) should be fairly small (single threaded, few gigs of memory, not too much disk usage, ...).

Most of the work that Airflow does is orchestration, most tasks are assumed to be remote, waiting for the external system to say it's done with the work. Data transfer operators are a little bit more involved, getting a chunk of data from a system and moving it into another.

If you need to run huge jobs, I'd suggest to do it in a system that is made for that purpose and have Airflow call that system.

[afarbos]

Ok, so for example I could use airflow, calling rabbitmq to execute jobs on ec2 instances?

If I am right how do you do your dashboard/stats after? (with cpu percentage, etc)

And do you have any simple way to deploy on specific systems? If yes, which systems?

[mistercrunch]

What are you trying to do? Are you looking for a system like Nagios to monitor your servers?

What do you mean by specific systems?

[afarbos]

I am trying to use airflow to modelize my workflow and execute it on ec2 instances.

I mean, is it easier to use ec2, hadoop or another specific system with airflow?

[mistercrunch]

Well Airflow nodes run Airflow machines (should work with EC2 or whatever linux boxes you have, metal or virtual), and they can task to external systems (Hadoop, Hive, MySQL, HDFS, ...).

In a simple setup you can just have one Airflow box that runs the scheduler, web server and a worker process and even a MySQL or Postgres database on that same box.

In a more complex setup you can have an array of workers, a dedicated web server, a dedicated scheduler, and a dedicated database.

If you are interested in looking at the workload on these machines, use whatever your in-house ops tool is (Nagios, Datadog, ...).

[afarbos]

Ok, so that answer to one of my question: if I want a dashboard, I have to do it myself.

I will try do be more clear of what I want to do. I take a look to the CeleryExecutor and my question is: If I use it with rabbitmq as broker, does that do auto-scaling with your worker ?

[mistercrunch]

The celery executor sends task instances to be executed by workers. Workers have the number of execution slots you define and when they receive a message to run a task instance they proceed to do so.

If you add new workers, they start listening to the queue and will take on processing task instances as well. At Airbnb we currently have 6 workers with 128 slots on each, meaning we can process up to 768 task instances in parallel. If you have deployment automation like we do, you can kick off more workers and get more slots by clicking a few buttons.

Since most of these tasks are just sensors of Hive jobs, the workers don't do much work, that's why we're able to provision 128 slots on machine that have 8 vCPUs without worrying about saturating those machines. Each worker could probably have twice as much slots and we'd be fine.

[mistercrunch]

I just threw in an endpoint to refresh the DagBag:
81ab8fb
/admin/airflow/refresh_all

I'm not adding a link to it since it wouldn't work in a multi-threaded / multi-server environment, but you'll be able to use this hidden endpoint in the next release as a hack.