Add missing docs on using more than one `pool_slot`

[kaxil]

Describe the issue with documentation

We added a feature that allow more than one pool slots to be used by a single task in #7160

However, we currently have no docs on it.

We should add some docs in https://airflow.apache.org/docs/apache-airflow/stable/concepts/pools.html that covers:

• Example of an operator using it
• Use-cases when it might be helpful

[BasPH]

Subscribing to the thread because I wonder what the use case is for pool slots.

[eladkal]

Subscribing to the thread because I wonder what the use case is for pool slots.

Pools are used to limit concurrency with external services.
Not all tasks "weight" the same. Some tasks might require more resources from the external service - so for those heavy tasks you might want to occupy more slots of the pool to avoid scheduling more tasks. For example: allowing to run up to 2 small tasks or up to 1 big task.

[BasPH]

I get that, but what do pool slots give over creating a second pool with fewer slots?

[eladkal]

I get that, but what do pool slots give over creating a second pool with fewer slots?

Say we have:
Pool(pool='db', slots=2)
This pool protect a resource that can handle up to 2 units of computing power.

and we have:

with DAG(...) as dag:
    BashOperator(
        task_id='big_task',
        bash_command='python script.py',
        pool_slots=2,
        pool='db'
    )
    
    BashOperator(
        task_id='small_task1',
        bash_command='python script2.py',
        pool_slots=1,
        pool='db'
    )
 
 
 with DAG(...) as dag2:   
    BashOperator(
        task_id='small_task2',
        bash_command='python script3.py',
        pool_slots=1,
        pool='db'
    )

With this code the resource is protected according to requirements.
How would suggest to get the same functionality with 2 pools (without pool_slots)?

[potiuk]

Ah. How cool I did not know Airflow can do that :)

[BasPH]

Oh I see, you'd only be able to approximate a limit of 2 "units" without pool slots. Thanks for clarifying!

[PApostol]

I can take this one!

[potiuk]

Feel free!

[eladkal]

Fixed by #20257