Skip to content

Conversation

@kaxil
Copy link
Member

@kaxil kaxil commented Feb 24, 2021

closes #13799

Without it the migration from 1.10.14 to 2.0.0 can fail with following error for old TIs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
    self._run_scheduler_loop()
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1533, in _do_scheduling
    num_queued_tis = self._critical_section_execute_task_instances(session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1132, in _critical_section_execute_task_instances
    queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/session.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1034, in _executable_task_instances_to_queued
    if task_instance.pool_slots > open_slots:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

Workaround was to run manually:

UPDATE task_instance SET pool_slots = 1 WHERE pool_slots IS NULL;

This commit makes adds a DB migration to change the value to 1 for records with NULL value. And makes the column NOT NULLABLE.

This bug was caused by #7160


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Feb 24, 2021
@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

@kaxil kaxil force-pushed the fix-pool-slots branch 5 times, most recently from 3dbdc8b to 2c77258 Compare February 24, 2021 23:45
closes apache#13799

Without it the migration from 1.10.14 to 2.0.0 can fail with following error for old TIs:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
    self._run_scheduler_loop()
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1533, in _do_scheduling
    num_queued_tis = self._critical_section_execute_task_instances(session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1132, in _critical_section_execute_task_instances
    queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/session.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1034, in _executable_task_instances_to_queued
    if task_instance.pool_slots > open_slots:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
```

Workaround was to run manually:

```
UPDATE task_instance SET pool_slots = 1 WHERE pool_slots IS NULL;
```

This commit makes adds a DB migration to change the value to 1 for records with NULL value. And makes the column NOT NULLABLE.

This bug was caused by apache#7160
@kaxil kaxil merged commit f763b7c into apache:master Feb 25, 2021
@kaxil kaxil deleted the fix-pool-slots branch February 25, 2021 02:56
ashb pushed a commit that referenced this pull request Mar 19, 2021
closes #13799

Without it the migration from 1.10.14 to 2.0.0 can fail with following error for old TIs:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
    self._run_scheduler_loop()
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1533, in _do_scheduling
    num_queued_tis = self._critical_section_execute_task_instances(session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1132, in _critical_section_execute_task_instances
    queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/session.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1034, in _executable_task_instances_to_queued
    if task_instance.pool_slots > open_slots:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
```

Workaround was to run manually:

```
UPDATE task_instance SET pool_slots = 1 WHERE pool_slots IS NULL;
```

This commit makes adds a DB migration to change the value to 1 for records with NULL value. And makes the column NOT NULLABLE.

This bug was caused by #7160

(cherry picked from commit f763b7c)
@kaxil kaxil mentioned this pull request Apr 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduler crashes when unpausing some dags with: TypeError: '>' not supported between instances of 'NoneType' and 'int'

3 participants