Skip to content

Conversation

@phi-friday
Copy link
Contributor

Like setup and teardown, skip_if and run_if should also be removed.

sample dag code

from __future__ import annotations

from pendulum import datetime

from airflow.decorators import dag, task


@dag(start_date=datetime(2024, 1, 1), schedule=None, catchup=False)
def venv_skip_if() -> None:
    @task.skip_if(lambda context: False)
    @task.virtualenv()
    def echo() -> None:
        print("hello world")

    echo()


venv_skip_if()

before in venv

1b4bbddb0c17
*** Found local files:
***   * /root/airflow/logs/dag_id=venv_skip_if/run_id=manual__2024-08-28T12:29:04.469903+00:00/task_id=echo/attempt=1.log
[2024-08-28, 12:29:05 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2024-08-28, 12:29:05 UTC] {process_utils.py:186} INFO - Executing cmd: /usr/local/bin/python -m virtualenv /tmp/venv5qst192w --system-site-packages --python=python
[2024-08-28, 12:29:05 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO - created virtual environment CPython3.8.19.final.0-64 in 276ms
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -   creator CPython3Posix(dest=/tmp/venv5qst192w, clear=False, no_vcs_ignore=False, global=True)
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -     added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0
[2024-08-28, 12:29:06 UTC] {process_utils.py:194} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2024-08-28, 12:29:06 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venv5qst192w/bin/pip install -r /tmp/venv5qst192w/requirements.txt
[2024-08-28, 12:29:06 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:29:10 UTC] {process_utils.py:194} INFO - 
[2024-08-28, 12:29:10 UTC] {process_utils.py:194} INFO - [notice] A new release of pip is available: 24.1 -> 24.2
[2024-08-28, 12:29:10 UTC] {process_utils.py:194} INFO - [notice] To update, run: python -m pip install --upgrade pip
[2024-08-28, 12:29:10 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venv5qst192w/bin/python /tmp/venv-callwp2r9u9t/script.py /tmp/venv-callwp2r9u9t/script.in /tmp/venv-callwp2r9u9t/script.out /tmp/venv-callwp2r9u9t/string_args.txt /tmp/venv-callwp2r9u9t/termination.log /tmp/venv-callwp2r9u9t/***_context.json
[2024-08-28, 12:29:10 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO - Traceback (most recent call last):
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO -   File "/tmp/venv-callwp2r9u9t/script.py", line 18, in <module>
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO -     @task.skip_if(lambda context: False)
[2024-08-28, 12:29:13 UTC] {process_utils.py:194} INFO - NameError: name 'task' is not defined
[2024-08-28, 12:29:13 UTC] {taskinstance.py:3167} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/taskinstance.py", line 753, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/opt/airflow/airflow/models/taskinstance.py", line 719, in _execute_callable
    return ExecutionCallableRunner(
  File "/opt/airflow/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
  File "/opt/airflow/airflow/models/baseoperator.py", line 402, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/decorators/base.py", line 266, in execute
    return_value = super().execute(context)
  File "/opt/airflow/airflow/models/baseoperator.py", line 402, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/operators/python.py", line 509, in execute
    return super().execute(context=serializable_context)
  File "/opt/airflow/airflow/models/baseoperator.py", line 402, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/operators/python.py", line 240, in execute
    return_value = self.execute_callable()
  File "/opt/airflow/airflow/operators/python.py", line 900, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path)
  File "/opt/airflow/airflow/operators/python.py", line 593, in _execute_python_callable_in_subprocess
    execute_in_subprocess(
  File "/opt/airflow/airflow/utils/process_utils.py", line 175, in execute_in_subprocess
    execute_in_subprocess_with_kwargs(cmd, cwd=cwd, env=env)
  File "/opt/airflow/airflow/utils/process_utils.py", line 198, in execute_in_subprocess_with_kwargs
    raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venv5qst192w/bin/python', '/tmp/venv-callwp2r9u9t/script.py', '/tmp/venv-callwp2r9u9t/script.in', '/tmp/venv-callwp2r9u9t/script.out', '/tmp/venv-callwp2r9u9t/string_args.txt', '/tmp/venv-callwp2r9u9t/termination.log', '/tmp/venv-callwp2r9u9t/airflow_context.json']' returned non-zero exit status 1.
[2024-08-28, 12:29:13 UTC] {taskinstance.py:1211} INFO - Marking task as FAILED. dag_id=venv_skip_if, task_id=echo, run_id=manual__2024-08-28T12:29:04.469903+00:00, execution_date=20240828T122904, start_date=20240828T122905, end_date=20240828T122913
[2024-08-28, 12:29:13 UTC] {taskinstance.py:337} ▶ Post task execution logs

after in venv

5dae42dc6e21
*** Found local files:
***   * /root/airflow/logs/dag_id=venv_skip_if/run_id=manual__2024-08-28T12:39:47.502535+00:00/task_id=echo/attempt=1.log
[2024-08-28, 12:39:48 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2024-08-28, 12:39:48 UTC] {process_utils.py:186} INFO - Executing cmd: /usr/local/bin/python -m virtualenv /tmp/venvt7te04zf --system-site-packages --python=python
[2024-08-28, 12:39:48 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO - created virtual environment CPython3.8.19.final.0-64 in 266ms
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -   creator CPython3Posix(dest=/tmp/venvt7te04zf, clear=False, no_vcs_ignore=False, global=True)
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -     added seed packages: pip==24.1, setuptools==70.1.0, wheel==0.43.0
[2024-08-28, 12:39:48 UTC] {process_utils.py:194} INFO -   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2024-08-28, 12:39:48 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venvt7te04zf/bin/pip install -r /tmp/venvt7te04zf/requirements.txt
[2024-08-28, 12:39:48 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:39:52 UTC] {process_utils.py:194} INFO - 
[2024-08-28, 12:39:52 UTC] {process_utils.py:194} INFO - [notice] A new release of pip is available: 24.1 -> 24.2
[2024-08-28, 12:39:52 UTC] {process_utils.py:194} INFO - [notice] To update, run: python -m pip install --upgrade pip
[2024-08-28, 12:39:52 UTC] {process_utils.py:186} INFO - Executing cmd: /tmp/venvt7te04zf/bin/python /tmp/venv-call55vo1yul/script.py /tmp/venv-call55vo1yul/script.in /tmp/venv-call55vo1yul/script.out /tmp/venv-call55vo1yul/string_args.txt /tmp/venv-call55vo1yul/termination.log /tmp/venv-call55vo1yul/***_context.json
[2024-08-28, 12:39:52 UTC] {process_utils.py:190} INFO - Output:
[2024-08-28, 12:39:56 UTC] {process_utils.py:194} INFO - hello world
[2024-08-28, 12:39:56 UTC] {python.py:242} INFO - Done. Returned value was: None
[2024-08-28, 12:39:56 UTC] {taskinstance.py:337} ▶ Post task execution logs

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@phi-friday
Copy link
Contributor Author

I'm getting an error related to the open api, I'm not understanding why it's happening, could you please explain?

@phi-friday phi-friday force-pushed the fix-venv-skip-if branch 3 times, most recently from 4d2a20a to 739fecf Compare September 5, 2024 09:48
@eladkal eladkal added this to the Airflow 2.10.3 milestone Sep 21, 2024
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Sep 21, 2024
@potiuk potiuk merged commit 69b4592 into apache:main Oct 2, 2024
1 check passed
@phi-friday phi-friday deleted the fix-venv-skip-if branch October 2, 2024 04:01
joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
josix pushed a commit to josix/airflow that referenced this pull request Jan 15, 2025
potiuk pushed a commit that referenced this pull request Jan 15, 2025
Co-authored-by: phi-friday <phi.friday@gmail.com>
@potiuk potiuk modified the milestones: Airflow 2.10.3, Airflow 2.10.5 Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants