Refactor serve_logs with FastAPI by jason810496 · Pull Request #52581 · apache/airflow

jason810496 · 2025-06-30T16:17:46Z

closes: #52526
related: https://lists.apache.org/thread/hfr8q85rgr6knpp5wblbz301ysnmzhht

Why

Move Flask ( or FAB you said ) out of airflow-core dependency

What

Replace Flask's send_from_directory fastapi_app.mount with JWTAuthStaticFiles ( which inherent from FastAPI's StaticFiles and extend the existed authorization. reference from fastapi/fastapi#858 (comment) )

airflow-core/src/airflow/utils/serve_logs.py

jason810496

I though there isn't Flask'ssend_from_directory alternative in FastAPI.
So I implement the validation for file path in first try ( and the security check is angry ).

Fortunately I found the gist for having authorization for FastAPI's StaticFiles. ( linked in the PR description )

airflow-core/src/airflow/utils/serve_logs.py

potiuk · 2025-07-01T07:32:53Z

One other thing - I think we can get rid of those dependencies from airflow-core/pyproject.toml:

   # We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
    "flask>=2.1.1",
    # We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
    "gunicorn>=20.1.0",

jason810496 · 2025-07-02T16:17:36Z

Thanks @potiuk @pierrejeambrun for the review! I will resolve the comments tomorow.

potiuk · 2025-07-02T19:15:32Z

Thanks @potiuk @pierrejeambrun for the review! I will resolve the comments tomorow.

Cool!

airflow-core/src/airflow/utils/serve_logs/log_server.py

airflow-core/src/airflow/utils/serve_logs/core.py

jason810496 · 2025-07-03T16:24:45Z

One other thing - I think we can get rid of those dependencies from airflow-core/pyproject.toml:

   # We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
    "flask>=2.1.1",
    # We could get rid of flask and gunicorn if we replace serve_logs with a starlette + unicorn
    "gunicorn>=20.1.0",

Just found out that we can't remove gunicorn that we are still using gunicorn for Daemonized the API server if user pass --daemon flag.

airflow/airflow-core/src/airflow/cli/commands/api_server_command.py

Lines 87 to 91 in edf86df

    
           else: 
        
               if args.daemon: 
        
                   daemonize() 
        
                   log.info("Daemonized the API server process PID: %s", os.getpid())

I can raise another PR to add the daemonize util as a new airflow.util and remove the gunicorn package before merging this one.

potiuk · 2025-07-08T10:31:33Z

👀 :)

Fix test_invalid_characters_handled Refactor with StaticFiles

pierrejeambrun · 2025-07-09T14:36:13Z

Super cool!

* Refactor serve_logs with FastAPI Fix test_invalid_characters_handled Refactor with StaticFiles * Remove details for 403 forbidden response * Replace Gunicorn with Uvicorn * Fix uvicorn multiple workers error * Remove flask and gunicorn from airflow-core/pyproject.toml * Remove old serve_logs module and add gunicorn to spelling_wordlist

The `airflow scheduler` fails to start with this error when the logs directory doesn't exist: ``` RuntimeError: Directory '/root/airflow/logs' does not exist ``` This regression was introduced in [apache#52581](apache#52581) when serve_logs was refactored from Flask to FastAPI. **Module Import Order Dependency**: FastAPI's `StaticFiles` validates directory existence at app creation time, but this happens at **module import time** before Airflow's initialization sequence: ```python uvicorn.run("airflow.utils.serve_logs.log_server:app") # imports module ↓ app = create_app() # runs immediately at module level ↓ StaticFiles(directory=log_directory) # FAILS: directory doesn't exist yet ``` **Expected initialization order**: ``` settings.initialize() → configure_logging() → init_log_folder() → create_app() ``` **Actual order with eager loading**: ``` import log_server → create_app() → FAIL (directory doesn't exist) ``` --- Without this fix, we see following error when running `airflow standalone` fresh in an isolated container: Error 1: ``` scheduler | Traceback (most recent call last): scheduler | File "/.venv/bin/airflow", line 10, in <module> scheduler | sys.exit(main()) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/__main__.py", line 55, in main scheduler | args.func(args) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/cli.py", line 114, in wrapper scheduler | return f(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler scheduler | run_command_with_daemon_option( scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option scheduler | callback() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda> scheduler | callback=lambda: _run_scheduler_job(args), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 42, in _run_scheduler_job scheduler | with _serve_logs(args.skip_serve_logs), _serve_health_check(enable_health_check): scheduler | File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__ scheduler | return next(self.gen) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in _serve_logs scheduler | from airflow.utils.serve_logs import serve_logs scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/__init__.py", line 20, in <module> scheduler | from airflow.utils.serve_logs.log_server import create_app scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 160, in <module> scheduler | app = create_app() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 153, in create_app scheduler | JWTAuthStaticFiles(directory=log_directory, html=False), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 49, in __init__ scheduler | super().__init__(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/starlette/staticfiles.py", line 56, in __init__ scheduler | raise RuntimeError(f"Directory '{directory}' does not exist") scheduler | RuntimeError: Directory '/root/airflow/logs' does not exist ``` Error 2: This was because we had harcoded 2 workers! but this isn't needed ``` scheduler | Process SpawnProcess-1:53: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor scheduler | Process SpawnProcess-1:54: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor triggerer | Process SpawnProcess-1:53: triggerer | Traceback (most recent call last): triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap triggerer | self.run() triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run triggerer | self._target(*self._args, **self._kwargs) triggerer | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started triggerer | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage triggerer | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen triggerer | return io.open(fd, mode, buffering, encoding, *args, **kwargs) triggerer | OSError: [Errno 9] Bad file descriptor ``` This implementation follows the same pattern as Airflow's main API: | Component | Pattern | File | |-----------|---------|------| | **Main API** | `cached_app()` with global caching | `api_fastapi/app.py` | | **serve_logs** | `get_app()` with global caching | `utils/serve_logs/log_server.py` | Both use lazy initialization with global app caching, ensuring proper startup order while maintaining performance. --- **Fixes**: Scheduler startup failures with LocalExecutor **Related-to**: d8a3ad2

Remove workers parameter from `uvicorn.run()` in `serve_logs` to fix file descriptor errors when scheduler starts serve_logs as a subprocess. The original implementation used `workers=2` (copied from Gunicorn) but this caused multiprocessing issues in containerized environments. Also implemented lazy app loading via `get_app()` function for better initialization order and architectural consistency with main API. The primary fix addresses: OSError: [Errno 9] Bad file descriptor. Secondary improvement ensures proper initialization timing. This regression was introduced in [apache#52581](apache#52581) when serve_logs was refactored from Flask to FastAPI. --- Without this fix, we see following error when running `airflow standalone` fresh in an isolated container: Error 1: ``` scheduler | Traceback (most recent call last): scheduler | File "/.venv/bin/airflow", line 10, in <module> scheduler | sys.exit(main()) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/__main__.py", line 55, in main scheduler | args.func(args) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/cli.py", line 114, in wrapper scheduler | return f(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler scheduler | run_command_with_daemon_option( scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option scheduler | callback() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda> scheduler | callback=lambda: _run_scheduler_job(args), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 42, in _run_scheduler_job scheduler | with _serve_logs(args.skip_serve_logs), _serve_health_check(enable_health_check): scheduler | File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__ scheduler | return next(self.gen) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in _serve_logs scheduler | from airflow.utils.serve_logs import serve_logs scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/__init__.py", line 20, in <module> scheduler | from airflow.utils.serve_logs.log_server import create_app scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 160, in <module> scheduler | app = create_app() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 153, in create_app scheduler | JWTAuthStaticFiles(directory=log_directory, html=False), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 49, in __init__ scheduler | super().__init__(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/starlette/staticfiles.py", line 56, in __init__ scheduler | raise RuntimeError(f"Directory '{directory}' does not exist") scheduler | RuntimeError: Directory '/root/airflow/logs' does not exist ``` Error 2: This was because we had harcoded 2 workers! but this isn't needed ``` scheduler | Process SpawnProcess-1:53: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor scheduler | Process SpawnProcess-1:54: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor triggerer | Process SpawnProcess-1:53: triggerer | Traceback (most recent call last): triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap triggerer | self.run() triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run triggerer | self._target(*self._args, **self._kwargs) triggerer | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started triggerer | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage triggerer | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen triggerer | return io.open(fd, mode, buffering, encoding, *args, **kwargs) triggerer | OSError: [Errno 9] Bad file descriptor ```

Remove workers parameter from `uvicorn.run()` in `serve_logs` to fix file descriptor errors when scheduler starts serve_logs as a subprocess. The original implementation used `workers=2` (copied from Gunicorn) but this caused multiprocessing issues in containerized environments. Also implemented lazy app loading via `get_app()` function for better initialization order and architectural consistency with main API. The primary fix addresses: OSError: [Errno 9] Bad file descriptor. Secondary improvement ensures proper initialization timing. This regression was introduced in [#52581](#52581) when serve_logs was refactored from Flask to FastAPI. --- Without this fix, we see following error when running `airflow standalone` fresh in an isolated container: Error 1: ``` scheduler | Traceback (most recent call last): scheduler | File "/.venv/bin/airflow", line 10, in <module> scheduler | sys.exit(main()) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/__main__.py", line 55, in main scheduler | args.func(args) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/cli.py", line 114, in wrapper scheduler | return f(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler scheduler | run_command_with_daemon_option( scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option scheduler | callback() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda> scheduler | callback=lambda: _run_scheduler_job(args), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 42, in _run_scheduler_job scheduler | with _serve_logs(args.skip_serve_logs), _serve_health_check(enable_health_check): scheduler | File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__ scheduler | return next(self.gen) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in _serve_logs scheduler | from airflow.utils.serve_logs import serve_logs scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/__init__.py", line 20, in <module> scheduler | from airflow.utils.serve_logs.log_server import create_app scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 160, in <module> scheduler | app = create_app() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 153, in create_app scheduler | JWTAuthStaticFiles(directory=log_directory, html=False), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 49, in __init__ scheduler | super().__init__(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/starlette/staticfiles.py", line 56, in __init__ scheduler | raise RuntimeError(f"Directory '{directory}' does not exist") scheduler | RuntimeError: Directory '/root/airflow/logs' does not exist ``` Error 2: This was because we had harcoded 2 workers! but this isn't needed ``` scheduler | Process SpawnProcess-1:53: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor scheduler | Process SpawnProcess-1:54: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor triggerer | Process SpawnProcess-1:53: triggerer | Traceback (most recent call last): triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap triggerer | self.run() triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run triggerer | self._target(*self._args, **self._kwargs) triggerer | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started triggerer | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage triggerer | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen triggerer | return io.open(fd, mode, buffering, encoding, *args, **kwargs) triggerer | OSError: [Errno 9] Bad file descriptor ```

…e#55443) Remove workers parameter from `uvicorn.run()` in `serve_logs` to fix file descriptor errors when scheduler starts serve_logs as a subprocess. The original implementation used `workers=2` (copied from Gunicorn) but this caused multiprocessing issues in containerized environments. Also implemented lazy app loading via `get_app()` function for better initialization order and architectural consistency with main API. The primary fix addresses: OSError: [Errno 9] Bad file descriptor. Secondary improvement ensures proper initialization timing. This regression was introduced in [apache#52581](apache#52581) when serve_logs was refactored from Flask to FastAPI. --- Without this fix, we see following error when running `airflow standalone` fresh in an isolated container: Error 1: ``` scheduler | Traceback (most recent call last): scheduler | File "/.venv/bin/airflow", line 10, in <module> scheduler | sys.exit(main()) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/__main__.py", line 55, in main scheduler | args.func(args) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/cli.py", line 114, in wrapper scheduler | return f(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler scheduler | run_command_with_daemon_option( scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option scheduler | callback() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda> scheduler | callback=lambda: _run_scheduler_job(args), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 42, in _run_scheduler_job scheduler | with _serve_logs(args.skip_serve_logs), _serve_health_check(enable_health_check): scheduler | File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__ scheduler | return next(self.gen) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in _serve_logs scheduler | from airflow.utils.serve_logs import serve_logs scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/__init__.py", line 20, in <module> scheduler | from airflow.utils.serve_logs.log_server import create_app scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 160, in <module> scheduler | app = create_app() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 153, in create_app scheduler | JWTAuthStaticFiles(directory=log_directory, html=False), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 49, in __init__ scheduler | super().__init__(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/starlette/staticfiles.py", line 56, in __init__ scheduler | raise RuntimeError(f"Directory '{directory}' does not exist") scheduler | RuntimeError: Directory '/root/airflow/logs' does not exist ``` Error 2: This was because we had harcoded 2 workers! but this isn't needed ``` scheduler | Process SpawnProcess-1:53: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor scheduler | Process SpawnProcess-1:54: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor triggerer | Process SpawnProcess-1:53: triggerer | Traceback (most recent call last): triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap triggerer | self.run() triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run triggerer | self._target(*self._args, **self._kwargs) triggerer | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started triggerer | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage triggerer | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen triggerer | return io.open(fd, mode, buffering, encoding, *args, **kwargs) triggerer | OSError: [Errno 9] Bad file descriptor ```

Remove workers parameter from `uvicorn.run()` in `serve_logs` to fix file descriptor errors when scheduler starts serve_logs as a subprocess. The original implementation used `workers=2` (copied from Gunicorn) but this caused multiprocessing issues in containerized environments. Also implemented lazy app loading via `get_app()` function for better initialization order and architectural consistency with main API. The primary fix addresses: OSError: [Errno 9] Bad file descriptor. Secondary improvement ensures proper initialization timing. This regression was introduced in [#52581](apache/airflow#52581) when serve_logs was refactored from Flask to FastAPI. --- Without this fix, we see following error when running `airflow standalone` fresh in an isolated container: Error 1: ``` scheduler | Traceback (most recent call last): scheduler | File "/.venv/bin/airflow", line 10, in <module> scheduler | sys.exit(main()) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/__main__.py", line 55, in main scheduler | args.func(args) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/cli.py", line 114, in wrapper scheduler | return f(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function scheduler | return func(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 52, in scheduler scheduler | run_command_with_daemon_option( scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option scheduler | callback() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 55, in <lambda> scheduler | callback=lambda: _run_scheduler_job(args), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 42, in _run_scheduler_job scheduler | with _serve_logs(args.skip_serve_logs), _serve_health_check(enable_health_check): scheduler | File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__ scheduler | return next(self.gen) scheduler | File "/.venv/lib/python3.10/site-packages/airflow/cli/commands/scheduler_command.py", line 62, in _serve_logs scheduler | from airflow.utils.serve_logs import serve_logs scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/__init__.py", line 20, in <module> scheduler | from airflow.utils.serve_logs.log_server import create_app scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 160, in <module> scheduler | app = create_app() scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 153, in create_app scheduler | JWTAuthStaticFiles(directory=log_directory, html=False), scheduler | File "/.venv/lib/python3.10/site-packages/airflow/utils/serve_logs/log_server.py", line 49, in __init__ scheduler | super().__init__(*args, **kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/starlette/staticfiles.py", line 56, in __init__ scheduler | raise RuntimeError(f"Directory '{directory}' does not exist") scheduler | RuntimeError: Directory '/root/airflow/logs' does not exist ``` Error 2: This was because we had harcoded 2 workers! but this isn't needed ``` scheduler | Process SpawnProcess-1:53: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor scheduler | Process SpawnProcess-1:54: scheduler | Traceback (most recent call last): scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap scheduler | self.run() scheduler | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run scheduler | self._target(*self._args, **self._kwargs) scheduler | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started scheduler | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage scheduler | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen scheduler | return io.open(fd, mode, buffering, encoding, *args, **kwargs) scheduler | OSError: [Errno 9] Bad file descriptor triggerer | Process SpawnProcess-1:53: triggerer | Traceback (most recent call last): triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap triggerer | self.run() triggerer | File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run triggerer | self._target(*self._args, **self._kwargs) triggerer | File "/.venv/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 73, in subprocess_started triggerer | sys.stdin = os.fdopen(stdin_fileno) # pragma: full coverage triggerer | File "/usr/local/lib/python3.10/os.py", line 1030, in fdopen triggerer | return io.open(fd, mode, buffering, encoding, *args, **kwargs) triggerer | OSError: [Errno 9] Bad file descriptor ``` GitOrigin-RevId: 1d9e6c0fc96bb6b507481b4d721c7f0c350d6981

jason810496 self-assigned this Jun 30, 2025

jason810496 added the full tests needed We need to run full set of tests for this PR to merge label Jun 30, 2025

github-advanced-security bot found potential problems Jun 30, 2025

View reviewed changes

jason810496 force-pushed the refactor/logging/replace-flask-serve-log-with-fastapi branch from 8be599f to 99651ed Compare June 30, 2025 18:00

jason810496 requested review from pierrejeambrun and potiuk June 30, 2025 18:01

jason810496 force-pushed the refactor/logging/replace-flask-serve-log-with-fastapi branch from 99651ed to b5facdf Compare June 30, 2025 18:02

jason810496 marked this pull request as ready for review June 30, 2025 18:02

jason810496 commented Jun 30, 2025

View reviewed changes

airflow-core/src/airflow/utils/serve_logs.py Show resolved Hide resolved

airflow-core/src/airflow/utils/serve_logs.py Show resolved Hide resolved

airflow-core/src/airflow/utils/serve_logs.py Outdated Show resolved Hide resolved