dag-processor failed to start in docker

[Taragolis]

Apache Airflow version

2.3.0 (latest released)

What happened

Standalone DagProcessor which run in Apache Airflow Production Docker Image failed with error

airflow-dag-processor_1  | 
airflow-dag-processor_1  | Traceback (most recent call last):
airflow-dag-processor_1  |   File "/home/airflow/.local/bin/airflow", line 8, in <module>
airflow-dag-processor_1  |     sys.exit(main())
airflow-dag-processor_1  |   File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 38, in main
airflow-dag-processor_1  |     args.func(args)
airflow-dag-processor_1  |   File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 51, in command
airflow-dag-processor_1  |     return func(*args, **kwargs)
airflow-dag-processor_1  |   File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
airflow-dag-processor_1  |     return f(*args, **kwargs)
airflow-dag-processor_1  |   File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_processor_command.py", line 80, in dag_processor
airflow-dag-processor_1  |     manager.start()
airflow-dag-processor_1  |   File "/home/airflow/.local/lib/python3.7/site-packages/airflow/dag_processing/manager.py", line 475, in start
airflow-dag-processor_1  |     os.setpgid(0, 0)
airflow-dag-processor_1  | PermissionError: [Errno 1] Operation not permitted

This error not happen if directly run in host system by airflow dag-processor

Seems like this issue happen because when we run in Apache Airflow Production Docker Image airflow process is session leader, and according to man setpgid

ERRORS
     setpgid() will fail and the process group will not be altered if:
     [EPERM]            The process indicated by the pid argument is a session leader.

What you think should happen instead

dag-processor should start in docker without error

How to reproduce

1. Use simple docker-compose file which use official Airflow 2.3.0 image

# docker-compose-dag-processor.yaml

version: '3'

volumes:
  postgres-db-volume:
  aiflow-logs-volume:

x-airflow-common:
  &airflow-common
  image: apache/airflow:2.3.0-python3.7
  environment:
    AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR: 'True'
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:insecurepassword@postgres/airflow
  volumes:
    - aiflow-logs-volume:/opt/airflow/logs
    - ${PWD}/dags:/opt/airflow/dags
  user: "${AIRFLOW_UID:-50000}:0"
  extra_hosts:
    - "host.docker.internal:host-gateway"

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: insecurepassword
      POSTGRES_DB: airflow
    ports:
      - 55432:5432
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: unless-stopped

  airflow-upgrade-db:
    <<: *airflow-common
    command: ["db", "upgrade"]
    depends_on:
      postgres:
        condition: service_healthy

  airflow-dag-processor:
    <<: *airflow-common
    command: dag-processor
    restart: unless-stopped
    depends_on:
      airflow-upgrade-db:
        condition: service_completed_successfully

2. docker-compose -f docker-compose-dag-processor.yaml up

Operating System

macOS Monterey 12.3.1

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

Docker: 20.10.12
docker-compose: 1.29.2

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct