Bug: API Trigger Results In Task Instances With NULL State In DB (v2.10.5)

by ADMIN 75 views

Bug: API Trigger results in Task Instances with NULL state in DB (v2.10.5)

Apache Airflow is a powerful workflow management platform that enables users to create, schedule, and monitor complex workflows. However, like any complex software, Airflow is not immune to bugs and issues. In this article, we will discuss a bug that was reported in Airflow version 2.10.5, where API triggers result in task instances with NULL state in the database.

When triggering a DAG run using the Airflow REST API (e.g., via a script calling the /dags/{dag_id}/dagRuns endpoint), the task instances created in the database (task_instance table) are inserted with a state of NULL instead of a runnable state like 'queued' or 'scheduled'. This prevents the Airflow scheduler from ever identifying these tasks as ready for execution, leaving the DAG run stuck indefinitely in the 'queued' state in the UI.

Analysis of PostgreSQL logs confirms the INSERT INTO task_instance query generated by Airflow is missing the state and queued_dttm columns from the list of columns being inserted. This is a critical issue that needs to be addressed to ensure that task instances are created with the correct state and queued_dttm values.

To reproduce this issue, follow these steps:

Set up Airflow

  • Install Apache Airflow v2.10.5 in a Python virtual environment with PostgreSQL as the database backend and LocalExecutor.
  • Ensure the database is initialized (airflow db init).
  • Configure Airflow:
    • Update your airflow.cfg (or environment variables) to correctly point sql_alchemy_conn to your PostgreSQL database.
    • Set executor = LocalExecutor.
    • Set the core logging level to DEBUG: [core] logging_level = DEBUG.

Configure PostgreSQL Verbose Logging

  • Edit your postgresql.conf file (/etc/postgresql/16/main/postgresql.conf or similar).
  • Set log_connections = on.
  • Set log_disconnections = on.
  • Set log_min_duration_statement = 0ms (or log_statement = 'all').
  • Save the file and reload or restart the PostgreSQL service (sudo systemctl reload postgresql or sudo systemctl restart postgresql).

Add a Test DAG

  • Place a simple DAG file designed for API triggers into your dags_folder. This DAG should have schedule_interval=None and define at least one task.

Start Airflow Components

  • Start the Airflow scheduler: airflow scheduler (or using your systemd service sudo systemctl start airflow-scheduler).
  • Start the Airflow webserver: airflow webserver (or using your systemd service sudo systemctl start airflow-webserver).

Trigger the DAG via API

  • Use a tool like curl or a Python script (like your main.py) to trigger the DAG using the Airflow REST API endpoint /api/v1/dags/{dag_id}/dagRuns. You'll need to pass any params your DAG expects in the request body.

Observe Behavior

  • Immediately after triggering, check the Airflow UI. The new DAG run will appear and stay in the queued state.
  • Watch the Airflow scheduler logs (journalctl -u airflow-scheduler.service -f). Look for DEBUG - No tasks to consider for execution. messages.
  • Watch the PostgreSQL logs (sudo tail -f /path/to/postgresql.log). Look for the INSERT INTO task_instance statement generated for the new run ID.
  • Use psql to query the task_instance table for the new run ID and check the state column (SELECT task_id, state, run_id FROM task_instance WHERE run_id = 'your_new_run_id';).

The DAG run remains stuck in the 'queued' state. Scheduler logs show No tasks to consider for execution. PostgreSQL logs show the INSERT INTO task_instance statement is missing the state and queued_dttm columns. Direct database query shows the task instances have state=NULL.

Distributor ID: Ubuntu Description: Ubuntu 24.04.2 LTS Release: 24.04 Codename: noble

  • apache-airflow-providers-amazon 9.4.0
  • apache-airflow-providers-common-compat 1.5.1
  • apache-airflow-providers-common-io 1.2.0
  • apache-airflow-providers-common-sql 1.24.0
  • apache-airflow-providers-fab 1.5.3
  • apache-airflow-providers-ftp 3.7.0
  • apache-airflow-providers-http 4.8.0
  • apache-airflow-providers-imap 3.5.0
  • apache-airflow-providers-postgres 5.10.0
  • apache-airflow-providers-smtp 2.0.1
  • apache-airflow-providers-sqlite 3.7.0

Official Apache Airflow Helm Chart

No response

No response

  • [ ] Yes I am willing to submit a PR!
  • [x] I agree to follow this project's Code of Conduct
    Bug: API Trigger results in Task Instances with NULL state in DB (v2.10.5) - Q&A

A: When triggering a DAG run using the Airflow REST API, the task instances created in the database are inserted with a state of NULL instead of a runnable state like 'queued' or 'scheduled'. This prevents the Airflow scheduler from ever identifying these tasks as ready for execution, leaving the DAG run stuck indefinitely in the 'queued' state in the UI.

A: To reproduce this issue, follow these steps:

  1. Set up Airflow with PostgreSQL as the database backend and LocalExecutor.
  2. Configure Airflow to correctly point sql_alchemy_conn to your PostgreSQL database.
  3. Set the core logging level to DEBUG.
  4. Configure PostgreSQL verbose logging.
  5. Add a test DAG with schedule_interval=None and define at least one task.
  6. Start the Airflow scheduler and webserver.
  7. Trigger the DAG via API using a tool like curl or a Python script.
  8. Observe the behavior in the Airflow UI, scheduler logs, and PostgreSQL logs.

A: The DAG run remains stuck in the 'queued' state. Scheduler logs show No tasks to consider for execution. PostgreSQL logs show the INSERT INTO task_instance statement is missing the state and queued_dttm columns. Direct database query shows the task instances have state=NULL.

A: The operating system affected is Ubuntu 24.04.2 LTS. The Airflow provider versions affected are:

  • apache-airflow-providers-amazon 9.4.0
  • apache-airflow-providers-common-compat 1.5.1
  • apache-airflow-providers-common-io 1.2.0
  • apache-airflow-providers-common-sql 1.24.0
  • apache-airflow-providers-fab 1.5.3
  • apache-airflow-providers-ftp 3.7.0
  • apache-airflow-providers-http 4.8.0
  • apache-airflow-providers-imap 3.5.0
  • apache-airflow-providers-postgres 5.10.0
  • apache-airflow-providers-smtp 2.0.1
  • apache-airflow-providers-sqlite 3.7.0

A: No, this issue is not related to the deployment method. It is a bug in Airflow 2.10.5 that affects API triggers.

A: Yes, if you are willing to submit a PR, please follow the project's Code of Conduct and submit a PR with the fix.

A: The next step is to submit a PR with the fix and wait for review and approval from the Airflow community.