Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dag runs keep in running state (job not send to celery?) #860

Closed
JordyMoos opened this issue Jan 11, 2016 · 1 comment
Closed

Dag runs keep in running state (job not send to celery?) #860

JordyMoos opened this issue Jan 11, 2016 · 1 comment

Comments

@JordyMoos
Copy link
Contributor

Hi,

My dag seem to stop running because active dags is reached.
If i go to the Dag Runs page then i see that there are 16 dag runs in the running state.
But they do not show up in celery.

After checking the airflow-scheduler logs i notice something odd but i do not know what to do.

First let me show you the logs for a good dag run:

[2016-01-08 06:32:16,057] {jobs.py:455} INFO - Getting list of tasks to skip for active runs.
[2016-01-08 06:32:16,059] {jobs.py:470} INFO - Checking dependencies on 1 tasks instances, minus 0 skippable ones
--> [2016-01-08 06:32:16,062] {base_executor.py:34} INFO - Adding to queue: airflow run bier print_date 2015-07-12T00:00:00 --pickle 42 --local
[2016-01-08 06:32:16,073] {jobs.py:633} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-01-08 06:32:16,074] {jobs.py:636} INFO - Loop took: 0.065772 seconds
[2016-01-08 06:32:16,075] {models.py:222} INFO - Finding 'running' jobs without a recent heartbeat
[2016-01-08 06:32:16,075] {models.py:228} INFO - Failing jobs without heartbeat after 2016-01-08 06:30:01.075953
--> [2016-01-08 06:32:16,118] {celery_executor.py:62} INFO - [celery] queuing ('bier', 'print_date', datetime.datetime(2015, 7, 12, 0, 0)) through celery, queue=control
[2016-01-08 06:32:21,014] {jobs.py:507} INFO - Prioritizing 0 queued jobs

Here we see that the run went to celery.

But if i now check a dag that is still in the running state:

[2016-01-08 06:32:36,062] {jobs.py:455} INFO - Getting list of tasks to skip for active runs.
[2016-01-08 06:32:36,064] {jobs.py:470} INFO - Checking dependencies on 2 tasks instances, minus 0 skippable ones
--> [2016-01-08 06:32:36,072] {base_executor.py:34} INFO - Adding to queue: airflow run bier print_date 2015-07-16T00:00:00 --pickle 46 --local
[2016-01-08 06:32:36,085] {jobs.py:633} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-01-08 06:32:36,085] {jobs.py:636} INFO - Loop took: 0.076642 seconds
[2016-01-08 06:32:36,086] {models.py:222} INFO - Finding 'running' jobs without a recent heartbeat
[2016-01-08 06:32:36,087] {models.py:228} INFO - Failing jobs without heartbeat after 2016-01-08 06:30:21.087194
[2016-01-08 06:32:41,012] {jobs.py:507} INFO - Prioritizing 0 queued jobs
[2016-01-08 06:32:41,021] {models.py:197} INFO - Importing /usr/local/lib/airflow/dags/bier.py
[2016-01-08 06:32:41,025] {models.py:284} INFO - Loaded DAG <DAG: bier>
[2016-01-08 06:32:41,046] {models.py:2156} INFO - Checking state for <DagRun bier @ 2015-07-14 00:00:00: scheduled__2015-07-14T00:00:00, externally triggered: False>
[2016-01-08 06:32:41,047] {models.py:2156} INFO - Checking state for <DagRun bier @ 2015-07-16 00:00:00: scheduled__2015-07-16T00:00:00, externally triggered: False>
[2016-01-08 06:32:41,049] {models.py:2156} INFO - Checking state for <DagRun bier @ 2015-07-17 00:00:00: scheduled__2015-07-17T00:00:00, externally triggered: False>
[2016-01-08 06:32:41,051] {jobs.py:455} INFO - Getting list of tasks to skip for active runs.
[2016-01-08 06:32:41,052] {jobs.py:470} INFO - Checking dependencies on 3 tasks instances, minus 0 skippable ones
--> [2016-01-08 06:32:41,060] {base_executor.py:34} INFO - Adding to queue: airflow run bier print_date 2015-07-17T00:00:00 --pickle 47 --local
[2016-01-08 06:32:41,068] {jobs.py:633} INFO - Done queuing tasks, calling the executor's heartbeat
[2016-01-08 06:32:41,068] {jobs.py:636} INFO - Loop took: 0.061148 seconds
[2016-01-08 06:32:41,069] {models.py:222} INFO - Finding 'running' jobs without a recent heartbeat
[2016-01-08 06:32:41,070] {models.py:228} INFO - Failing jobs without heartbeat after 2016-01-08 06:30:26.070181
[2016-01-08 06:32:46,013] {jobs.py:507} INFO - Prioritizing 0 queued jobs

We see here that 2 dags are being queued but there is no log about sending it to celery.

Does anyone know what is going on?

Thanks in advance!

Jordy

@basebase
Copy link

Will you solve it?
Thanks!

@kaxil kaxil closed this as completed Mar 20, 2020
potiuk added a commit that referenced this issue Jan 19, 2021
* Adds automated user creation in the production image

This PR implements automated user creation for the production image
controlled by environment variables.

This is a solution for anyone who would like to make a quick test
of the production image and would like to:

* init/upgrade the DB automatically
* create a user

This is particularly useful for internal SQLite db initialization
but can also be used to initialize the user in docker-compose
or similar cases where there is no equivalent of init containers
that are usually used to perform the initialization.

Closes #860
kaxil pushed a commit that referenced this issue Jan 21, 2021
* Adds automated user creation in the production image

This PR implements automated user creation for the production image
controlled by environment variables.

This is a solution for anyone who would like to make a quick test
of the production image and would like to:

* init/upgrade the DB automatically
* create a user

This is particularly useful for internal SQLite db initialization
but can also be used to initialize the user in docker-compose
or similar cases where there is no equivalent of init containers
that are usually used to perform the initialization.

Closes #860

(cherry picked from commit bc026cf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants