Skip to content

Commit

Permalink
Remove executor error unit test
Browse files Browse the repository at this point in the history
This unit test was designed to trap unusual errors when setting up an
Executor and therefore relied on being able to create just such an
error. The previous version relied on the fact that dag_ids are fragile
but moving to deferred dag assignment fixed that fragility and “broke”
the unit test. The only other solution I’ve found so far is to take
advantage of the fact that the `pool` attribute is accessed exactly
twice when running a task and putting a 1/0 payload in an overloaded
`pool` property. But that’s too fragile to make a unit test because no
one will be able to figure out why accessing `pool` elsewhere in
airflow makes this unit test fail. For the time being, I’m removing the
unit test.
  • Loading branch information
jlowin committed Apr 12, 2016
1 parent fb0c577 commit 9b6c84d
Show file tree
Hide file tree
Showing 2 changed files with 0 additions and 84 deletions.
49 changes: 0 additions & 49 deletions tests/dags/test_raise_executor_error.py

This file was deleted.

35 changes: 0 additions & 35 deletions tests/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,41 +71,6 @@ def test_backfill_examples(self):
ignore_first_depends_on_past=True)
job.run()

def test_trap_executor_error(self):
"""
Test that errors setting up tasks (before tasks run) are caught.
Executors run tasks with the `airflow run` command. If a task runs,
its state (success, failure, or other) is stored in the database and
`airflow run` exits without error. However, if an error is raised
before the task runs, then the task won't be able to update its status.
This can put the executor into an infinite loop of trying to run the
task.
To counteract that, the executor traps errors coming from
`airflow run` (essentially looking for returncode != 0).
This unit test creates such an error by trying
to run a subdag whose dag_id has changed and therefore can't be
found. If the trap is working properly, the error will be caught
and the Backfill will report failures. If the trap is not working,
the job will run infinitely (the unit test uses a timeout to protect
against that case).
Test for https://github.com/airbnb/airflow/pull/1220
"""
dag = self.dagbag.get_dag('test_raise_executor_error')
dag.clear()
job = BackfillJob(
dag=dag,
start_date=DEFAULT_DATE,
end_date=DEFAULT_DATE)
# run with timeout because this creates an infinite loop if not
# caught
def run_with_timeout():
with timeout(seconds=30):
job.run()
self.assertRaises(AirflowException, run_with_timeout)

def test_backfill_pooled_tasks(self):
"""
Test that queued tasks are executed by BackfillJob
Expand Down

0 comments on commit 9b6c84d

Please sign in to comment.