Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A dag's schedule interval can no longer be an instance of dateutils.relativedelta #19416

Closed
1 of 2 tasks
rcheatham-q opened this issue Nov 4, 2021 · 1 comment · Fixed by #19418
Closed
1 of 2 tasks
Labels
affected_version:2.2 Issues Reported for 2.2 area:core kind:bug This is a clearly a bug

Comments

@rcheatham-q
Copy link

Apache Airflow version

2.2.1 (latest released)

Operating System

debian

Versions of Apache Airflow Providers

apache-airflow==2.2.1
apache-airflow-providers-amazon==2.3.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.0.0
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-jira==2.0.1
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.2.0

Deployment

Other Docker-based deployment

Deployment details

Dask executor, custom-built Docker images, postgres 12.7 backend

What happened

I upgraded Airflow from 2.0.2 to 2.2.1, and some DAGs I have that used dateutils.relativedelta objects as schedule intervals stopped running

What you expected to happen

The code for the schedule_interval parameter of the DAG constructor indicates that a relativedelta object is allowed, so I expected the DAG to be correctly parsed and scheduled.

How to reproduce

Create a DAG that has a relativedelta object as its schedule interval, and it will not appear in the UI or be scheduled.

Anything else

Here is the code that causes the failure within the PR where it was introduced: link

Here are the logs for the exception, found in the scheduler logs for the file that contains the offending DAG

ERROR   | {dagbag.py:528} - 'relativedelta' object has no attribute 'total_seconds'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 515, in collect_dags
    found_dags = self.process_file(filepath, only_if_updated=only_if_updated, safe_mode=safe_mode)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 298, in process_file
    found_dags = self._process_modules(filepath, mods, file_last_changed_on_disk)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 401, in _process_modules
    dag.timetable.validate()
  File "/usr/local/lib/python3.9/site-packages/airflow/timetables/interval.py", line 274, in validate
    if self._delta.total_seconds() <= 0:
AttributeError: 'relativedelta' object has no attribute 'total_seconds'

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@rcheatham-q rcheatham-q added area:core kind:bug This is a clearly a bug labels Nov 4, 2021
@rcheatham-q rcheatham-q changed the title A dag's scheduler interval can no longer be an instance of dateutils.relativedelta A dag's schedule interval can no longer be an instance of dateutils.relativedelta Nov 4, 2021
@kaxil
Copy link
Member

kaxil commented Nov 5, 2021

cc @uranusjr @ashb

@eladkal eladkal added the affected_version:2.2 Issues Reported for 2.2 label Nov 5, 2021
kaxil pushed a commit that referenced this issue Nov 5, 2021
…val`` (#19418)

Also add relativedelta to timetable test cases to ensure this does not
regress.

Fix #19416
jedcunningham pushed a commit that referenced this issue Nov 5, 2021
…val`` (#19418)

Also add relativedelta to timetable test cases to ensure this does not
regress.

Fix #19416

(cherry picked from commit b590cc8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:2.2 Issues Reported for 2.2 area:core kind:bug This is a clearly a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants