datetime in parameters cause "TypeError: Object of type set is not JSON serializable" #19519

ajbosco · 2021-11-10T15:52:48Z

Apache Airflow version

2.2.2rc1

Operating System

Debian Buster (docker)

Versions of Apache Airflow Providers

n/a

Deployment

Other 3rd-party Helm chart

Deployment details

No response

What happened

The DAG parses and runs fine on 2.1.3, but when we upgrade to the latest version it fails to import:

Broken DAG: [/usr/local/airflow/dags/test_dag.py] Traceback (most recent call last):
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

What you expected to happen

No response

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

jedcunningham · 2021-11-10T15:56:51Z

This should be fixed in 2.2.2 via #19267. Can you test in 2.2.2rc1 and report your results in #19515? Thanks.

jedcunningham · 2021-11-10T16:14:41Z

@ajbosco, can you provide a DAG that reproduces this? The DAGs I have handy that use set and datetime params work.

dstandish · 2021-11-10T16:50:48Z

@dag.task(params={"a": {1, 2, 3}, "b": datetime.now()})
def datetime_param(value):
    print(value)

datetime_param("{{ params.a }} | {{ params.b }}")

could not repro with this task

potiuk · 2021-11-10T16:51:44Z

Try pendulum datetime @dstandish

jedcunningham · 2021-11-10T16:52:28Z

pendulum.datetime worked for me 🤷‍♂️

dstandish · 2021-11-10T16:52:36Z

no luck

@dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
def datetime_param(value):
    print(value)


datetime_param("{{ params.a }} | {{ params.b }}")

potiuk · 2021-11-10T16:53:25Z

We need example then @ajbosco :)

kaxil · 2021-11-10T16:55:12Z

Yup, let's just wait until then.

Also please double-check with the Airflow version in the Footer of the webserver or airflow CLI to confirm that you are using 2.2.2rc1

ajbosco · 2021-11-10T18:04:49Z

Welp....I think there was some issue in the database I was using. I started over from scratch and cannot reproduce this error either. My bad!

kaxil · 2021-11-10T18:06:18Z

@ajbosco Awesome, can you change your vote on the mailing list too, please :)

dstandish · 2021-11-10T18:17:27Z

@ajbosco, it's not impossible that you encountered some kind of issue with upgrade, deserializing dags that were serialized prior to 2.2.0.

If that's what you were seeing, it would be really good if you could let us know. I.e. we try to be backwards compatible but if there was an issue then that would be great if we could try to repro and fix. If you think that might be what was going on and you can repro LMK. I can try to check this too. What version are you upgrading from?

potiuk · 2021-11-10T18:26:08Z

Also I think we have a new "dags reserialize" coming - #19471 - which is really good idea because the "serialized" form of DAGs is pretty safe to delete and recreate.

I just wonder if this is not something that we should always do after/during upgrade "just in case". It might slow-down initial dag refresh, but at least we can be sure that everything in serialized dags is "correct".

potiuk · 2021-11-10T18:26:45Z

Exception being likely DAGs tha have been deleted already.

dstandish · 2021-11-10T18:27:53Z

I just wonder if this is not something that we should always do after/during upgrade "just in case". It might slow-down initial dag refresh, but at least we can be sure that everything in serialized dags is "correct".

This I agree with completely

JoranDox · 2022-09-23T09:12:24Z

Looks like it's back? Or at least something very similar (in this case a timedelta not being json serializable). Should I create a new ticket for this or just add my log here?

Ooops! Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)
find similar issues using:
- GitHub Discussions
- GitHub Issues
- Stack Overflow
- the usual search engine you use on a daily basis
if you run Airflow on a Managed Service, consider opening an issue using the service support channels
if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a bug report.
Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.9.14
Airflow version: 2.3.4+astro.2
Node: nebular-instrument-9608-webserver-6955c95644-t2d9z

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/site-packages/airflow/www/auth.py", line 46, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py", line 117, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py", line 80, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/www/views.py", line 2984, in graph
return self.render_template(
File "/usr/local/lib/python3.9/site-packages/airflow/www/views.py", line 709, in render_template
return super().render_template(
File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/baseviews.py", line 322, in render_template
return render_template(
File "/usr/local/lib/python3.9/site-packages/flask/templating.py", line 147, in render_template
return _render(app, template, context)
File "/usr/local/lib/python3.9/site-packages/flask/templating.py", line 130, in _render
rv = template.render(context)
File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.9/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/usr/local/lib/python3.9/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
{% from 'appbuilder/loading_dots.html' import loading_dots %}
File "/usr/local/lib/python3.9/site-packages/airflow/www/templates/airflow/dag.html", line 36, in top-level template code
{% set execution_date_arg = request.args.get('execution_date') %}
File "/usr/local/lib/python3.9/site-packages/astronomer/airflow/version_check/templates/astro-baselayout.html", line 1, in top-level template code
{% extends airflow_base_template %}
File "/usr/local/lib/python3.9/site-packages/airflow/www/templates/airflow/main.html", line 21, in top-level template code
{% from 'airflow/_messages.html' import show_message %}
File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 50, in top-level template code
{% block tail %}
File "/usr/local/lib/python3.9/site-packages/airflow/www/templates/airflow/graph.html", line 137, in block 'tail'
let taskInstances = {{ task_instances|tojson }};
File "/usr/local/lib/python3.9/site-packages/jinja2/filters.py", line 1688, in do_tojson
return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)
File "/usr/local/lib/python3.9/site-packages/jinja2/utils.py", line 658, in htmlsafe_json_dumps
dumps(obj, **kwargs)
File "/usr/local/lib/python3.9/site-packages/flask/json/provider.py", line 230, in dumps
return json.dumps(obj, **kwargs)
File "/usr/local/lib/python3.9/json/init.py", line 234, in dumps
return cls(
File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/json.py", line 108, in _default
raise TypeError(f"Object of type '{obj.class.name}' is not JSON serializable")
TypeError: Object of type 'timedelta' is not JSON serializable

dstandish · 2022-09-23T09:25:40Z

it's not clear that this this is from params. can you include minimal dag file example

JoranDox · 2022-09-23T12:07:38Z

Huh, strangely, without changing anything config-wise, it now works. I did encounter it on a dag that just changed (I added a task, which hadn't run yet). Could that have caused it?

dstandish · 2022-09-23T21:13:01Z

dunno 🤷

ajbosco added area:core kind:bug This is a clearly a bug labels Nov 10, 2021

jedcunningham closed this as completed Nov 10, 2021

jedcunningham reopened this Nov 10, 2021

jedcunningham mentioned this issue Nov 10, 2021

Sets in parameters cause "TypeError: Object of type set is not JSON serializable" #19096

Closed

2 tasks

ajbosco closed this as completed Nov 10, 2021

uranusjr mentioned this issue Nov 11, 2021

Always re-serialise DAGs after an upgrade #19533

Closed

2 tasks

potiuk mentioned this issue Nov 18, 2021

Trigger DAG re-serialization after upgrade #19637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

datetime in parameters cause "TypeError: Object of type set is not JSON serializable" #19519

datetime in parameters cause "TypeError: Object of type set is not JSON serializable" #19519

ajbosco commented Nov 10, 2021 •

edited by jedcunningham

Loading

jedcunningham commented Nov 10, 2021

jedcunningham commented Nov 10, 2021 •

edited

Loading

dstandish commented Nov 10, 2021 •

edited

Loading

potiuk commented Nov 10, 2021

jedcunningham commented Nov 10, 2021

dstandish commented Nov 10, 2021 •

edited

Loading

potiuk commented Nov 10, 2021

kaxil commented Nov 10, 2021

ajbosco commented Nov 10, 2021

kaxil commented Nov 10, 2021

dstandish commented Nov 10, 2021 •

edited

Loading

potiuk commented Nov 10, 2021

potiuk commented Nov 10, 2021

dstandish commented Nov 10, 2021

JoranDox commented Sep 23, 2022

Python version: 3.9.14
Airflow version: 2.3.4+astro.2
Node: nebular-instrument-9608-webserver-6955c95644-t2d9z

dstandish commented Sep 23, 2022

JoranDox commented Sep 23, 2022

dstandish commented Sep 23, 2022

datetime in parameters cause "TypeError: Object of type set is not JSON serializable" #19519

datetime in parameters cause "TypeError: Object of type set is not JSON serializable" #19519

Comments

ajbosco commented Nov 10, 2021 • edited by jedcunningham Loading

Apache Airflow version

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

What happened

What you expected to happen

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

jedcunningham commented Nov 10, 2021

jedcunningham commented Nov 10, 2021 • edited Loading

dstandish commented Nov 10, 2021 • edited Loading

potiuk commented Nov 10, 2021

jedcunningham commented Nov 10, 2021

dstandish commented Nov 10, 2021 • edited Loading

potiuk commented Nov 10, 2021

kaxil commented Nov 10, 2021

ajbosco commented Nov 10, 2021

kaxil commented Nov 10, 2021

dstandish commented Nov 10, 2021 • edited Loading

potiuk commented Nov 10, 2021

potiuk commented Nov 10, 2021

dstandish commented Nov 10, 2021

JoranDox commented Sep 23, 2022

Python version: 3.9.14 Airflow version: 2.3.4+astro.2 Node: nebular-instrument-9608-webserver-6955c95644-t2d9z

dstandish commented Sep 23, 2022

JoranDox commented Sep 23, 2022

dstandish commented Sep 23, 2022

ajbosco commented Nov 10, 2021 •

edited by jedcunningham

Loading

jedcunningham commented Nov 10, 2021 •

edited

Loading

dstandish commented Nov 10, 2021 •

edited

Loading

dstandish commented Nov 10, 2021 •

edited

Loading

dstandish commented Nov 10, 2021 •

edited

Loading

Python version: 3.9.14
Airflow version: 2.3.4+astro.2
Node: nebular-instrument-9608-webserver-6955c95644-t2d9z