Python3 pandas json encoding problem

[LilithWittmann]

I tried to use the charts function of airflow but every query I run returns an internal server error. I think it has something to do with json encoding of the pandas object.
My system:

• centos 7
• python 3.4
• airflow 1.5.1
• postgresql 9.4

Traceback (most recent call last):
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask_admin/base.py", line 68, in inner
    return self._run_view(f, *args, **kwargs)
  File "/var/www/airflow/env/lib/python3.4/site-packages/flask_admin/base.py", line 359, in _run_view
    return fn(self, *args, **kwargs)
  File "/var/www/airflow/env/lib/python3.4/site-packages/airflow/www/app.py", line 112, in decorated_function
    return f(*args, **kwargs)
  File "/var/www/airflow/env/lib/python3.4/site-packages/airflow/www/utils.py", line 96, in view_func
    return f(*args, **kwargs)
  File "/var/www/airflow/env/lib/python3.4/site-packages/airflow/www/app.py", line 607, in chart_data
    response=json.dumps(payload, indent=4, default=date_handler),
  File "/usr/lib64/python3.4/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.4/json/encoder.py", line 194, in encode
    chunks = list(chunks)
  File "/usr/lib64/python3.4/json/encoder.py", line 422, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib64/python3.4/json/encoder.py", line 396, in _iterencode_dict
    yield from chunks
  File "/usr/lib64/python3.4/json/encoder.py", line 396, in _iterencode_dict
    yield from chunks
  File "/usr/lib64/python3.4/json/encoder.py", line 317, in _iterencode_list
    yield from chunks
  File "/usr/lib64/python3.4/json/encoder.py", line 430, in _iterencode
    yield from _iterencode(o, _current_indent_level)
  File "/usr/lib64/python3.4/json/encoder.py", line 427, in _iterencode
    raise ValueError("Circular reference detected")
ValueError: Circular reference detected

This is the object that can't be encoded:

{'columns': [{'title': 'year'}, {'title': 'price'}], 'data': [[2000.0, 9900.2], [2010.0, 76666.2], [2011.0, 3432.3], [2012.0, 5444.2], [2013.0, 2345.332], [2014.0, 34444.33], [2015.0, 414.4027888224663]], 'index': [0, 1, 2, 3, 4, 5, 6]}

[mistercrunch]

Weird, how can json have a circular reference? it makes no sense to me... But with a bit of googling it looks like it's coming from the date_handler function:

http://stackoverflow.com/questions/14249115/serializing-output-to-json-valueerror-circular-reference-detected

[mistercrunch]

This may help:
#420

[LilithWittmann]

Ok after I found some other issues related to python3 I switched back to python2 /o\ (but there it works…)

[jlowin]

@LilithWittmann could you please create issues for any Py3 problems you encounter? We'll try to squash them.

[LilithWittmann]

@jlowin Ok ( e.g. #422, #414 )

[jlowin]

Thanks! Unfortunately mysql has been problematic for Py3 in the past but I don't use it here so I have trouble testing for compatibility. Let's see if anyone can help.

[jlowin]

Believe this was closed with #427, please reopen if not.