You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When Transformer recognizes the type to be str, it will convert the (sub) object to str type. The issue is, if such (sub) object's original type is dict, the current method of converting to str produces JSON incompatible string:
When Transformer recognizes the type to be str, it will convert the (sub) object to str type. The issue is, if such (sub) object's original type is dict, the current method of converting to str produces JSON incompatible string:
singer-python/singer/transform.py
Line 288 in 6472683
This results in the conversion from a dict
to
instead of
str(data)
conversion seems to produce problems with escape characters as well.I am wondering if it is acceptable to replace
str(data)
withjson.dumps(data)
One may argue that the tap should fully specify the schema so that the (sub)object is written out as dict. However, many of the Rest API often includes the field whose schema is not static.
An example is Github API's event object. event.payload is a (dict) object, but the schema depends on the event type.
https://docs.github.com/en/free-pro-team@latest/developers/webhooks-and-events/github-event-types#event-object-common-properties
In fact, I discovered this issue while I was debugging tap-github's usage of Transformer:
https://github.com/singer-io/tap-github/blob/master/tap_github/__init__.py#L361
If the str conversion was done through json.dumps, it would have been possible to parse JSON in the target datastore such as BigQuery and Redshift.
The text was updated successfully, but these errors were encountered: