Stitch integration error

[SeanFLynch]

Hi,

We're getting the following errror with the zendesk chat stitch integration:

Error persisting data for table "shortcuts": Record 0 did not conform to schema:
#: #: no subschema matched out of the total 2 subschemas
#: expected: null, found: JSONObject
#: extraneous key [scope] is not permitted

Any help with this would be much appreciated?

Sean.

[KAllan357]

Hi @SeanFLynch,

I think we might need a little more information here.

The first part indicates to me that some key in the shortcuts data maps to a JSON object that the schema is not supporting.

The second error is clear - the JSON Schema for the shortcuts table does not support the "scope" field. We should add more schema in this area - https://github.com/singer-io/tap-zendesk-chat/blob/master/tap_zendesk_chat/schemas/shortcuts.json#L25.

I think it should look like:

"scope": {
      "type": [
        "null",
        "string"
      ]
    }

[SeanFLynch]

@KAllan357 Here are the full logs:

2018-08-07 00:02:32,735Z main - INFO Running tap-zendesk-chat version 0.1.12 and target-stitch version 1.7.4
2018-08-07 00:02:32,884Z main - INFO Starting tap to discover schemas: tap-env/bin/tap-zendesk-chat --config /tmp/tap_discover_config.json --discover
2018-08-07 00:02:33,276Z tap - INFO HTTP request to "departments" endpoint took 0.214s, returned status code 200
2018-08-07 00:02:33,468Z tap - INFO HTTP request to "account" endpoint took 0.191s, returned status code 403
2018-08-07 00:02:33,468Z tap - INFO Ignoring 403 from account endpoint - this must be an integrated Zendesk account. This endpoint will be excluded from discovery.
2018-08-07 00:02:33,502Z main - INFO Tap exited normally.
2018-08-07 00:02:33,504Z main - INFO Saving list of discovered streams
2018-08-07 00:02:33,594Z main - INFO Saving structure of stream agents
2018-08-07 00:02:33,633Z main - INFO Saving structure of stream chats
2018-08-07 00:02:33,677Z main - INFO Saving structure of stream shortcuts
2018-08-07 00:02:33,688Z main - INFO Saving structure of stream triggers
2018-08-07 00:02:33,700Z main - INFO Saving structure of stream bans
2018-08-07 00:02:33,708Z main - INFO Saving structure of stream departments
2018-08-07 00:02:33,716Z main - INFO Saving structure of stream goals
2018-08-07 00:02:33,727Z main - INFO Getting list of selected properties
2018-08-07 00:02:33,784Z main - DEBUG Getting initial state
2018-08-07 00:02:33,789Z main - INFO Starting tap: tap-env/bin/tap-zendesk-chat --config /tmp/tap_config.json --properties /tmp/properties.json --catalog /tmp/catalog.json --state /tmp/tap_state.json
2018-08-07 00:02:33,796Z main - INFO Starting target: target-env/bin/target-stitch --config /tmp/target_config.json
2018-08-07 00:02:34,073Z tap - INFO Using chat_search_interval_days: 14
2018-08-07 00:02:34,191Z target - INFO Using Stitch import URL https://api.stitchdata.com/v2/import/batch
2018-08-07 00:02:34,309Z tap - INFO HTTP request to "chats" endpoint took 0.233s, returned status code 200
2018-08-07 00:02:34,669Z tap - INFO HTTP request to "chats" endpoint took 0.358s, returned status code 200
2018-08-07 00:02:34,687Z tap - INFO replicated 40 records from "chats" endpoint
2018-08-07 00:02:34,972Z tap - INFO HTTP request to "chats" endpoint took 0.283s, returned status code 200
2018-08-07 00:02:35,264Z tap - INFO HTTP request to "chats" endpoint took 0.290s, returned status code 200
2018-08-07 00:02:35,271Z tap - INFO replicated 13 records from "chats" endpoint
2018-08-07 00:02:35,271Z tap - INFO Using chat_search_interval_days: 14
2018-08-07 00:02:35,491Z tap - INFO HTTP request to "chats" endpoint took 0.218s, returned status code 200
2018-08-07 00:02:35,690Z tap - INFO HTTP request to "chats" endpoint took 0.198s, returned status code 200
2018-08-07 00:02:35,692Z tap - INFO replicated 1 records from "chats" endpoint
2018-08-07 00:02:35,693Z target - INFO Sending batch with 54 messages for table chats to https://api.stitchdata.com/v2/import/batch
2018-08-07 00:02:35,975Z tap - INFO HTTP request to "shortcuts" endpoint took 0.281s, returned status code 200
2018-08-07 00:02:36,487Z main - INFO State update: adding bookmarks.agents.offset.id = null
2018-08-07 00:02:36,487Z main - INFO State update: adding bookmarks.chats.chat.end_timestamp = "2018-08-06T18:07:26Z"
2018-08-07 00:02:36,487Z main - INFO State update: adding bookmarks.chats.offline_msg.timestamp = "2018-08-03T10:19:49Z"
2018-08-07 00:02:36,487Z main - INFO State update: adding bookmarks.chats.offset.chat.next_url = null
2018-08-07 00:02:36,487Z main - INFO State update: adding bookmarks.chats.offset.offline_msg.next_url = null
2018-08-07 00:02:36,488Z main - INFO State update: adding currently_syncing = "chats"
2018-08-07 00:02:36,502Z tap - INFO replicated 436 records from "shortcuts" endpoint
2018-08-07 00:02:36,502Z target - INFO Sending batch with 436 messages for table shortcuts to https://api.stitchdata.com/v2/import/batch
2018-08-07 00:02:36,615Z target - CRITICAL Error persisting data for table "shortcuts": Record 0 did not conform to schema:
2018-08-07 00:02:36,615Z target - CRITICAL #: #: no subschema matched out of the total 2 subschemas
2018-08-07 00:02:36,616Z target - CRITICAL #: expected: null, found: JSONObject
2018-08-07 00:02:36,616Z target - CRITICAL #: extraneous key [scope] is not permitted
2018-08-07 00:02:36,764Z tap - INFO HTTP request to "triggers" endpoint took 0.260s, returned status code 200
2018-08-07 00:02:36,769Z tap - INFO replicated 67 records from "triggers" endpoint
2018-08-07 00:02:37,037Z tap - INFO HTTP request to "bans" endpoint took 0.267s, returned status code 200
2018-08-07 00:02:41,652Z main - WARNING Thread tap did not finish within timeout
2018-08-07 00:02:41,653Z main - ERROR Target exited abnormally with status 1. Terminating tap.
2018-08-07 00:02:41,656Z main - INFO Exit status is: Discovery succeeded. Tap failed with code -15. Target failed with code 1 and error message: "Error persisting data for table "shortcuts": Record 0 did not conform to schema:
#: #: no subschema matched out of the total 2 subschemas
#: expected: null, found: JSONObject
#: extraneous key [scope] is not permitted".

I'm not sure if you get much further with this?

[KAllan357]

Thanks. I actually think that adding the field to the schema I mentioned above would get things going. Would you be willing to make a PR here?

Basically add to the top level of the shortcuts schema:

"scope": {
      "type": [
        "null",
        "string"
      ]
    }

[SeanFLynch]

@KAllan357 my knowledge of javascript is zero, so there'll almost certainly be a bracket out of place if I made a PR. Might be better if you or someone else does it...

[KAllan357]

@SeanFLynch no problem. I should have this released today.

[KAllan357]

Released in 0.1.13

[SeanFLynch]

@KAllan357 we have the same error again:

Error persisting data for table "shortcuts": Record 0 did not conform to schema:
#: #: no subschema matched out of the total 2 subschemas
#: expected: null, found: JSONObject
#: extraneous key [id] is not permitted

Any ideas?

[TomaszE]

Hi @KAllan357
to add to above : error is changing

Now we have:

Error persisting data for table "shortcuts": Record 0 did not conform to schema:
#: #: no subschema matched out of the total 2 subschemas
#: expected: null, found: JSONObject
#: 2 schema violations found
#: extraneous key [id] is not permitted
#: extraneous key [departments] is not permitted