🐛 Source Shopify: unavailable balance transactions causes 404 error

[marcosmarxm]

Current Behavior

When using the Shopify source with a store in a country that does not support balance transactions, the whole sync fails.

Expected Behavior

The absence of balance transactions should not break the sync.

This Github issue is synchronized with Zendesk:

Ticket ID: #1778
Priority: normal
Group: Community Assistance Engineer
Assignee: Nataly Merezhuk

Original ticket description:

• Is this your first time deploying Airbyte?: No
• OS Version / Instance: Ubuntu
• Memory / Disk: 14GB RAM, 20GB
• Deployment: Docker on GCP GCE
• Airbyte Version: 0.39.39-alpha
• Source name/version: Shopify 0.1.37
• Destination name/version: BigQuery 1.1.11
• Step: Happens during the sync
• Description: Facing a problem with the description of it mentioning “Failure Origin: source, Message: Something went wrong within the source connector” when connecting Shopify to BigQuery. The logs are uploaded for reference. I don’t understand the problem because I have Admin permission to Shopify hence I would not come into problem when calling the API for the Shopify. Please help. Thanks.
  logs-24.txt (422.4 KB)

[Discourse post]

[marcosmarxm]

Comment made from Zendesk by Nataly Merezhuk on 2022-08-02 at 16:35:

Hi @thongseng188,

I found a related post with someone using the same Shopify connector version and getting the same error:
https://discuss.airbyte.io/t/failed-to-pull-data-from-shopify/1549

This is the error I see in your logs:
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://montigo-bottle.myshopify.com/admin/api/2021-07/shopify_payments/balance/transactions.json?limit=250&order=id+asc&since_id=0

Could you please try to make a manual cURL call using your API token to hit this endpoint and see the headers/response from Shopify? If the issue persists then, I will internally link your post to this GitHub issue:
#14065

[marcosmarxm]

Comment made from Zendesk by Marcos Marx on 2022-08-03 at 06:27:

@natalyjazzviolin hello, I found out the problem already. The source connector always stops at requests.exceptions.HTTPError: 404 Client Error: Not Found for url: ``https://montigo-bottle.myshopify.com/admin/api/2021-07/shopify_payments/balance/transactions.json?limit=250&order=id+asc&since_id=0 .

The reason is in Malaysia’s Shopify store (or in some other countries not mentioned here link) do not support balance transactions on Shopify Payment. Other tables are not ingested when encountered with this problem. Hence it failed. Tried the manual cURL as well, the same result.

Regards,
Seng.

[Discourse post]

[marcosmarxm]

Comment made from Zendesk by Nataly Merezhuk on 2022-08-03 at 15:51:

Thanks for the clarification! I'll escalate this to GitHub as this seems to be an oversight regarding using the Shopify connector in countries that do not support balance transactions!