"service error" on available_drops check and claim operation since 14:00:00 GMT-3

[IRhoAias]

16:04:58: Fatal error encountered:
16:04:58:
16:04:58: Traceback (most recent call last):
16:04:58: File "main.py", line 155, in main
16:04:58: File "twitch.py", line 587, in run
16:04:58: File "twitch.py", line 747, in _run
16:04:58: File "twitch.py", line 1657, in bulk_check_online
16:04:58: File "asyncio\tasks.py", line 571, in _wait_for_one
16:04:58: File "twitch.py", line 1387, in gql_request
16:04:58: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]
16:04:58:
16:04:58: Exiting...
16:04:58:
16:04:58: Application Terminated.
16:04:58: Close the window to exit the application.

This error stops occurring when I exclude “Lost Ark” from the list.

[DevilXD]

Here's one of mine that I just got during claiming:

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 155, in main
    await client.run()
  File "C:\...\TwitchDropsMiner\twitch.py", line 587, in run
    await self._run()
  File "C:\...\TwitchDropsMiner\twitch.py", line 648, in _run
    await drop.claim()
  File "C:\...\TwitchDropsMiner\inventory.py", line 259, in claim
    result = await super().claim()
             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\...\TwitchDropsMiner\inventory.py", line 142, in claim
    result = await self._claim()
             ^^^^^^^^^^^^^^^^^^^
  File "C:\...\TwitchDropsMiner\inventory.py", line 158, in _claim
    response = await self._twitch.gql_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\...\TwitchDropsMiner\twitch.py", line 1387, in gql_request
    raise GQLException(response_json['errors'])
exceptions.GQLException: [{'message': 'service error', 'path': ['claimDropRewards']}]

The claim did succeed, so I couldn't catch it happening, but the result of it was an error anyway. I'll need to investigate these further. I can't reproduce this for Lost Ark specifically, at least when I checked it just now. The AvailableDrops GQL hash hasn't changed for me (yet?).

EDIT: It's been a while since the claim operation hash was updated, which could be the cause for this. If anyone would be able to get their hands on the updated hash (if it was updated), that'd be helpful.

[imranh2]

Happening for me as well, excluding Lost Ark works as a fix.

[DevilXD]

Just to be sure, I've rechecked this on my side when the above message was about 20 minutes old, and haven't encountered an error - Lost Ark campaigns and channels loaded in just fine. Whatever causes the error, hasn't arrived to my Twitch account yet. It seems like Twitch is doing one of those gradual rollouts again, and thus some people are using the new version, while others are still on the old version.

We need to wait. Based on the past events, all it takes to fix these is just a new AvailableDrops GQL hash, that would need to be grabbed by someone running the newer version. Either way, the rollouts usually last a day or two, sometimes three, so I'm going to get my hands on it eventually anyway.

[Luckz]

Seems I haven't completed the "Aegir Awakens" campaign for Lost Ark yet.
When I started writing this post, I was unable to start up TDM if I don't exclude Black Desert.
Now by the time I'm done writing, I have to additionally exclude Lost Ark too.

19:24:33: Traceback (most recent call last):
19:24:33:   File "C:\Users\{TDMPATH}\main.py", line 155, in main
19:24:33:     await client.run()
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 587, in run
19:24:33:     await self._run()
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 747, in _run
19:24:33:     await self.bulk_check_online(acl_channels)
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 1657, in bulk_check_online
19:24:33:     response_list = await coro
19:24:33:                     ^^^^^^^^^^
19:24:33:   File "C:\Users\{PythonStuff}\envs\TwitchPIP\Lib\asyncio\tasks.py", line 631, in _wait_for_one
19:24:33:     return f.result()  # May raise f.exception().
19:24:33:            ^^^^^^^^^^
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 1387, in gql_request
19:24:33:     raise GQLException(response_json['errors'])
19:24:33: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]
19:24:33: 
19:24:33: Exiting...

But this is just #415 again anyway, innit?

[skmedix]

We need to wait. Based on the past events, all it takes to fix these is just a new AvailableDrops GQL hash, that would need to be grabbed by someone running the newer version.

Can you provide any methods to grab latest GQL hash? I may look into it

[DevilXD]

@skmedix It's a tedious task of opening Twitch pages while having web developer tools open (F12 on most browsers). On the network calls tab, find the POST requests done to https://gql.twitch.tv/gql, and on the "request" tab you should see the JSON sent to the service. It'll contain the operation name and sha256 hash of it like so (hash on the right):

It's a bit advanced technical things. Doing all this requires some basic knowledge of how to use the web dev tools, as well as where to look for, and even what Twitch pages to open. "AvailableDrops" GQL call is made only when you open a stream (any channel will do), inventory and campaigns pages let you drop their respective hashes, claim drop GQL operation happens only during the claim process, so you need to catch the request when clicking on the claim button, and that requires some manual finishing mining of a drop (I'll try doing this later today or tomorrow).

[skmedix]

@DevilXD I only asked because I missed it the first time I looked (I didn't open a stream, but I directly went to the inventory). With a simple matcher it's actually super easy, and I can confirm that the hash for DropsHighlightService_AvailableDrops didn't change, and matches one in constants.py. Edit: I will try to catch DropsPage_ClaimDropRewards too.

[imranh2]

This has now started working again without Lost Ark in the exlusions list.

[DevilXD]

Yeah, I've checked Lost Ark this morning again, and still no errors or other issues. It seems that the rollout was possibly reverted, due to the errors popping up. We'll need to monitor the situation though.

[DevilXD]

@IRhoAias That's a "content restricted in region" error, not something related to this issue. Handling JSON output out of the watch endpoint is on my TODO list for a while now. See: #586.

[DevilXD]

Since this has stopped being an issue, I guess it can be closed now.

[DevilXD]

Turns out it's back: #599. I'm reopening this one, to keep the discussion up.

[Luckz]

tldr: Had to block Lost Ark again.

[DevilXD]

The annoying part of this is that the bulk check logic specifically batches requests together. That means that, even if I would add any kind of logging to this, it'd narrow it down to up to 20 channels. Rectifying the error is also hard to do, given that the gql_request method raises the exception for an error in at least one of the 20 responses it receives in the batch, making isolating the particular case quite hard.

The only thing I can think of here, would be to rework the gql_request method a little, to allow for more granular output. I'll start working on this. It'll let us determine which channel causes the issue at the very least.