Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"service error" on available_drops check and claim operation since 14:00:00 GMT-3 #590

Open
IRhoAias opened this issue Oct 24, 2024 · 32 comments
Labels
Bug Something isn't working

Comments

@IRhoAias
Copy link

16:04:58: Fatal error encountered:
16:04:58:
16:04:58: Traceback (most recent call last):
16:04:58: File "main.py", line 155, in main
16:04:58: File "twitch.py", line 587, in run
16:04:58: File "twitch.py", line 747, in _run
16:04:58: File "twitch.py", line 1657, in bulk_check_online
16:04:58: File "asyncio\tasks.py", line 571, in _wait_for_one
16:04:58: File "twitch.py", line 1387, in gql_request
16:04:58: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]
16:04:58:
16:04:58: Exiting...
16:04:58:
16:04:58: Application Terminated.
16:04:58: Close the window to exit the application.

This error stops occurring when I exclude “Lost Ark” from the list.

@DevilXD DevilXD changed the title Fatal error encountered since 14:00:00 GMT-3 "service error" on available_drops check and claim operation since 14:00:00 GMT-3 Oct 25, 2024
@DevilXD
Copy link
Owner

DevilXD commented Oct 25, 2024

Here's one of mine that I just got during claiming:

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 155, in main
    await client.run()
  File "C:\...\TwitchDropsMiner\twitch.py", line 587, in run
    await self._run()
  File "C:\...\TwitchDropsMiner\twitch.py", line 648, in _run
    await drop.claim()
  File "C:\...\TwitchDropsMiner\inventory.py", line 259, in claim
    result = await super().claim()
             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\...\TwitchDropsMiner\inventory.py", line 142, in claim
    result = await self._claim()
             ^^^^^^^^^^^^^^^^^^^
  File "C:\...\TwitchDropsMiner\inventory.py", line 158, in _claim
    response = await self._twitch.gql_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\...\TwitchDropsMiner\twitch.py", line 1387, in gql_request
    raise GQLException(response_json['errors'])
exceptions.GQLException: [{'message': 'service error', 'path': ['claimDropRewards']}]

The claim did succeed, so I couldn't catch it happening, but the result of it was an error anyway. I'll need to investigate these further. I can't reproduce this for Lost Ark specifically, at least when I checked it just now. The AvailableDrops GQL hash hasn't changed for me (yet?).

EDIT: It's been a while since the claim operation hash was updated, which could be the cause for this. If anyone would be able to get their hands on the updated hash (if it was updated), that'd be helpful.

@DevilXD DevilXD added the Bug Something isn't working label Oct 25, 2024
@imranh2
Copy link

imranh2 commented Oct 25, 2024

Happening for me as well, excluding Lost Ark works as a fix.

@DevilXD
Copy link
Owner

DevilXD commented Oct 25, 2024

Just to be sure, I've rechecked this on my side when the above message was about 20 minutes old, and haven't encountered an error - Lost Ark campaigns and channels loaded in just fine. Whatever causes the error, hasn't arrived to my Twitch account yet. It seems like Twitch is doing one of those gradual rollouts again, and thus some people are using the new version, while others are still on the old version.

We need to wait. Based on the past events, all it takes to fix these is just a new AvailableDrops GQL hash, that would need to be grabbed by someone running the newer version. Either way, the rollouts usually last a day or two, sometimes three, so I'm going to get my hands on it eventually anyway.

@Luckz
Copy link
Contributor

Luckz commented Oct 25, 2024

Seems I haven't completed the "Aegir Awakens" campaign for Lost Ark yet.
When I started writing this post, I was unable to start up TDM if I don't exclude Black Desert.
Now by the time I'm done writing, I have to additionally exclude Lost Ark too.

19:24:33: Traceback (most recent call last):
19:24:33:   File "C:\Users\{TDMPATH}\main.py", line 155, in main
19:24:33:     await client.run()
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 587, in run
19:24:33:     await self._run()
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 747, in _run
19:24:33:     await self.bulk_check_online(acl_channels)
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 1657, in bulk_check_online
19:24:33:     response_list = await coro
19:24:33:                     ^^^^^^^^^^
19:24:33:   File "C:\Users\{PythonStuff}\envs\TwitchPIP\Lib\asyncio\tasks.py", line 631, in _wait_for_one
19:24:33:     return f.result()  # May raise f.exception().
19:24:33:            ^^^^^^^^^^
19:24:33:   File "C:\Users\{TDMPATH}\twitch.py", line 1387, in gql_request
19:24:33:     raise GQLException(response_json['errors'])
19:24:33: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]
19:24:33: 
19:24:33: Exiting...

But this is just #415 again anyway, innit?

@skmedix
Copy link

skmedix commented Oct 25, 2024

We need to wait. Based on the past events, all it takes to fix these is just a new AvailableDrops GQL hash, that would need to be grabbed by someone running the newer version.

Can you provide any methods to grab latest GQL hash? I may look into it

@DevilXD
Copy link
Owner

DevilXD commented Oct 25, 2024

@skmedix It's a tedious task of opening Twitch pages while having web developer tools open (F12 on most browsers). On the network calls tab, find the POST requests done to https://gql.twitch.tv/gql, and on the "request" tab you should see the JSON sent to the service. It'll contain the operation name and sha256 hash of it like so (hash on the right):
picture

It's a bit advanced technical things. Doing all this requires some basic knowledge of how to use the web dev tools, as well as where to look for, and even what Twitch pages to open. "AvailableDrops" GQL call is made only when you open a stream (any channel will do), inventory and campaigns pages let you drop their respective hashes, claim drop GQL operation happens only during the claim process, so you need to catch the request when clicking on the claim button, and that requires some manual finishing mining of a drop (I'll try doing this later today or tomorrow).

@skmedix
Copy link

skmedix commented Oct 25, 2024

@DevilXD I only asked because I missed it the first time I looked (I didn't open a stream, but I directly went to the inventory). With a simple matcher it's actually super easy, and I can confirm that the hash for DropsHighlightService_AvailableDrops didn't change, and matches one in constants.py. Edit: I will try to catch DropsPage_ClaimDropRewards too.

@imranh2
Copy link

imranh2 commented Oct 26, 2024

This has now started working again without Lost Ark in the exlusions list.

@DevilXD
Copy link
Owner

DevilXD commented Oct 26, 2024

Yeah, I've checked Lost Ark this morning again, and still no errors or other issues. It seems that the rollout was possibly reverted, due to the errors popping up. We'll need to monitor the situation though.

@IRhoAias

This comment was marked as off-topic.

@DevilXD
Copy link
Owner

DevilXD commented Oct 27, 2024

@IRhoAias That's a "content restricted in region" error, not something related to this issue. Handling JSON output out of the watch endpoint is on my TODO list for a while now. See: #586.

@DevilXD
Copy link
Owner

DevilXD commented Oct 28, 2024

Since this has stopped being an issue, I guess it can be closed now.

@DevilXD DevilXD closed this as completed Oct 28, 2024
@DevilXD
Copy link
Owner

DevilXD commented Nov 1, 2024

Turns out it's back: #599. I'm reopening this one, to keep the discussion up.

@DevilXD DevilXD reopened this Nov 1, 2024
@Luckz
Copy link
Contributor

Luckz commented Nov 1, 2024

tldr: Had to block Lost Ark again.

@DevilXD
Copy link
Owner

DevilXD commented Nov 2, 2024

The annoying part of this is that the bulk check logic specifically batches requests together. That means that, even if I would add any kind of logging to this, it'd narrow it down to up to 20 channels. Rectifying the error is also hard to do, given that the gql_request method raises the exception for an error in at least one of the 20 responses it receives in the batch, making isolating the particular case quite hard.

The only thing I can think of here, would be to rework the gql_request method a little, to allow for more granular output. I'll start working on this. It'll let us determine which channel causes the issue at the very least.

@DevilXD
Copy link
Owner

DevilXD commented Nov 2, 2024

#600 points towards 40+ games being tracked as mining candidates, as the source of the error. To be fair, I've never imagined someone would actively be playing (and want drops for) that many games at the same time, I'm not sure if it's even possible time-wise to keep up with the campaigns, even if running the miner 24/7.

As a temporary mitigation for anyone affected, I'd suggest trimming down your priority lists, and excluding games you care the least about. Also, anyone affected should comment here their priority list length, if using the "priority list only" priority mode, or include their dump file otherwise. The number of linked games with campaigns to them would be enough, but that may be hard to determine in the other priority modes.

d4cfcb5 has been pushed to increase the GQL rate limit a bit, which should help to mitigate the issue somewhat. I'd still recommend trimming down on the games you won't need though.

@JourneyOver
Copy link

JourneyOver commented Nov 2, 2024

I'm not sure if it's even possible time-wise to keep up with the campaigns, even if running the miner 24/7.

It's def possible, as I'm near 40 games myself (currently at 37 and adding more every so often whenever I see a game I might be interested in playing or am playing) and rarely (like once in a blue moon type of situation so it's very very rare) ever miss a drop with the miner running 24/7.

Anyways I've yet to run into this error myself even with games like Lost Ark in my list which seems to be a main candidate of causing this issue for some people, albeit it was mined completely on the day that the campaign started for that game so maybe that could be why who really knows.

@bitterbutt
Copy link

bitterbutt commented Nov 12, 2024

Not using the priority list at all and have received the claimDropRewards service error across Rainbow Six Siege and Black Ops 6 today anytime a drop is to be claimed. My priority mode is 'Ending soonest', if it matters.

Unsure if GQL hash is still needed. But leaving my dev tools open for now to see if I can get something useful.

Edit
Hash for DropsHighlightService_AvailableDrops in my browser is same as in code currently.
eff13f4a43157238e40b4cd74b0dac3a41b5f8fb31de1a3b19347fae84e60b92

Hash for DropsPage_ClaimDropRewards in my browser is same as in code currently.
a455deea71bdc9015b78eb49f4acfbce8baa7ccbedd28e549bb025bd0f751930

@kgaff
Copy link

kgaff commented Nov 13, 2024

I've had it twice for rust in the last few hours, with only 2 games in the list. I sadly didn't think to record the errors but I will keep an eye out for them again. Just in case of the bug others have mentioned above, I have added lost ark to the ignore list.

@kgaff
Copy link

kgaff commented Nov 13, 2024

11:22:30: Fatal error encountered: 11:22:30: 11:22:30: Traceback (most recent call last): 11:22:30: File "main.py", line 155, in main 11:22:30: File "twitch.py", line 587, in run 11:22:30: File "twitch.py", line 648, in _run 11:22:30: File "inventory.py", line 259, in claim 11:22:30: File "inventory.py", line 142, in claim 11:22:30: File "inventory.py", line 158, in _claim 11:22:30: File "twitch.py", line 1387, in gql_request 11:22:30: exceptions.GQLException: [{'message': 'service error', 'path': ['claimDropRewards']}] 11:22:30: 11:22:30: Exiting... 11:22:30: 11:22:30: Application Terminated. 11:22:30: Close the window to exit the application.

@spayn
Copy link

spayn commented Nov 13, 2024

Same error ('claimDropRewards') happens for me every time miner claims a reward, works fine after I restart application until the next reward is claimed.

@ParkerSaint
Copy link

ParkerSaint commented Nov 14, 2024

I am constantly having this issue as well, here is the traceback I got:

05:54:31 PM: 
05:54:31 PM: Traceback (most recent call last):
05:54:31 PM:   File "/tmp/.mount_TwitchE4JBvH/usr/src/main.py", line 155, in main
05:54:31 PM:     await client.run()
05:54:31 PM:   File "/tmp/.mount_TwitchE4JBvH/usr/src/twitch.py", line 587, in run
05:54:31 PM:     await self._run()
05:54:31 PM:   File "/tmp/.mount_TwitchE4JBvH/usr/src/twitch.py", line 747, in _run
05:54:31 PM:     await self.bulk_check_online(acl_channels)
05:54:31 PM:   File "/tmp/.mount_TwitchE4JBvH/usr/src/twitch.py", line 1657, in bulk_check_online
05:54:31 PM:     response_list = await coro
05:54:31 PM:   File "/tmp/.mount_TwitchE4JBvH/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
05:54:31 PM:     return f.result()  # May raise f.exception().
05:54:31 PM:   File "/tmp/.mount_TwitchE4JBvH/usr/src/twitch.py", line 1387, in gql_request
05:54:31 PM:     raise GQLException(response_json['errors'])
05:54:31 PM: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]
05:54:31 PM: 
05:54:31 PM: Exiting...
05:54:31 PM: 
05:54:31 PM: Application Terminated.
05:54:31 PM: Close the window to exit the application.

@DevilXD
Copy link
Owner

DevilXD commented Nov 16, 2024

Has any of you checked if the drop is actually claimed in the inventory page? If not, what happens when you try to claim it yourself? Or does it display a message like "Expired" or smth like that?

@DevilXD DevilXD mentioned this issue Nov 17, 2024
@dragongrafdiscord
Copy link

well since i would guess i need to move my answers to here
every drop i farmed was already claimed

@bitterbutt
Copy link

I haven't actually experienced the issue in the past couple days. So no news to report here. Odd it went away.

@GitCshes
Copy link

Hello, I've experienced this issue in the morning and now no longer experience it. I'm not a coder so I don't know what exactly is wrong but I have an idea of what could be the cause.

This morning after experiencing the issue I decreased the amount of games listed on my priority list, which solved the problem. Later on in the day once some drops were completed I added back the previous games I had removed onto the list with no issues.
Whenever this problem occurs the Status was at "Gathering channels..." before crashing, with the only difference between now and then being that some of the drops were complete and thus no longer needed a channel listed.

Could it be possible that something in the program is being overloaded with the number of channels it is tracking and listing, thus crashing the game? I could be wrong, but I would assume if each campaign was limited to only listing up to 10 channels (currently some list 20+) and if offline channels were removed and then replaced, it could prevent the program from crashing.

@DevilXD
Copy link
Owner

DevilXD commented Nov 18, 2024

Could it be possible that something in the program is being overloaded with the number of channels it is tracking and listing, thus crashing the game? I could be wrong, but I would assume if each campaign was limited to only listing up to 10 channels (currently some list 20+) and if offline channels were removed and then replaced, it could prevent the program from crashing.

The limit is applied after the list of channel candidates is processed. The candidates are collected based on the selected games and campaigns present (no campaign, no channels collected for it's game). For campaigns that do not define an ACL, a list of like 20-30 live channels is collected. For campaigns with an ACL, the entire ACL is collected - and some campaigns define really long ACLs. Ravendawn campaigns alone define over 1000 channels, 5 times the amount the miner cares about and can support. With the current GQL rate limit of 10/1, checking the status of all those channels takes 100 seconds at best. If you add many more games to this, the reload time becomes longer and longer, which is okay assuming it wouldn't eventually crash, yet it still does sometimes.

#554 (comment) has a list of games that define really long ACLs, having which as a mining candidate will cause the most trouble with the miner. The issue itself also explains what happens if the online status check is skipped entirely.

I can see an optimization that would avoid double-checking the channels that were already tracked by the miner up to the reload point, but that'll only reduce the to-check list by ~200 channels. The miner was never designed nor expected to handle that many channels, and I don't really know what to do. All of those channels need to be processed to semi-guarantee correct functioning and proper fallback, when other channels decide to go offline.

@dragongrafdiscord
Copy link

oh it works for me again buuut specially now claiming rust everytime its done with one it just gives this error:
08:46:27: Fatal error encountered: 08:46:27: 08:46:27: Traceback (most recent call last): 08:46:27: File "main.py", line 155, in main 08:46:27: File "twitch.py", line 587, in run 08:46:27: File "twitch.py", line 637, in _run 08:46:27: File "twitch.py", line 1468, in fetch_inventory 08:46:27: File "asyncio\tasks.py", line 571, in _wait_for_one 08:46:27: File "twitch.py", line 1425, in fetch_campaigns 08:46:27: File "twitch.py", line 1387, in gql_request 08:46:27: exceptions.GQLException: [{'message': 'PersistedQueryNotFound'}] 08:46:27: 08:46:27: Exiting... 08:46:28: 08:46:28: Application Terminated. 08:46:28: Close the window to exit the application.
i will see if it works with no interuption if i leave twitch open with a drops claimer extention....

@DevilXD
Copy link
Owner

DevilXD commented Nov 18, 2024

"PersistedQueryNotFound" implies the "DropCampaignDetails" GQL operation needs a new hash. I'll get on this in a few hours once I'm home.

@GitCshes
Copy link

All of those channels need to be processed to semi-guarantee correct functioning and proper fallback, when other channels decide to go offline.

I would assume people order their priority list based on what they want most vs. least. Would it help if instead of searching all channels at once, it would only search the first x channels of the priority list and add more games lower in the priority list over time as the upper one's campaigns are complete? It wouldn't be necessary to keep track of the 5 games at the very bottom when the 5 games at the top are active and already take several hours to complete. In cases where all the channels of the upper campaigns are offline, you can then start checking lower campaigns until one that is online is found.

@DevilXD
Copy link
Owner

DevilXD commented Nov 18, 2024

You're not wrong, but you're also talking about something that would require a substantial rewrite of the miner's internals, something that I'd only consider doing for #220. Also, everything works just fine if you're not trying to cram in 40+ games into this thing. At this point, it makes more sense to me to put a straight up hard limit on how many games you can add to the miner, and that's it.

@GitCshes
Copy link

You're not wrong, but you're also talking about something that would require a substantial rewrite of the miner's internals, something that I'd only consider doing for #220. Also, everything works just fine if you're not trying to cram in 40+ games into this thing. At this point, it makes more sense to me to put a straight up hard limit on how many games you can add to the miner, and that's it.

As long as some of the campaigns are previously claimed the error doesn't appear (currently I have everything added with no problems because most of the things I actually want are already claimed). Instead of having a hard limit you could just give a warning to decrease the amount of games listed to users. (hopefully, that will also stop new users from coming on here).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests