Fatal error crash: [{'message': 'service error', 'path': ['game']}]

[Unclepotap4ik]

The program crashes in few seconds after opening with text:

07:32:37: Fatal error encountered:
07:32:37:
07:32:37: Traceback (most recent call last):
07:32:37: File "main.py", line 158, in main
07:32:37: File "twitch.py", line 764, in run
07:32:37: File "twitch.py", line 916, in _run
07:32:37: File "twitch.py", line 1709, in get_live_streams
07:32:37: File "twitch.py", line 1566, in gql_request
07:32:37: exceptions.MinerException: GQL error: [{'message': 'service error', 'path': ['game']}]

Try to reinstall, but that doesn't work

[DevilXD]

Hello. "service error" signifies a problem with Twitch internals, not the miner. These usually go away on their own after some time passes, once Twitch fixes the issue.

Which games you have added to your priority list?

[Asiimov2077]

This error also occurs when I farm RUST drop.

[DevilXD]

@daeeros Auto-restart makes no sense for a project like this, as it's going to run into the very same error right away, and all you end up with, is the miner spamming the Twitch internal API with errors - a very sure way of encouraging them to take more and further steps to bot-proof their system and essentially kill this project. If auto-restart would be a good idea, I'd implement it ages ago.

Please don't do this.

[someC0d3r]

Can confirm:

09:56:48: Fatal error encountered:
09:56:48: 
09:56:48: Traceback (most recent call last):
09:56:48:   File "C:\Users\Administrator\Desktop\TwitchDropsMiner\TwitchDropsMiner-master (1)\TwitchDropsMiner-master\main.py", line 158, in main
09:56:48:     await client.run()
09:56:48:   File "C:\Users\Administrator\Desktop\TwitchDropsMiner\TwitchDropsMiner-master (1)\TwitchDropsMiner-master\twitch.py", line 764, in run
09:56:48:     await self._run()
09:56:48:   File "C:\Users\Administrator\Desktop\TwitchDropsMiner\TwitchDropsMiner-master (1)\TwitchDropsMiner-master\twitch.py", line 916, in _run
09:56:48:     new_channels.update(await self.get_live_streams(game))
09:56:48:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
09:56:48:   File "C:\Users\Administrator\Desktop\TwitchDropsMiner\TwitchDropsMiner-master (1)\TwitchDropsMiner-master\twitch.py", line 1709, in get_live_streams
09:56:48:     response = await self.gql_request(
09:56:48:                ^^^^^^^^^^^^^^^^^^^^^^^
09:56:48:   File "C:\Users\Administrator\Desktop\TwitchDropsMiner\TwitchDropsMiner-master (1)\TwitchDropsMiner-master\twitch.py", line 1566, in gql_request
09:56:48:     raise MinerException(f"GQL error: {response_json['errors']}")
09:56:48: exceptions.MinerException: GQL error: [{'message': 'service error', 'path': ['game']}]
09:56:48: 
09:56:48: Exiting...
09:56:48: 
09:56:48: Application Terminated.
09:56:48: Close the window to exit the application.

On latest master version

Infos:

pip updated
requirements updated

[Vadson159]

@DevilXD
I guess topics like this will occur from time to time anyway. Because of the very same errors I was unlucky to skip a few Halo Infinite drop campaigns that were running in my nighttime and I couldn't restart miner manually. Maybe it would be a good idea to output this error for the user as "twitch having troubles right now, waiting..." or something like that, and make it auto restart in a minute or two to not spam twitch API that hard.

[DevilXD]

When a "service error" occurs, there's nothing you can do to help - restarting won't fix it. This is an internal Twitch error that cannot be "handled" by any other way, than just waiting (even a couple of hours) and then restarting the miner. Handling this in a more graceful way was tracked by #347 for some time, but then it got closed due to the issue being no longer present (until again now).

Regardless, I've noticed that unrequested application crashes could use some more notification noise to them, which caecef4 should help with.

I'll try to debug this for Rust specifically in the mean time. So far I haven't been able to reproduce it, Rust starts mining just fine for me.

[someC0d3r]

@DevilXD

Can you tell what this method call does? Does it log to a file by any chance? If so I maybe can provide you my response JSON :/

[wlk3773208]

In the past, this error would only occur occasionally and the frequency was very low. But this time, the frequency of the error is very frequent. This error will definitely occur here within 2 hours. Could it be caused by someone abusing the mechanism? For example, divide 100 systems on a powerful server and then use it for 100 accounts at the same time?

[DevilXD]

Can you tell what this method call does? Does it log to a file by any chance? If so I maybe can provide you my response JSON :/

@someC0d3r That's a debug logger call. You'd need to run the miner with the GQL debug flag: --debug-gql. Also use --log to log everything to a file.

The logging system was never fleshed out enough for me to consider using it over "live" debugging via the in-built VSCode debugger, but I guess it's worth a shot.

A word of caution though: note that the resulting log may include some sensitive information about your Twitch account, that'd allow a 3rd party to authorize as you, which could result in "unpleasant" consequences (general "your account has been hacked or hijacked" / potential loss of account). I never built into it any system that'd filter those out. The log file should thus either be thoroughly examined by you, to remove any traces of sensitive information (you have to have the knowledge of what to look for), and any omission means risking the safety of your Twitch account, which means that some "more personal" way of giving me the log file would be strongly preferred. I personally have zero to no interest accessing someone else's account, but if you don't trust me with the log file either, I'd suggest simply not sending one at all.

If you have Discord, I'm available under @devilxd. Otherwise, you can email me the log file to [email protected].

[DevilXD]

@wlk3773208 I've seen others setup true account farms, that'd run this miner (among other ones) on thousands of accounts. This is considered not only Twitch API abuse, but also an abuse of this project, which I do not support in the slightest. For a more descriptive journey about this, you can check out the project goals: https://github.com/DevilXD/TwitchDropsMiner#project-goals All of this contributes to Twitch putting out preventive measures, that makes it harder to run a personal, single-account miner like this one. But there isn't much I can do to combat this, unfortunately.

Regarding the issue, it's most likely an extra condition on Twitch side, which I'm yet to discover. Since I can't reproduce the error myself, there isn't much I can do for now. I'll keep trying, I guess.

[boypipe]

@wlk3773208 I've seen others setup true account farms, that'd run this miner (among other ones) on thousands of accounts. This is considered not only Twitch API abuse, but also an abuse of this project, which I do not support in the slightest. For a more descriptive journey about this, you can check out the project goals: https://github.com/DevilXD/TwitchDropsMiner#project-goals All of this contributes to Twitch putting out preventive measures, that makes it harder to run a personal, single-account miner like this one. But there isn't much I can do to combat this, unfortunately.

Regarding the issue, it's most likely an extra condition on Twitch side, which I'm yet to discover. Since I can't reproduce the error myself, there isn't much I can do for now. I'll keep trying, I guess.

It happens when running on VM and using the manual built solution (adding a line of code) provided by another person in issue #386

[DevilXD]

Please tell me it's not verify=False, because that's a pretty terrible idea / """solution""". A better idea would be to pip install either certifi or truststore to supply the missing certificates instead.

[Unclepotap4ik]

Looks like problem is gone. I have only one crash this morning, and then program works very well

[DevilXD]

I bet it was just Twitch themselves messing something up internally on their side, and this being the byproduct. I should really get to some proper handling for "service error" specifically. This isn't one where retrying after mere seconds makes any sense, the miner will need to essentially enter idle and wait until it's over. I mentioned this already under #347, but it got closed without adding said handling, so that was pretty bad on my part.

[Zhanyee]

I'm having this problem for around 2 months now, is there a solution?

[JourneyOver]

There def seems to be an issue with Destiny 2 today. In the past two hours, three separate tickets have been closed as duplicates of the same problem, and I'm encountering the same error—I'm sure others are too.

For now, the only solution is to either add Destiny 2 to the exclusion list or remove it from the priority list if you're using "priority list only" mode. Doing either seems to resolve the issue for the time being.

[Taizunz]

Just started happening again today

00:41:08: Fatal error encountered:
00:41:08: 
00:41:08: Traceback (most recent call last):
00:41:08:   File "main.py", line 155, in main
00:41:08:   File "twitch.py", line 587, in run
00:41:08:   File "twitch.py", line 747, in _run
00:41:08:   File "twitch.py", line 1656, in bulk_check_online
00:41:08:   File "asyncio\tasks.py", line 571, in _wait_for_one
00:41:08:   File "twitch.py", line 1387, in gql_request
00:41:08: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]
00:41:08: 
00:41:08: Exiting...

[dogodogowoof]

Can confirm adding Destiny 2 to my exclusion list worked. thx @JourneyOver

[DevilXD]

I've put more effort into investigating this issue, by checking how the Twitch website handles this. I found out that the DropCampaignDetails GQL call on the website has a different hash than the miner was using, so I've updated it: f11e334

Please let me know if that resolves the issue for now.

[JourneyOver]

So far it looks to be good! :D

[Luckz]

I've put more effort into investigating this issue, by checking how the Twitch website handles this. I found out that the DropCampaignDetails GQL call on the website has a different hash than the miner was using, so I've updated it: f11e334

Please let me know if that resolves the issue for now.

On e79ab9d, if I don't block Destiny 2, I still get:

18:55:30: Traceback (most recent call last):
18:55:30:   File "<path>\main.py", line 155, in main
18:55:30:     await client.run()
18:55:30:   File "<path>\twitch.py", line 587, in run
18:55:30:     await self._run()
18:55:30:   File "<path>\twitch.py", line 747, in _run
18:55:30:     await self.bulk_check_online(acl_channels)
18:55:30:   File "<path>\twitch.py", line 1656, in bulk_check_online
18:55:30:     response_list = await coro
18:55:30:                     ^^^^^^^^^^
18:55:30:   File "<path>\Lib\asyncio\tasks.py", line 631, in _wait_for_one
18:55:30:     return f.result()  # May raise f.exception().
18:55:30:            ^^^^^^^^^^
18:55:30:   File "<path>\twitch.py", line 1387, in gql_request
18:55:30:     raise GQLException(response_json['errors'])
18:55:30: exceptions.GQLException: [{'message': 'service error', 'path': ['channel', 'viewerDropCampaigns']}]

[JourneyOver]

hmm wonder why you are still able to run into the issue, as I tested the latest commit on an account with just destiny 2 in the priority list as well as testing with all three different priority modes and haven't ran back into the issue anymore since these last 2 commits so I'm wondering if you have something else possibly going on..

[DevilXD]

Twitch seems to have some kind of a rolling deployment going on. Some people experience a crash, some not. Some get the updated hash value, some are still left running the old one. Eventually though, the change is migrated to all users. The timeframe for this appears to be at least a few days, I'd say 2-3.

The error seems to be originating from the bulk status check now. I'll investigate further.

EDIT: The error pointed at the "AvailableDrops" section of the bulk status check, so I've checked the GQL hash, and it was outdated as well. Please try again with the latest master.

[JourneyOver]

Feels like we really need to find a way to update the hashes automatically some how periodically or something just so that we stop running into issues due to a hash being out of date xD Makes me wonder if there are any other hashes that are out of date that we don't know about yet.

[dogodogowoof]

@DevilXD now working for me with f128e91 AND its mining the destiny 2 drop.

[DevilXD]

@dogodogowoof Cheers for verifying =)

@JourneyOver Outdated hashes don't do anything, until Twitch starts to change around their internals. Once they do so, they usually change the query around as well, and then hash changes. The result of this is either the service crashing when using the old hash (service error), or the query stops being cached on the server side (PersistedQueryNotFound).

I don't really know where the hashes come from, but I know that getting that information is nigh impossible. I know it's the raw GQL query being "compressed" into a hash form, where both the client and the server know the actual query, and by sending the hash itself, both can relay the information of what the client wants, and what the server is supposed to respond with, with minimal information being actually sent as the request. Technically, I could send raw GQL queries without hashes, but that'd only get rid of the "PersistedQueryNotFound" error, and it'd still crash when Twitch would change around their internal structures.

The only solution would be to do the impossible and analyze the web client to extract hashes out of it, but the problem is that it's minified JS with some quite cryptic pieces of code, or it's straight up encrypted/mangled JS that specifically targets being hard to reverse-engineer. The simplest solution for now is to just update the hashes manually, as they don't really change that often. For example, the last time the AvailableDrops hash was updated, beyond yesterday, was 17 months ago: 5b8b6c7

Other examples beyond the updates from the last 2 days:

• Campaigns hash last updated 3 months ago: 9768635
• Campaign details hash last updated 3 months ago, then 11 months ago: 0f3a784, 5167d95

All of the above operations, plus the "Inventory" hash, were updated 20 months ago before that: bfd3893

If you'd like to investigate on your own, here are the two main files I'm pretty sure are responsible for the Twitch web client functionality:

• Core file, seems to contain some kind of query definitions: https://assets.twitch.tv/assets/core-d2bdb1143480a8de70ab.js
• "p.js", which is the actual file that sends out the GQL queries (URL implies it's a generated file that changes every time): https://k.twitchcdn.net/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/p.js

[JourneyOver]

Hmm I might take a look at things when I have some free time to really dig down into it, but yea I guess if they don't change as often as I thought they was going to then it might just be pointless.

[skapytek]

For me this error occurs every time im starting twitchdropsminer

[Luckz]

For me this error occurs every time im starting twitchdropsminer

Temporarily add Lost Ark and Black Desert to your exclusion list?

[skapytek]

For me this error occurs every time im starting twitchdropsminer

Temporarily add Lost Ark and Black Desert to your exclusion list?

still the same error