APKMirror: Set up custom user agent to bypass Cloudflare protection

[ryester19]

Prerequisites

I did see #957 and #1097, but those are closed

Describe the feature

The APKMirror source is pretty much useless to me in its current state. I always receive the forbidden error for APKMirror links, regardless of network or VPN I'm connected to. Using curl confirms that this is because of Cloudflare protection

I downloaded APKUpdater and saw that their implementation of APKMirror checking is working just fine on the same phone. Digging deeper into their issues section, I found rumboalla/apkupdater#16, where an APKMirror developer suggested setting a custom user agent as part of their agreement. Changing the user agent to "APKUpdater-v3.0.3" in curl now allows me to retrieve the RSS feed for any app on APKMirror

Seeing as though the APKMirror devs were willing to make an exception for APKUpdater, it shouldn't be too hard to request the same treatment for Obtainium

[ImranR98]

Hey @archon810 would it be possible for Obtainium to use the user agent workaround? APKMirror is a track-only source so users do have to visit your page to download apps. Relevant: #44

[DwainZwerg]

Hello @ryester19 ,
@akramer-zibra is working on a PR to avoid few problems with websites that are protected by Cloudflare.

[Grishnackh]

The PR #1411 has been merged. It sets the user agent for GitLab only though, not for APKMirror or other sources.

[DwainZwerg]

The PR #1411 has been merged. It sets the user agent for GitLab only though, not for APKMirror or other sources.

@akramer-zibra I think it would definitely make sense to extend this change to all sources. I thought you had?

[akramer-zibra]

Hey @DwainZwerg , yes and no. #1411 added a referer header. This was enough to bypass the cloudflare protection in the case with Gitlab and the AuroraStore app. There was no need to send a User-Agent header aswell.

That is why this issue seems to be a bit different here. As @ryester19 described above cloudflare may require a User-Agent, but I can double check this on my local dev setup.

By the way: Obtainium has already a feature to configure specific HTTP request headers. But this feature is currently only implemented for HTML sources. This could be a solution for this issue here aswell.

UPDATE: I can't reproduce this issue, sorry. I think the point is as Imran said above: The APKMirror source version is "track-only" in the latest Obtainium version (1.0.5), which means that the user has to visit the download page manually and Obtainium does only track version updates, but does not retrieve any .apk files.