Solve issues with gitlab .apk files and cloudflare protection

[akramer-zibra]

There occur issues if releases refer to external hosted .apk files. In some cases (e.g. Aurora Store) download is not possible because cloudflare protection gives "forbidden" error. This patch will add a "referer" header to the download request to pacify cloudflare protection service.

Notice: In case of the Aurora Store project, you still need a Gitlab API token, to retrieve latest .apk release. Obtainium either retrieves .apk data from Gitlab API (token necessary) or from tags-page, which has no information about the latest .apk url.

Related to: #1397, #1389, #1384, #1382, #1381, #1380, #1359, #854, #785, #697

Tested with Android 14 in an emulated AVD.

[ImranR98]

Great 👍

Just to confirm - were you still getting blocked without the Referrer header? I haven't been able to test this because I'm not longer getting blocked by Cloudflare either way (not sure if they removed it or just selective about IPs).

[Disqu3-mirOir-qt]

hello @ImranR98 fixed side AuroraStore cf:

https://gitlab.com/AuroraOSS/AuroraStore/-/issues/1052

[Grishnackh]

@Disqu3-mirOir-qt Can you please provide information on the mentioned solution to the Cloudflare issue, i.e. any link to a PR or commit? In the linked ticket above the maintainer only states that you fixed the issue, unfortunately not how this has been fixed.

[akramer-zibra]

I can confirm the cloudflare fix. The .apk dowload from auroraoss.com now works inside Obtainium. Tested with Android 14, Obtainium 1.0.2 version and a configured Gitlab API token. Think we can close this PR if anyone else can confirm this?

[DwainZwerg]

Think we can close this PR if anyone else can confirm this?

We should integrate the PR anyway. There are certainly other sites that are protected with Cloudflare and with which Obtainium has problems.

[Grishnackh]

I agree, @DwainZwerg. This might solve issues for other sources, i.e. #1387.

[akramer-zibra]

There are certainly other sites that are protected with Cloudflare and with which Obtainium has problems.

So you think this solution should be more generic?

Seems not a bad idea to send always a user-agent with http requests 🤔

[DwainZwerg]

Seems nt a bad idea to send always a user-agent with http requests.

I think it could be an option to opt in for each app individually in the respective additional app options.

[mjdNjhNJ]

The solution about tokens refers to practice of generating personal access token on GitLab or to use some public app-specific one?

[Grishnackh]

@mjdNjhNJ It's related to personal access tokens, see https://github.com/ImranR98/Obtainium/wiki#gitlab.

[mjdNjhNJ]

@Grishnackh Thanks. Confirming that everything works as expected with personal access token (read_api), Aurora Store successfully updated to 4.4.2.

[ImranR98]

So you think this solution should be more generic?

@akramer-zibra let me know if you still want the PR to be merged (after the one review comment is addressed) or if you plan on making something more generic. I don't think there's any harm in adding a referrer header by default for all requests (just not sure if that would be enough to bypass CF protection anyways).

[akramer-zibra]

Hey @ImranR98 . I changed the host in the referer header as you mentioned. Sure this change makes the code more consistent.

Tested this with Android 14 on an emulated AVD:

• When I add Aurora Store gitlab repositoy Then Obtainium shows AuroraStore 4.3.5 as latest version And When I click install Then Obtainium installs this version (4.3.5).
• Given a Gitlab API token in the settings When I add Aurora Store gitlab repository Then Obtainium shows a Warning prompt "The App source ... but the release package comes from ..." And When I click continue Then Obtainium shows AuroraStore 4.4.2 as latest version And When I click install Then Obtainium installs this version (4.4.2).

I think this PR is ready to merge.

As some people mentioned above, there could be a more generic solution to this: for example a On-Off-switch within the app specific options to enable ("opt-in") sending HTTP headers. I would prefer to handle this within a separate PR, because I think there might be a need for further discussion on this.

Nevertheless the Gitlab scraping seems still having issues when the .apk information on the release-page and tags-page differ. Using the Gitlab API for .apk retrieval solves this problem, but not every Obtainium user might have a Gitlab.com account. So I think there also might be a need for some code enhancements concerning this. I will endeavour to make a PR for this in the near future.