Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Web Scraping to Enable AuroraStore Downloads (Through the GitLab Source) #854

Closed
luciobortoletto opened this issue Sep 4, 2023 · 8 comments
Closed

Improve Web Scraping to Enable AuroraStore Downloads (Through the GitLab Source) #854

luciobortoletto opened this issue Sep 4, 2023 · 8 comments
Labels
enhancement New feature or request

Comments

@luciobortoletto
Copy link

luciobortoletto commented Sep 4, 2023 •
edited
Loading

Prerequisites
My request is not part of an existing issue.

Describe the bug
When I try to install AuroraStore (https://gitlab.com/AuroraOSS/AuroraStore) It gives me a Forbidden error.

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'Add App'
  2. Tap on 'App Source URL'
  3. Write https://gitlab.com/AuroraOSS/AuroraStore
  4. Tap'Add'
  5. See error

Screenshots and Logs
Screenshot_20230904-140532
`2023-09-04 13:19:53.288: info: This is the first ever run of Obtainium.

2023-09-04 13:19:53.289: info: Update interval was set to 360 (reason: last check was 1970-01-01 01:00:00.000).

2023-09-04 13:19:59.665: info: Update interval was set to 60 (reason: setting changed).

2023-09-04 13:42:02.324: error: Forbidden

2023-09-04 13:42:07.747: error: Forbidden

2023-09-04 13:42:18.612: info: Update interval was set to 60 (reason: last check was 1970-01-01 01:00:00.000).

2023-09-04 13:42:29.473: error: Forbidden

2023-09-04 13:42:48.557: error: Importato App

2023-09-04 13:43:19.330: error: Forbidden

2023-09-04 13:45:33.368: error: Forbidden

2023-09-04 13:46:18.926: error: Importato App

2023-09-04 13:46:27.365: error: Forbidden: [828490272]

2023-09-04 13:47:42.816: error: Forbidden: [828490272]

2023-09-04 13:47:47.889: info: Update interval was set to 60 (reason: last check was 1970-01-01 01:00:00.000).

2023-09-04 13:47:52.680: error: Forbidden: [828490272]

2023-09-04 13:48:44.771: error: Servono credenziali aggiuntive (in Impostazioni)

2023-09-04 13:53:23.525: error: Forbidden

2023-09-04 13:53:45.838: error: Forbidden

2023-09-04 13:54:05.636: error: Forbidden

2023-09-04 13:54:08.959: error: Forbidden

2023-09-04 13:54:10.497: error: Forbidden

2023-09-04 13:54:17.139: error: Forbidden

2023-09-04 13:54:27.770: error: Importato App

2023-09-04 13:54:35.446: error: Forbidden: [828490272]

2023-09-04 13:56:23.219: error: Forbidden

2023-09-04 13:57:36.673: error: Forbidden

2023-09-04 13:58:06.099: error: Forbidden

2023-09-04 14:04:10.621: info: Update interval was set to 60 (reason: last check was 1970-01-01 01:00:00.000).

2023-09-04 14:04:22.833: error: Forbidden

2023-09-04 14:05:00.336: error: Importato App

2023-09-04 14:05:03.667: error: Forbidden: [828490272]

2023-09-04 14:05:16.646: error: Forbidden: [828490272]

2023-09-04 14:05:19.830: error: Forbidden: [828490272]

2023-09-04 14:10:11.430: error: Forbidden: [828490272]

2023-09-04 14:11:08.079: info: Update interval was set to 60 (reason: last check was 1970-01-01 01:00:00.000).`

Please complete the following information:

  • Device: Pixel 7
  • OS: Android 13
  • Obtainium Version: v1.14.10-beta

Additional context
This phone Is new and on my old phone it has never happened.
@luciobortoletto luciobortoletto added the bug Something isn't working label Sep 4, 2023
@ImranR98
Copy link
Owner

ImranR98 commented Sep 4, 2023

Although the source is GitLab, the APKs are actually hosted on auroraoss.com. It looks like they've added some CloudFlare protection to prevent web scraping. It is probably possible to bypass, but no combination of headers I tried seems to work.

This is not exactly a bug in Obtainium since the GitLab Source does still work fine.

@ImranR98 ImranR98 added enhancement New feature or request and removed bug Something isn't working labels Sep 4, 2023
@ImranR98 ImranR98 changed the title Can't install AuroraStore from GitLab Improve Web Scraping to Enable AuroraStore Downloads (Through the GitLab Source) Sep 4, 2023
@ImranR98
Copy link
Owner

ImranR98 commented Sep 4, 2023

For some reason the following curl command works: curl -A "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36" --output A.apk https://auroraoss.com/AuroraStore/Stable/AuroraStore_4.3.1.apk

But setting the exact same user agent in the Flutter HTTP client doesn't work.

@ImranR98 ImranR98 added the help wanted Extra attention is needed label Sep 6, 2023
@ImranR98 ImranR98 mentioned this issue Sep 12, 2023
Closed
@Marocco2
Copy link
Contributor

For some reason the following curl command works: curl -A "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36" --output A.apk https://auroraoss.com/AuroraStore/Stable/AuroraStore_4.3.1.apk

But setting the exact same user agent in the Flutter HTTP client doesn't work.

Kinda like a random idea but did you check if your http library does inject some arbitrary headers? May led to some nasty surprises

@ImranR98
Copy link
Owner

Kinda like a random idea but did you check if your http library does inject some arbitrary headers? May led to some nasty surprises

Only if you don't specify headers yourself, which I did

@ImranR98 ImranR98 mentioned this issue Jan 11, 2024
Open
@ceskyDJ
Copy link

ceskyDJ commented Jan 11, 2024

Hmm, interesting. Co when you catch request from Flutter and from curl, they are the same? Aren't here some different library for HTTP requests for Dart?

@ImranR98
Copy link
Owner

I don't know if they're exactly the same, but the host, user-agent, and accept headers are. Yeah I could try a different library.

@ImranR98
Copy link
Owner

Just realized this is a duplicate of #709 (the only difference being that this one goes through GitLab - the actual problem is with the Aurora site).

@ImranR98 ImranR98 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 12, 2024
@ImranR98 ImranR98 removed the help wanted Extra attention is needed label Jan 12, 2024
@DwainZwerg DwainZwerg mentioned this issue Jan 31, 2024
Closed
@ThreeDeeJay ThreeDeeJay mentioned this issue Feb 8, 2024
Closed
This was referenced Feb 10, 2024
Closed
Closed
akramer-zibra added a commit to akramer-zibra/Obtainium that referenced this issue Feb 19, 2024
@akramer-zibra
Set referer header for gitlab app source context
cfa4c68 
There occur issues if a gitlab project refers in its release data to external hosted .apk file. In some cases (e.g. Aurora Store) download is not possible because cloudflare protection gives "forbidden" error. The referer header seems to pacify this cloudflare protection. Tested with Android 14 in an AVD emulator.

Related to: ImranR98#1397, ImranR98#1389, ImranR98#1384, ImranR98#1382, ImranR98#1381, ImranR98#1380, ImranR98#1359, ImranR98#854, ImranR98#785, ImranR98#697
This was referenced Feb 19, 2024
Closed
Merged
@DwainZwerg
Copy link
Contributor

Hello @luciobortoletto ,
according to https://gitlab.com/AuroraOSS/AuroraStore/-/issues/1052#note_1778388859, the Aurora Store devs have fixed the problem. The download of AuroraStore should work normally again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Marocco2 @ImranR98 @ceskyDJ @luciobortoletto @DwainZwerg