Add new mapping for google play changes

[pvomhoff]

I fixed the gplay.app function to be compatible with the old AND new Google Play Store Version.

It looks like that Google doesnt provide all of the information as before the change.
I wasnt able to map the following properties:

• size
• editorsChoice
• features

[Mudrakula]

hey! Assign @facundoolano as a reviewer please

[danheim]

Hi @facundoolano
Could you please take a look, it's emergency

[Lghher]

Looks like it's work for gplay.app, but how to solve the problem of gplay.search。This function (gplay.search) always returns null list

[kitsunde]

FYI.

If you need it installed you can do:

"google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/545/head"

Additionally, upgrading this broke because cheerio because of issues relating to cheeriojs/cheerio#2545 cheerio rc.11. I recommend adding:

"cheerio": "1.0.0-rc.10"

[ronald-hove]

FYI.

If you need it installed you can do:

"google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/545/head"

Additionally, upgrading this broke because cheerio because of issues relating to cheeriojs/cheerio#2545 cheerio rc.11. I recommend adding:

"cheerio": "1.0.0-rc.10"

This works - 2022/05/27

[maciejmackowiak]

Hi @pvomhoff
Is the mapping always working for you?
I've noticed that sometimes the correct data for example for version is in ['ds:5', 1, 2, 140, 0, 0, 0] instead of the ds:4 block

For example if you check the source of those pages:
https://play.google.com/store/apps/details?id=com.mojang.minecraftpe&gl=us
https://play.google.com/store/apps/details?id=com.mojang.minecraftpe&gl=tw

Or am I'm missing something?

[pvomhoff]

Hi @maciejmackowiak

Hmm you found an interesting problem. It looks like that Google has a different mapping for some countries.
The ds:4 mapping works great for me in all European and American countries. Some Asian countries have a complete different mapping 🤔
The ds5 only works is specific countries.

To solve this issue we need to introduce multiple mapping depending on the country.

Does the ds4 mapping works for you with the country US?

[pvomhoff]

@kitsunde Makes sense. I pinned the cherrio version till the problem is fixed

[facundoolano]

instead of having each property an array of two elements, I suggest that you have separate mapping objects, your code choses the correct one at the beginning and the rest remains the same with whatever mapping was selected.

This will also help considering that it's not only legacy vs new but also mapping values change by country

[pvomhoff]

Ahh good point 👍

I split the mapping in a general.v1 and in a general.v2 file. Later on we can also have different mappings by country here.

[baguse]

@pvomhoff i was try with your code, but the search result still return empty array

[pvomhoff]

@baguse This PR fixes the gplay.app function only.
Unfortunately the other functions like gplay.search and ´gplay.list` need some more attention 😞

[baguse]

@pvomhoff sadly, i need the search feature too

[baguse]

@pvomhoff im so sorry if maybe i spam in here, but can you help me to fix the search issue. I will donate some money if you fix that immediately.

[Giszmo]

I suspect the rollout will be finished before this update hits npm. Yesterday I got a 100% error rate with 1857 apps. Maybe not worth keeping lib/mapping/app/general.v1.js around.

That said, using "google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/545/head" I had a 100% success rate today.

[andrew-curry]

FYI.

If you need it installed you can do:

"google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/545/head"

Additionally, upgrading this broke because cheerio because of issues relating to cheeriojs/cheerio#2545 cheerio rc.11. I recommend adding:

"cheerio": "1.0.0-rc.10"

how to do "google-play-scraper": "git+https://github.com/facundoolano/google-play-scraper#pull/545/head",i dont know where to make this change

[kitsunde]

In your package.json, if you still have problems this is more of a beginners question for StackOverflow since it's a basic question about npm. See for example https://stackoverflow.com/questions/17509669/how-to-install-an-npm-package-from-github-directly

You can read about how npm works here https://docs.npmjs.com/cli/v6/configuring-npm/package-json#git-urls-as-dependencies

[ysb]

Hi,
I've created new appList mappings, I can make a PR after this merged.
The biggest problem is there is no category top charts page on play.google.com. Or I couldn't find. If we can find cluster links of category top charts, I can make the revision on code.

[pvomhoff]

@ysb Ohh cool. I was also looking in the the list mapping. I think saw a way via an ajax call to get the top charts.
Happy to discuss the implementation with you. 🙂

[ysb]

@pvomhoff Sure, I think new pagination (batchExecute endpoint) is OK. I just need a start point for category top charts. There is a slider on category pages but it's limited with 45. There is no token for next page.

[pvomhoff]

@ysb Yeah thats exactly the AJAX endpoint i was referring to. For me it looks like that they actually limited the top charts to just 45 entries. 😞
The category request work well with the cursor token

[ysb]

@pvomhoff Yeah, I believe it should be somewhere but we couldn't find yet. I didn't get the point why they hide such a useful information for regular visitors.

[tayhalla]

Looks like some of the list URL path construction changed too. For ex, a list call w collection: TOP_FREE, category: SPORTS used to direct to https://play.google.com/store/apps/top/category/SPORTS?hl=us&gl=US

But the new URL on the redesign is: https://play.google.com/store/apps/category/SPORTS?hl=en_US&gl=US

It seems they eliminated the collection from the path entirely. Would repairing the list construction be in scope for this PR?

[ChayaGel]

try to take this fix to my project,
but fails:
UnhandledPromiseRejectionWarning: Error: Error requesting Google Play:Invalid URL: undefined/store/apps/details?id=...
what is it?

(and search not work, get "[]")

[ChayaGel]

try to take this fix to my project, but fails: UnhandledPromiseRejectionWarning: Error: Error requesting Google Play:Invalid URL: undefined/store/apps/details?id=... what is it?

(and search not work, get "[]")

I see that BASE_URL= undefined
In the meantime, I entered the address into BASE_URL manually and it works,
It is not clear why it does not come from the configuration
Maybe someone can peek at it?

And the search does not work for me - returns an empty array
Is it supposed to work now? Is this a bug in me or has it not been fixed yet?

[pavi2410]

Working good! https://github.com/pavi2410/PlayBadges

[mobile-challenges]

Hi, I test this fix in an internal tool of my job and works fine!!!  👏🏻

[NathanManning]

Not sure if it's within the scope of this PR but search still isn't working.

[andrew-curry]

is it still works now? i got [] from both app() and list()

[dave-filion]

Yeah still not getting back a list of apps from the query... not seeing any errors, but also not getting any data back.

[dave-filion]

Yeah still not getting back a list of apps from the query... not seeing any errors, but also not getting any data back.

[facundoolano]

closing this as it should be fixed by #557

[kitsunde]

closing this as it should be fixed by #557

This also contains a fix for cheerio as the latest version 1.0.0-rc11 breaks #545 (comment), could we get that into main?

[facundoolano]

do you mind sending a separate PR with just that change?

[YairLeviMevorach]

Again, any fix?