feat(store): specify links to navigate to between product searches #1542

neatchee · 2020-12-29T09:01:42Z

...to reduce likelihood of bot detection/captcha

Description

Fixes #1504
When querying a store, if deterrentLinks has been defined, pick one at random and navigate there then wait 3 seconds. This should help reduce the likelihood of being sent to a captcha on subsequent page loads from the same store.
Defined in store model as 'captchaDeterrent' with parameters:
hardLinks string[]
searchUrl string
searchTerms string[]

Testing

Add the following to amazon.ts (already added in pr)

	captchaDeterrent: {
		hardLinks: ['https://www.amazon.com/Amazon-Video/b/?ie=UTF8&node=2858778011&ref_=nav_cs_prime_video',
			'https://www.amazon.com/alm/storefront?almBrandId=VUZHIFdob2xlIEZvb2Rz&ref_=nav_cs_whole_foods_in_region',
			'https://www.amazon.com/gp/goldbox?ref_=nav_cs_gb'],
		searchUrl: 'https://www.amazon.com/s?k=%%s&i=todays-deals&ref=nb_sb_noss_2',
		searchTerms: ['goober', 'dungeons+and+dragons']
	},

set HEADLESS=false in dotenv
npm start run

New dependencies

None

gigi2006 · 2020-12-29T12:35:07Z

Hey @neatchee it is only amazon US?
Please make it for all Amazon Stores 👍

neatchee · 2020-12-30T00:08:34Z

Hey @neatchee it is only amazon US?
Please make it for all Amazon Stores 👍

This is just the framework and one store setup for it. There will be a lot of work required to add the deterrent links for all the different stores that need it.

undecided2013 · 2020-12-30T17:16:25Z

I merged your changes into my cloned main locally and ran it for a a while. After an hour or so, Amazon captcha started coming up and it did not go away.

p.s. I did not do headless=false, I used true, what is the impact of that?

neatchee · 2020-12-30T21:36:18Z

I merged your changes into my cloned main locally and ran it for a a while. After an hour or so, Amazon captcha started coming up and it did not go away.

p.s. I did not do headless=false, I used true, what is the impact of that?

Headless=true is the default, so it does nothing.

This change only includes a very basic list of links to use for Amazon. Even with a comprehensive list, this change will not protect 100% against captcha. It will still depend on how frequently you query Amazon pages. This is meant to reduce the likelihood by some reasonable amount, not prevent captchas entirely.

undecided2013 · 2020-12-30T23:43:41Z

I merged your changes into my cloned main locally and ran it for a a while. After an hour or so, Amazon captcha started coming up and it did not go away.
p.s. I did not do headless=false, I used true, what is the impact of that?

Headless=true is the default, so it does nothing.

This change only includes a very basic list of links to use for Amazon. Even with a comprehensive list, this change will not protect 100% against captcha. It will still depend on how frequently you query Amazon pages. This is meant to reduce the likelihood by some reasonable amount, not prevent captchas entirely.

Do you think more links would help? Also, I have tweaked the delay between requests to be longer but I guess that doesn't matter to Amazon, their 'cache' of lookups maybe be a hashmap with no expiration on the keys

jef · 2021-01-03T07:09:57Z

I'll look at this in length tomorrow (probably). Thank you @neatchee!

neatchee

Maybe I just don't understand git but...this was merged into main but isn't showing up in main's commit history???

…uct searches to reduce likelihood of bot detection/captcha When querying a store, if deterrentLinks has been defined, pick one at random and navigate there then wait 3 seconds

jef · 2021-01-10T18:03:46Z

Maybe I just don't understand git but...this was merged into main but isn't showing up in main's commit history???

I don't think so! Perhaps misread?

jef · 2021-01-10T18:06:07Z

I wonder for searchTerms if we have a master list of random words rather per store? What do you think of that?

Cool idea though, I like it!

neatchee · 2021-01-10T19:13:23Z

I like the ability to go per-store incase they get wise to searches that return no results, but I'd totally be happy with a default list to use if one isn't specified for the store

…

On Sun, Jan 10, 2021, 10:06 AM Jef LeCompte ***@***.***> wrote: I wonder for searchTerms if we have a master list of random words rather per store? What do you think of that? Cool idea though, I like it! — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH6B7UHZEA4W7QR2EE3SZHUBZANCNFSM4VM7JF7A> .

jef · 2021-01-10T20:37:08Z

Hmm... I guess there are only so many retailers that carry generic products. Perhaps this is fine. No worries! We can stick with this.

jef

Overall looks good, just a suggestion and comment.

Thanks @neatchee!

src/config.ts

jef · 2021-01-10T20:39:01Z

src/store/lookup.ts

+			setTimeout(() => {
+				// Do nothing
+			}, 3000);


How come we want this 3 second timeout?

This is basically an arbitrary "wait after page load so we don't look as bot-like". I was being cautious about us hitting a search page and then immediately bouncing to another page as soon as the search results are loaded. I wanted to at least KINDA look like someone actually viewed the search results hehe.

That makes sense to me :)

Co-authored-by: Jef LeCompte <[email protected]>

kangelovski · 2021-01-14T10:51:18Z

It seems to work. Tried it against german amazon pages.
Here is my amazon-de.ts - captchaDeterrent part:

captchaDeterrent: { hardLinks: [ 'https://www.amazon.de/Amazon-Video/b?ie=UTF8&node=3010075031', 'https://www.amazon.de/dp/B00NTQ6K7E?ref_=nav_em_adbl_nav_sl_link1_0_2_11_2', 'https://www.amazon.de/gp/goldbox?ref_=nav_cs_de' ], searchTerms: ['hirschhausen', 'matrix', 'tefal', 'playstation', 'fifa', 'xbox'], searchUrl: 'https://www.amazon.de/s?k=%%s&i=todays-deals&ref=nb_sb_noss_2' },

neatchee · 2021-01-14T10:58:25Z

What happens if you have headless true? I've tested it with headless and didn't have any issue.

…

On Thu, Jan 14, 2021, 2:51 AM kangelovski ***@***.***> wrote: It seems to work. Tried it against german amazon pages. Notice: It only works with HEADLESS set to false. Good work! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH3PEO5AWRD36FV4IW3SZ3EDLANCNFSM4VM7JF7A> .

kangelovski · 2021-01-14T11:23:14Z

What happens if you have headless true? I've tested it with headless and
didn't have any issue.

EDIT: I've changed some network routing and added some searchTerms and hardLinks, and it does work with headless=true too. (Probably some kind of side effect caused by older bot checks). I will monitor it a little longer.

OLD Comment: The CAPTCHA notification reoccur if HEADLESS is set to true. If HEADLESS is set to false, no CAPTCHA notification occurs.

jontaru · 2021-01-14T16:11:05Z

Testing this and I am still getting captcha with headlless false for amazon-ca, was it merged into the main branch?

kangelovski · 2021-01-14T16:42:30Z

Testing this and I am still getting captcha with headlless false for amazon-ca, was it merged into the main branch?

It's not merged yet. You would need to modify four files by yourself. Instead of amazon.ts, you would need to modify the amazon-ca.ts file.

neatchee · 2021-01-14T18:05:37Z

Oh, yeah, this won't completely defend against captcha, especially if your IP has already been flagged.

…

On Thu, Jan 14, 2021, 3:23 AM kangelovski ***@***.***> wrote: What happens if you have headless true? I've tested it with headless and didn't have any issue. … <#m_-2948378581425577092_> On Thu, Jan 14, 2021, 2:51 AM kangelovski *@*.***> wrote: It seems to work. Tried it against german amazon pages. Notice: It only works with HEADLESS set to false. Good work! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1542 (comment) <#1542 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABA3WH3PEO5AWRD36FV4IW3SZ3EDLANCNFSM4VM7JF7A . The CAPTCHA notification reoccur. If HEADLESS is set to false, no CAPTCHA notification occurs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WHYVG33DMH6NN3I7UCTSZ3H3DANCNFSM4VM7JF7A> .

kangelovski · 2021-01-15T08:40:24Z

I only received CAPTCHAs again. If I switch to non-headless and back to headless again, the CAPTCHAs disappear. I can imagine the changed viewport of the browser triggered the bot-recognation back to "human user"... Maybe we can randomize the viewport parameters too? (I would recommend, that viewport-size is getting changed between each random 60 seconds / 20 minutes)

neatchee · 2021-01-15T09:15:54Z

As I mentioned, this alone won't stop captcha. There are a larger number of factors that contribute to whether or not you get sent to a captcha. This is just one tool in the toolbox and does not guarantee that you won't see a captcha. The most important thing with captcha is request frequency. The fewer stores, the fewer models/products, the more likely you are to get a captcha.

…

On Fri, Jan 15, 2021, 12:40 AM kangelovski ***@***.***> wrote: I only received CAPTCHAs again. If I switch to non-headless and back to headless again, the CAPTCHAs disappear. I can imagine the changed viewport of the browser triggered the bot-recognation back to "human user"... Maybe we can randomize the viewport parameters too? (I would recommend, that viewport-size is getting changed between each random 60 seconds / 20 minutes) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH2LTPHKVQ6HKH3GO5TSZ75QPANCNFSM4VM7JF7A> .

kangelovski · 2021-01-15T09:35:20Z

As I mentioned, this alone won't stop captcha. There are a larger number of factors that contribute to whether or not you get sent to a captcha. This is just one tool in the toolbox and does not guarantee that you won't see a captcha. The most important thing with captcha is request frequency. The fewer stores, the fewer models/products, the more likely you are to get a captcha.

Please don't get me wrong. I like that tool! I don't criticize the occurence of captcha in my test. I'm just trying to support you with my insights.

jef · 2021-01-17T13:24:18Z

Okay, I'm going to merge this in now. Thank you @neatchee for working on this and answering my questions!

kakashi84 · 2021-01-21T11:26:53Z

hi, in a some links amazon there is write captcha .i must do something?

neatchee · 2021-01-21T11:40:50Z

For help please visit the discord server and ask in the #help channel

…

On Thu, Jan 21, 2021, 3:27 AM kakashi84 ***@***.***> wrote: hi, in a some links amazon there is write captcha .i must do something? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WHYJOLTXN7NWMWSHNMTS3AFQ3ANCNFSM4VM7JF7A> .

kakashi84 · 2021-01-21T12:25:18Z

For help please visit the discord server and ask in the #help channel
…
On Thu, Jan 21, 2021, 3:27 AM kakashi84 @.***> wrote: hi, in a some links amazon there is write captcha .i must do something? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1542 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABA3WHYJOLTXN7NWMWSHNMTS3AFQ3ANCNFSM4VM7JF7A .

witch server?

neatchee · 2021-01-21T19:38:29Z

Check the "looking for help" part of this page: https://jef.codes/streetmerchant/

…ef#1542) Fixes jef#1504 When querying a store, if `deterrentLinks` has been defined, pick one at random and navigate there then wait 3 seconds. This should help reduce the likelihood of being sent to a captcha on subsequent page loads from the same store.

neatchee requested a review from jef as a code owner December 29, 2020 09:01

neatchee commented Jan 6, 2021

View reviewed changes

neatchee closed this Jan 9, 2021

neatchee force-pushed the main branch from 3339e0d to 9c2ea8d Compare January 9, 2021 12:20

Specify links (including search + terms) to navigate to between prod…

56276be

…uct searches to reduce likelihood of bot detection/captcha When querying a store, if deterrentLinks has been defined, pick one at random and navigate there then wait 3 seconds

neatchee reopened this Jan 9, 2021

jef reviewed Jan 10, 2021

View reviewed changes

jef and others added 3 commits January 12, 2021 10:41

Merge branch 'main' into main

70f5c26

Merge branch 'main' into main

e017630

Update src/config.ts

8bc1f55

Co-authored-by: Jef LeCompte <[email protected]>

kangelovski approved these changes Jan 14, 2021

View reviewed changes

jef approved these changes Jan 17, 2021

View reviewed changes

Merge branch 'main' into main

654b7d7

jef enabled auto-merge (squash) January 17, 2021 13:25

jef changed the title ~~feat(store): Specify links (including search + terms) to navigate to between product searches...~~ feat(store): specify links to navigate to between product searches Jan 17, 2021

jef merged commit 0982774 into jef:main Jan 17, 2021

vutting4221 mentioned this pull request Jul 31, 2022

[Snyk] Fix for 1 vulnerabilities vutting4221/streetmerchant#82

Open

github-actions bot mentioned this pull request Oct 21, 2022

chore(main): release 3.7.0 #3060

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(store): specify links to navigate to between product searches #1542

feat(store): specify links to navigate to between product searches #1542

neatchee commented Dec 29, 2020 •

edited

Loading

gigi2006 commented Dec 29, 2020

neatchee commented Dec 30, 2020

undecided2013 commented Dec 30, 2020 •

edited

Loading

neatchee commented Dec 30, 2020

undecided2013 commented Dec 30, 2020

jef commented Jan 3, 2021

neatchee left a comment •

edited

Loading

jef commented Jan 10, 2021

jef commented Jan 10, 2021

neatchee commented Jan 10, 2021 via email

jef commented Jan 10, 2021

jef left a comment

jef Jan 10, 2021

neatchee Jan 12, 2021

jef Jan 14, 2021

kangelovski commented Jan 14, 2021 •

edited

Loading

neatchee commented Jan 14, 2021 via email

kangelovski commented Jan 14, 2021 •

edited

Loading

jontaru commented Jan 14, 2021 •

edited

Loading

kangelovski commented Jan 14, 2021 •

edited

Loading

neatchee commented Jan 14, 2021 via email

kangelovski commented Jan 15, 2021

neatchee commented Jan 15, 2021 via email

kangelovski commented Jan 15, 2021

jef commented Jan 17, 2021

kakashi84 commented Jan 21, 2021

neatchee commented Jan 21, 2021 via email

kakashi84 commented Jan 21, 2021

neatchee commented Jan 21, 2021

feat(store): specify links to navigate to between product searches #1542

feat(store): specify links to navigate to between product searches #1542

Conversation

neatchee commented Dec 29, 2020 • edited Loading

Description

Testing

New dependencies

gigi2006 commented Dec 29, 2020

neatchee commented Dec 30, 2020

undecided2013 commented Dec 30, 2020 • edited Loading

neatchee commented Dec 30, 2020

undecided2013 commented Dec 30, 2020

jef commented Jan 3, 2021

neatchee left a comment • edited Loading

Choose a reason for hiding this comment

jef commented Jan 10, 2021

jef commented Jan 10, 2021

neatchee commented Jan 10, 2021 via email

jef commented Jan 10, 2021

jef left a comment

Choose a reason for hiding this comment

jef Jan 10, 2021

Choose a reason for hiding this comment

neatchee Jan 12, 2021

Choose a reason for hiding this comment

jef Jan 14, 2021

Choose a reason for hiding this comment

kangelovski commented Jan 14, 2021 • edited Loading

neatchee commented Jan 14, 2021 via email

kangelovski commented Jan 14, 2021 • edited Loading

jontaru commented Jan 14, 2021 • edited Loading

kangelovski commented Jan 14, 2021 • edited Loading

neatchee commented Jan 14, 2021 via email

kangelovski commented Jan 15, 2021

neatchee commented Jan 15, 2021 via email

kangelovski commented Jan 15, 2021

jef commented Jan 17, 2021

kakashi84 commented Jan 21, 2021

neatchee commented Jan 21, 2021 via email

kakashi84 commented Jan 21, 2021

neatchee commented Jan 21, 2021

neatchee commented Dec 29, 2020 •

edited

Loading

undecided2013 commented Dec 30, 2020 •

edited

Loading

neatchee left a comment •

edited

Loading

kangelovski commented Jan 14, 2021 •

edited

Loading

kangelovski commented Jan 14, 2021 •

edited

Loading

jontaru commented Jan 14, 2021 •

edited

Loading

kangelovski commented Jan 14, 2021 •

edited

Loading