Selenium-Profiles was working up until yesterday - any ideas to get around this new strange bot detection?

[wirriejoe]

Hi all,

Probably a long shot, but I am trying to scrape Hyatt's website for points availabilities

https://www.hyatt.com/shop/service/rooms/roomrates/dpsas?spiritCode=dpsas&rooms=1&adults=2&checkinDate=2024-01-05&checkoutDate=2024-01-07&kids=0

Selenium-Profiles has been working like a charm for the past couple of weeks, but yesterday, the site changed its bot detection, so that Selenium-Profiles now returns data with some fields changed to null.

Selenium-Profiles page load:

Manual page load:

Has anyone seen anything like this before? Any advice on how to get around this?

Here's my Selenium-Profile init:

load_dotenv(find_dotenv())
profile = json.loads(os.getenv('SELENIUM_PROFILE_3'))

profile["proxy"] = {
  "proxy": super_proxy_url
}

options = ChromeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
# options.add_argument(f"--proxy-server={super_proxy_url}")
driver = Chrome(profile=profile, options=options,uc_driver=False,injector_options=True, seleniumwire_options=True)
driver.execute_cdp_cmd('Runtime.disable',{})
injector = driver.profiles.injector

[kaliiiiiiiiii]

@wirriejoe
You can try with:

driver = Chrome(profile={}, options=options,injector_options=True,driverless_options=True)

[wirriejoe]

Same problem. The page loads, but the values I'm looking for are null

Values exist when I load manually in Google Chrome

[wirriejoe]

@kaliiiiiiiiii curious if you had a chance to take a look? Is there anything I can do on my end to help debug as well?

[kaliiiiiiiiii]

@wirriejoe Not yet, I'll give you a feedback

[wirriejoe]

Hmm interestingly enough, I ran your selenium_detector.py test and it looks like Selenium is being detected. Any idea how to bypass these detections?

[kaliiiiiiiiii]

@wirriejoe Wait In my env it passes. Can you provide the url to the script you used?

[wirriejoe]

@kaliiiiiiiiii here's my test script

from selenium_profiles.webdriver import Chrome
from selenium_profiles.profiles import profiles
from selenium.webdriver.common.by import By  # locate elements
from selenium.webdriver import ChromeOptions
# Start Driver

profile = profiles.Windows()  # or .Android

options = ChromeOptions()
# options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument('ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument('lang=zh-CN,zh,zh-TW,en-US,en')
options.add_argument('--disable-gpu')
options.add_argument("disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
# options.add_argument(f"--proxy-server={super_proxy_url}")
driver = Chrome(profile={}, options=options, uc_driver=False, injector_options=True, seleniumwire_options=True)
# driver = Chrome(profile={}, options=options, uc_driver=True, driverless_options=True, injector_options=True)
# driver.execute_cdp_cmd('Runtime.disable',{})
# driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
injector = driver.profiles.injector

def click(x,y):
    driver.execute_cdp_cmd("Input.dispatchMouseEvent",{"type":"mousePressed", "x":x, "y":y})
    driver.execute_cdp_cmd("Input.dispatchMouseEvent", {"type": "mouseReleased", "x": x, "y": y})


driver.get("https://hmaker.github.io/selenium-detector/")
driver.execute_cdp_cmd("Runtime.disable",{})


driver.find_element(By.CSS_SELECTOR, '#chromedriver-token').send_keys(
    driver.execute_script('return window.token')
    )
driver.find_element(By.CSS_SELECTOR, '#chromedriver-asynctoken').send_keys(
        driver.execute_async_script('window.getAsyncToken().then(arguments[0])')
    )
driver.find_element(By.CSS_SELECTOR, '#chromedriver-test').click()
input('Test finished, check the result on the browser page.\nPress enter to exit...')
driver.quit()

[kaliiiiiiiiii]

@wirriejoe going the url:https://www.hyatt.com/shop/service/rooms/roomrates/dpsas?spiritCode=dpsas&rooms=1&adults=2&checkinDate=2024-01-05&checkoutDate=2024-01-07&kids=0 directly in chrome geives me the same results (doens't pass)

[wirriejoe]

@kaliiiiiiiiii this is interesting. I just checked in 4 different browsers and the behavior was different

1. Chrome: passed
2. Chrome Incognito: failed
3. Firefox: failed
4. Safari: failed

Let me know if you're able to repro

[wirriejoe]

I just tried clearing my cookie on Chrome and it now fails as well. It looks like I had a cookie saved from previously that was helping me pass.

[kaliiiiiiiiii]

@wirriejoe I suppose that answers your question. Youl'll have to navigate to the main page first for collecting the cookie

[wirriejoe]

@kaliiiiiiiiii Ah, I seem to have found the problem. It looks like they changed the endpoint so "&rateFilter=woh" now needs to be passed into the request URL in order to see the points rates. Weirdly, my browser was showing me a false positive because of an old cookie. I've now fixed it and things are working as intended. Thanks so much for your help!

[kaliiiiiiiiii]

Closing as it seems to be resolved