[Bug] My software stucks on browser.close with plugin-stealth 2.7.x version #421

Dam998 · 2021-02-04T22:45:46Z

My software stucks on browser.close with last version (puppeteer-extra-plugin-stealth)

I'm using more than 1 browser open at the same time, and when I'm trying to close 1 my software stucks.
this has never happened with version 2.6.7 or before

my code:

var pages = await this.browser.pages()
for (let i = 0; i < pages.length; i++) {
    await pages[i].close();
}
await this.browser.close();

Tested with 2.7.x version

if i downgrade to 2.6.7 everything works fine

berstend · 2021-02-04T22:52:33Z

How are you opening the browsers, .launch or .connect?

Dam998 · 2021-02-04T22:53:16Z

.launch

berstend · 2021-02-04T22:54:00Z

Could you remove the user-agent-override evasion and test if it works then?

const pluginStealth = require('puppeteer-extra-plugin-stealth')()
pluginStealth.enabledEvasions.delete('user-agent-override')
puppeteer.use(pluginStealth)

berstend · 2021-02-04T22:55:35Z

The issue might be related to #413 - we save some state there and added functionality depending on how the browser is launched

Dam998 · 2021-02-04T22:58:12Z

if I remove user-agent-override is working

can I ask you why? if this is correct and nothing change at the end
or if you suggest me to launch browser in a different way

berstend · 2021-02-04T23:10:16Z

So it works without the user-agent-override evasion?

The reason is to narrow down the problem, we made a larger update to the evasion so it makes sense the issue is somewhere there.

If removing user-agent-override fixes the issue in 2.7.x then you just need to wait for us to develop a fix, until then you need to use the older version for your use-case.

berstend · 2021-02-04T23:14:57Z

@Dam998 did you mean opening multiple browsers or multiple pages in a single browser?

Your words indicate the former, your code the latter

Dam998 · 2021-02-04T23:17:40Z

opening multiple browsers (more browsers, each browser with 2 pages)

I created a class that on start() open a browser and then close it at the end, this is why

Niek · 2021-02-04T23:18:20Z

@Dam998 do you also mind sharing a minimal reproducible code sample so we can check?

berstend · 2021-02-04T23:18:36Z

Are you able to make a small self-contained test file? this helps us reproduce the issue quickly. Thanks!

Dam998 · 2021-02-04T23:18:58Z

I'll try, give me some minutes

Dam998 · 2021-02-05T00:31:50Z

I reproduced it with this simple script:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const locateChrome = require('locate-chrome');
const delay = require('delay');

(async () => {
    console.log('start')
    for (var i = 0; i < 20; i++) {
        (async () => {
            console.log('start')
            try {
                var chromeOptions = {
                    headless: false,
                    ignoreDefaultArgs: ['--disable-extensions'],
                    ignoreHTTPSErrors: true,
                    args: [
                        '--disable-web-security',
                        '--disable-dev-shm-usage',
                        '--no-sandbox',
                        '--disable-setuid-sandbox',
                        '--disable-gpu',
                        '--ignore-certificate-errors',
                        '--enable-features=NetworkService'
                    ],
                };

                let chromePath = await locateChrome()
                chromeOptions.executablePath = chromePath;

                puppeteer.use(StealthPlugin())
                const browser = await puppeteer.launch(chromeOptions);
                const page = await browser.newPage();
                
                await page.goto('https://github.com/berstend/puppeteer-extra/issues/421', { 'waitUntil': 'domcontentloaded', timeout: 60000 });
                await delay(13000);

                var pages = await browser.pages()
                for (let j = 0; j < pages.length; j++) {
                    await pages[j].close();
                }
                await browser.close();
            }
            catch (err) {
                console.log(err)
            }
        })();
        await delay(10000);
    }
})();

this script just open more browsers and then close them continusly
with 2.6.7 version everything works good, with 2.7.x after the first browser close, the script freezes

and with 2.7.x if you try to do CTRL + C in terminal it not wll close, with 2.6.7 yes

berstend · 2021-02-05T00:36:48Z

Thanks! can't test right now but does it make a difference if you move puppeteer.use(StealthPlugin()) outside of the loop? It's not necessary to do that 20 times, once is enough. :-)

Dam998 · 2021-02-05T00:39:19Z

nothing change

this is only an example, I made this little mistake here but nothing change if I move it outside of the loop

berstend · 2021-02-05T00:39:59Z

Gotcha. Thanks for the test case, we'll look into it. :-)

Niek · 2021-02-18T14:23:20Z

I cannot reproduce this issue, your test script works fine for me @Dam998. I only got some memleak warnings but that's to be expected:

$ node test.js
start
start
start
start
start
start
start
start
start
start
(node:70151) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:70151) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:70151) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:70151) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit
start
start
start
start
start
start
start
start
start
start
start

@Dam998 can you run the following script and paste the output in a comment?

npx envinfo@latest --system --binaries --npmPackages '*(puppeteer*|playwright*|automation-extra*|@extra*)'

Dam998 · 2021-02-18T21:48:58Z

@Niek

Dam998 · 2021-02-18T21:54:38Z

I tested again my script with 2.7.4 version and I'm getting same error
with 2.6.7 works fine

berstend · 2021-02-19T11:01:39Z

@Dam998 pretty sure this is related to you using Windows in one way or another - our core team is all on POSIX.

Are you using WSL to run this stuff?

berstend · 2021-02-19T11:04:20Z

The issue is most likely somewhere in the user-data-dir plugin:
https://github.com/berstend/puppeteer-extra/blob/master/packages/puppeteer-extra-plugin-user-data-dir/index.js

Can you run your script with DEBUG=puppeteer-extra-plugin:user-data-dir myscript.js?

Dam998 · 2021-02-19T16:38:41Z

with version 2.7.4:

$ DEBUG=puppeteer-extra-plugin:user-data-dir node testBug.js

start
start
  puppeteer-extra-plugin:user-data-dir initialized {
  deleteTemporary: true,
  deleteExisting: false,
  files: [],
  folderPath: 'C:\\Users\\dpapi\\AppData\\Local\\Temp',
  folderPrefix: 'puppeteer_dev_profile-'
} +0ms
  puppeteer-extra-plugin:user-data-dir created custom dir C:\Users\dpapi\AppData\Local\Temp\puppeteer_dev_profile-sCe9ey +5ms
  puppeteer-extra-plugin:user-data-dir Wrote file C:\Users\dpapi\AppData\Local\Temp\puppeteer_dev_profile-sCe9ey\Default\Preferences +12ms
start
  puppeteer-extra-plugin:user-data-dir created custom dir C:\Users\dpapi\AppData\Local\Temp\puppeteer_dev_profile-O8kLVd +10s
  puppeteer-extra-plugin:user-data-dir Wrote file C:\Users\dpapi\AppData\Local\Temp\puppeteer_dev_profile-O8kLVd\Default\Preferences +1ms
  puppeteer-extra-plugin:user-data-dir onClose +5s
  puppeteer-extra-plugin:user-data-dir removeUserDataDir +1ms

with version 2.7.4 everything stuck, the terminal too doesn't responde, I need always to force close it
but only if I reproduce this situation, if I use only 1 browser at a time there are no problems

Niek · 2021-02-20T10:17:20Z

@berstend what do you think about bumping fs-extra to @9? See also: https://github.com/jprichardson/node-fs-extra/blob/master/CHANGELOG.md

I think this might be the main cause: jprichardson/node-fs-extra#804

berstend · 2021-02-20T11:15:04Z

Yeah, bumping fs-extra might fix this issue

Niek · 2021-03-14T20:49:37Z

@Dam998 Please check with the latest master branch.

KhalilMohammad · 2021-08-16T16:39:38Z

I get this issue if I use puppeteer-extra with puppeteer-page-proxy.
It freezes on browser.close()

alamothe · 2021-09-02T10:56:04Z

@k-funk @KhalilMohammad this issue is Windows specific, please create a separate issue. Thanks!

LinkTree3 · 2021-09-08T15:35:57Z

I am seeing the same issue right now.
I am on windows and when I add 'puppeteer-extra-plugin-stealth' and use puppeteer.use(StealthPlugin()) I can't use ctrl+c to terminate the process. :(

Is there any solution for this?

I ended up reverting to 2.6.7 as well. It works for me for now. :)

berstend · 2021-09-13T07:34:25Z

Alright, to summarize what we know so far:

The reason this happens in version 2.7 and up is that we started using the puppeteer-extra-plugin-user-data-dir plugin in the user-agent-override evasion (unfortunately that's necessary to properly spoof locales)

As to why the user-data-dir plugin causes this behavior is not entirely clear yet, but my gut tells it must be this code:

puppeteer-extra/packages/puppeteer-extra-plugin-user-data-dir/index.js

Lines 111 to 116 in 0049d60

    
           async onClose() { 
        
             debug('onClose') 
        
             if (this.shouldDeleteDirectory) { 
        
               this.deleteUserDataDir() 
        
             } 
        
           }

Can someone experiencing this issue monkey patch their ./node_modules/puppeteer-extra-plugin-user-data-dir/index.js file and comment out this onClose handler to see if this fixes it?

berstend · 2021-09-13T07:37:45Z

more specifically the onClose handler will result in this sync call:

puppeteer-extra/packages/puppeteer-extra-plugin-user-data-dir/index.js

Lines 67 to 76 in 0049d60

    
           deleteUserDataDir() { 
        
             debug('removeUserDataDir') 
        
             try { 
        
               // We're doing it sync to improve chances to cleanup 
        
               // correctly in the event of ultimate disaster. 
        
               fse.rmdirSync(this._userDataDir, { recursive: true }) 
        
             } catch (e) { 
        
               debug(e) 
        
             } 
        
           }

LinkTree3 · 2021-09-13T07:39:44Z

Alright, to summarize what we know so far:

The reason this happens in version 2.7 and up is that we started using the puppeteer-extra-plugin-user-data-dir plugin in the user-agent-override evasion (unfortunately that's necessary to properly spoof locales)

As to why the user-data-dir plugin causes this behavior is not entirely clear yet, but my gut tells it must be this code:

puppeteer-extra/packages/puppeteer-extra-plugin-user-data-dir/index.js

Lines 111 to 116 in 0049d60

async onClose() {

debug('onClose')

if (this.shouldDeleteDirectory) {

this.deleteUserDataDir()

}

}

Can someone experiencing this issue monkey patch their ./node_modules/puppeteer-extra-plugin-user-data-dir/index.js file and comment out this onClose handler to see if this fixes it?

Hey thank you so much for looking into this! Yes I'll gladly check this and report. :)

LinkTree3 · 2021-09-13T07:44:47Z

Yes your suggestion worked! :)

Now how do we fix it permanently without needing to resort to monkey patching?

berstend · 2021-09-13T07:48:38Z

@LinkTree3 Cool, thanks for testing :)

Would you be able to test if this change makes a difference (commenting the onClose handler back in)?
https://github.com/berstend/puppeteer-extra/pull/502/files

(basically replacing rmdirSync with rmSync)

dev-hyperweb · 2021-09-13T07:49:17Z

it clearly. this issued stuck in onClose event

so if im not wrong, even empty event still stuck
in now, I fixed with onDisconnected event to use for my project

LinkTree3 · 2021-09-13T07:52:14Z

rmSync

This did not work.

berstend · 2021-09-13T07:54:00Z

rmSync

This did not work.

Ok, one last test :)

fs.rm(this._userDataDir, { recursive: true, force: true }, console.log)

LinkTree3 · 2021-09-13T07:54:05Z

~~YUP This did the trick! :)~~

berstend · 2021-09-13T08:00:44Z

YUP This did the trick! :)

Sweet! Thanks for testing 👍 So we know that (for whatever reason) removing the temporary profile folder synchronously can cause issues on windows, whereas doing it asynchronously works fine.

I'm gonna prepare a fix + new version within the next days, as time permits.

LinkTree3 · 2021-09-13T08:02:00Z

YUP This did the trick! :)

Sweet! Thanks for testing 👍 So we know that (for whatever reason) removing the temporary profile folder synchronously can cause issues on windows, whereas doing it asynchronously works fine.

I'm gonna prepare a fix + new version within the next days, as time permits.

Thank you so much! looking forward to it :)

LinkTree3 · 2021-09-13T08:21:57Z

berstend I am so sorry, but it seems that I might have had a mistake while testing it with
fs.rm(this._userDataDir, { recursive: true, force: true }, console.log)

Now when I try it again it seems to not work :(

I am getting an EBUSY error and a bunch of "null" when I try to exit

[Error: EBUSY: resource busy or locked, unlink 'C:\Users[username]\AppData\Local\Temp\puppeteer_dev_profile-1BHM2o\lockfile'] {
errno: -4082,
code: 'EBUSY',
syscall: 'unlink',
path: 'C:\Users\[username]\AppData\Local\Temp\puppeteer_dev_profile-1BHM2o\lockfile'
}

berstend · 2021-09-13T08:46:51Z

@LinkTree3 no worries :) could you try if changing onClose to onDisconnected (with the original fse.rmdirSync code) does make a difference in your case?
https://github.com/berstend/puppeteer-extra/pull/530/files

LinkTree3 · 2021-09-13T09:11:35Z

@LinkTree3 no worries :) could you try if changing onClose to onDisconnected (with the original fse.rmdirSync code) does make a difference in your case?
https://github.com/berstend/puppeteer-extra/pull/530/files

Yes this one seems to work! I tested it a couple times to make sure :)

I also added a console log right before and after the fse.rmdirSync call and they both fired after the puppeteer browser closed.

berstend · 2021-09-13T19:26:31Z

@LinkTree3 awesome, we're gonna go with this fix then

berstend · 2021-09-17T00:48:05Z

This should be fixed now through #530 (thanks @dev-hyperweb 👍 )

Successfully published:
 - [email protected]
 - [email protected]
 - [email protected]

cc @LinkTree3 @Dam998

berstend · 2022-07-08T08:36:43Z

More properly fixed in #656

Dam998 added issue: bug report A bug has been reported needs triage labels Feb 4, 2021

Dam998 changed the title ~~[Bug] My software stucks on browser.close with 2.7.x version~~ [Bug] My software stucks on browser.close with plugin-stealth 2.7.x version Feb 4, 2021

Niek added a commit that referenced this issue Feb 22, 2021

Upgrade fs-extra, possible fix for #421

d4735e9

Niek mentioned this issue Feb 22, 2021

chore: Upgrade fs-extra, possible fix for #421 #436

Merged

berstend pushed a commit that referenced this issue Mar 14, 2021

chore: Upgrade fs-extra, possible fix for #421 (#436)

c408808

berstend added the ongoing-research label Sep 13, 2021

berstend mentioned this issue Sep 13, 2021

fs.rmdirSync() deprecated, use rmSync() #502

Closed

berstend added work-in-progress This is currently being worked on and removed ongoing-research labels Sep 13, 2021

dev-hyperweb mentioned this issue Sep 13, 2021

fix(plugin-user-data-dir): Use onDisconnect instead of onClose #530

Merged

berstend closed this as completed in 713f9fa Sep 17, 2021

This was referenced Oct 20, 2021

[Bug] puppeteer-extra-plugin:user-data-dir.onClose called twice and causes Error. Also possible memory leak with Plugins #559

Closed

[Bug] puppeteer-extra-plugin:user-data-dir.onDisconnected hangs when trying to delete a profile. #581

Closed

josep11 mentioned this issue Dec 29, 2021

[Bug] Chromium crashes on browser.close() when using plugin-stealth 2.9.0 #601

Closed

[Bug] My software stucks on browser.close with plugin-stealth 2.7.x version #421

[Bug] My software stucks on browser.close with plugin-stealth 2.7.x version #421

Comments

Dam998 commented Feb 4, 2021 • edited Loading

berstend commented Feb 4, 2021

Dam998 commented Feb 4, 2021

berstend commented Feb 4, 2021

berstend commented Feb 4, 2021

Dam998 commented Feb 4, 2021

berstend commented Feb 4, 2021

berstend commented Feb 4, 2021

Dam998 commented Feb 4, 2021

Niek commented Feb 4, 2021

berstend commented Feb 4, 2021

Dam998 commented Feb 4, 2021

Dam998 commented Feb 5, 2021

berstend commented Feb 5, 2021

Dam998 commented Feb 5, 2021

berstend commented Feb 5, 2021

Niek commented Feb 18, 2021 • edited Loading

Dam998 commented Feb 18, 2021

Dam998 commented Feb 18, 2021

berstend commented Feb 19, 2021

berstend commented Feb 19, 2021

Dam998 commented Feb 19, 2021

Niek commented Feb 20, 2021 • edited Loading

berstend commented Feb 20, 2021

Niek commented Mar 14, 2021

KhalilMohammad commented Aug 16, 2021

alamothe commented Sep 2, 2021

LinkTree3 commented Sep 8, 2021 • edited Loading

berstend commented Sep 13, 2021 • edited Loading

berstend commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021 • edited Loading

berstend commented Sep 13, 2021

dev-hyperweb commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021

berstend commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021 • edited Loading

berstend commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021 • edited Loading

berstend commented Sep 13, 2021

LinkTree3 commented Sep 13, 2021 • edited Loading

berstend commented Sep 13, 2021

berstend commented Sep 17, 2021

berstend commented Jul 8, 2022

Dam998 commented Feb 4, 2021 •

edited

Loading

Niek commented Feb 18, 2021 •

edited

Loading

Niek commented Feb 20, 2021 •

edited

Loading

LinkTree3 commented Sep 8, 2021 •

edited

Loading

berstend commented Sep 13, 2021 •

edited

Loading

LinkTree3 commented Sep 13, 2021 •

edited

Loading

LinkTree3 commented Sep 13, 2021 •

edited

Loading

LinkTree3 commented Sep 13, 2021 •

edited

Loading

LinkTree3 commented Sep 13, 2021 •

edited

Loading