waitForNavigation is never resolved

[qnoox]

Hello,

Issue:
Tried passing a waitUntil option after a while there is an error.

Error Message:
Uncaught ExtractrIo\Rialto\Exceptions\Node\FatalException: Navigation Timeout Exceeded: 30000ms exceeded.

Code:
$page->waitForNavigation([ 'waitUntil' => 'networkidle0', ]);

Why was the waitUntil used:
Wanted to add in a wait as i am taking screenshots of a lot of pages and some screenshot does not
return the data for that page as it does not have time to load 100%.

Might be related not sure:
puppeteer/puppeteer#257

Might you have any other solutions to this issue?

Thank you in advance.

[nesk]

Without a code example, the only thing I can think of is to increase the navigation timeout:

$puppeteer = new Puppeteer([
    'read_timeout' => 65, // In seconds
]);

$puppeteer->launch()->newPage()->goto($url, [
    'timeout' => 60000, // In milliseconds
]);

Please, provide a reproducible example if your issue is not solved by this code.

[eldair]

@nesk could you please provide an example of working waitForNavigation ?

$puppeteer = new Puppeteer([
    'read_timeout' => 300,
]);

$browser = $puppeteer->launch();
$page = $browser->newPage();
$page->waitForNavigation();
$page->goto('https://google.com');
$browser->close();

This snippet always throws the same error as reported above

[nesk]

Ooh, thanks @eldair, and sorry @qnoox, there is a real issue here. I don't know how I didn't see it before…

Here's what's happening internally: when an instruction is sent to Node, it is executed with the await keyword, which will make the process stay on hold until the promise returned by the instruction is resolved.

It's working for the majority of the cases, but here the waitForNavigation method should be executed before goto, however the promise returned by waitForNavigation will be resolved only once the navigation is done. We have a chicken and egg problem here…

Maybe I should provide a modifier allowing to execute an instruction without await? Something like this:

$page = (new Puppeteer)->launch()->newPage();

// The promise is returned instead of being awaited, due to the "lazy" modifier.
$navigationPromise = $page->lazy->waitForNavigation();

$page->goto('https://google.com');

$navigationPromise->then(function() {
    var_dump('Navigation done!');
});

What do you think of this?

Until it's fixed, you can use the options of the goto method to wait properly for the navigation.

[liamcharmer]

Did this ever get solved @nesk

[jonnywilliamson]

I'm starting to run into this problem a lot now too.

[dfuse-dev]

any workaround for this? I click on a button which opens a new page in a new tab but browser pages doesnt update with this new one

[feralheart]

I'm starting to run into this problem, too

[leocavalcante]

For those looking for a workaround, I came out with this little jerry-rig with the help of Symfonys DomCrawler and CSS Selector, works for me at least.

Just wrap newPage:

$page = new WaitPageDecorator($browser->newPage());

Then you can:

$page->waitContext()
    ->waitFor('selector.in-the[new=page]')
    ->tap('selector.in-the[new=page]');

class WaitPageDecorator
{
    private Page $page;
    private int $throttleThreshold;
    private string $lastExecContextId;

    public function __construct(Page $page, int $throttleThreshold = 2222500)
    {
        $this->page = $page;
        $this->throttleThreshold = $throttleThreshold;
    }

    public function __call($name, $arguments)
    {
        if (substr($name, 0, 4) === 'wait') {
            return call_user_func_array([$this, $name], $arguments);
        }

        return call_user_func_array([$this->page, $name], $arguments);
    }

    public function waitContext(): self
    {
        $execContextId = $this->waitExecContextId();

        if (!isset($this->lastExecContextId)) {
            $this->lastExecContextId = $execContextId;
        }

        while ($execContextId === $this->lastExecContextId) {
            $this->waitThrottle();
            $execContextId = $this->waitExecContextId();
        }

        return $this;
    }

    public function waitFor(string $selector, int $timeout = 30): self
    {
        // TODO: Implement timeout

        while (!$this->waitContains($selector)) {
            $this->waitThrottle();
        }

        return $this;
    }

    public function waitEval(string $selector, string $jsFuncBody): string
    {
        return $this->page->querySelectorEval($selector, JsFunction::createWithParameters(['elem'])->body($jsFuncBody));
    }

    public function waitContains(string $selector): bool
    {
        $bodyHtml = $this->waitEval('body', 'return elem.innerHTML');
        $crawler = new Crawler($bodyHtml);
        return $crawler->filter($selector)->count() > 0;
    }


    private function waitExecContextId(): string
    {
        /** @var ExecutionContext $context */
        $context = $this->page->mainFrame()->executionContext();
        return $context->getResourceIdentity()->uniqueIdentifier();
    }

    private function waitThrottle(): void
    {
        usleep($this->throttleThreshold);
    }
}

Maybe you will need to adjust the throttling threshold.

[imtiazwazir]

@leocavalcante, thank you but your example is not working

[ThaDaVos]

Any ETA for a fix?

[defaultpage]

@nesk: Maybe I should provide a modifier allowing to execute an instruction without await

That would be great. We could use the Amp's event loop and write code like this:

public function testExample()
{
    $puppeteer = new Puppeteer();

    $browser = yield $puppeteer->launch();
    $page = yield $browser->newPage();

    $page->click('html form button');
    yield $page->waitForNavigation();

    yield $browser->close();
});

[tshakah]

I also get this when I click on something and then wait for the new page to load.

In puppeteer you can wrap several commands in an await (see https://stackoverflow.com/a/52212395) - could we have a function that takes a closure and then that becomes a promise in JS?

[stanolacko]

I solved this by some trick.

1. Add some random class name to < body > element
2. Do click where you want
3. Make loop which check if < body > still has added class name
4. Here need to made some timeout check for not crash puppeteer browser instance