Crawler not catching all links?

[huglester]

For example, script won't find any childs here:

./vendor/bin/centipede run http://www.altevents.darbai.webas.lt/
200 http://www.altevents.darbai.webas.lt/

Why so? there are more links on that page. but it only checks the mainpage?

I have checked the UrlExtractor class, the extract() method receives valid html,
but DOMDocument does not extract the links. Maybe we should use symfony/dom-crawler ?

Thanks!

[umpirsky]

@huglester Given url is under construction atm. Can you give fresh example please? Thanks.

[huglester]

@umpirsky please try now:)

[umpirsky]

@huglester

[huglester]

Very strange. Have you tried pressing F5?

[umpirsky]

@huglester Ctrl + F5 did the trick.

After commenting out https://github.com/umpirsky/centipede-crawler/blob/master/src/Centipede/Extractor/UrlExtractor.php#L13 I got:

PHP Warning:  DOMDocument::loadHTML(): AttValue: " expected in Entity
PHP Warning:  DOMDocument::loadHTML(): Couldn't find end of Start Tag meta in Entity

Looks like there is invalid html.