You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using this package right (more than 500k pages) now and it would be wonderful to have those features:
Write sitemap while crawling. That way user knows everithing is working. At the moment I can only see it's crawling looking at the webserver log. But I have no Idea if, when it ends, it will really write the sitemap or not. And it can took hours (or days).
Use a log with already fetched urls (or the sitemap itself) with an option to restart without doing it from the beginning. But don't know if this is possible due the need to crawl anyway to be sure to include new pages. But if the user knows there are not new pages and he just needed to interrupt the crawler he could restart from (let say) the last 100 crawlder urls.
The text was updated successfully, but these errors were encountered:
I'm using this package right (more than 500k pages) now and it would be wonderful to have those features:
The text was updated successfully, but these errors were encountered: