[SITES] patchwork.kernel.org

[palfrey]

First please check that it is really an issue with the library, and not some special case of website:

[x] There is no paywall
[x] You do not have to be logged in to see the articles
[x] You tried using a common browser user agent in your configuration / call
[x] The website is not in the list of well known problematic sites

Your report as follows:

Website that does not parse correctly:

https://patchwork.kernel.org/

Some sample urls that I have tried

https://patchwork.kernel.org/project/linux-kselftest/cover/[email protected]/

The exact code i used to test this articles/website

  article = Article(url, fetch_images=False, follow_meta_refresh=True)
  article.download()
  article.parse()

** What parts of the article are missing / not parsed correctly **

[ ] Title
[x] Text Content
[ ] Publication Date
[ ] Authors
[ ] Images
[ ] Movies
[ ] Other, please specify:

Other information, remarks, messages, etc:

[gzanfardino]

Doesn't really look like a newspaper, is it within the project scope to support this kind of websites?

[AndyTheFactory]

The problem is that all replies are in divs with "comment" class

The module removes all these since in most news sites these are down in the comments section, and thus removed from the article body

These classes are defined here

newspaper4k/newspaper/cleaners.py

Lines 25 to 38 in c5e4170

	self.remove_nodes_re = (
	"^side$|combx|retweet|mediaarticlerelated|menucontainer|"
	"navbar|storytopbar-bucket|utility-bar|inline-share-tools"
	"|comment|PopularQuestions|contact|foot|footer|Footer|footnote"
	"|cnn_strycaptiontxt|cnn_html_slideshow|cnn_strylftcntnt"
	"|links|meta$|shoutbox|sponsor"
	"|tags|socialnetworking|socialNetworking|cnnStryHghLght"
	"|cnn_stryspcvbx|^inset$|pagetools|post-attributes"
	"|welcome_form|contentTools2|the_answers"
	"|communitypromo|runaroundLeft|subscribe|vcard|articleheadings"
	"|date|^print$|popup|author-dropdown|tools|socialtools|byline"
	"|konafilter|KonaFilter|breadcrumbs|^fn$|wp-caption-text"
	"|legende|ajoutVideo|timestamp|js_replies"
	)