Stop parsing HTML with regex

[elad661]

Parsing HTML with regex is ugly and leads to a lot of problems, yet we do that too much.

We should either use a dedicated HTML parser (say lxml with cssselect or libsoup) or switch to JSON-based APIs wherever we can.

[elad661]

If we take fuckingweather as an example, using cssselect would change

    page = web.get("http://thefuckingweather.com/?where=%s" % (text))
    re_mark = re.compile('<p class="remark">(.*?)</p>')
    results = re_mark.findall(page)

to

    page = lxml.html.parse("http://thefuckingweather.com/?where=%s" % (text)).getroot()
    result = page.cssselect('p.remark')[0].text

Which is simpler, and will work even if more attributes would be added to the <p>

Therefor I'm in favor of introducing cssselect as a dependency.

[elad661]

I guess we can close this.