[roosterteeth] Added new extractor

[ngld]

This extractor allows you to download single videos are whole seasons from roosterteeth.com, achievementhunter.com and fun.haus.

The RoosterteethShowIE allows you to filter videos using a simple regex filter (the second test case contains an example). I've added that feature since I found no other way to do this with YTDL's own filters. I hope you don't mind.

This resolves #6371.

[dstftw]

Broken on python 2:

py26yt "http://roosterteeth.com/show/red-vs-blue#;season=.* 1$" -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'http://roosterteeth.com/show/red-vs-blue#;season=.* 1$', u'-v']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2015.08.09
[debug] Git HEAD: 5e879ff
[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-73993-g8a17335, ffprobe N-73993-g8a17335, rtmpdump 2.4
[debug] Proxy map: {}
Traceback (most recent call last):
  File "youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "C:\Dev\git\youtube-dl\master\youtube_dl\__init__.py", line 410, in main
    _real_main(argv)
  File "C:\Dev\git\youtube-dl\master\youtube_dl\__init__.py", line 400, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\Dev\git\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1653, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "C:\Dev\git\youtube-dl\master\youtube_dl\YoutubeDL.py", line 655, in extract_info
    ie_result = ie.extract(url)
  File "C:\Dev\git\youtube-dl\master\youtube_dl\extractor\common.py", line 286, in extract
    return self._real_extract(url)
  File "C:\Dev\git\youtube-dl\master\youtube_dl\extractor\roosterteeth.py", line 51, in _real_extract
    ep_filter = compat_urllib_parse.parse_qs(params)
AttributeError: 'module' object has no attribute 'parse_qs'

Doesn't work for me at all:

py34yt "http://roosterteeth.com/show/red-vs-blue#;season=.* 1$" -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['http://roosterteeth.com/show/red-vs-blue#;season=.* 1$', '-v']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2015.08.09
[debug] Git HEAD: 5e879ff
[debug] Python version 3.4.3 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-73993-g8a17335, ffprobe N-73993-g8a17335, rtmpdump 2.4
[debug] Proxy map: {}
[RoosterteethShow] red-vs-blue: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are usin
g the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Dev\git\youtube-dl\master\youtube_dl\extractor\common.py", line 325, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Dev\git\youtube-dl\master\youtube_dl\YoutubeDL.py", line 1860, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\python\python343\lib\urllib\request.py", line 469, in open
    response = meth(req, response)
  File "C:\python\python343\lib\urllib\request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\python\python343\lib\urllib\request.py", line 507, in error
    return self._call_chain(*args)
  File "C:\python\python343\lib\urllib\request.py", line 441, in _call_chain
    result = func(*args)
  File "C:\python\python343\lib\urllib\request.py", line 587, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

UPD: serves me with 403 captcha.

[dstftw]

I don't think URL is a good place for some custom extractor-specific filters that's not supported by the website itself. It should be generic or should not at all. There is a --match-title to filter by title.

[ngld]

I've fixed the first error and removed my own filter. I'm not sure what caused the 403 error in your case. Can you try again with -v --dump-pages?

--match-filter won't work in this case since the titles don't include the season number. I've tried to add a season field to the playlist item but the RoosterteethIE extractor doesn't have access to it and its result replaces the playlist entry.

[dstftw]

-v is already present, --dump-pages won't help since it's 403.
It's a captcha page:

<html>

<head>

<title>Rooster Teeth &middot; Argh!</title>

<style type='text/css'>body,td{cursor:default;}body{background:#fff;color:#000;font:12px Arial , Helvetica , sans-serif;}h2{color:#222;}td{font-size:11px;line-height:150%;vertical-align:top;}a{font-size:11px;text-decoration:none;line-height:150%;color:#c2262b;cursor:pointer;}a:hover{text-decoration:underline;}.secret{font-size:11px;color:#eee;}</style>

</head>

<body>

<center>

<img width="300" height="194" src="data:image/gif;base64,R0lGODlhLAHCALMAAObm5rcfLJybmmZjYCAVFfvzpXAZH7GqdrW0skxJR8zMymU1OYKAfTMwLgAAAP///yH5BAAAAAAALAAAAAAsAcIAAAT/8MlJq7046827/2AojmRpnmiqrmzrvnAsz3Rt33iu73zv/8CgcEgsGo/IpHLJ/Cia0Kg0mHhOr9jsizHQer9gEMJhDZvPYQChAUC739eEowuv25FjB+PO7wMbDnp+g4Q2eXNthYqLKwOBDg1ljJOUHwCAjwOJlZydFoeBBAMIm56mlAKPqg2jpTuup7E1DKq1ogIKsDMKCQmyvze0tcMECQMMAgIICssAuiaXDgjA1DSpw9jYa8bIpLklqb7V4zEIBNno6Q5rA7gganPk8i8KmOr32A0JDJIY1wLzArIAIAefwXwD+lFwJEigwxQCzh2cqCrBNAuY9jzcCI2BRIoU/y1SUCBRHMeTIwAwaPARpMEqEhQ8aoCyJgkAAuy5vEdgD7xAz2wK1aBAwACdO7NZtHdxqNN3CBgkQJo0lCo6T7OKGdCyqq2gWrMCiDrgaFevw5qGDVs0wVm06bCuddoWrksCYOcGHFvQ7k61eh+qpOqXotzAARH0LVwVL+LEixl7Bfh4nAKGku2arCxL5dvMSR33kMqPsxDPoCVr5GGPQV7TMnB+Tu11Mw0ACr5J6Jvg9Z2xypq5iUg7tUIUySbgliqxWCJ7bCatLIsp0pCxN1cWp30YIqSps6eBIkD5DO7cz44VWF/ggBzbP4z5nnB59na70VW0A2CQwJOuorRDiv8XirHCRQPlxXQAewwCchwPajQwoAY4RXZfaglCs8YDFmYDUIePiPLgEr0owKBHkgzAIIOOrBaETOsg840zzAjg1oU4wlfCOR/i0wVhtmTIBAIqrljAVInkZOR6B8RDBEmqrEGAfTimJhoKGbXFEjpVGESTFEUaecCU4C24ZJPdAQEAZlW2OcyIIQizzj5SpQMkMW3wkh8SvCzJHgMMmMkeAiuKgsedbt4HJwgwZkbTIYv+oJiffo65jqCGJoEAooli2AhouZW0hAAMnDiAoEbKAWgD7PXEhALaHTRlp0m5eEI0jPUYiJBFHFBqe5B4hGqDZDwQpqtQFEXdOczqYwz/LiDS6iMLUPrlS6MO8EpEVOsBIl4Clc7BBXsN2BoFjc5sQpC0SaUJDVd25aKTuUVwWwAgAAFKqXunkutuFuaw2+4LY3ExVVI3qkLvtuC2t03DlBpZzBlyCuzSv9SyWdXCQ/TJ3gHDRkyujlesa3GtN0Q7UaRAQCxyxAQAhkU9J1clMwzVbtxEmC9XinGynNasDssdccoAAjamo60R+/T8Mci+ljVfEjkLDdKVLvBCZbbKEZZQFBGFCTXIH5s1pYE3T0Gz1TuRjJzKtbgYcChfTwHcAL0k45EDC1Y3YRpBs43O0iVsClKCvBwzNRRqlNqkNG5oLLisi3Ow5k5Y36Fe/wG0EA3FNZOD5PYIxCVFOBqTrhezeVuHjo3nHFTc9iAe33s6mK6D9LMlcHs5yCV/7o5T2kOAkjs+/qWwNlqZ1wE8k19iUN+Uu/sQ+PGPcMzo9QYR70YDZo553GXrGBBAAAZo7wPo2N+zZwlVw2VdHwlgChj5BJh//vnIEoFr+/e4HQfih5/KYaFpqsMKkcq3vwYaYHVEkB0As1G9DfwkMxXUQuPYI4djyCF/DQwh+kang/9NMBvvSwn3dgK7K9DiV5YKhP5EKEIIBoF9J3wdCiQomQzi4DLqo5BiArGPgoCQhkg0gA9n0Lv2kQcFxmOeB1dxHSIxq38Uyo1R3MKsWv/MEIlgbN4OsJXDWgjQgivMhkTKtQleBCJ6PVjOlI5ogIZYACdSYcmUDMBHPjKLj2AM5P6UGATJldFJJzAkSLyhC0fAUQewOgcgQ1jHcuFNHw1YQAMmSck/CvKT53ugAVtgwkNCYpQVICBIHnnHTOUAJ3r8oggfSMtP1jF/sgRlILG4Axya0nsfaOI9SPiA0twAlnsEIydByUcG6vKZ6BNjykypMOVpZltcOSIN+wjNBzpgmdDUZbF4QMZDEtMDilwlKgeiGG3OEpyC9KY7wwlNV+qAhydMHgr44xdg7uIo8BxkLuMpSXoatIGsvEEaBRdED+BzJ0tcwWU2qUxuPtP/mwM9aD39+YIo5jCFN1no0HKgp4yiL6C7TKZGVzrCHaQTgGfkgEe9AlIYtDOQtbzoHE3K0ovWNDatc905HYpBGmjNpDll5h952tON5mCmE+SoB0SKj4aSoD5InZJOndnUrhpgqJ+iJlhlSpuYgsAjSF3HVpnaVYPWsYUaoqYdd0ibsXaAF2mdZxI92da+6m+uM3goAOGaAWFibp0V8EwfZYlRW94SpX3V6AIAugbEgsCwk0voTYJ62BSsiQAnreE34/lArUbWgWVZwErJg5sFTMmyHihnDiOaAajahbZ3TIAfGTvaiqr0tA7UCEE06p9cwIuwInhp+3B7AcFi7hG9/0nBAHbrQK5SsrRsbesC1KVaUDJrf9lyhm4hN4MLmpK5FsAs8paxDBVE5JYNLOg7cQlcgVLyfjz15iP0V0eA9pcGzl2uCkoJF6t64BIB8ORjBUpLyHrVIlGZIUgHsE394lKGoQShRJB7YM6+0cMXsmsGZMs82E7AHKFcR3Ma7ODTbvfE+tMRALqbYVvwD8OD/OYaOBw7w9iWVvo8wY8HpgIUO9B82e2pErkww/RVAAF8TNt09UuMHIdCIn8V8YFX2IUAd8rAGvDyTsxKIdDW95MLKIMCFoBkhaDoAiZLxxfn+NsH8jgDvkyHT6i6HWmCgM/4IHNhkxxZ+Mw4AAuA8//8lJMTg+jSngO+3hMfMGRpofcB5s2MVDdg5PoO1ADjgzScjeLhJHONBZWGRBnE3Kkgk+4+gr5AAjwtAGcwoMn/KgsGVHIwkBB6DZvuQOAOo143MZfAoAEzBqZ7WgOoZc2hJJ4yLpDngxA6wQhaQbVXcT/sxZpD67iQsi1AEo0qcZlOTqxqX2wJVmPj2gmuWwoQxUsJbNtqwZaASnABYszdWbeCXACbKQkQlTRZFwowgGYvEMmqwDsAp06BYGHSXCcil8SZ0fJIzByATH4xOrdu4Gagne7mYswo1EHLw9+qAlXGSAPFtku/f7oBd6PlzlxB3zSYjb7ybCraqeTj0rD/A+eYo+Ph2DbxA4C0aAz0W+Yzb2GmQXOwcafymwZIxKzRVwooIxoWqci3G/3Cj5VH/AQdsuGIgYy5mLpcMmw4CgrkEL1DJ7oCAjAAuysgkgykS1lPR4c+KRxOB2g8sZJjbQfuTZFZHeQoSUFQXuYm7gfcuZhH2UTedaSPXT/IjYHnyaqvPaXLx/mNtf6AzeOywliFZh/KqJHBAL0TmovBGGrBG8MNXKHQGyRBeb8oeU1guBC5JgTKnUjxDVInuabFswiSC6xEcDRmcMH3jw+6Luv4bWcgrS/F+NufZb48fCzQ+bU4fAXKoom7KgSWbeo7BbwOSkSGwEZbgsR+SjD1/9l5GGnoNwwmRhBHMxIIJ1zMAC9tshq6UUzZVVk38XcmhnGH8wAexgwBWAv5lgFSEV0SEBUYIBWQFyUpRhtqN30SQHiCVHrb4hfOcIH9J1eX1jV2tCYKUX7WhX33oBBfE3IrqHZBMHEHQRMxmA3ToINss3AiMBjG0AtkAigHEiIDhYRK01x6x1Z1tIE0kE4JwHhXIQEeBhDJl0N+Bj8I0AxjlwkCgERUqBQUt37bd3ZDEC1J0z1gOBEAkWonpHRlZnipB20h1IbEMA24QW7Z9V9GEIO9wVmJEIaYJognc3kdYAz6RmmyBImrACc+iFOGxydv4R9FWAvR03q7kYGP8P9t7yAO5mAUlGQlrqAum6hMiIAE1QYQFFgLWEGKlGaKgWB1yHcRCTdLtAEYBIg0AxdPgFUEhqSKB1EeXDYBtMc2vugEJhGM1QUazmEBUxaHMzgDVKFPergJmGUSq5c70+hQCnSJmfFEzGABNGZL9ncEU+dzL7EQE5EfoQhAEoGKHkCJMTFnGacuiZVkiKgEUTRyBiEX7nYl5Wg15bIrQFUG9Ic+6wgWE8mJ/HgDGoM1t5gJ80cRZYBsExQJPCIDpGJ5rsEAVlYY+mAuyZAn7xhGw6cE8UOMrZMg+aiBFOCFuUMAxTROMAATasBJgghBMqNbC5AAMYlE65CRwRA3FxD/OICxQjgZjdKCFzIhiYVlBSpIkX7Rfru2lMrEgk1gXu4yhoAxhh75kabkk5fAhxcwChKwiX4BgbVFkKcUBRXjNmPYDzy5lvZ4SDRxGTOQEN7HX8zjGirRD7kRE2K5TbOYLB/hgdSGD67QkV+YWJgYYsV0jh2wB1Fhjdh3fPrWD7c2WY8pWk6ZA7xhkTzxityjI3poNV3QgDBwfFKhP77nTzynVE3XBHKyO0PWPGrZiXFZRp75AdM2lOdTFbxSCgBAaHVEmlEwN2DZhyOFd0NYWDmkhSNwGbmgks3ZePsgfvQBlr35g6v5VBJhYL1zg5Sza1aJI2XYAmUxFbo5hLdj/w561wBxSJl6+Uaw44Xvdz1K+HZs041O0AURUYIB5AHB90znkJwktTeTxig7KGv9ARaYeTLeCQ5PAGVUFiL7kBtI8yCK0UapKYxAyARTUS7muWXqcDN/aXsP8JfsYqNb8IG5oUVukXoeYA4LUGsIsKIOdA5veC6w5Vy80qH1+ZO5s55xUnCuEKMa4HVX+GiQIKUzow7a0jpPCqWZBZe8UzCxxSuxaEtISqZYgCjaopbz0ZD38aHIEXd+qQtsxGik93KeIFgLk2qeg6NVQqEmMBVR0XXdQXLIMADHyEyBcJ2d8Hbuwk8Z6gSbiTBsGgKXoA+HcRSSsInXVkeQQKd1EP8tJCOSOvkOxVkcOloDKKZ3VnBoX6UM6blVkgcMGLdwfTkCy9cprWoDPPdVXHBkbsUsXAoHIOKTGFBp3okaiQKgVPBFSHekwZKpbjBTrkYBOUmnFRJ/1goNFMVSfrSlDgEixPOeEGF0IEGo9BCubiWq8icQM8UxQ6aVlqeAjkKqRtVbFyVP8rYRPOQ2oRimIdCr8nOs5QAIWNgc+/CttHMWSvhSBBsn+CGXUnA59LVYpRUKXWivnSBB2bqT6TCxICCo6eCxOVA6xMAKVmoTBLY0b6esWSM/DuuqKCcguVCznOBLoxMtShhXtQEbbhAZfpZOP2sC0aivQtuCtTAiD6X/fjXHE6a6tJEDlcuKDuxKbu6DT1lLtUKgEz/lcihbAc7VDtgAtV57BMbzDP1HsiMgWGeoDc8ytmnLA2xCPISBtpOoFLuoDhahs3ULVICJEdigoBRCGNHRobcauFggJ9LUIUq7AXnmgT/RC/eZMFFyDIASuYx7Gx+RNhbitiJARuHHd3mpre2EDr/aueuTPRhAFXpLFC0Rsj/JMZejRpzLugSDCSRjIV2bSmD7DFlpQYpUb7o7BNcgRhZCt/q2GH6mEotnJ+2AHsd7GhJxM0gRuyFoC8lpWMUwvdXrA8JgLpmGsMqBpDoBrSEQc8WADIAbviPxqFobJd+aCgW3GOpr/zlpGG6Cp0bMC78Y0EEWgEOG21ybUTFHm1jOmwvM8I258H0tkcAArAK00B2yk7vamCGU50NgKwlTJxfAUSf/O8EfCR9sor27xnAfYVXXgCA0ig1ShsEkzHDxuHSqMMIaADph6kgj8hNlEQo4PMNrx5gfUcAiYA/VMwanwxB7UBC/KcQ1gBMDHCLvu72PwDLxegF5AJohAqRQ3AOY8bsUOxO2CWd5wcQSoBOiIMNf7AFrVMWvOwycioqYQBkEJMZtfKWn+AO3SFuUyoAWYr55HMDy+wMBxgrg0bAcYETQ6aM1PMgd9UZwXHF3YUzyCZFOh8eQTAGAQLs4IKczcQzKAIebQEluQTzItODJt0Eq8xklGeIRm9wDqfDENqCyHIs3vQAepjMSpxzLn2B4kzwBOty+uoAbUYG5asQS+6DKvnwbjmDE79C+pDA1pzcMlHmGzVxCR2HJLnWOBcISDaBHgcDG2WwJLYsD56ypbUAjulfOgUHO7hzP8jzP9FzP9nzP+JzP+vwAEQAAOw==">

<br />

<div>

<form class="challenge-form" id="challenge-form" action="/cdn-cgi/l/chk_captcha" method="get">
<script type="text/javascript" src="/cdn-cgi/scripts/cf.challenge.js" data-type="custom"  data-ray="214e099963600ca7" async></script>
<noscript id="cf-captcha-bookmark" class="cf-captcha-info">
  <iframe src="//www.google.com/recaptcha/api/noscript?k=6LeT6gcAAAAAAAZ_yDmTMqPH57dJQZdQcu6VFqog" height="300" width="500" frameborder="0"></iframe>
  <input type="hidden" name="recaptcha_response_field" value="manual_challenge">
  <label for="manual_recaptcha_challenge_field">Enter confirmation code after solving challenge above</label>
  <textarea id="manual_recaptcha_challenge_field" name="recaptcha_challenge_field" rows="3" cols="40"></textarea>
  <button type="submit" class="cf-captcha-submit">Submit</button>
</noscript>
</form>


</div>



<br />

<br />

</center>

</body>

</html>

[dstftw]

By the way, technically you are free to build titles in any way.

[ngld]

That captcha page is generated by CloudFlare. Strange, I've never encountered it myself.
I'm not sure how to handle that page... Does it work after solving the captcha in your browser?

I'd like to add the season to an episode's title but the video page doesn't include the required information and I don't see a way to pass the season from RoosterteethShowIE to RoosterteethIE.

[dstftw]

Yes, I can browse it in browser it after solving the captcha. Workaround is to pass cookies exported from browser to youtube-dl. But there is nothing can be done in extractor.
For passing custom data smuggle_url and unsmuggle_url can be used.

[ngld]

Alright, now you can use --match-filter '^Season 2:' if you only want episodes from the second season.
You can also use the season in the filename template (i.e. -o '%(season)s/%(raw_title)s.%(ext)s').
I'm quite happy with this solution. Thanks for helping me out.

The extractor now uses the native HLS implementation. Anything else I should change?

[jaimeMF]

You should directly do:

res = {
    '_type': 'url_transparent',
   'url': 'https://youtube.com/watch?v=' + meta['youtubeKey'],
   'id': video_id,
}

[ngld]

I've updated the extractor and rebased the fork. Anything else I should change?

[yan12125]

Since #8497 landed, please move changes in youtube_dl/extractors/__init__.py to youtube_dl/extractors/extractors.py. Check step 5 of the new developer instructions for more information.

[antdude]

Still waiting for the fixes? I still can't download like from https://roosterteeth.com/episode/death-battle-season-5-doctor-strange-vs-doctor-fate-marvel-vs-dc ... :(

[ngld]

This extractor was for the old site. The video embeds have changed and this code won't work anymore.

[ngld]

@antdude This is the PR you're looking for: #16105