[Roosterteeth] Update extractor to support new site. #16105
Conversation
|
I came here looking for this precise fix and I can confirm it works for me. Is it usual for a PR like this to sit unmerged for 3 weeks? |
|
@DIzFer Thanks for confirming that it works! It's been working for me, but I don't watch Rooster Teeth videos very often anymore, so I haven't really been able to test my fix extensively. I don't know how long a PR usually sits waiting for a review or merge, but I imagine the maintainers simply haven't had the time to check this one yet. @dstftw seems to be the most active (or only?) developer of youtube-dl, so I would guess this PR has just gotten lost in the flood of issues and pull requests the project gets every week. |
|
bump because this is still unmerged on the master branch |
youtube_dl/extractor/roosterteeth.py
Outdated
|
|
||
| if len(video_response.get('data', [])) == 0: | ||
| raise ExtractorError('Unable to download video information') | ||
| video_attributes = video_response.get('data')[0].get('attributes') |
There was a problem hiding this comment.
I'm not sure what 'Same' is meant to be referencing, but I removed the check and error, and changed video_attributes declaration to use try_get. If this is not what you meant for me to change, I can revert it.
|
I've (hopefully) made the requested changes. Awaiting further review. |
|
Thanks for this, seems to be working on every video I've tried so far. One critique would be that the thumbnails it fetches are tiny (240ish), even though there are 720-1080p thumbnails available on the video page, minor thing but would be great if someone could fix. |
|
@RichardHancock It should now download a larger image for the thumbnail. |
|
@ddmgy Thanks that works great. Seems to download with incorrect image file extension for some videos, but that's Rooster Teeth's end as I had this issue when manually downloading them. Thanks for doing this, works just as good as the old website extractor now, Hopefully someone can get this merged in soon |
|
@RichardHancock Glad to hear it works. Would you mind telling me which videos are giving the wrong extension for thumbnails? I'll see if I can fix it. |
|
@ddmgy Seems to be pretty random, only ones I can find doing it right now are recent Camp Camp episodes. I assume one departments just uploading them wrong. I don't really think it's anything that you need to fix, it's just their mistake. I'll post again if I find other series doing it. Thanks |
… download the largest available image now.
|
@RichardHancock Hmm, the thumbnails returned for that Camp Camp episode are a mix of PNGs and JPGs, and I'm not sure which set of those youtube-dl is actually grabbing.
Well, you can still use --write-all-thumbnails and manually select which image you want to keep, but using --write-thumbnail should at least choose to download only the largest available thumbnail now. I wish there was a proper way to choose which thumbnail you want to download, but I'm not seeing any available options for that. Thanks for your feedback on this. I don't use thumbnails, so it probably never would have come to my attention that I was grabbing the wrong size. |
|
As I no longer have a FIRST subscription with roosterteeth.com, I will be unable to make any further changes to this pull request, in the unlikely event that it is reviewed again or considered for merging. If anyone wants to take over maintenance of this pull request, leave a comment here so we can figure out a way to transfer it over. |
|
@ddmgy I would be willing to take it over if needed. |
|
The test for the extractor is failing at the moment, |
|
Hi, any ways this could support bonus episodes? For example https://roosterteeth.com/bonus-feature/rwby-adam-short (volume 6, adam character) |
Fixed test for new rooster teeth site
|
@dstftw |
|
Any updates to this or the other PR for presumably the same problem (#17843)? It's pretty inconvenient using the API workaround described in #16094, so I'd encourage @dstftw or other maintainers to merge this... Given that the Rooster Teeth extractor doesn't work at all in the current master branch, I would think there's not as much worry about breaking it with changes like these... |
|
[2019-03-10 00:31:52] Rome Reginelli:
Any updates to this or the other PR for presumably the same problem (#17843)? It's pretty inconvenient using the API workaround described in #16094, so I'd encourage @dstftw or other maintainers to merge this...
Given that the Rooster Teeth extractor doesn't work at all in the current master branch, I would think there's not as much worry about breaking it with changes like these...
Not much to break and this PR works, I have it on my youtube-dl installation (thanks to gentoo allowing to just drop a patchfile) for quite a while.
|
|
FWIW I used this change to download stuff... so, it does work... It'd be really nice to just merge this..................... .. .. . . . . . . . . . . . . . . |
|
The requested changes seem pretty pedantic too - just merge it 🚀 |
|
Weren't the requested changes made anyway? Or at least, it doesn't seem to me like there's been any review since #16105 (comment). |
|
Please merge. |
|
[2019-05-09 12:41:13-0700] jonnyrobbie:
I don't know - I'm still having problems:
```
./bin/youtube-dl --verbose https://roosterteeth.com/episode/rwby-volume-6-1
[debug] System config: []
[debug] User config: ['--no-mtime', '--write-sub', '--sub-format', 'best', '--embed-subs', '--add-metadata', '--xattrs', '--mark-watched']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://roosterteeth.com/episode/rwby-volume-6-1']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 3.7.3 (CPython) - Linux-5.0.13-arch1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3
[debug] Proxy map: {}
[RoosterTeeth] rwby-volume-6-1: Downloading webpage
ERROR: Unable to extract m3u8 URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
ie_result = ie.extract(url)
File "/usr/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 529, in extract
ie_result = self._real_extract(url)
File "/usr/lib/python3.7/site-packages/youtube_dl/extractor/roosterteeth.py", line 115, in _real_extract
raise ExtractorError('Unable to extract m3u8 URL')
youtube_dl.utils.ExtractorError: Unable to extract m3u8 URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
```
Weird, here it just works:
```
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://roosterteeth.com/episode/rwby-volume-6-1']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.30
[debug] Python version 3.6.5 (CPython) - Linux-4.19.27-gentoo-r1-x86_64-AMD_FX-tm-8350_Eight-Core_Processor-with-gentoo-2.6
[debug] exe versions: ffmpeg 4.1.1, ffprobe 4.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[RoosterTeeth] rwby-volume-6-1: Downloading video information (1/2)
[RoosterTeeth] rwby-volume-6-1: Downloading video information (2/2)
[RoosterTeeth] rwby-volume-6-1: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://rtv3-roosterteeth.akamaized.net/store/777dcdfb41acc2cebc48a69a94ab4460-f6785b63/ts/f6785b63-hls_1080p-store-777dcdfb41acc2cebc48a69a94ab4460.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 339
[download] Destination: Argus Limited-29129.mp4
[download] 100% of 864.13MiB in 02:40
[debug] ffmpeg command line: ffprobe -show_streams 'file:Argus Limited-29129.mp4'
[ffmpeg] Fixing malformed AAC bitstream in "Argus Limited-29129.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Argus Limited-29129.mp4' -c copy -f mp4 -bsf:a aac_adtstoasc 'file:Argus Limited-29129.temp.mp4'
```
The patch I picked up for it is available at (I didn't modify it but
the code in the PR might have changed in the meantime):
https://hacktivis.me/git/etc_portage.work/patches/net-misc/youtube-dl/8b703800ecb-pull-16105-rooster-teeth.diff
|
yeah, that was my bad. I cloned the repos and switched the branches, and then just run the bin/ executable, which just imports the system vanilla version of ytdl. I should have run it with After realizing that, it worked fine. |
|
Any update on if this is getting merged in to the official build? Or if it still works? |
|
I've tried that a few days ago and it still worked for me. But I was simply downloading single episodes by the simple episode url. Maybe some more advanced querries still cause issues? I don't know. |
|
The URL scheme for Rooster Teeth appears to have changed from roosterteeth.com/episode/foo to roosterteeth.com/watch/foo. When using one of the new /watch/ URLs the extractor doesn’t recognize it for the RoosterTeeth extractor and then the generic one fails. Even with the new URL simply changing /watch/ to /episode/ allows it to continue to work. |
|
I'm trying to use this version and I'm getting a 403 error: youtube-dl --verbose https://roosterteeth.com/episode/backwardz-compatible-2019-sadism-or-masochism |
It looks like that video is behind the FIRST subscription and you’ve not supplied a login so as to access FIRST content. Try supplying a username with --username and entering your password when prompted. You could also use a cookie file if the previous method doesn’t work |
Ah I see! Thanks for that. I have clearly misunderstood ddmgy's description - I had thought that her extractor meant that I didn't have to log in at all. I'll try this then! |
Please follow the guide below
xinto all the boxes [ ] relevant to your pull request (like that [x])Before submitting a pull request make sure you have:
In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:
What is the purpose of your pull request?
Description of your pull request and other information
This updates the extractor for Roosterteeth.com to support the recent changes to the website. It now uses the officially provided API rather than scraping HTML. Addresses issue #16094.
There are some issues that need to be fixed before it can be merged.
Videos that do not require log in can be downloaded with no issue, but attempting to log in fails. I have not yet found a way to log in through the new API. The old log in URL stills exists, and can be used to log in through a browser. However, the form is now loaded through Javascript after the page loads, so youtube-dl seems unable to grab it.Logging in and downloading FIRST-only videos now works.The changes I made don't look great, as I just wanted to get it mostly working before hopefully getting feedback on the log in issue. I'll be going through and fixing things up over the weekend.I've cleaned things up and made user-facing strings less generic. I'm open to any suggestions for further changes.Logging in is done by POSTing username and password to an authorization URL, and an access token is returned.
This access token is a cookie in the browser, so it could be saved as a cookie here, but I'm not sure how to do that, or how to check if a saved cookie is expired. I'll look more in to it.Barring any requested changes, this seems ready to be merged.
Edit: The access token is a standard JWT that is valid for 8 hours after logging in. I have implemented saving the access token as a cookie and re-downloading it after it expires, so users who download multiple videos in a short time will not have to log in every time.