Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[funimation] Extract more format info from site #9515

Closed
wants to merge 2 commits into from

Conversation

Parent5446
Copy link

Use the JSON playlist information provided by Funimation to extract more
attributes about the stream formats, such as season and episode
number. Also fixed the returned video information to use the 'series'
key rather than the 'artist' key, the latter of which is meant for
albums.

Also, determine whether the user supplied the subtitle or English dub
URL based on the query parameters, and set the language preference based
on that and the site-provided stream info.

query = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
preference = None
if query.get('watch') is not None and len(query.get('watch')) > 0:
preference = query.get('watch')[-1]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be simplified to query.get('watch', [None])[-1].

@dstftw
Copy link
Collaborator

dstftw commented May 17, 2016

Add some tests for series related fields.

Also, determine whether the user supplied the subtitle or English dub
URL based on the query parameters, and set the language preference based
on that and the site-provided stream info.

Add a test for that also.

Squash commits when done.

@Parent5446
Copy link
Author

Thanks for the review. I'll make the changes once I have a chance.

@Parent5446
Copy link
Author

@dstftw did I miss anything? I don't see any further comments.

.format(playlist.get('showId'), pagenum * 20))
request.add_header('User-Agent', user_agent)
episode_list = self._download_json(
request, display_id, 'Downloading episode list from {}'.format(pagenum * 20))['main']
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{} does not work in python 2.6.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compatibility kicks me in the ass again. Will fix quickly. (And sorry for not testing 2.6. I forgot on the last commit.)

Use the JSON playlist information provided by Funimation to extract more
attributes about the stream formats, such as season and episode
number. Also fixed the returned video information to use the 'series'
key rather than the 'artist' key, the latter of which is meant for
albums.

Also, determine whether the user supplied the subtitle or English dub
URL based on the query parameters, and set the language preference based
on that and the site-provided stream info.
Add new InfoExtractor for Funimation series and video pages to make
playlists of all videos in a series, similar to the existing
CrunchyrollShowPlaylistIE class.

Also adjusted the tests for both Funimation extractors to match pages
that actually exist.
@Parent5446
Copy link
Author

@dstftw Friendly ping

@Starsam80
Copy link
Contributor

Nice work @Parent5446! However, I noticed a few issues when using this patch:

  • %(episode_number)s always has a .0 at the end.
$ python -m youtube_dl --get-filename --output "%(series)s/%(episode_number)s - %(episode)s.%(ext)s" http://www.funimation.com/shows/my-hero-academia/videos/official/izuku-midoriya-origin
My Hero Academia/1.0 - Izuku Midoriya - Origin.mp4
  • Playlists don't work. Different results with and without cookies.
$ python -m youtube_dl --verbose --output "%(series)s/%(episode_number)s - %(episode)s.%(ext)s" http://www.funimation.com/shows/my-hero-academia
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '--output', '%(series)s/%(episode_number)s - %(episode)s.%(ext)s', 'http://www.funimation.com/shows/my-hero-academia']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.05.10
[debug] Git HEAD: e82d98c
[debug] Python version 3.5.2 - Linux-4.7.4-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 3.1.3, ffprobe 3.1.3, rtmpdump 2.4
[debug] Proxy map: {}
[funimation:playlist] my-hero-academia: Downloading series webpage
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/__init__.py", line 421, in main
    _real_main(argv)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/__init__.py", line 411, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/YoutubeDL.py", line 1736, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/YoutubeDL.py", line 676, in extract_info
    ie_result = ie.extract(url)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/common.py", line 341, in extract
    return self._real_extract(url)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/funimation.py", line 295, in _real_extract
    webpage = self._download_webpage(request, display_id, 'Downloading series webpage')
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/funimation.py", line 32, in _download_webpage
    return super(FunimationBaseIE, self)._download_webpage(*args, **kwargs)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/common.py", line 501, in _download_webpage
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/common.py", line 408, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/common.py", line 388, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/YoutubeDL.py", line 1946, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/utils.py", line 779, in http_open
    req)
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1146, in _send_request
    self.putheader(hdr, value)
  File "/usr/lib/python3.5/http/client.py", line 1082, in putheader
    if _is_illegal_header_value(values[i]):
TypeError: expected string or bytes-like object

$ python -m youtube_dl --verbose --cookies ~/Anime/fun_cookies.txt --output "%(series)s/%(episode_number)s - %(episode)s.%(ext)s" http://www.funimation.com/shows/my-hero-academia
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '--cookies', '/home/starsam80/Anime/fun_cookies.txt', '--output', '%(series)s/%(episode_number)s - %(episode)s.%(ext)s', 'http://www.funimation.com/shows/my-hero-academia']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.05.10
[debug] Git HEAD: e82d98c
[debug] Python version 3.5.2 - Linux-4.7.4-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 3.1.3, ffprobe 3.1.3, rtmpdump 2.4
[debug] Proxy map: {}
[funimation:playlist] my-hero-academia: Downloading series webpage
ERROR: Unable to extract players data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/YoutubeDL.py", line 676, in extract_info
    ie_result = ie.extract(url)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/common.py", line 341, in extract
    return self._real_extract(url)
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/funimation.py", line 301, in _real_extract
    webpage, 'players data'),
  File "/home/starsam80/temp/temp_ytdl/youtube_dl/extractor/common.py", line 644, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract players data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants