[tver] Add support for TVer #26662

tsukumijima · 2020-09-21T21:28:56Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.

TVer is Japan's largest free distribution service for TV programs, operated mainly by five Japanese TV stations.
It distributes various programs such as Japanese anime, drama, news, and variety shows, and has more than 10 million active users.
This pull request was opened to make TVer videos available for download on youtube-dl.

Note

TVer is a service for Japan, so I set _GEO_COUNTRIES = ['JP'] to disguise the X-Fowarded-For header as a Japanese IP, but it may still not be available for download from outside Japan.
The test from test / test_download.py doesn't seem to handle encrypted HLS videos correctly and says if pycryptodome is not installed, an error "ERROR: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg" will occur.
Also, if pycryptodome is installed, an error "Value Error: Data must be padded to 16 byte boundary in CBC mode" will occur. I'm not sure why.
To avoid this, I've added a'skip'to _TESTS to skip the test (although most TVer videos aren't available after a week or so, so after a while the test itself doesn't make sense...)
I an using Google Translate. The text may be unnatural.

dstftw · 2020-09-24T00:21:26Z

youtube_dl/extractor/tver.py

+        {
+            'url': 'https://tver.jp/corner/f0056997',  # 'corner'
+            'md5': 'aac4e681dcdb775fc44497da4f7bdd05',  # MD5 hash of a short video downloaded by running youtube-dl with the --test option
+            'info_dict': {
+                'id': 'f0056997',  # TVer ID
+                'display_id': 'ref:kanokari_10',  # Brightcove ID
+                'ext': 'mp4',
+                'title': '彼女、お借りします　第10話「友達の彼女」-トモカノ-',
+                'description': 'バイトの初任給を何に使おうか考える和也だったが、ふと栗林のことが脳裏をよぎる。最近栗林の様子がおかしいと、木部から話を聞いていたのだ。ボーッとしていたり、女性不信のつぶやきをしているという。和也は意を決して、栗林を呼び出すことに。翌日、栗林が和也を待っていると──「駿君、だよね？」。待ち合わせ場所にやって来たのは、千鶴だった……！',
+                'thumbnail': 'https://cf-images.ap-northeast-1.prod.boltdns.net/v1/jit/5102072605001/900216cc-2e97-4c19-93bb-1a531de358d6/main/1920x1080/12m18s37ms/match/image.jpg',
+                'duration': 1476.075,
+                'timestamp': 1599554409,
+                'upload_date': '20200908',
+                'uploader_id': '5102072605001',
+            },
+            'skip': 'Running from test_download.py doesn\'t seem to be able to handle encrypted HLS videos',
+        },
+        {
+            'url': 'https://tver.jp/episode/76799350',  # 'episode'
+            'md5': 'ad893db02b8a3e949216c463af7ce51e',  # MD5 hash of a short video downloaded by running youtube-dl with the --test option
+            'info_dict': {
+                'id': '76799350',  # TVer ID
+                'display_id': '2366_2365_4533',  # Brightcove ID
+                'ext': 'mp4',
+                'title': '港時間　#49　神奈川県／リビエラシーボニアマリーナ　9月18日(金)放送分',
+                'description': '【毎週金曜 よる12時15分から放送】\n\n日本のヨット文化 を育んできた三浦半島の西海岸、小網代湾にあるリビエラシーボニアマリーナ。昨年から始まったSailGPの日本チームを率いるヨット界のレジェンドに会いました。',
+                'thumbnail': 'https://cf-images.ap-northeast-1.prod.boltdns.net/v1/jit/4394098883001/904361ca-40d3-4028-8478-8916b9a0ff49/main/1920x1080/58s80ms/match/image.jpg',
+                'duration': 116.16,
+                'timestamp': 1600052421,
+                'upload_date': '20200914',
+                'uploader_id': '4394098883001',
+            },
+            'skip': 'Running from test_download.py doesn\'t seem to be able to handle encrypted HLS videos',
+        },


Remove duplicate tests.

It has been deleted.

dstftw · 2020-09-24T00:21:31Z

youtube_dl/extractor/tver.py

+from .brightcove import BrightcoveNewIE
+
+
+class TVerIE(BrightcoveNewIE):


Do not inherit brightcove, you must delegate instead.

TVer makes use of Brightcove, but some entities, such as descriptions, had to be extracted from TVer's site.
I didn't know how to overwrite the entity retrieved by BrightcoveNewIE in such a case, so I ended up with such a dirty implementation.
In the return value, I found that it was possible to delegate in the correct way by setting '_type' to 'url_transparent' and setting the entity retrieved by TVerIE such as 'description', so I rewrote it significantly.
I didn't do enough preliminary research ... I'm sorry.

dstftw · 2020-09-24T00:21:41Z

youtube_dl/extractor/tver.py

+                'ext': 'mp4',
+                'title': '半沢直樹(新シリーズ)　第1話 子会社VS銀行!飛ばされた半沢の新たな下剋上が始まる',
+                'description': '大和田(香川照之)の不正を糾弾し､子会社へ出向を命じられた半沢直樹(堺雅人)は､東京セントラル証券営業企画部長に｡ある日1500億円超の買収案件が舞い込むが…｡',
+                'thumbnail': 'https://cf-images.ap-northeast-1.prod.boltdns.net/v1/jit/4031511847001/37b5f176-3989-48d9-81d1-4688e80c5531/main/1920x1080/34m10s16ms/match/image.jpg',


No exact URLs.

dstftw · 2020-09-24T00:21:45Z

youtube_dl/extractor/tver.py

+                'display_id': 'ref:hanzawa_naoki---s2----323-001',  # Brightcove ID
+                'ext': 'mp4',
+                'title': '半沢直樹(新シリーズ)　第1話 子会社VS銀行!飛ばされた半沢の新たな下剋上が始まる',
+                'description': '大和田(香川照之)の不正を糾弾し､子会社へ出向を命じられた半沢直樹(堺雅人)は､東京セントラル証券営業企画部長に｡ある日1500億円超の買収案件が舞い込むが…｡',


dstftw · 2020-09-24T00:21:57Z

youtube_dl/extractor/tver.py

+
+    def _real_extract(self, url):
+
+        # extract video id


Remove all useless comments.

Obviously excessive comments have been removed.
However, I think that the code will be difficult to understand if there are too few comments, so I left the comments that I felt necessary.

dstftw · 2020-09-24T00:22:40Z

youtube_dl/extractor/tver.py

+        if self._downloader.params.get('verbose', False):
+            self.to_screen('Video Information: %s' % video_info)
+            self.to_screen('Brightcove Account ID: %s' % brightcove_account_id)
+            self.to_screen('Brightcove Video ID: %s' % brightcove_video_id)
+            self.to_screen('Brightcove URL: %s' % brightcove_url)


Remove all debug garbage.

It has been deleted.

dstftw · 2020-09-24T00:22:46Z

youtube_dl/extractor/tver.py

+            self.to_screen('Brightcove URL: %s' % brightcove_url)
+
+        # evacuate _VALID_URL
+        _VALID_URL = self._VALID_URL


dstftw · 2020-09-24T00:23:03Z

youtube_dl/extractor/tver.py

+        # temporarily replace _VALID_URL
+        # prevent _VALID_URL from being the URL of Tver when executing the parent class's _real_extract () method
+        self._VALID_URL = r'https?://players\.brightcove\.net/(?P<account_id>\d+)/(?P<player_id>[^/]+)_(?P<embed>[^/]+)/index\.html\?.*(?P<content_type>video|playlist)Id=(?P<video_id>\d+|ref:[^&]+)'


dstftw · 2020-09-24T00:23:11Z

youtube_dl/extractor/tver.py

+        # get video information
+        info_dict = super(TVerIE, self)._real_extract(brightcove_url)


dstftw · 2020-09-24T00:23:33Z

youtube_dl/extractor/tver.py

+        # undo _VALID_URL
+        self._VALID_URL = _VALID_URL
+
+        # TVer ID
+        info_dict['id'] = video_id
+        # Brightcove ID
+        info_dict['display_id'] = brightcove_video_id
+        # select large thumbnail
+        info_dict['thumbnail'] = info_dict.get('thumbnail').replace('160x90', '1920x1080')
+        # desctiption
+        info_dict['description'] = description


Noway. See other extractors on how to delegate properly.

I think I fixed all the parts you pointed out.
Thank you for your review.

…d 'uploader' keys to extraction information

…On Demand) hosted by its own distribution system

tsukumijima · 2020-10-16T17:59:39Z

I'm sorry to be busy, Would you please review it again?
Or should I reopen the PR?

…P Error 404

…tver

Paun · 2020-12-03T10:10:06Z

Oh finally!!

I am very glad to see this will be supported soon.

Nice work.

tsukumijima · 2020-12-03T14:09:19Z

The merged changes are a bit different from the code I wrote, but in any case, I'm happy to see TVer support added to youtube-dl.
I haven't confirmed if it works yet, but as far as I can see the code, it should work.

tsukumijima · 2020-12-03T14:23:38Z

I've read the code and it's certainly more verbose and nice than my code.
I also admire the ability to write this code in just a few hours after receiving your request.

tsukumijima added 3 commits September 22, 2020 03:50

[tver] Add new extractor

9f45837

[tver] fix the code pointed out in flake8

e57602b

[tver] fix to more verbose code

a109b5f

tsukumijima marked this pull request as draft September 21, 2020 21:30

tsukumijima marked this pull request as ready for review September 21, 2020 21:30

dstftw requested changes Sep 24, 2020

View reviewed changes

dstftw added do-not-merge pending-fixes labels Sep 24, 2020

tsukumijima added 3 commits September 27, 2020 18:30

[tver] Significantly improved extraction logic and added 'creator' an…

bf6f104

…d 'uploader' keys to extraction information

[tver] Added support for videos distributed by Fuji TV (FOD, Fuji TV …

bcae9d7

…On Demand) hosted by its own distribution system

Merge branch 'master' into feature/tver

fd07d48

tsukumijima requested a review from dstftw September 29, 2020 21:35

tsukumijima added 8 commits October 26, 2020 02:48

[tver] Update supportedsites.md

e9c481f

[tver] Fix incorrect case

dbda3bc

Merge branch 'master' of gitlab.com:dstftw/youtube-dl into feature/tver

6605124

Merge branch 'master' of gitlab.com:dstftw/youtube-dl into feature/tver

154f3a8

[tver] Fix issue where some videos would fail to download with an HTT…

c7ccb21

…P Error 404

Merge branch 'master' of github.com:ytdl-org/youtube-dl into feature/…

6ec08d5

…tver

Merge branch 'master' of github.com:ytdl-org/youtube-dl into feature/…

4125d7d

…tver

Merge branch 'master' of github.com:ytdl-org/youtube-dl into feature/…

d2479b4

…tver

remitamine closed this in 64554c1 Dec 2, 2020

ThirumalaiK pushed a commit to ThirumalaiK/youtube-dl that referenced this pull request Jan 28, 2021

[tver] Add new extractor (closes ytdl-org#26662)(closes ytdl-org#27284)

f41794b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tver] Add support for TVer #26662

[tver] Add support for TVer #26662

tsukumijima commented Sep 21, 2020 •

edited

Loading

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020

dstftw Sep 24, 2020

tsukumijima Sep 29, 2020 •

edited

Loading

tsukumijima commented Oct 16, 2020

Paun commented Dec 3, 2020

tsukumijima commented Dec 3, 2020

tsukumijima commented Dec 3, 2020

		from .brightcove import BrightcoveNewIE


		class TVerIE(BrightcoveNewIE):

		# get video information
		info_dict = super(TVerIE, self)._real_extract(brightcove_url)

[tver] Add support for TVer #26662

[tver] Add support for TVer #26662

Conversation

tsukumijima commented Sep 21, 2020 • edited Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

Note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsukumijima Sep 29, 2020 • edited Loading

Choose a reason for hiding this comment

tsukumijima commented Oct 16, 2020

Paun commented Dec 3, 2020

tsukumijima commented Dec 3, 2020

tsukumijima commented Dec 3, 2020

tsukumijima commented Sep 21, 2020 •

edited

Loading

tsukumijima Sep 29, 2020 •

edited

Loading