[francetv] Add fallback video url extraction (closes ytdl-org#22561) #23000
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fallback on another API endpoint when no video formats are found. Closes #22561
|
works fine here ! with the patch: restrict-filenames seems to not be applied Tested with version 2019.12.25 |
06Games
approved these changes
Feb 4, 2020
|
I can confirm that this pull request works perfectly!
This pull request should really be merged! |
|
I can confirm this fixes an issue on some videos of france.tv ( https://www.france.tv/france-4/les-as-de-la-jungle-a-la-rescousse/1404197-les-brousse-brothers.html in my case) |
|
Just noticed this pull request after working on fix, I re-factored quite a bit so I'm not sure whether it would be acceptable but I also added MPD manifests support and used the fallback to retrieve some meta-data. I haven't opened a pull request because this one looks a lot cleaner but I'm leaving my version here in case you want to check it out. |
|
@oilarabla Nice work! Your version seems fine, maybe you could split the first commit if that bothers the maintainers but otherwise... I also updated my first post with some additional details about the mobile vs desktop manifests. |
|
Works like a charm for me. I was unable to download many Ninjago episodes without this patch. |
9 tasks
|
Closing in favor of #27047. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please follow the guide below
xinto all the boxes [ ] relevant to your pull request (like that [x])Before submitting a pull request make sure you have:
In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:
What is the purpose of your pull request?
Issue
The current extractor for FranceTV relies on the following JSON endpoint to retrieve video URLs https://sivideo.webservices.francetelevisions.fr/tools/getInfosOeuvre/v2/. These URLs are then matched based on their extension. Unfortunately, the associated field in the returned manifest is
nullwith results outlined in issue #22561 (and for reasons that would necessitate further investigation).As an example, the following URL exhibits this behavior: https://www.france.tv/france-5/echappees-belles/echappees-belles-saison-13/1066323-l-alsace-de-village-en-village.html.
Fix
In case no video URLs can be retrieved, this commit falls back on the subsequent endpoint https://player.webservices.francetelevisions.fr/v1/videos/. To prevent a 400 status code with an "Origin unknown" message, the following query parameters are mandatory:
device_type(e.g.mobile)browser(e.g.firefox)Details
Retrieve the video ID from a URL, for instance,
5da51221-d6c7-4de4-ba5b-46a657ad5a32:The new API answers depending on the
device_type(maybe somebrowservalues get special treatment, but evenytdlworks fine):curl 'https://player.webservices.francetelevisions.fr/v1/videos/5da51221-d6c7-4de4-ba5b-46a657ad5a32?device_type=desktop&browser=firefox'{ "video": { "workflow": "token-akamai", "token": "https://hdfauthftv-a.akamaihd.net/esi/TA?format=json&url=https%3A%2F%2Fcloudreplayfrancetv.akamaized.net%2F1daedb0b835f5%2F236202674_france-domtom_TA.ism%2Fmanifest.mpd", "duration": 2705, "embed": false, "format": "dash", "offline_rights": false, "is_live": false, "drm": false, "drm_type": null, "license_type": null, "player_verification": false, "is_DVR": false, "spritesheets": [ "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead10.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead11.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead12.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead13.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead14.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead15.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead16.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead17.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead18.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead19.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead110.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead111.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead112.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead113.jpeg", "https://staticftv-a.akamaihd.net/videospritesheets/202036/5f538b0bdead1/236202674-5f538b0bdead114.jpeg" ], "is_startover_enabled": false, "coming_next": { "timecode": 2695, "duration": 8 }, "url": "https://cloudreplayfrancetv.akamaized.net/1daedb0b835f5/236202674_france-domtom_TA.ism/manifest.mpd", "captions": [] }, "meta": { "id": "5da51221-d6c7-4de4-ba5b-46a657ad5a32", "plurimedia_id": "236202674", "title": "Tout compte fait", "additional_title": "Une maison plus saine et plus écolo", "pre_title": null, "broadcasted_at": "2020-09-05T14:06:45+02:00", "image_url": "https://sivideo.webservices.francetelevisions.fr/staticftv/ref_emissions//2020-09-05/PDM_236202674.jpg" }, "streamroot": { "enabled": true, "content_id": "https://cloudreplayfrancetv.akamaized.net/1daedb0b835f5/236202674_france-domtom_TA.ism/manifest.mpd", "property": "replay", "license": "6fa93815-2dc9-4df6-9967-3bb108d14bb5" }, "markers": { "analytics": { "isTrackingEnabled": false }, "estat": { "crmID": null, "dom": null, "level1": "france2", "level2": "toutcomptefait", "level3": "info", "level4": "INTEGRAL", "level5": null, "serial": "260060207982", "streamGenre": "web", "streamName": "Tout compte fait-05/09/2020-236202674", "streamDuration": 2705, "newLevel1": "05/09/2020", "newLevel2": "5da51221-d6c7-4de4-ba5b-46a657ad5a32", "newLevel3": "francetv", "newLevel4": "S00_E00", "newLevel5": "Une maison plus saine et plus écolo", "newLevel6": "online", "newLevel7": null, "newLevel8": "magazinedinformation", "newLevel9": null, "newLevel10": null, "newLevel11": null, "newLevel12": null, "newLevel13": null, "newLevel14": null, "newLevel15": null, "mediaContentId": "236202674", "mediaDiffMode": "TVOD", "mediaChannel": "825", "netMeasurement": null }, "npaw": { "customDimension1": null, "customDimension2": "web-desktop", "customDimension3": "toutcomptefait", "customDimension4": "france2", "customDimension5": "info", "customDimension6": "Intégrale", "customDimension7": null, "customDimension8": "single" }, "pub": { "csid": "www.france.tv", "caid": "5da51221-d6c7-4de4-ba5b-46a657ad5a32", "afid": "137378050", "sfid": "6963749", "midroll_timecode": [], "isPreview6h": false, "mediaTailorUrl": null, "pollingInterval": -1 } } }On mobile, the manifest is HLS m3u8, whereas on desktop, it is DASH MPD.
Desktop vs mobile diff
The manifests are both generated by Unified Streaming Platform, and seem to contain the same tracks, with no bitrate or format variations:
manifest.mpdmaster.m3u8My implementation relies on the mobile manifest as a fallback, but @oilarabla rewrote the extraction logic #23000 (comment), which might be more appropriate if the API change is permanent.
Need examples!
I have no idea of the scale of this issue, so if you have some examples of non-working videos, please post them below (or even write some script to find them).