Use oEmbed for YouTube Shorts

[SuperSandro2000]

Based on https://core.trac.wordpress.org/changeset/53106

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

[clokep]

Do you have a before / after of the result of what gets changed by including this? (And an example URL?)

[SuperSandro2000]

Before it displayed the youtube login page and now the preview like for any other youtube. For testing any shorts url suffices.

[MatMaul]

Would it be possible to test with the official Youtube oEmbed descriptor from here ?

If it works well I think we should rather use this one.

Is it possible we crossed each other at FOSDEM recently ? I talked with someone about that, that may have been you, or not :)

[clokep]

Would it be possible to test with the official Youtube oEmbed descriptor from here ?

We would want to double check it doesn't regress anything on those other endpoints.

[SuperSandro2000]

Would it be possible to test with the official Youtube oEmbed descriptor from here ?

The only differences is that mine also adds http and theirs add discovery: true. I don't know oembed if that makes a difference but it sounds like it should not to me.

Is it possible we crossed each other at FOSDEM recently ? I talked with someone about that, that may have been you, or not :)

Not, it's not.

[SuperSandro2000]

I tried adding the oembed providers.json and ran into the following problems:

• entries without schemes crash synapse, can be filtered with jq '[ ..|objects|select(.endpoints[0]|has("schemes")) ]'
• none http(s?) schemes are not supported Unsupported oEmbed scheme (spotify) for pattern: spotify:*

Edit:

Solved that with more jq magic jq '[ ..|objects| select(.endpoints[0]|has("schemes")) | .endpoints[0].schemes=([ .endpoints[0].schemes[]|select(.|contains("http")) ]) ]'

[clokep]

I tried adding the oembed providers.json and ran into the following problems:

I think we don't want to include the whole file, there are some providers which give much worse previews with oEmbed (we do use autodiscovery though 🤷 ).

[SuperSandro2000]

So, whats happening with this PR now?

[clokep]

So, whats happening with this PR now?

I think it just never got added back to the queue for some reason...

[clokep]

Before it displayed the youtube login page and now the preview like for any other youtube. For testing any shorts url suffices.

For testing I used https://www.youtube.com/shorts/usofnFoFNN4 (ignore the somewhat janky formatting below -- I have a custom test jig that just splats the preview URLs to an HTML page...)

Before this branch I got:

Open Graph response

{
  "og:site_name": "YouTube",
  "og:url": "https://www.youtube.com/shorts/usofnFoFNN4",
  "og:title": "Meet Kevin Bacon, the dog who grabs a beer for you! #JimmyFallon #FallonTonight",
  "og:image:width": 480,
  "og:image:height": 360,
  "og:type": "video.other",
  "og:video:url": "https://www.youtube.com/embed/usofnFoFNN4",
  "og:video:secure_url": "https://www.youtube.com/embed/usofnFoFNN4",
  "og:video:type": "text/html",
  "og:video:width": 113,
  "og:video:height": 200,
  "og:app:name:iphone": "YouTube",
  "og:app:id:iphone": "544007664",
  "og:app:name:ipad": "YouTube",
  "og:app:id:ipad": "544007664",
  "og:app:url:iphone": "vnd.youtube://www.youtube.com/shorts/usofnFoFNN4?feature=applinks",
  "og:app:url:ipad": "vnd.youtube://www.youtube.com/shorts/usofnFoFNN4?feature=applinks",
  "og:app:name:googleplay": "YouTube",
  "og:app:id:googleplay": "com.google.android.youtube",
  "og:app:url:googleplay": "https://www.youtube.com/shorts/usofnFoFNN4",
  "og:player": "https://www.youtube.com/embed/usofnFoFNN4",
  "og:player:width": "405",
  "og:player:height": "720",
  "og:description": "About\n\nPress\n\nCopyright\n\nContact us\n\nCreators\n\nAdvertise\n\nDevelopers\n\nTerms\n\nPrivacy\n\nPolicy & Safety\n\nHow YouTube works\n\nTest new features\n\nNFL Sunday Ticket\n\n© 2023 Google LLC",
  "og:image": "http://localhost:8080/_matrix/media/v3/thumbnail/localhost:8480/2023-04-10_LlaqneEPkPXgudwk?height=96&width=96",
  "og:image:type": "image/jpeg",
  "matrix:image:size": 8993
}

On this branch I got:

Open Graph response

{
  "og:url": "https://www.youtube.com/shorts/usofnFoFNN4",
  "og:title": "Meet Kevin Bacon, the dog who grabs a beer for you! #JimmyFallon #FallonTonight",
  "og:site_name": "YouTube",
  "og:type": "video.other",
  "og:video:width": 113,
  "og:video:height": 200,
  "og:image:width": 480,
  "og:image:height": 360,
  "og:image": "http://localhost:8080/_matrix/media/v3/thumbnail/localhost:8480/2023-04-10_pVipqwdHrIpbvpnE?height=96&width=96",
  "og:image:type": "image/jpeg",
  "matrix:image:size": 8993
}

---

The differences I see are:

• Without this change we get a garbage description field. ✅
• With this change the Open Graph response loses a bunch of info (in particular the true embed URL). ❌

But it sounds like you don't see any response without the oEmbed configuration? I wonder if this is a difference between region (I'm in the US).

I'm unsure if any clients actually use the video bits of the Open Graph response, unfortunately. 😢

[SuperSandro2000]

My homeserver is located in Germany and I only get the below picture for the link you posted.

I am not sure how to generate the oembed result without youtube's oembed url.

 ➜ curl -s "https://www.youtube.com/oembed/?format=json&url=https://www.youtube.com/shorts/usofnFoFNN4" | jq .
{
  "author_name": "The Tonight Show Starring Jimmy Fallon",
  "author_url": "https://www.youtube.com/@fallontonight",
  "height": 200,
  "html": "<iframe width=\"200\" height=\"113\" src=\"https://www.youtube.com/embed/usofnFoFNN4?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen title=\"Meet Kevin Bacon, the dog who grabs a beer for you! #JimmyFallon #FallonTonight\"></iframe>",
  "provider_name": "YouTube",
  "provider_url": "https://www.youtube.com/",
  "thumbnail_height": 360,
  "thumbnail_url": "https://i.ytimg.com/vi/usofnFoFNN4/hq2.jpg",
  "thumbnail_width": 480,
  "title": "Meet Kevin Bacon, the dog who grabs a beer for you! #JimmyFallon #FallonTonight",
  "type": "video",
  "version": "1.0",
  "width": 113
}

[clokep]

I am not sure how to generate the oembed result without youtube's oembed url.

Oh the JSON I put there was by calling the preview URL endpoint of a local Synapse, so something like:

$ curl --header "Authorization: Bearer YOUR-ACCESS-TOKEN" "http://example.com/_matrix/media/r0/preview_url?url=https://www.youtube.com/shorts/usofnFoFNN4" | json_pp

Anyway, I 100% agree this is a problem. I think some ways forward might be:

1. Accept this PR as-is, downside is that it loses some information if the HTML is available. This information is likely not used by many clients.
2. Reject this PR, downside is that URL previews don't work in Europe (?) and that the regions it works in give some garbage data in the description.
3. Treat the URLs in the included oEmbed provider list the same way as autodiscovered URLs (meaning we hit the original URL as well as the one from the provider list and combine the results). (This probably will be somewhere in-between 1 & 2: it will still have garbage data in the description, but will hopefully have some valid information in all regions.)

I'm not sure of the best way forward, but have a feeling 2 is not an option. Do you (or others) have thoughts?

[SuperSandro2000]

Not really sure. Maybe this could be detected and saved that the native way is not yielding good results? But that would add more code that's also hard to test.

[clokep]

Well it doesn't seem like others have opinions on this. 😄

I think the thing to do is likely to:

1. Merge this as is so we can at least give minimal support for previews everywhere.
2. Update the oEmbed code to always request the oEmbed and the original URL in a follow-up.

Does that sound like it would be best of all worlds (minus making an additional request)?

[clokep]

Sorry for the delay on this, sounds like there's general agreement with the above so I'll mash the button.

[SuperSandro2000]

Thanks!