[bug] Loading a video/audio as an assets does not work.

[firstdorsal]

Describe the bug

Loading a video as an asset does not work.

I followed the guide here: https://tauri.studio/docs/api/js/modules/tauri

It works fine with image files (png/jpg etc.) but not with videos.
I tried loading the video https://www.w3schools.com/html/mov_bbb.mp4 directly from the URL, and this worked without problems.
But using the asset:///home/paul/Downloads/mov_bbb.mp4 URL this does not work
Loading images from the same folder does work.

In the network panel you can see that the request failed but no further reason is specified

The loading of assets this way is IMO not very good explained and when it fails, for some reason, it is very difficult to find out why.

Thank you for creating Tauri, it is awesome!

Reproduction

No response

Expected behavior

No response

Platform and versions

Operating System - Arch Linux, version Rolling Release X64

Node.js environment
  Node.js - 17.7.1
  @tauri-apps/cli - 1.0.0-rc.6
  @tauri-apps/api - 1.0.0-rc.2

Global packages
  npm - 8.5.4
  pnpm - Not installed
  yarn - 1.22.17

Rust environment
  rustup - 1.24.3
  rustc - 1.59.0
  cargo - 1.59.0
  toolchain - stable-x86_64-unknown-linux-gnu 

App directory structure
/dist
/node_modules
/src-tauri
/src
/.git
/public

App
  tauri - 1.0.0-rc.4
  tauri-build - 1.0.0-rc.4
  tao - 0.6.4
  wry - 0.13.3
  build-type - build
  CSP - default-src tauri: asset: https://asset.localhost
  distDir - ../dist
  devPath - http://localhost:3000/
  framework - React

Stack trace

No response

Additional context

No response

[red010182]

This config works for me.

    "allowlist": {
      ...
      "protocol": {
        "asset": true,
        "assetScope": ["**"]
      }
    },
    "security": {
      "csp": "default-src 'self'; img-src 'self' asset: https://asset.localhost; media-src 'self' asset: https://asset.localhost"
    }

---

Edit:

Don't do this. It's dangerous: #3735 (comment)

[FabianLars]

As i said in the other issue, you shouldn't configure the scope like this (Especially since we are working on a fix).
You're better of by doing it more like this:

1. Configure it correctly and as strict as possible.
2. Use dialogs and file-drop events to dynamically extend it.
3. Use rust to manually extend the scope at runtime as a last resort.

About scope errors: It should be logged to the terminal (in tauri dev) and we return a 403 response (with an empty body) on scope related errors

[firstdorsal]

This is not a scope issue. My scope already looks like this:

 "protocol": {
      "all": false,
      "asset": true,
      "assetScope": ["*"]
},

For my project this is necessary as the file explorer that i'm building requires the capability to, you know, display files :p.
Images do work via asset://, text like files and pdfs are loaded through rust but videos won't load with the assets protocol.

[FabianLars]

Ah whoops, didn't see that you were 2 different people, sorry.

[red010182]

Did you try adding media-src in csp? Add media-src 'self' asset: https://asset.localhost works for me.

[firstdorsal]

@red010182
The default-src already does allow assets: default-src tauri: asset: https://asset.localhost
I tried it anyway but it is not working. Also loading assets from the web wouldn't work if that was the problem.

[tedbyron]

@firstdorsal do you have a reproduction? I have a video conversion app I'm making and I'm able to display user-uploaded videos (from tauri dialog) in a <video> element with the following config:

{
  // ...
  "protocol": {
    "asset": true
  },
  // ...
  "security": {
    "csp": "default-src: 'self'; media-src 'self' asset:"
  }
}

[lucasfernog]

I can reproduce this issue, seems to be a webkit2gtk problem.

[dzervas]

Is there a workaround that we can use? Or we just wait for a proper fix?

[firstdorsal]

@tedbyron

from dialog

If you mean using the "Native" browser dialog to select a video then this is not related I think.

@lucasfernog great, good to know! Is this working on Win/Mac?

[lucasfernog]

I never had issues on Windows and macOS. And I don't know about a workaround :/

[lucasfernog]

@wusyong was investigating this issue, I think it's a problem with latest webkit2gtk.

[joehillen]

Is https://asset.localhost/<filepath> supposed to work also? I get "Connection refused" with that but nothing at all with asset://.

[joehillen]

Also, is there an upstream bug report? I can't find one.

[FabianLars]

Is https://asset.localhost/<filepath> supposed to work also? I get "Connection refused" with that but nothing at all with asset://.

asset:// is Linux and macOS and https://asset.localhost/ is Windows.

[keiya01]

I tried to open this page on wry like the following.

let webview = WebViewBuilder::new(window)?.with_url("https://yari-demos.prod.mdn.mozit.cloud/en-US/docs/Web/HTML/Element/video/_sample_.single_source.html")?;

But this shows the empty video screen .

Screen.Recording.2022-07-30.at.19.41.28.mov

edit: oops, this is my mistakes. webkitgtk seems to need to install gstreamer...

[keiya01]

Maybe issue is here for webkitgtk
https://bugs.webkit.org/show_bug.cgi?id=146351

[lucasfernog]

It's weird 'cause I was sure it worked on previous webkit2gtk releases :(

[Robinerra]

Loading audio with the asset protocol is also broken

[wusyong]

It seems fixed in 2.36.4.
https://webkitgtk.org/2022/07/28/webkitgtk2.36.5-released.html
I plan to add headers support again when I have some time. Will test this too.

[loicortola]

Hi all,
I don't know if it helps, I have just tested a basic video sample using the asset protocol. With a regular up-to-date ubuntu release, using webkitgtk2.36.6.
Videos are still not playing on my station. Hopefully the next webkitgtk releases will have it fixed :)

[trippjoe]

Commenting the function name so others can find this issue when they search for convertFileSrc.

[aaalloc]

For the people who are on Linux, images can be loaded through asset protocol (#1438) but not for videos and audio due to webkit lib issue (see https://bugs.webkit.org/show_bug.cgi?id=146351, last email). Someone on tauri discord found a workaround for audio file (this maybe applicable to other type of file), you just have to load the file with fs and create a Blob which you create an DataURL with it

Example :

  import { readBinaryFile } from "@tauri-apps/api/fs"
  const [urlAudio, setUrlAudio] = useState<string>("")
  return (
    <>
      <Button
        variant="ghost"
        onClick={() => {
          const filePath = .....
          void readBinaryFile(filePath)
            .catch((err) => {
              console.error(err)
            })
            .then((res) => {
              const fileBlob = new Blob([res as ArrayBuffer], { type: "audio/mpeg" })
              const reader = new FileReader()
              reader.readAsDataURL(fileBlob)
              const url = URL.createObjectURL(fileBlob)
              setUrlAudio(url)
            })
        }}
        className="w-full justify-start font-normal"
      >
        ok
      </Button>
      <audio controls>
        <source src={urlAudio} type="audio/mpeg" />
      </audio>
    </>
  )

You just need to setup tauri.conf.json permission for letting fs reading files with "scope": ["**"]

EDIT : you could do even better with a simple fetch, you will have to add linux-protocol-headers to the features of tauri your cargo.toml file

...
tauri = { version = "1.4", features = [...., "linux-protocol-headers"] }
....

And the code simplifies to

          const filePath = "/home/yanovskyy/Musique/Shinigami.m4a"
          fetch(convertFileSrc(filePath))
            .then((res) => {
              // create blob url from response
              res
                .blob()
                .then((blob) => {
                  const url = URL.createObjectURL(blob)
                  setUrlAudio(url)
                })
                .catch((err) => {
                  console.error(err)
                })
            })
            .catch((err) => {
              console.error(err)
            })

This, I think, is the best workaround for small video/audio file this moment because readBinaryFile slow down the process and there is memory problem. If you have big files (up to GB), well rip ...

[foodornt]

This is quite annoying
I hope webkit lib solves it soon, although would still be problematic for users who haven't updated the lib

[amosjyng]

I'm not sure if this is the same bug, but I'm unable to play bundled audio files. I'm using SvelteKit and doing

<audio controls>
  <source src="/switch.ogg" type="audio/ogg" />
</audio>

and placed the corresponding sound file in src-svelte/static/switch.ogg. I can play this sound on yarn tauri dev but not the release build. Other assets such as font files or images in the static directory load just fine.

If instead I set the audio source to be some random file from https://en.wikipedia.org/wiki/Category:Wikipedia_Ogg_files, then that external sound file plays successfully even on the release build.

If instead I place the file in src-svelte/src/lib/sounds/switch.ogg and try to programmatically play the sound:

<script lang="ts">
  import clickSound from "$lib/sounds/switch.ogg";
  ...
  function playClick() {
    const audio = new Audio(clickSound);
    audio.play();
  }
</script>

<button
  ...
    on:click={playClick}
  >
    ...
</button>

I get the error

0:00:10.166424993 20275 0x559a88e1aa00 WARN            uridecodebin gsturidecodebin.c:1409:gen_source_element:<uridecodebin0> error: No URI handler implemented for "tauri".

where console.log(clickSound) gives me

tauri://localhost/_app/immutable/assets/switch.326f3a0f.ogg

convertFileSrc doesn't appear to help either.

[thewh1teagle]

Does it happens in Apple webkit as well?
https://bugs.webkit.org/show_bug.cgi?id=146351

Looks very related to the bug in Linux
The deeper issue: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3021

Looks like both Apple webkit and webkitgtk uses gstreamer
So it most likely an issue in gstreamer

Update:
It happens in MacOS Webkit as well!

[FabianLars]

gstreamer is only used on Linux. The issue in your case should be that you're providing a plain file path while it needs the asset: protocol (via convertFileSrc).

[thewh1teagle]

@FabianLars
Looks like apple's webkit does uses gstreamer.

Source/WebCore/platform/mediastream/gstreamer

[FabianLars]

That codebase is shared between apple, webkitgtk, and wpe (and more?). So having a gstreamer folder doesn't mean much for the macos port, especially if we take a look at what other folders are in that mediastream folder.

[thomaskvnze]

Guess I stumbled into the same issue.

I have build an app with tauri and sveltekit for the raspberry pi. Everything works so far except playing an embedded mp4 video in the bundled app for the pi.

The video works on all other platforms in their respective bundled version. On the raspi, I see the play button crossed out. There is no error in the console. I can see that the source section in the inspector contains the video. The src link uses the tauri protocol in the bundled version. When I change the src url in the inspector dom to the same video hosted on a server, it plays the video. I assume this is the same custom schema protocol issue.

[superkelvint]

I'm running into this issue on Linux (MP3 audio playback works with dev but not release build), except its happening not just for assets but also remote MP3 URL playback.

Does anyone know of a workaround on Linux?

[niraj-khatiwada]

Any updates on this? I've been holding off a feature in Linux that allows video play from local asset. Can anyone from the team please respond if there are any alternatives?

[FabianLars]

Still the same alternatives/workarounds as mentioned earlier in this issue:

1. Load small files via fs functions as binary
2. Large files require a http server on the rust side (or a sidecar in your language of choice i guess) so that the file will be loaded over http instead.

Because we do not want to add a http server into tauri itself (assuming nobody changed their minds), we're stuck with this until we have an alternative for webkitgtk.

[ZerdoX-x]

Does anyone know of a workaround on Linux?

@Siirko @superkelvint please take a look at references to this issue, they are available in the discussion above. Please do not bump same issue with same questions.

luxuereal/Alexandria_EBook_Tauri@1050664
tranxuanthang/lrcget@0a2fe99
vjousse/pomodorolm@61e9055

this might be helpful.

---

@niraj-khatiwada

Any updates on this?

https://bugs.webkit.org/show_bug.cgi?id=146351 Here is the original bug report and discussion in webkit bug tracker. Tauri didn't do anything on it's side yet, and I don't think it can do for now. As it will break one of the core feature: working without running local server.

[niraj-khatiwada]

Does anyone know of a workaround on Linux?

@Siirko @superkelvint please take a look at references to this issue, they are available in the discussion above. Please do not bump same issue with same questions.

luxuereal/Alexandria_EBook_Tauri@1050664 tranxuanthang/lrcget@0a2fe99 vjousse/pomodorolm@61e9055

this might be helpful.

@niraj-khatiwada

Any updates on this?

https://bugs.webkit.org/show_bug.cgi?id=146351 Here is the original bug report and discussion in webkit bug tracker. Tauri didn't do anything on it's side yet, and I don't think it can do for now. As it will break one of the core feature: working without running local server.

Thanks for the information. I didn't realize it was the issue on webgtk gstreamer itself. I understand that Tauri cannot do anything about it. Looks like it's already been 9 years since the issue was raised and I don't think it is going to be fixed anytime soon. So I'll just spin up a http server as a workaround.

[GiorgioBaldelli]

A quick workaround that I resorted to for audio files:

let sound = new Audio("data:audio/mpeg;base64,<insert base64 converted audio string>");
sound.currentTime = 0;
sound.play();

Less than ideal, but does the job for small/short audio files.

[ImUrX]

cursed way to fix this issue:

const blobUrl = URL.createObjectURL(await fetch(asset).then(res => res.blob()))
const sound = new Audio(blobUrl)

[loziniak]

Perhaps it could be worked around by Tauri by allowing to register custom response generator under a path of default scheme? Something like tauri::Builder::default().register_uri_path("/videos", |_app, request| { ... , where all requests going to /videos would be handled by custom code?

Could WebviewBuilder::on_web_resource_request be the solution?

[FabianLars]

This is exactly how the default tauri and asset protocols work and the root cause of the issue here so yet another protocol with the same implementation would not help anyone.

[loziniak]

I understand. I assumed this is possible: tauri-apps/wry#1087
And that in release mode files are not loaded all by custom protocol, like tauri: .

[FabianLars]

Yeah that would be cool but only windows supports modifying arbitrary http requests/responses like that (and it's also how the current custom protocol implementation works on windows). The current macOS and Linux webviews only support monitoring requests.