graphics: play a movie or a video #110

hajimehoshi · 2015-01-27T18:22:02Z

No description provided.

hajimehoshi · 2019-09-17T14:56:08Z

I'm not sure how feasible this would be 🤔

KeitoTobi1 · 2021-04-20T06:10:28Z

MP4 Library.
https://github.com/abema/go-mp4

hajimehoshi · 2021-04-20T06:25:20Z

Thanks, but I think this doesn't include codecs.

Zyko0 · 2021-04-20T08:50:08Z

Is going ffmpeg route an option ? I've seen quite a few go bindings, but I'm afraid this would add some CGO again, idk if it's an issue

hajimehoshi · 2021-04-20T09:39:27Z

Cgo is not an option especially for Windows, unfortunately. Another issue is that ffmpeg is LGPL and even if we had pure Go version of ffmpeg, this would conflict with Ebiten's license.

hajimehoshi · 2022-06-07T23:28:13Z

https://github.com/CrazyInfin8/mpg-go

mrg0lden · 2022-07-23T08:20:51Z

One side note regarding CGo issues, I think some can be mitigated using WASM based bindings and pure Go WASM runtime like this one:
https://github.com/tetratelabs/wazero

For most C libraries which are portable, I think this approach works.

SolarLune · 2022-07-24T17:11:26Z

https://github.com/CrazyInfin8/mpg-go

This seems like a fine approach. Playing back a variety of formats would be ideal, but is understandably difficult to manage - it might be fine to simply create a script or tool to invoke FFMPEG on the developer's computer and convert videos to the correct format. Either this could be distributed as part of ebitengine's toolset, or simply mentioned on a wiki / knowledgebase for the developer to run as a script (like the Linux game building script).

One side note regarding CGo issues, I think some can be mitigated using WASM based bindings and pure Go WASM runtime like this one: https://github.com/tetratelabs/wazero

For most C libraries which are portable, I think this approach works.

I'm not knowledgeable on WebAssembly, so forgive me if I'm not understanding how this works, but if I understand you correctly, your idea is to compile FFMPEG into a web assembly binary (?), use wazero to run the binary, use bindings to call FFMPEG in the web assembly runtime to decode videos, and finally get that data and play the audio and display the frames?

mrg0lden · 2022-07-24T18:58:31Z

@SolarLune Yes, I'm not sure how different the performance will be. (The WASM runtime compiles the code to machine code, also the execution part is written in assembly, so I assume it is performant, or at least as performant as CGo)
I tried it on one C library, and the least I got is portability and stability. So I think at least these two benefits do exist.

hajimehoshi · 2022-10-23T16:18:28Z

https://github.com/gen2brain/mpeg

gen2brain · 2022-10-25T21:16:56Z

Hi, I added an Ebitengine example here https://github.com/gen2brain/mpeg/blob/main/examples/player-eb/main.go. There are some issues though, the audio is crackling. I tried to play the SDL example with the S16 format I added and it works fine there. Also, if I try to seek, the memory will go crazy, 1G and more. I added some fake Seek so I can also seek an audio player, but that didn't change anything.

hajimehoshi · 2022-10-26T04:42:40Z

The latest Ebitengine v2.5 should mitigate the issue, but I failed go mod tidy

$ git diff
diff --git a/examples/player-eb/go.mod b/examples/player-eb/go.mod
index 61c0163..b1f869e 100644
--- a/examples/player-eb/go.mod
+++ b/examples/player-eb/go.mod
@@ -6,7 +6,7 @@ go 1.19
 
 require (
        github.com/gen2brain/mpeg v0.1.0
-       github.com/hajimehoshi/ebiten/v2 v2.4.8
+       github.com/hajimehoshi/ebiten/v2 62cbe99a274dd783ace736b842cbd2353e16af19 
        github.com/jfbus/httprs v0.0.0-20190827093123-b0af8319bb15
 )

$ go mod tidy
go: downloading github.com/hajimehoshi/ebiten/v2 v2.5.0-alpha.4.0.20221025043734-62cbe99a274d
player-eb imports
        github.com/hajimehoshi/ebiten/v2 imports
        github.com/hajimehoshi/ebiten/v2/internal/ui imports
        golang.org/x/mobile/app imports
        golang.org/x/exp/shiny/driver/gldriver: ambiguous import: found package golang.org/x/exp/shiny/driver/gldriver in multiple modules:
        golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56 (/Users/hajimehoshi/go/pkg/mod/golang.org/x/[email protected]/shiny/driver/gldriver)
        golang.org/x/exp/shiny v0.0.0-20221025133541-111beb427cde (/Users/hajimehoshi/go/pkg/mod/golang.org/x/exp/[email protected]/driver/gldriver)
player-eb imports
        github.com/hajimehoshi/ebiten/v2 imports
        github.com/hajimehoshi/ebiten/v2/internal/ui imports
        golang.org/x/mobile/app imports
        golang.org/x/exp/shiny/screen: ambiguous import: found package golang.org/x/exp/shiny/screen in multiple modules:
        golang.org/x/exp v0.0.0-20190731235908-ec7cb31e5a56 (/Users/hajimehoshi/go/pkg/mod/golang.org/x/[email protected]/shiny/screen)
        golang.org/x/exp/shiny v0.0.0-20221025133541-111beb427cde (/Users/hajimehoshi/go/pkg/mod/golang.org/x/exp/[email protected]/screen)

Related: #2376

tinne26 · 2022-10-27T16:03:56Z

@gen2brain I made a couple tests, and the audio issues may not be directly related to Ebitengine. I used some code to capture the audio being served by mpg, then saving it to a file at the end, and then using another Ebitengine program to play the captured audio in isolation, without video nor your mpeg package or anything... and it has the same artifacts. The capture was a simple wrapper that did only this:

n, err := mpgAudioReader.Read(buffer)
rawAudio = append(rawAudio, buffer[0 : n]...)

So I'd suspect something with the S16 format, or maybe Ebitengine trying to read too far ahead in advance at the start (it tends to issue two initial buffer reads) and that causing some issue with data not being yet ready (or not, haven't really checked how your audio streaming code works, and I also played around with SetAudioLeadTime and didn't help, so maybe not...). I also tested that it wasn't anything related to the number of channels or LSB vs MSB. Didn't find anything there either.

Regarding Seek behavior, I replaced ebiten.IsKeyPressed with inpututil.IsKeyJustPressed, and it seemed to work ok for me. So maybe you didn't notice it was retriggering many times per second due to IsKeyPressed returning true for as long as you keep pressing the key.

By the way, awesome work, this is kind of a big deal for Ebitengine, so you are making a lot of people happy!

gen2brain · 2022-10-27T16:29:17Z

@tinne26 Thanks, I did not know about IsKeyJustPressed, I didn't use Ebitengine before, and I suspected something is wrong there because I could not toggle fullscreen, I will change that soon. About the S16, I don't know, as I said I tested with SDL (you can choose the audio format there) and tried the example and it worked, so I ruled that out. It can also be related to the issue that was pointed out, about lock/unlock before Read, because the decoder will definitely read from its own Reader during decoding.

gen2brain · 2022-10-27T16:56:19Z

Yes, sorry, just to point out this, the decoder will preallocate the memory for the audio sample, and it should be used in the callback function, i.e. write the received sample to buffer. That is usually nice and enough for many libraries. In this case, I just added a bytes.Reader that reads that memory and seeks to begin if the length is 0, that is probably not the right way. There is also the other way, to call manually DecodeAudio()/DecodeVideo() instead of Decode() and sync audio/video manually, but I also didn't figure out where and how should I use those instead.

tinne26 · 2022-10-27T17:38:05Z

So, if I understand correctly, the issue is that when the callbacks are set, data should be consumed there. The fact that we are using both the audio reader directly and Decode() means we are interacting with the audio decoder from two different places and that's what causes the artifacts, probably due to races or similar on the underlying buffers, or the direct reads jumping to the start again accidentally.

If that's correct, that's great, as it means both projects are working properly and we only need to figure out the best way to sync things in Ebitengine. Don't worry if you are more unfamiliar with Ebitengine, with all that info we can totally figure it out by ourselves if you need to focus on something else.

gen2brain · 2022-10-27T18:11:52Z

Well, yes, the Decode() is called with the tick seconds, usually 1/60, it will call Video/Audio callbacks any number of times, for this to work you must SetAudioLeadTime() to the duration of the buffer, that would be mpeg.SamplesPerFrame/samplerate, you should set the Ebitengine buffer size to the same duration. Also, SetVideoEnabled/SetAudioEnabled functions are all related to Decode(). In the Ebitenengine case, the audio callback function doesn't do anything, it is defined only so Decode() could decode audio (doesn't work without callback). With DecodeVideo/DecodeAudio, they just decode and return one single audio/video frame.

Also, just to mention, this is of course rewrite of the C library, everything is single-threaded, there are no go routines, etc. Both audio and video constantly check if there are more bits with the has() function that will try to load more from the reader if not available. This I guess is a perfect case for some producer/consumer pattern or something like that, but for now I didn't touch that. I did learned a lot but I am not expert here, so any help is welcome.

hajimehoshi · 2022-10-28T08:21:30Z

@gen2brain Do you have any suggestion about this issue? #110 (comment)

gen2brain · 2022-10-28T08:34:36Z

@hajimehoshi No idea, mpeg package only uses the standard library. I get the same, I wanted to try and build a master without modules, but then the problem is v2.

hajimehoshi · 2022-10-28T08:40:31Z

I'd avoid nested go.mod if possible. You can move the examples to other repositories or simply remove child go.mod files. Both should not break backward compatibility

gen2brain · 2022-10-28T08:51:43Z

Ok, I removed modules, tidy now works, thanks! I also switched to inpututil.

tinne26 · 2022-10-28T23:44:44Z

I made a small package to make it easier to manage this on Ebitengine: https://github.com/tinne26/mpegg. Still a work in progress, but it works.

The key idea is to let Ebitengine handle the audio, decoding from the mpeg as much as Ebitengine requests, and then using the audio.Player.Current() position to figure out the expected video position. When a frame is requested, we decode the video until we reach that position. I need to add some compensation for the audio, as there's always some latency due to drivers and stuff, but there don't seem to be any major problems.

I can help with the standalone example for gen2brain too, but to point to the critical code in that new package:

The Read() method on audio_adapter.go connects the mpeg audio decoding to Ebitengine's audio player.
The CurrentFrame() method on player.go makes the video catch up to the audio position.

hajimehoshi · 2022-10-29T06:52:53Z

@tinne26 Awesome! I could play the same video as @gen2brain's test video without noises!

I found an error after finishing the video:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1001612cc]

goroutine 8 [running]:
github.com/tinne26/mpegg.(*audioAdapter).Read(0xc0000a2360, {0xc00050a000, 0x44e8, 0x44e8})
        /Users/hajimehoshi/ebitengine-games/mpegg/audio_adapter.go:83 +0x32c
github.com/hajimehoshi/ebiten/v2/audio.(*timeStream).Read(0xc0004a8060, {0xc00050a000?, 0xc000526008?, 0x10014e920?})
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/ebiten/[email protected]/audio/player.go:334 +0xcf
github.com/hajimehoshi/oto/v2/internal/mux.(*playerImpl).readSourceToBuffer(0xc000506000)
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/oto/[email protected]/internal/mux/mux.go:451 +0x152
github.com/hajimehoshi/oto/v2/internal/mux.(*Mux).loop(0xc00002c200)
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/oto/[email protected]/internal/mux/mux.go:82 +0x1ce
created by github.com/hajimehoshi/oto/v2/internal/mux.New
        /Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/oto/[email protected]/internal/mux/mux.go:46 +0x105
exit status 2

gen2brain · 2022-10-29T22:11:52Z

@tinne26 Feel free to modify whatever you need directly in the library. I added a lot of helper functions like RGBA(), YCbCr(), Bytes(), Pixels(), memory is preallocated for different audio formats, etc. This is one case where you definitely don't want to allocate new memory and convert something. Currently, there are no new allocations when decoding (well, anything can happen when seeking). Reader() is added just for Ebitenegine, feel free to remove it, change it, or whatever.

tinne26 · 2022-10-30T08:54:53Z

I'll open an issue on your mpeg repository instead if that's fine with you and we can debate it there. This issue is already resolved and both mpeg and Ebitengine are doing fine, but it's true that there may be some room for improving ease of use for a couple use-cases on mpeg, and they are not related to Ebitengine itself, but rather to the "allow audio to be arbitrarily buffered and then have video catching up to a specific point in time" approach. We could make a much more compact single-file example for Ebitengine from that (or for anything else that wishes to use a similar approach).

hajimehoshi · 2023-09-18T14:40:32Z

At least I want to create an example to play a video (probably with https://github.com/tinne26/mpegg and/or https://github.com/gen2brain/mpeg).

Closes #110

hajimehoshi · 2024-04-11T11:36:03Z

@tinne26 Based on your https://github.com/tinne26/mpegg, I've created a simple example, thanks!

https://github.com/hajimehoshi/ebiten/tree/main/examples/video

hajimehoshi added the feature label Jan 27, 2015

hajimehoshi changed the title ~~Show a movie~~ graphics: Show a movie Feb 21, 2016

hajimehoshi changed the title ~~graphics: Show a movie~~ graphics: Play a movie Sep 10, 2017

hajimehoshi mentioned this issue Jul 26, 2019

video support? #905

Closed

hajimehoshi added the request label Jul 26, 2019

hajimehoshi added the help-wanted label Sep 17, 2019

hajimehoshi changed the title ~~graphics: Play a movie~~ graphics: play a movie or a video Jun 7, 2022

hajimehoshi added this to the v2.7.0 milestone Sep 18, 2023

hajimehoshi modified the milestones: v2.7.0, v2.8.0 Mar 10, 2024

hajimehoshi added a commit that referenced this issue Apr 11, 2024

examples: add an example to play a video

1f3a80e

Closes #110

hajimehoshi added a commit that referenced this issue Apr 11, 2024

examples: add an example to play a video

a78a68c

Closes #110

hajimehoshi closed this as completed in d3befbf Apr 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

graphics: play a movie or a video #110

graphics: play a movie or a video #110

hajimehoshi commented Jan 27, 2015

hajimehoshi commented Sep 17, 2019

KeitoTobi1 commented Apr 20, 2021 •

edited

Loading

hajimehoshi commented Apr 20, 2021

Zyko0 commented Apr 20, 2021

hajimehoshi commented Apr 20, 2021 •

edited

Loading

hajimehoshi commented Jun 7, 2022

mrg0lden commented Jul 23, 2022

SolarLune commented Jul 24, 2022 •

edited

Loading

mrg0lden commented Jul 24, 2022

hajimehoshi commented Oct 23, 2022

gen2brain commented Oct 25, 2022

hajimehoshi commented Oct 26, 2022 •

edited

Loading

tinne26 commented Oct 27, 2022 •

edited

Loading

gen2brain commented Oct 27, 2022

gen2brain commented Oct 27, 2022

tinne26 commented Oct 27, 2022 •

edited

Loading

gen2brain commented Oct 27, 2022

hajimehoshi commented Oct 28, 2022

gen2brain commented Oct 28, 2022

hajimehoshi commented Oct 28, 2022

gen2brain commented Oct 28, 2022

tinne26 commented Oct 28, 2022 •

edited

Loading

hajimehoshi commented Oct 29, 2022

gen2brain commented Oct 29, 2022

tinne26 commented Oct 30, 2022

hajimehoshi commented Sep 18, 2023 •

edited

Loading

hajimehoshi commented Apr 11, 2024

graphics: play a movie or a video #110

graphics: play a movie or a video #110

Comments

hajimehoshi commented Jan 27, 2015

hajimehoshi commented Sep 17, 2019

KeitoTobi1 commented Apr 20, 2021 • edited Loading

hajimehoshi commented Apr 20, 2021

Zyko0 commented Apr 20, 2021

hajimehoshi commented Apr 20, 2021 • edited Loading

hajimehoshi commented Jun 7, 2022

mrg0lden commented Jul 23, 2022

SolarLune commented Jul 24, 2022 • edited Loading

mrg0lden commented Jul 24, 2022

hajimehoshi commented Oct 23, 2022

gen2brain commented Oct 25, 2022

hajimehoshi commented Oct 26, 2022 • edited Loading

tinne26 commented Oct 27, 2022 • edited Loading

gen2brain commented Oct 27, 2022

gen2brain commented Oct 27, 2022

tinne26 commented Oct 27, 2022 • edited Loading

gen2brain commented Oct 27, 2022

hajimehoshi commented Oct 28, 2022

gen2brain commented Oct 28, 2022

hajimehoshi commented Oct 28, 2022

gen2brain commented Oct 28, 2022

tinne26 commented Oct 28, 2022 • edited Loading

hajimehoshi commented Oct 29, 2022

gen2brain commented Oct 29, 2022

tinne26 commented Oct 30, 2022

hajimehoshi commented Sep 18, 2023 • edited Loading

hajimehoshi commented Apr 11, 2024

KeitoTobi1 commented Apr 20, 2021 •

edited

Loading

hajimehoshi commented Apr 20, 2021 •

edited

Loading

SolarLune commented Jul 24, 2022 •

edited

Loading

hajimehoshi commented Oct 26, 2022 •

edited

Loading

tinne26 commented Oct 27, 2022 •

edited

Loading

tinne26 commented Oct 27, 2022 •

edited

Loading

tinne26 commented Oct 28, 2022 •

edited

Loading

hajimehoshi commented Sep 18, 2023 •

edited

Loading