Use of video_set_callbacks

[tgaugry]

Hello,
i have been trying to get images frame per frame from an rtsp stream.
I can get the stream in vlc window, but i cannot get the frame data.
I read that i should use the video_set_callbacks function; however, instance crash if i set a callback.
This is a part of the code I've been using :

import vlc, ctypes

pl = vlc.MediaPlayer('rtsp://addr')

@vlc.CallbackDecorators.VideoLockCb
def _lockcb(opaque, planes):
        print "lock"
        return ctypes.create_string_buffer('\000')

p = ctypes.POINTER(ctypes.c_int)()
pl.video_set_callbacks(_lockcb, None, None, p)

pl.play()

Did i do something wrong ? Or maybe it isn't supported in the python binding ?

Thanks !

[oaubert]

It should work, but I could not test all bindings and especially callbacks which can be trickier to automatically convert from C to python. If you found a solution, please post it here.

[tgaugry]

I imagine there is a magic function somewhere that does the trick... But i didn't found it.
I did not find a solution with VLC; I had to switch to ffmpeg + Popen.
The resulting code is extremely quirky, sadly, so I would still like to find a solution using VLC's video_set_callbacks.

[daniel5gh]

Started playing with this today and it turns out that the ctypes callback definition for VideoLockCb is incorrect. I looked at the original definition here.

See CorrectVideoLockCb in the code below. This code spits out some frames to PNGs using PIL during the first 10 seconds.

Once I started using the correct definition I was able to set the pointer to the buffer I allocated as the documentation instructed:

Depending on the video chroma, one or three pixel planes of
adequate dimensions must be returned via the second parameter. Those
planes must be aligned on 32-bytes boundaries.

\param planes start address of the pixel planes (LibVLC allocates the array
            of void pointers, this callback must initialize the array) [OUT]

With chroma RV32 we only need to set the first plane (planes[0])

No actual locking is being done, which may or may not be a problem (because GIL?) :) Also writing the images in the display callback is a bad idea, but good enough to show this works.

import vlc
import ctypes
import time
import sys

from PIL import Image

pl = vlc.MediaPlayer('videofile.mp4')

VIDEOWIDTH = 1920
VIDEOHEIGHT = 1080

# size in bytes when RV32
size = VIDEOWIDTH * VIDEOHEIGHT * 4
# allocate buffer
buf = (ctypes.c_ubyte * size)()
# get pointer to buffer
buf_p = ctypes.cast(buf, ctypes.c_void_p)

# global frame (or actually displayed frame) counter
framenr = 0

# vlc.CallbackDecorators.VideoLockCb is incorrect
CorrectVideoLockCb = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p))


@CorrectVideoLockCb
def _lockcb(opaque, planes):
    print("lock", file=sys.stderr)
    planes[0] = buf_p


@vlc.CallbackDecorators.VideoDisplayCb
def _display(opaque, picture):
    print("display {}".format(framenr))
    global framenr
    if framenr % 24 == 0:
        # shouldn't do this here! copy buffer fast and process in our own thread, or maybe cycle
        # through a couple of buffers, passing one of them in _lockcb while we read from the other(s).
        img = Image.frombuffer("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf, "raw", "BGRA", 0, 1)
        img.save('img{}.png'.format(framenr))
    framenr += 1


vlc.libvlc_video_set_callbacks(pl, _lockcb, None, _display, None)
pl.video_set_format("RV32", VIDEOWIDTH, VIDEOHEIGHT, VIDEOWIDTH * 4)

pl.play()
time.sleep(10)

[daniel5gh]

The unlock definition also seemed incorrect. This works:

CorrectVideoUnlockCb = ctypes.CFUNCTYPE(
    ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p))

I am by no means experienced with ctypes so this could very well be horribly wrong, but it does seem to work.

Also now with proper lock on buf1 acquired and released in the callbacks, I am doing this in the main thread:

    with buf1_lock:
        ctypes.memmove(buf2, buf1, size)
    img = Image.frombuffer("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf2, "raw", "BGRA", 0, 1)
    img.save('img{}.png'.format(framenr))

Seems to work great! I am planning to use it for something totally different than the (slow) PIL saving, but this proofs the concept nicely.

[adnanaddewala]

I get bus error (core dumped) when I run the following code.. What would be the issue?

`import vlc
import ctypes
import time
import sys
import cv2
import numpy
from PIL import Image

vlcInstance = vlc.Instance()
player = vlcInstance.media_player_new()
player.set_mrl("rtsp://root:[email protected]/axis-media/media.amp")

VIDEOWIDTH = 1920
VIDEOHEIGHT = 1080

size = VIDEOWIDTH * VIDEOHEIGHT * 4

buf = (ctypes.c_ubyte * size)()

buf_p = ctypes.cast(buf, ctypes.c_void_p)

framenr = 0

CorrectVideoLockCb = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p))

@CorrectVideoLockCb
def _lockcb(opaque, planes):
#print("lock", file=sys.stderr)
planes[0] = buf_p

@vlc.CallbackDecorators.VideoDisplayCb
def _display(opaque, picture):
global framenr
if framenr % 24 == 0:
img = Image.frombuffer("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf, "raw", "BGRA", 0, 1)
opencvImage = cv2.cvtColor(numpy.array(img), cv2.COLOR_RGB2BGR)
cv2.imwrite("imgcv{}.png".format(framenr),opencvImage)
framenr += 1

vlc.libvlc_video_set_callbacks(player, _lockcb, None, _display, None)
player.video_set_format("RV64", VIDEOWIDTH, VIDEOHEIGHT, VIDEOWIDTH * 4)

player.play()
time.sleep(10)`

[daniel5gh]

What would be the issue?

I don't know, but what catches my eye is that rv64 value for chroma, I doubt that is a thing, for one it's not listed here. Also note the documentation snippet I quoted (or linked here) saying that you must allocate memory for one or three planes depending on the chroma.

[George3256]

Я получаю ошибку шины (ядро сбрасывается), когда я запускаю следующий код .. В чем будет проблема?

`импорт vlc
импорт ctypes
время
импорта import sys
import cv2
import numpy
from PIL import Image

vlcInstance = vlc.Instance ()
player = vlcInstance.media_player_new ()
player.set_mrl ("rtsp: // root: [email protected]/axis-media/media.amp")

VIDEOWIDTH = 1920
VIDEOHEIGHT = 1080

размер = VIDEOWIDTH * VIDEOHEIGHT * 4

buf = (ctypes.c_ubyte * size) ()

buf_p = ctypes.cast (buf, ctypes.c_void_p)

framenr = 0

CorrectVideoLockCb = ctypes.CFUNCTYPE (ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER (ctypes.c_void_p))

@CorrectVideoLockCb
def _lockcb (непрозрачный, плоскости):
#print ("lock", file = sys.stderr)
плоскости [0] = buf_p

@ vlc.CallbackDecorators.VideoDisplayCb
def _display (непрозрачный, картинка):
глобальный framenr
if framenr% 24 == 0:
img = Image.frombuffer ("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf, "raw", "BGRA", 0, 1)
opencvImage = cv2.cvtColor (numpy.array (img), cv2.COLOR_RGB2BGR)
cv2.imwrite ("imgcv {}. Png" .format (framenr), opencvImage)
framenr + = 1

vlc.libvlc_video_set_callbacks (player, _lockcb, None, _display, None)
player.video_set_format ("RV64", VIDEOWIDTH, VIDEOHEIGHT, VIDEOWIDTH * 4)

player.play ()
time.sleep (10) `

how to add this code in class python ?
it bothers me that I cannot add the self parameter to these functions in python

[daniel5gh]

Here you are working with ctypes and callbacks from native code, because of this the function must be callable from C and have a specific signature, one that doesn't include a self reference to some python object. Nothing it stopping you from wrapping this in some class as long as the callback function matches the expected function signature.

All this is not in the scope of this issue and I suggest you read up on python in general and ctypes in specific.

Also, since this issue is closed I assume the bug is fixed and you don't need the code you pasted.

[George3256]

Many thanks. You have indicated the right path for me. I found what I need.

[fpoltronieri]

The unlock definition also seemed incorrect. This works:

CorrectVideoUnlockCb = ctypes.CFUNCTYPE(
    ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p))

I am by no means experienced with ctypes so this could very well be horribly wrong, but it does seem to work.

Also now with proper lock on buf1 acquired and released in the callbacks, I am doing this in the main thread:

    with buf1_lock:
        ctypes.memmove(buf2, buf1, size)
    img = Image.frombuffer("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf2, "raw", "BGRA", 0, 1)
    img.save('img{}.png'.format(framenr))

Seems to work great! I am planning to use it for something totally different than the (slow) PIL saving, but this proofs the concept nicely.

Great! Can you please share this version of the code?

[daniel5gh]

Great! Can you please share this version of the code?

Everything you need is in this thread, however if it helps you I dug up the code from when I worked on this and added the relevant part to a gist https://gist.github.com/daniel5gh/36f6a802451c02d3434f9aab730c1828

[fpoltronieri]

Thank you very much for the gist :)

[oaubert]

I do not see the reason for the VideoUnlockCb patching, since it is correctly defined in the vlc module. And I just committed a fix for the VideoLockCb so it should be OK now too.
The code you have would be nice to have in the standard distribution. I am thinking of defining a vlc.contrib package which could hold such reusable contributions. Would you be OK if I integrate the gist as vlc.contrib.GlSurface?

[daniel5gh]

I do not see the reason for the VideoUnlockCb patching, since it is correctly defined in the vlc module.

The code in the gist is a direct copy of the code I wrote back in 2017

Would you be OK if I integrate the gist as vlc.contrib.GlSurface

Absolutely!

Looking at it again, it might be better to keep references to the callback functions on the surface instance itself. In my code the user of Surface did that to prevent python GC from deleting those while they were still referenced by the player.

I was just being pragmatic back then and I have limited experience with this, but should there be some Py_INCREF or equivalent going on before passing a ref to the callback to native code?

I just ran into this GC deletion because I created the callback function in the scope/closure of get_libvlc_lock_callback and I think I did that because the callback signature can't include self. Maybe defining the callback on the class and returning a partial function which includes self could work too.

[oaubert]

Yes, keeping reference of the callbacks could be done at the module level, but it introduces complexity (and I imagine corner cases) so for the moment, I chose the easy path for me. I prepared the code (which I would put in the examples/ directory), could you have a look and see if you can improve it before I commit it:
https://gist.github.com/oaubert/55db79388634e86ab61585642fd74bd8

[daniel5gh]

I approve and left some small comments on the gist.

[dexception]

Is there anyway to reduce CPU usage for the following code ? Any other efficient way to convert it to opencv ? Can't we do it directly without using PIL ?

def getOpencvFrame(buf):
    pil_image = Image.frombuffer("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf, "raw", "BGRA", 0, 1)
    img = cv2.cvtColor(np.array(pil_image), cv2.COLOR_RGB2BGR)
    return img

[daniel5gh]

Is there anyway to reduce CPU usage for the following code ? Any other efficient way to convert it to opencv ? Can't we do it directly without using PIL ?

I don't know, I suggest you ask on stack overflow, this seems off topic for this issue.

[rho-sk]

Is there anyway to reduce CPU usage for the following code ? Any other efficient way to convert it to opencv ? Can't we do it directly without using PIL ?

def getOpencvFrame(buf):
    pil_image = Image.frombuffer("RGBA", (VIDEOWIDTH, VIDEOHEIGHT), buf, "raw", "BGRA", 0, 1)
    img = cv2.cvtColor(np.array(pil_image), cv2.COLOR_RGB2BGR)
    return img

as_np = np.ndarray(shape=(VIDEOHEIGHT, VIDEOWIDTH, 4), dtype=np.ubyte, buffer=buf)
cv2.imshow('image', as_np)

30% of CPU processing down :-)

[dexception]

@rho-sk
Nice.. Next step would be to port this conversion to GPU. That would really reduce CPU usage.