Python integration with sender and receiver pipeline

[himansingh]

Hello everyone, thank you for this great product.

I am trying to utilize UltraGrid to send and receive video frames from a USB camera using a Python script with minimum latency.

Machine1 (Edge Device):
Connected to a USB camera -> UltraGrid sender streams the camera feed

Machine2 (Placed in a server room and is part of an on-premise cluster):
Receives the feed -> processes individual frames in python using opencv

Any help in integrating UltraGrid with python opencv would be very much appreciated.

[himansingh]

Alternatively, would it be better to provide a filesink mechanism, like Gstreamer, to write the frames on disk?

[MartinPulec]

hi @himansingh,

there is multiple options how to achieve this – basically you'd need to use some of auxiliary displays, eg.:

a. unix_sock – output will be written as RGB to specified UNIX socket, from which it can be read with the Python
b. file display – perhaps less eligible - you can output to some output FIFO file (eg. NUT) and then read inside other process
c. dump display – dumps just plain RAW files to filesystem

Personally, I'd suggest either a or c. a is quite straightforward, for c you'd perhaps want to use something like:

-d dump:/tmp/dir --param decoder-use-codec=RGB 

Then it would dump raw RGB files to /tmp/dir (00000001.rgb and on) which you can directly process (and you'd perhaps also like to delete processed files).

[himansingh]

@MartinPulec Thank you for the quick response. I did try the -d dump option earlier but must have done something wrong.

The dump display option has solved the issue. Don't have much experience with unix sockets but will try to implement that and send an update.

[himansingh]

@MartinPulec,

The suggested commands were running smoothly until now. My team just noticed a random change in the index of the file at the start of the receiver command.

System Information
OS: Ubuntu 20.04
CPU: AMD RYZEN 6000 Series
Ultragrid version: UltraGrid-1.8.6-x86_64.AppImage

Description
I was testing UltraGrid using the following commands on the same system.

Sender Command:
./UltraGrid-1.8.6-x86_64.AppImage -t file:/path/to/.mp4:codec=RGB

Receiver Command:
./UltraGrid-1.8.6-x86_64.AppImage -d dump:/path/to/folder --param decoder-use-codec=RGB

Expected Behaviour
On the start of the receiver command, the file name should start from 0000001 namely 0000001.rgb

Observation
Usually, the file name starts from index 1, namely 0000001.rgb but at times we have observed that the start index changes randomly.

Steps to replicate:
This is a random behaviour so it is very difficult to replicate but I am mentioning the steps that caused the issue for me:

1. Start sender and received
2. Stop receiver
3. stop sender
4. Wait for some time (say 10 mins)
5. Start sender
6. start receiver

This is a random behaviour so it's very difficult to replicate. We have observed this just a couple of times out of hundreds of runs.

[MartinPulec]

Thanks for reporting, we've looked through the code and I believe that the frame with the index 1 will be always output if there is the input.

Can I ask what is the first index value? Is it some small number or anything entirely random? Is the sequence continuous? The incorrect directory listing would be ideal. Also UG might be useful for debugging.

Steps to replicate:

Thanks, but I'd need more about the steps, namely steps 1.-4. Is there some specific reason why those are needed? The thing is that if you remove the dump directory before step 6, the behavior should not be affected by the steps 1.-4. It it would, it implies that there is some insane state, probably on filesystem (eventually network state, eg. NAT

[MartinPulec]

PS: (for the record) I've tested with following script and I didn't manage (yet) to reproduce the problem:

script.sh (can be split to sender and receiver if needed)

#!/bin/sh -eu
sleep_min=3 # must allow at least first dump creation
sleep_max=5
sleep_span=$((sleep_max-sleep_min))

if [ -e dump ]; then
  echo "'dump' exists, remove first!"
  exit 1
fi

last_pid=
atexit() {
  if [ $last_pid ]; then
    kill $last_pid
    echo "Killing $last_pid!"
    kill $last_pid
  fi
}
trap atexit INT

while :
do
  sleep_dur=$(awk 'BEGIN{srand(); print int(rand() * '$sleep_span');}')
  UltraGrid-continuous-x86_64.AppImage -t testcard:c=RGB -d dump:dump &
  last_pid=$!
  sleep $((sleep_min + sleep_dur))
  kill $!
  wait $!
  ls dump
  if [ ! -f dump/00000001.* ]
  then
    echo ERROR - file index 1 not found!
    break
  fi
  rm dump/0*.rgb dump/video.info
done

But of course, it doesn't emulate much timing issues, if there is some race-condition.

Important: If you intend to use the script, make sure to download recent UltraGrid, because when the AppImage was run in background, killing the AppImage left the UG processes running! Also be very cautious, because the script uses rm etc.