bcm2835 v4l2 missing timestamps only in H.264 #1836
Labels
Waiting for external input
Waiting for a comment from the originator of the issue, or a collaborator.
Waiting for internal comment
Waiting for comment from a member of the Raspberry Pi engineering team
Comments
|
I've just tried this using v4l2-ctl and setting the driver One thought is that the header bytes are likely to arrive at the kernel later than the timestamp on the first frame, so we may well have a backwards jump there. Header bytes in the above will be at 109.773225, whilst the first encoded frame is at 109.558918. Is that what is upsetting ffmpeg? The driver does have the kernel start time stored, so could insert that very easily instead. (Having been looking at the video encoder over the last couple of days, it seems that if using inline headers then those all come out with a 0 timestamp too. I don't know what the best thing to do in that situation is. Possibly store the last frame timestamp and reuse it) |
|
I don't know what upsets ffmpeg. I just did h264 bcm2835-v4l2 debug: ffmpeg says for the first frame: bcm2835-v4l2 debug with mjpeg: now ffmpeg gets actual values |
|
In short: timestamps are missing/corrupted for the first frame, subsequent frames are ok. |
|
It looks like there is an issue on both sides. vs extended kernel log All those kernel log timestamps look valid to me. The interesting bit is that the kernel reports delivering buffers of size 27, 19924, 2382, 3834. ffmpeg is reporting frames of 19951, 2382, 3834. It has combined the header bytes into the first frame, and I suspect it has dropped the timestamp in the process. http://lists.ffmpeg.org/pipermail/ffmpeg-devel-irc/2016-March/003424.html from 13:38:02 CET onwards appears to indicate that there is an issue in ffmpeg if handling raw H264 streams. I'll look into making sure that the V4L2 timestamps are correct (mostly done already), but I'm not going to go debugging FFmpeg in any detail. |
|
Thanks @6by9 for all the work and the quick response! I'll try your modifications when they are ready and file a bug with ffmpeg if needed. I built my ffmpeg from the master git branch because I couldn't find a binary distribution. (Compiling times on the pi 3 are now bearable with I also tried libav's avconv, but couldn't get it to work at all with v4l2. |
|
I'll dig out my diffs for V4L2 and push them to my fork of the kernel branch if you fancy rebuilding them. I have been working against the 4.9 tree instead of 4.4, but that shouldn't make a big difference. As it happens I pulled down and kicked off a build of ffmpeg before I left the office. Hopefully it'll build successfully and be waiting for me on Monday. |
|
Sorry, I'm going to have to say this appears to be FFMPEG mishandling header bytes from V4L2 devices. (I've just added the code to correctly signal key frames through V4L2 in case FFMPEG was needing to see those, and that is why the second buffer has flags 0x2009) Looks to be a tad strange - that duration looks very like AV_NOPTS_VALUE (0x8000000000000000) converted to seconds, but the start time is also the timestamp of the first P-frame, ignoring the header bytes and I-frame. I'll push my driver updates (all kernel side), but then I'm going to have to park this waiting on more information. |
|
Updates pushed to https://github.com/6by9/linux rpi-4.9.y branch. |
6by9
added a commit
to 6by9/linux
that referenced
this issue
Feb 13, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi#1836 Signed-off-by: Dave Stevenson <[email protected]>
|
Ok I'll try 4.9 I noticed that the timestamps generated by the bcm2835 driver are boot-time based, while the other devices all use UNIX time. Is that also a bug? |
Nope. All our buffers have the V4L2_BUF_FLAG_TIMESTAMP MONOTONIC flag set, which means they are with respect to clock_gettime(CLOCK_MONOTONIC).
The unspecified starting point happens to be boot time. |
6by9
added a commit
to 6by9/linux
that referenced
this issue
Feb 21, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi#1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 1, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
pelwell
pushed a commit
that referenced
this issue
Mar 2, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
Noltari
pushed a commit
to Noltari/linux
that referenced
this issue
Mar 8, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi/linux#1836 Signed-off-by: Dave Stevenson <[email protected]>
ED6E0F17
pushed a commit
to ED6E0F17/linux
that referenced
this issue
Mar 12, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi#1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 12, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 17, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 18, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 23, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 27, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
fuzeman
pushed a commit
to fuzeman/linux-ubuntu-zesty
that referenced
this issue
Mar 30, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi/linux#1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Mar 31, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 2, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 10, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 13, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 18, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 22, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 24, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 27, 2017
H264 header come off VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jun 17, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jun 17, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jun 26, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jun 26, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
phhusson
pushed a commit
to phhusson/dkms-rpi-codec
that referenced
this issue
Jun 26, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi/linux#1836 Signed-off-by: Dave Stevenson <[email protected]>
rsglobal
pushed a commit
to GloDroid/glodroid_forks
that referenced
this issue
Jun 26, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi/linux#1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jul 1, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jul 1, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jul 13, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jul 13, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jul 20, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Aug 10, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
rsglobal
pushed a commit
to GloDroid/glodroid_forks
that referenced
this issue
Aug 12, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi/linux#1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Aug 12, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Aug 15, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Aug 15, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Aug 19, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Aug 19, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 1, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 1, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 11, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 15, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 28, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Oct 2, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Oct 7, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Oct 16, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Oct 19, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Nov 4, 2020
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. #1836 Signed-off-by: Dave Stevenson <[email protected]>
jai-raptee
pushed a commit
to jai-raptee/iliteck1
that referenced
this issue
Apr 30, 2024
…stamp H264 header come from VC with 0 timestamps, which means they get a strange timestamp when processed with VC/kernel start times, particularly if used with the inline header option. Remember the last frame timestamp and use that if set, or otherwise use the kernel start time. raspberrypi/linux#1836 Signed-off-by: Dave Stevenson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When configured to capture H.264 video from the Raspberry Camera Module v2.1 through the video4linux2 driver, ffmpeg complains that "Timestamps are unset", causing sync issues when combined with audio coming from an other device. As you can see from the ffmpeg output, "This is deprecated and will stop working in the future. Fix your code to set the timestamps properly"
Timestamps are correct when capturing MJPEG instead of H.264. I first tried with the default kernel (4.4.38-v7+) and after upgrading to bleeding edge (4.4.47-v7+) the problem persists. Here is some timestamp debug data I got from
ffmpeg -debug_ts -vcodec h264 -i /dev/video0 -c:v copy -t 0.1 timestamps.mkv:In MJPEG I get correct timestamps (and they seem to sync quite well when I add USB audio). From
ffmpeg -debug_ts -vcodec mjpeg -i /dev/video0 -c:v copy -t 0.1 timestamps.mkv:Default kernel:
After
rpi-updateThe text was updated successfully, but these errors were encountered: