Handle truncated PES packets #37

tmm1 · 2022-01-21T22:04:52Z

Simple fix for #35

Unfortunately this is breaking some tests which are creating and operating on malformed packets.

--- FAIL: TestParsePESData (0.00s)
    --- FAIL: TestParsePESData/with_header (0.00s)
        data_pes_test.go:407:
            	Error Trace:	data_pes_test.go:407
            	Error:      	Not equal:
            	            	expected: &astits.PESData{Data:[]uint8{0x64, 0x61, 0x74, 0x61}, Header:(*astits.PESHeader)(0x1381690)}
            	            	actual  : &astits.PESData{Data:[]uint8{0x64, 0x61, 0x74, 0x61, 0x73, 0x74, 0x75, 0x66, 0x66}, Header:(*astits.PESHeader)(0xc0000534a0)}

            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,4 +1,4 @@
            	            	 (*astits.PESData)({
            	            	- Data: ([]uint8) (len=4) {
            	            	-  00000000  64 61 74 61                                       |data|
            	            	+ Data: ([]uint8) (len=9) {
            	            	+  00000000  64 61 74 61 73 74 75 66  66                       |datastuff|
            	            	  },
            	            	@@ -62,3 +62,3 @@
            	            	   }),
            	            	-  PacketLength: (uint16) 67,
            	            	+  PacketLength: (uint16) 9,
            	            	   StreamID: (uint8) 1
            	Test:       	TestParsePESData/with_header
--- FAIL: TestParseData (0.00s)
    data_test.go:47:
        	Error Trace:	data_test.go:47
        	Error:      	Not equal:
        	            	expected: []*astits.DemuxerData{(*astits.DemuxerData)(0xc000073770)}
        	            	actual  : []*astits.DemuxerData{(*astits.DemuxerData)(0xc000073720)}

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -24,4 +24,4 @@
        	            	   PES: (*astits.PESData)({
        	            	-   Data: ([]uint8) (len=4) {
        	            	-    00000000  64 61 74 61                                       |data|
        	            	+   Data: ([]uint8) (len=9) {
        	            	+    00000000  64 61 74 61 73 74 75 66  66                       |datastuff|
        	            	    },
        	            	@@ -85,3 +85,3 @@
        	            	     }),
        	            	-    PacketLength: (uint16) 67,
        	            	+    PacketLength: (uint16) 9,
        	            	     StreamID: (uint8) 1
        	Test:       	TestParseData
FAIL
FAIL	github.com/asticode/go-astits	0.008s
FAIL

Specifically, the with_header PES test advertises a packet length of 67 even though it only has 9 bytes of data:

go-astits/data_pes_test.go

Lines 258 to 265 in 473f66c

    
           "with_header", 
        
           func(w *astikit.BitsWriter, withStuffing bool, withCRC bool) { 
        
           	packetLength := 67 
        
           	stuffing := []byte("stuff") 
        
           	if !withStuffing { 
        
           		packetLength -= len(stuffing) 
        
           	}

go-astits/data_pes_test.go

Lines 329 to 332 in 473f66c

    
           stuffing := []byte("stuff") 
        
           w.Write([]byte("data")) // Data 
        
           if withStuffing { 
        
           	w.Write(stuffing) // Stuffing

I tried to fix these tests but they make a lot of assumptions and reuse fixtures like pesWithHeaderBytes() which creates quite the rabbit hole.

tmm1 · 2022-01-21T22:07:33Z

Simple fix for #35

My first attempt was more complex: fancybits@a55f10a

That approach also works, but the same tests fail for the same reason.

coveralls · 2022-01-21T22:11:51Z

Coverage increased (+0.03%) to 77.24% when pulling b1ef54e on tmm1:truncated-pes-adjust into 473f66c on asticode:master.

asticode · 2022-01-23T10:24:22Z

Specifically, the with_header PES test advertises a packet length of 67 even though it only has 9 bytes of data

Are you sure this is an error? 🤔 packet length is supposed to indicate the number of bytes after this field, optional header included.

asticode · 2022-01-23T10:28:57Z

Regarding the content of the PR, I don't really like the idea of ignoring the problem and rewriting the packet length.

As you suggested here, I would implement the CorruptedPacketHandler logic and make the corrupted packet go through it.

eric · 2022-11-15T00:00:42Z

Fixed the issue with the tests not passing.

asticode · 2022-11-16T08:22:12Z

Fixed the issue with the tests not passing.

Thanks @eric, but as I mentioned in my previous comment, I'd rather go the CorruptedPacketHandler route in this case 👍

ayhangenc · 2023-06-07T12:36:50Z

Hi,

I'd like to share my case which might be related, I set up a muxer from a live stream into srt mpegts via astits.newmuxer and during muxing I have error messages both on go app and ffplay(receiver);
SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 185 bytes of payload: only 182 is available...
on app log and then
panic: runtime error: slice bounds out of range [1319:1316]
goroutine 12 [running]:
bufio.(*Writer).Write(0xc00029e100, {0xc0002801e1?, 0x10014c675?, 0x10014c675?}) /usr/local/go/src/bufio/bufio.go:670 +0x1c8
github.com/asticode/go-astikit.(*BitsWriter).write(0xc0002a42d0, {0xc0002801e1, 0x1, 0x1}) /Users/user/go/pkg/mod/github.com/asticode/[email protected]/binary.go:142 +0x3d
on console...
also on ffplay;
[mpegts @ 0x7fae05714180] Packet corrupt (stream = 0, dts = 6652800).
[h264 @ 0x7fae06a3d280] concealing 2720 DC, 2720 AC, 2720 MV errors in P frame
[aac @ 0x7fae068416c0] channel element 0.5 is not allocated f=0/0

I just wonder how to overcome this? What would be CorruptPacketHandler case for muxer?

Thanks in advance

AyhG

asticode · 2023-06-13T07:08:55Z

@ayhangenc could you share the next 10 or 20 lines of the panic stack?

ayhangenc · 2023-06-13T08:33:47Z

Hi @asticode ,

here is the complete dump of the log screen for the app.
it'a a basic rtsp to srt gateway. test source is ffmpeg and the srt consumer is ffplay currently.
I have a lot of error messages on ffplay while video is frozen before panic. Panic is not instant though, it takes a while.
hope this helps..
thnak you.

the error message on ffplay is this:

...
[mpegts @ 0x7fbf19f16c00] changing packet size to 192    0B f=0/0   
[mpegts @ 0x7fbf19f16c00] changing packet size to 188    0B f=0/0   
[mpegts @ 0x7fbf19f16c00] changing packet size to 192    0B f=0/0   
[mpegts @ 0x7fbf19f16c00] changing packet size to 188    0B f=0/0   
[mpegts @ 0x7fbf19f16c00] changing packet size to 204    0B f=0/0   
[mpegts @ 0x7fbf19f16c00] changing packet size to 188    0B f=0/0   
1906.04 A-V:520.879 fd= 123 aq=    0KB vq=    0KB sq=    0B f=0/0   
...

and the app log:

% go run .
2023/06/13 10:39:55 loading config files from config folder ./config!...
2023/06/13 10:39:55 Running RTSP Client!... live2
2023/06/13 10:39:55 Running RTSP Client!... live1
2023/06/13 10:39:55 Socket  connected!... &{129186975 false 0xc000236090 127.0.0.1 5808 map[blocking:0 latency:250 mode:caller payloadsize:1316 streamid:CAM1 transtype:live] 2 1456 -1}
2023/06/13 10:39:55 Socket  connected!... &{129186974 false 0xc0002b0090 127.0.0.1 5809 map[blocking:0 latency:250 mode:caller payloadsize:1316 streamid:CAM2 transtype:live] 2 1456 -1}
2023/06/13 10:39:55 IDR Not Present Error: 
2023/06/13 10:39:55 Encoding video access units into muxer error: IDR Not Present Error... 
2023/06/13 10:39:55 IDR Not Present Error: 
2023/06/13 10:39:55 Encoding video access units into muxer error: IDR Not Present Error...
2023/06/13 10:39:55 IDR Not Present Error: 
2023/06/13 10:39:55 Encoding video access units into muxer error: IDR Not Present Error... 
...
2023/06/13 10:40:09 IDR Not Present Error: 
2023/06/13 10:40:09 Encoding video access units into muxer error: IDR Not Present Error...  
2023/06/13 10:40:09 IDR Not Present Error: 
2023/06/13 10:40:09 Encoding video access units into muxer error: IDR Not Present Error...  
2023/06/13 10:40:16 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 272 bytes of payload: only 182 is available...  
2023/06/13 10:40:16 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:40:16 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:40:16 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 293 bytes of payload: only 182 is available...  
2023/06/13 10:40:16 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:40:16 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:40:44 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 333 bytes of payload: only 182 is available...  
2023/06/13 10:40:44 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:40:44 Encoding audio access units into muxer error: SRT Send Error... 
2023/06/13 10:42:04 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 356 bytes of payload: only 184 is available...  
2023/06/13 10:42:04 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:42:04 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:42:14 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 187 bytes of payload: only 182 is available...  
2023/06/13 10:42:14 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:42:14 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:42:14 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 187 bytes of payload: only 182 is available...  
2023/06/13 10:42:14 SRT Audio ErrorChannel active... 1001
 
2023/06/13 10:42:14 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:43:08 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 183 bytes of payload: only 182 is available...  
2023/06/13 10:43:08 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:43:08 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:43:19 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 193 bytes of payload: only 182 is available...  
2023/06/13 10:43:19 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:43:19 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:43:45 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 143 bytes of payload: only 140 is available...  
2023/06/13 10:43:45 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:43:45 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:43:55 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 236 bytes of payload: only 184 is available...  
2023/06/13 10:43:55 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:43:55 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 350 bytes of payload: only 182 is available...  
2023/06/13 10:43:55 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:43:55 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:43:55 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:44:17 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 321 bytes of payload: only 184 is available...  
2023/06/13 10:44:17 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:44:17 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:44:34 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 310 bytes of payload: only 182 is available...  
2023/06/13 10:44:34 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:44:34 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:44:34 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 305 bytes of payload: only 182 is available...  
2023/06/13 10:44:34 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:44:34 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:44:39 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 197 bytes of payload: only 182 is available...  
2023/06/13 10:44:39 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:44:39 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:44:39 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 55 bytes of payload: only 30 is available...  
2023/06/13 10:44:39 SRT Audio ErrorChannel active... 1001
 
2023/06/13 10:44:39 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:44:39 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 258 bytes of payload: only 182 is available...  
2023/06/13 10:44:39 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:44:39 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:45:03 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 229 bytes of payload: only 184 is available...  
2023/06/13 10:45:03 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:45:03 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 343 bytes of payload: only 182 is available...  
2023/06/13 10:45:03 Encoding audio access units into muxer error: SRT Send Error...   
2023/06/13 10:45:03 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:45:03 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:45:13 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 245 bytes of payload: only 184 is available...  
2023/06/13 10:45:13 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:45:13 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:46:08 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 219 bytes of payload: only 184 is available...  
2023/06/13 10:46:08 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:46:08 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:46:23 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 213 bytes of payload: only 182 is available...  
2023/06/13 10:46:23 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:46:23 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:46:55 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 222 bytes of payload: only 182 is available...  
2023/06/13 10:46:55 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:46:55 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:46:55 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 142 bytes of payload: only 68 is available...  
2023/06/13 10:46:55 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:46:55 Encoding video access units into muxer error: SRT Send Error...  
2023/06/13 10:47:48 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 185 bytes of payload: only 184 is available...  
2023/06/13 10:47:48 SRT Audio ErrorChannel active... 1001
 
2023/06/13 10:47:48 Encoding audio access units into muxer error: SRT Send Error...  
2023/06/13 10:48:44 SRT MPEGTS muxer write SRT Video error: writePacket: can't write 156 bytes of payload: only 153 is available...  
2023/06/13 10:48:44 SRT Video ErrorChannel active...  127.0.0.1
2023/06/13 10:48:44 Encoding video access units into muxer error: SRT Send Error... 
2023/06/13 10:48:44 SRT MPEGTS muxer write SRT Audio error: writePacket: can't write 175 bytes of payload: only 22 is available...  
2023/06/13 10:48:44 SRT Audio ErrorChannel active... 1002
 
2023/06/13 10:48:44 Encoding audio access units into muxer error: SRT Send Error...  
panic: runtime error: slice bounds out of range [120:0]

goroutine 52 [running]:
bytes.(*Buffer).Write(0x3?, {0xc000280318?, 0xc00004a970?, 0x10039f53b?})
        /usr/local/go/src/bytes/buffer.go:172 +0xd6
github.com/asticode/go-astikit.(*BitsWriter).write(0xc000298320, {0xc000280318, 0x1, 0x1})
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/binary.go:142 +0x3d
github.com/asticode/go-astikit.(*BitsWriter).flushBsCache(...)
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/binary.go:169
github.com/asticode/go-astikit.(*BitsWriter).writeFullByte(0x0?, 0x20?)
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/binary.go:179 +0x7e
github.com/asticode/go-astikit.(*BitsWriter).writeByteSlice(0xc000298320, {0xc000599180, 0xa8, 0xc00004aaa0?})
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/binary.go:130 +0x89
github.com/asticode/go-astikit.(*BitsWriter).Write(0xc000298320?, {0x1003e5cc0?, 0xc00004aa50?})
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/binary.go:75 +0xa5
github.com/asticode/go-astits.writePESData(0x100405d00?, 0xc000282570?, {0xc000599180?, 0x1aa?, 0x1aa?}, 0x8?, 0xb6?)
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/data_pes.go:465 +0xb3
github.com/asticode/go-astits.(*Muxer).WriteData(0xc0002846e0, 0xc00004acb8)
        /Users/anyuser/go/pkg/mod/github.com/asticode/[email protected]/muxer.go:230 +0x4dd
X.7/gateway.(*SRT).Encode(0xc0000e2580, {0xc0003c9aa0, 0x4, 0x10038d28a?}, {0x0?, 0x10005687f?, 0x10004e552?}, 0x7db35b0a00)
        /Users/anyuser/GolandProjects/X.7/gateway/srt.go:175 +0x10cd
X.7/gateway.(*Gateway).RunRTSP.func1(0xc000292140?)
        /Users/anyuser/GolandProjects/X.7/gateway/rtsp.go:127 +0x191
github.com/bluenviron/gortsplib/v3.(*clientFormat).readRTPUDP(0xc0002983c0, 0xc000282870?)
        /Users/anyuser/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/client_format.go:117 +0x185
github.com/bluenviron/gortsplib/v3.(*clientMedia).readRTPUDPPlay(0xc000296480, {0xc000176000, 0xc00004af20?, 0x5c1})
        /Users/anyuser/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/client_media.go:291 +0xae
github.com/bluenviron/gortsplib/v3.(*clientUDPListener).runReader(0xc0002b80e0)
        /Users/anyuser/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/client_udp_listener.go:195 +0x1e5
created by github.com/bluenviron/gortsplib/v3.(*clientUDPListener).start
        /Users/anyuser/go/pkg/mod/github.com/bluenviron/gortsplib/[email protected]/client_udp_listener.go:153 +0xaa
exit status 2

asticode · 2023-06-13T08:44:52Z

Can you open an issue including test data which would allow us to reproduce the problem so that it could be traced more easily?

pes: read available PES data when truncated

b1ef54e

Add extra empty bytes to the buffer to make tests pass

8d2fa2e

asticode force-pushed the master branch 2 times, most recently from 106bbe5 to 0df190a Compare July 27, 2023 09:41

Merge remote-tracking branch 'upstream/master' into truncated-pes-adjust

9fc774d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle truncated PES packets #37

Handle truncated PES packets #37

tmm1 commented Jan 21, 2022

tmm1 commented Jan 21, 2022 •

edited

Loading

coveralls commented Jan 21, 2022

asticode commented Jan 23, 2022

asticode commented Jan 23, 2022

eric commented Nov 15, 2022

asticode commented Nov 16, 2022

ayhangenc commented Jun 7, 2023

asticode commented Jun 13, 2023

ayhangenc commented Jun 13, 2023 •

edited

Loading

asticode commented Jun 13, 2023

	"with_header",
	func(w *astikit.BitsWriter, withStuffing bool, withCRC bool) {
	packetLength := 67
	stuffing := []byte("stuff")

	if !withStuffing {
	packetLength -= len(stuffing)
	}

	stuffing := []byte("stuff")
	w.Write([]byte("data")) // Data
	if withStuffing {
	w.Write(stuffing) // Stuffing

Handle truncated PES packets #37

Are you sure you want to change the base?

Handle truncated PES packets #37

Conversation

tmm1 commented Jan 21, 2022

tmm1 commented Jan 21, 2022 • edited Loading

coveralls commented Jan 21, 2022

asticode commented Jan 23, 2022

asticode commented Jan 23, 2022

eric commented Nov 15, 2022

asticode commented Nov 16, 2022

ayhangenc commented Jun 7, 2023

asticode commented Jun 13, 2023

ayhangenc commented Jun 13, 2023 • edited Loading

asticode commented Jun 13, 2023

tmm1 commented Jan 21, 2022 •

edited

Loading

ayhangenc commented Jun 13, 2023 •

edited

Loading