fix(331): fix json conversion #366

alxbl · 2021-10-26T01:25:16Z

This PR addresses several issues with the pyrdp-convert refactor:

Missing scapy imports leading to exceptions when converting
Bad __iter__ implementation for ExportedPDUStream resulting in improper cleanup
Missing handler cleanup call in PCAPConverter

Among other things, it closes #331.

Tested on both encrypted pcaps and ExportedPDU pcaps.

alxbl · 2021-10-26T01:29:32Z

I had to force push this because I accidentally included commits for the notify2 changes.

pyrdp/convert/PCAPConverter.py

pyrdp/convert/ExportedPDUStream.py

obilodeau

I gave this another good look this morning and I think it's good to go. Now I'm going to test it on some pcaps and replays and advise.

obilodeau · 2021-10-28T13:56:13Z

I see what you mean now by the fact that you felt this is broken. I removed the global exception handler because it would hide errors but in the pcap conversion codepath the fact that the generator raises a StopIterator does make conversions always fail with a stack trace. I'm glad you caught this before the next release.

alxbl · 2021-10-28T15:54:00Z

I think this PR fixes the StopIteration throw by making ExportedPDUStream behave like a proper iterator. We should be able to keep the global exception handler. Technically we could get rid of the except StopIteration bit.

I may have a few extra modifications to this in my other PRs. I kinda worked on all of them in parallel so they got a bit mixed :O

obilodeau · 2021-11-26T19:27:20Z

Right now this patch doesn't work for me.

Replay to JSON:

$ pyrdp-convert.py long-nla-session-test.pyrdp -f json -o json-fix-from-replay 
[*] Converting 'long-nla-session-test.pyrdp' to JSON
100% (691 of 691) |####################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Succesfully wrote '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-replay-long-nla-session-test.json'

$ ls -lh json-fix-from-replay-long-nla-session-test.json 
-rw-r--r-- 1 olivier olivier 83K Nov 26 14:21 json-fix-from-replay-long-nla-session-test.json

Pcap to JSON:

$ pyrdp-convert.py long-nla-session-test.pcap -f json -o json-fix-from-pcap -s long-nla-session-test.secrets 
[*] Analyzing PCAP 'long-nla-session-test.pcap' ...
    - 192.168.0.11 -> 192.168.0.11: TLS, master secret available (!)
    - 127.0.0.1 -> 127.0.0.1: TLS, master secret available (!)
    - 192.168.0.11 -> 192.168.0.11: TLS, master secret available (!)
    - 127.0.0.1 -> 127.0.0.1: TLS, master secret available (!)
    - 192.168.0.11 -> 192.168.0.11: TLS, master secret available (!)
    - 127.0.0.1 -> 127.0.0.1: TLS, master secret available (!)
[*] Processing 192.168.0.11 -> 192.168.0.11
100% (25 of 25) |######################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-pcap/'
[*] Processing 127.0.0.1 -> 127.0.0.1
100% (24 of 24) |######################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-pcap/'
[*] Processing 192.168.0.11 -> 192.168.0.11
100% (25 of 25) |######################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-pcap/'
[*] Processing 127.0.0.1 -> 127.0.0.1
100% (23 of 23) |######################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-pcap/'
[*] Processing 192.168.0.11 -> 192.168.0.11
  5% (92 of 1617) |####                                                                                | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 18

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
  8% (138 of 1617) |#######                                                                            | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
  9% (150 of 1617) |#######                                                                            | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Error PDU payload length: expected = 226, length = 28644
 10% (173 of 1617) |########                                                                           | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: Invalid X224 Error PDU payload length: expected = 113, length = 47553

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 12% (207 of 1617) |##########                                                                         | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 17% (276 of 1617) |##############                                                                     | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 24% (390 of 1617) |####################                                                               | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 30% (494 of 1617) |#########################                                                          | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 36% (597 of 1617) |##############################                                                     | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 40% (656 of 1617) |#################################                                                  | Elapsed Time: 0:00:01 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 243, length = 35530

[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 248

[-] Failed to handle data, continuing anyway: Invalid X224 Disconnect Request payload length: expected = 21, length = 48006
 46% (746 of 1617) |######################################                                             | Elapsed Time: 0:00:01 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
100% (1617 of 1617) |##################################################################################| Elapsed Time: 0:00:01 Time:  0:00:01

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-pcap/'
[*] Processing 127.0.0.1 -> 127.0.0.1
  2% (51 of 2357) |#                                                                                   | Elapsed Time: 0:00:00 ETA:  00:00:00
[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 96
  5% (118 of 2357) |####                                                                               | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 131, length = 19833
  7% (184 of 2357) |######                                                                             | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: Invalid X224 Error PDU payload length: expected = 187, length = 31997

[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 102

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 73, length = 59562

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
  8% (201 of 2357) |#######                                                                            | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Disconnect Request payload length: expected = 90, length = 24239

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 10% (251 of 2357) |########                                                                           | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 12% (285 of 2357) |##########                                                                         | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 14% (352 of 2357) |############                                                                       | Elapsed Time: 0:00:00 ETA:   0:00:02
[-] Failed to handle data, continuing anyway: Invalid X224 Error PDU payload length: expected = 174, length = 31792

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 23% (552 of 2357) |###################                                                                | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 29% (686 of 2357) |########################                                                           | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 31% (753 of 2357) |##########################                                                         | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: Invalid X224 Connection Request payload length: expected = 23, length = 35359
 34% (820 of 2357) |############################                                                       | Elapsed Time: 0:00:00 ETA:   0:00:01
[-] Failed to handle data, continuing anyway: Invalid X224 Connection Request payload length: expected = 236, length = 24939

[-] Failed to handle data, continuing anyway: Invalid X224 Disconnect Request payload length: expected = 213, length = 34803

Traceback (most recent call last):
  File "/home/olivier/Documents/gosecure/src/pyrdp/pyrdp/convert/PCAPConverter.py", line 47, in process
    self.processStream(startTimeStamp, stream)
  File "/home/olivier/Documents/gosecure/src/pyrdp/pyrdp/convert/PCAPConverter.py", line 110, in processStream
    for data, timeStamp, src, _dst in progressbar(stream):
  File "/home/olivier/Documents/gosecure/src/pyrdp/venv/lib/python3.9/site-packages/progressbar/shortcuts.py", line 10, in progressbar
    for result in progressbar(iterator):
  File "/home/olivier/Documents/gosecure/src/pyrdp/venv/lib/python3.9/site-packages/progressbar/bar.py", line 547, in __next__
    value = next(self._iterable)
  File "/home/olivier/Documents/gosecure/src/pyrdp/pyrdp/convert/TLSPDUStream.py", line 84, in decryptTLSStream
    record = packet[TLS]
  File "/home/olivier/Documents/gosecure/src/pyrdp/venv/lib/python3.9/site-packages/scapy/packet.py", line 1344, in __getitem__
    raise IndexError("Layer [%s] not found" % name)
IndexError: Layer [TLS] not found

[-] Failed: Layer [TLS] not found

$ ls -lh json-fix-from-pcap/
total 28K
-rw-r--r-- 1 olivier olivier   26 Nov 26 14:24 20211102154021_192.168.0.11-192.168.0.11.json
-rw-r--r-- 1 olivier olivier   26 Nov 26 14:24 20211102154022_127.0.0.1-127.0.0.1.json
-rw-r--r-- 1 olivier olivier   26 Nov 26 14:24 20211102154217_192.168.0.11-192.168.0.11.json
-rw-r--r-- 1 olivier olivier   26 Nov 26 14:24 20211102154218_127.0.0.1-127.0.0.1.json
-rw-r--r-- 1 olivier olivier   26 Nov 26 14:24 20211102154225_192.168.0.11-192.168.0.11.json
drwxr-xr-x 2 olivier olivier 4.0K Nov 26 14:24 certs
drwxr-xr-x 2 olivier olivier 4.0K Nov 26 14:24 files

L7 PDU Pcap to JSON:

$ pyrdp-convert.py long-nla-session-test-l7pdu.pcap -f json -o json-fix-from-l7-pcap 
[*] Analyzing PCAP 'long-nla-session-test-l7pdu.pcap' ...
    - 192.168.0.11 -> 192.168.0.11: plaintext
[*] Processing 192.168.0.11 -> 192.168.0.11
100% (1510 of 1510) |##################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/json-fix-from-l7-pcap/'


$ ls -lh json-fix-from-
json-fix-from-l7-pcap/                           json-fix-from-replay-long-nla-session-test.json
json-fix-from-pcap/                              
(venv) olivier@barachois:~/gosecure/research/projets/2021-10_pyrdp-video-precision$ ls -lh json-fix-from-l7-pcap/
total 12K
-rw-r--r-- 1 olivier olivier   50 Nov 26 14:25 20211102154229_192.168.0.11-192.168.0.11.json
drwxr-xr-x 2 olivier olivier 4.0K Nov 26 14:25 certs
drwxr-xr-x 2 olivier olivier 4.0K Nov 26 14:25 files

It might be that the session enumeration code relies only on IP addresses instead of IP:port tuples. I've seen this bug elsewhere in the redirection code.

obilodeau · 2021-12-02T23:05:41Z

I worked on this today. I'm not there yet but I'm making progress.

> TypeError: 'EDecimal' object cannot be interpreted as an integer I wasn't getting those error before, I think it might be python 3.10 related. Flooring gets rid of the error. That information was lost anyway.

Introduced an InetSocketAddress abstraction and adapted all code paths to it.

obilodeau · 2021-12-27T09:08:42Z

I think I finally fixed the issue(s). Tested many cases: TLS, PDUs, etc. Some failures left but I think it is in cases where we couldn't convert those streams anyway: pcap contains pyrdp-mitm <-> server flows in addition to the client <-> pyrdp-mitm downgraded one.

I'll let the tests run and sleep on it before merging.

obilodeau · 2021-12-27T16:55:36Z

Replay to JSON:

$ time pyrdp-convert.py long-nla-session-test.pyrdp -f json -o long-nla-session-test-json-fix-from-replay 
[*] Converting 'long-nla-session-test.pyrdp' to JSON
100% (691 of 691) |################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Succesfully wrote '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-replay-long-nla-session-test.json'

real	0m4.104s
user	0m4.081s
sys	0m0.663s

$ ls -l long-nla-session-test-json-fix-from-replay-long-nla-session-test.json  
-rw-r--r-- 1 olivier olivier 84957 Dec 27 11:24 long-nla-session-test-json-fix-from-replay-long-nla-session-test.json

Pcap to JSON:

$ time pyrdp-convert.py long-nla-session-test.pcap -f json -o long-nla-session-test-json-fix-from-pcap -s long-nla-session-test.secrets 
[*] Analyzing PCAP 'long-nla-session-test.pcap' ...
    - 192.168.0.11:34282 -> 192.168.0.11:3389 : TLS, master secret available (!)
    - 127.0.0.1:47944 -> 127.0.0.1:13389 : TLS, master secret available (!)
    - 192.168.0.11:34284 -> 192.168.0.11:3389 : TLS, master secret available (!)
    - 127.0.0.1:47946 -> 127.0.0.1:13389 : TLS, master secret available (!)
    - 192.168.0.11:34286 -> 192.168.0.11:3389 : TLS, master secret available (!)
    - 127.0.0.1:47948 -> 127.0.0.1:13389 : TLS, master secret available (!)
[*] Processing 192.168.0.11:34282 -> 192.168.0.11:3389
100% (25 of 25) |##################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pcap/'
[*] Processing 127.0.0.1:47944 -> 127.0.0.1:13389
100% (24 of 24) |##################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pcap/'
[*] Processing 192.168.0.11:34284 -> 192.168.0.11:3389
100% (25 of 25) |##################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pcap/'
[*] Processing 127.0.0.1:47946 -> 127.0.0.1:13389
100% (23 of 23) |##################################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pcap/'
[*] Processing 192.168.0.11:34286 -> 192.168.0.11:3389
100% (1617 of 1617) |##############################################| Elapsed Time: 0:00:02 Time:  0:00:02

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pcap/'
[*] Processing 127.0.0.1:47948 -> 127.0.0.1:13389
  3% (90 of 2357) |#                                               | Elapsed Time: 0:00:00 ETA:   0:00:06
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
  5% (135 of 2357) |##                                             | Elapsed Time: 0:00:00 ETA:   0:00:04
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 101

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Request payload length: expected = 173, length = 41748

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
  7% (175 of 2357) |###                                            | Elapsed Time: 0:00:00 ETA:   0:00:05
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Request payload length: expected = 29, length = 34996

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 120, length = 63995
  8% (195 of 2357) |###                                            | Elapsed Time: 0:00:00 ETA:   0:00:06
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Request payload length: expected = 234, length = 37146

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
  9% (214 of 2357) |####                                           | Elapsed Time: 0:00:00 ETA:   0:00:06
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 10% (247 of 2357) |####                                           | Elapsed Time: 0:00:00 ETA:   0:00:06
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 78

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 11% (270 of 2357) |#####                                          | Elapsed Time: 0:00:00 ETA:   0:00:06
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 37, length = 42080
 12% (292 of 2357) |#####                                          | Elapsed Time: 0:00:00 ETA:   0:00:06
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 15% (360 of 2357) |#######                                        | Elapsed Time: 0:00:01 ETA:   0:00:05
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'

[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 210
 16% (382 of 2357) |#######                                        | Elapsed Time: 0:00:01 ETA:   0:00:05
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 23% (562 of 2357) |###########                                    | Elapsed Time: 0:00:01 ETA:   0:00:04
[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 141, length = 51649
 29% (696 of 2357) |#############                                  | Elapsed Time: 0:00:01 ETA:   0:00:04
[-] Failed to handle data, continuing anyway: Invalid X224 Connection Confirm payload length: expected = 219, length = 33981
 31% (741 of 2357) |##############                                 | Elapsed Time: 0:00:01 ETA:   0:00:03
[-] Failed to handle data, continuing anyway: unsupported operand type(s) for 'in': 'int' and 'EnumMeta'
 35% (831 of 2357) |################                               | Elapsed Time: 0:00:01 ETA:   0:00:03
[-] Failed to handle data, continuing anyway: Invalid X224 Data PDU length indicator: expected = 2, indicator = 154

Traceback (most recent call last):
  File "/home/olivier/Documents/gosecure/src/pyrdp/pyrdp/convert/PCAPConverter.py", line 48, in process
    self.processStream(startTimeStamp, stream)
  File "/home/olivier/Documents/gosecure/src/pyrdp/pyrdp/convert/PCAPConverter.py", line 111, in processStream
    for data, timeStamp, src, _dst in progressbar(stream):
  File "/home/olivier/Documents/gosecure/src/pyrdp/venv/lib/python3.10/site-packages/progressbar/shortcuts.py", line 10, in progressbar
    for result in progressbar(iterator):
  File "/home/olivier/Documents/gosecure/src/pyrdp/venv/lib/python3.10/site-packages/progressbar/bar.py", line 547, in __next__
    value = next(self._iterable)
  File "/home/olivier/Documents/gosecure/src/pyrdp/pyrdp/convert/TLSPDUStream.py", line 86, in decryptTLSStream
    record = packet[TLS]
  File "/home/olivier/Documents/gosecure/src/pyrdp/venv/lib/python3.10/site-packages/scapy/packet.py", line 1344, in __getitem__
    raise IndexError("Layer [%s] not found" % name)
IndexError: Layer [TLS] not found

[-] Failed: Layer [TLS] not found

real	0m58.576s
user	0m57.499s
sys	0m1.284s

$ ls -l long-nla-session-test-json-fix-from-pcap/
total 132
-rw-r--r-- 1 olivier olivier     26 Dec 27 11:26 20211102154021_192.168.0.11:34282-192.168.0.11:3389.json
-rw-r--r-- 1 olivier olivier     26 Dec 27 11:26 20211102154022_127.0.0.1:47944-127.0.0.1:13389.json
-rw-r--r-- 1 olivier olivier     26 Dec 27 11:26 20211102154217_192.168.0.11:34284-192.168.0.11:3389.json
-rw-r--r-- 1 olivier olivier     26 Dec 27 11:27 20211102154218_127.0.0.1:47946-127.0.0.1:13389.json
-rw-r--r-- 1 olivier olivier 103722 Dec 27 11:27 20211102154225_192.168.0.11:34286-192.168.0.11:3389.json
drwxr-xr-x 2 olivier olivier   4096 Dec 27 11:26 certs
drwxr-xr-x 2 olivier olivier   4096 Dec 27 11:26 files
drwxr-xr-x 3 olivier olivier   4096 Dec 27 11:27 filesystems

The TLS error at the end is expected. It is happening on the TLS session between the MITM and the server, on which we don't do a protocol downgrade attack so we don't support all of it.

There are timestamp differences (looking at a diff) and there is 201 more events when converting from the pcap:

$ cat long-nla-session-test-json-fix-from-replay-long-nla-session-test.json | jq | grep "timestamp" | wc -l
907
$ cat long-nla-session-test-json-fix-from-pcap/20211102154225_192.168.0.11\:34286-192.168.0.11\:3389.json | jq | grep "timestamp" | wc -l
1108

From L7 PDU to JSON (one stream):

$ time pyrdp-convert.py long-nla-session-test-l7pdu.pcap -f json -o long-nla-session-test-json-fix-from-pdus  
[*] Analyzing PCAP 'long-nla-session-test-l7pdu.pcap' ...
    - 192.168.0.11:34286 -> 192.168.0.11:3389 : plaintext
[*] Processing 192.168.0.11:34286 -> 192.168.0.11:3389
100% (1510 of 1510) |##############################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pdus/'

real	0m6.239s
user	0m6.056s
sys	0m0.753s

$ ls -l long-nla-session-test-json-fix-from-pdus/
total 116
-rw-r--r-- 1 olivier olivier 103722 Dec 27 11:45 20211102154229_192.168.0.11:34286-192.168.0.11:3389.json
drwxr-xr-x 2 olivier olivier   4096 Dec 27 11:45 certs
drwxr-xr-x 2 olivier olivier   4096 Dec 27 11:45 files
drwxr-xr-x 3 olivier olivier   4096 Dec 27 11:45 filesystems

$ time pyrdp-convert.py long-nla-session-test-l7pdu-all-streams.pcap -f json -o long-nla-session-test-json-fix-from-pdus-all-streams  
[*] Analyzing PCAP 'long-nla-session-test-l7pdu-all-streams.pcap' ...
    - 0.20.0.4:1638404 -> 0.21.0.4:1703940 : plaintext
[*] Processing 0.20.0.4:1638404 -> 0.21.0.4:1703940
100% (1557 of 1557) |##############################################| Elapsed Time: 0:00:00 Time:  0:00:00

[+] Successfully wrote all files to '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/long-nla-session-test-json-fix-from-pdus-all-streams/'

real	0m5.705s
user	0m5.646s
sys	0m0.627s

Using on a multiple streams PDU is unsupported.

obilodeau

Last little adjustments, I'll merge soon

alxbl changed the title ~~Fix json conversion~~ fix(331): fix json conversion Oct 26, 2021

fix(331): Call conversion handler cleanup.

15b727a

alxbl force-pushed the fix-json-conversion branch from 7005f06 to 15b727a Compare October 26, 2021 01:28

obilodeau reviewed Oct 26, 2021

View reviewed changes

pyrdp/convert/PCAPConverter.py Outdated Show resolved Hide resolved

obilodeau reviewed Oct 26, 2021

View reviewed changes

pyrdp/convert/ExportedPDUStream.py Outdated Show resolved Hide resolved

fix: revert conversion success message for now.

348b3da

obilodeau reviewed Oct 28, 2021

View reviewed changes

obilodeau mentioned this pull request Nov 26, 2021

fix: Better success messages for pyrdp-convert #369

Merged

obilodeau added 2 commits December 27, 2021 02:00

Fixing type errors with high-precision pcaps

07a8d92

> TypeError: 'EDecimal' object cannot be interpreted as an integer I wasn't getting those error before, I think it might be python 3.10 related. Flooring gets rid of the error. That information was lost anyway.

Fixed many TLS decryption issues when src.ip == dst.ip

b45d92a

Introduced an InetSocketAddress abstraction and adapted all code paths to it.

Refactoring to regroup hardcoded PDU offsets and class renamed

ffbeb05

obilodeau reviewed Dec 27, 2021

View reviewed changes

obilodeau merged commit 4265d2b into GoSecure:master Dec 28, 2021

obilodeau mentioned this pull request Jan 7, 2022

feat(#313): Add support for conversion of multiple exported sessions. #368

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(331): fix json conversion #366

fix(331): fix json conversion #366

alxbl commented Oct 26, 2021

alxbl commented Oct 26, 2021

obilodeau left a comment

obilodeau commented Oct 28, 2021

alxbl commented Oct 28, 2021

obilodeau commented Nov 26, 2021

obilodeau commented Dec 2, 2021

obilodeau commented Dec 27, 2021

obilodeau commented Dec 27, 2021

obilodeau left a comment

fix(331): fix json conversion #366

fix(331): fix json conversion #366

Conversation

alxbl commented Oct 26, 2021

alxbl commented Oct 26, 2021

obilodeau left a comment

Choose a reason for hiding this comment

obilodeau commented Oct 28, 2021

alxbl commented Oct 28, 2021

obilodeau commented Nov 26, 2021

obilodeau commented Dec 2, 2021

obilodeau commented Dec 27, 2021

obilodeau commented Dec 27, 2021

obilodeau left a comment

Choose a reason for hiding this comment