TLS layer appears as Raw

[marcindulak]

Brief description

The expected TLS layer of "Client Hello" shows as Raw.

I'm using curl against an nginx server configured with a self-signed certiticate, and try to read the saved pcap with scapy.
I get the same problem with Raw layer from both nginx using TLS1.2 and TLS1.3. Wireshark shows the expected TLS layer.
Making curl https://www.wireshark.org results in the expected TLS1.3 layer when read with scapy.

I checked https://github.com/secdev/scapy/issues but didn't find an indication what the issue may be.
I tested also #4082 using gpotter2@e28c0a2.

Scapy version

8e116af

Python version

3.11.3

Operating system

Debian GNU/Linux 12 (bookworm)

Additional environment information

docker compose setup that includes client <-> router <-> server, all running Debian.

How to reproduce

1. Configure nginx

   sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/server.key -out /etc/ssl/certs/server.crt -subj '/CN=server'
   sudo cp server /etc/nginx/sites-enabled
   sudo service nginx restart

   using the following server file

   server {
    listen 8443 ssl;
    listen [::]:8443 ssl;
   
    ssl_certificate /etc/ssl/certs/server.crt;
    ssl_certificate_key /etc/ssl/private/server.key;
    ssl_protocols TLSv1.3;
   
    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;
   
    server_name server;
   
    location / {
             try_files $uri $uri/ =404;
        }
   }

2. On client, add sudo sh -c "echo 192.168.64.3 server >> /etc/hosts"

3. On client, start tcpdump

   sudo tcpdump -i eth0 -s0 -nxxX -w /tmp/server.pcap

4. From client, issue a curl request against the server

   echo > SSLKEYLOGFILE && SSLKEYLOGFILE=SSLKEYLOGFILE curl --cacert server.crt https://server:8443

5. On client, execute scapy script python script.py against the saved pcap

   from scapy.all import *
   
   load_layer('tls')
   conf.tls_session_enable = True
   conf.tls_nss_filename = 'SSLKEYLOGFILE'
   
   packets = rdpcap('server.pcap')
   print(packets[3].show())

6. If desired, create pcapng

   editcap --inject-secrets tls,SSLKEYLOGFILE server.pcap server.pcapng

Actual result

###[ Ethernet ]### 
  dst       = 02:42:c0:a8:30:02
  src       = 02:42:c0:a8:30:03
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = 569
     id        = 207
     flags     = DF
     frag      = 0
     ttl       = 64
     proto     = tcp
     chksum    = 0x4699
     src       = 192.168.48.3
     dst       = 192.168.64.3
     \options   \
###[ TCP ]### 
        sport     = 57146
        dport     = 8443
        seq       = 1342615260
        ack       = 3137762623
        dataofs   = 8
        reserved  = 0
        flags     = PA
        window    = 502
        chksum    = 0xf382
        urgptr    = 0
        options   = [('NOP', None), ('NOP', None), ('Timestamp', (4110525849, 588790177))]
###[ Raw ]### 
           load      = b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03p\x0bCD\xec\x05\xef]$\xd45\x9e\x1d\xfd\xa8\xcb\x8b`k\xb9\xe3f\x90\xbe@2\x00U\x83d\x8c\xfd ;\xb6\xbd\xd9\xd8\xb10Ycn\xf3Y\xcc\x11\xa8\xd4t\x8b\x8a\xb1\xfb\x015\xf6\xdbg\x9c\xfaF\xf9vR\x00>\x13\x02\x13\x03\x13\x01\xc0,\xc00\x00\x9f\xcc\xa9\xcc\xa8\xcc\xaa\xc0+\xc0/\x00\x9e\xc0$\xc0(\x00k\xc0#\xc0\'\x00g\xc0\n\xc0\x14\x009\xc0\t\xc0\x13\x003\x00\x9d\x00\x9c\x00=\x00<\x005\x00/\x00\xff\x01\x00\x01u\x00\x00\x00\x0b\x00\t\x00\x00\x06server\x00\x0b\x00\x04\x03\x00\x01\x02\x00\n\x00\x16\x00\x14\x00\x1d\x00\x17\x00\x1e\x00\x19\x00\x18\x01\x00\x01\x01\x01\x02\x01\x03\x01\x04\x00\x10\x00\x0e\x00\x0c\x02h2\x08http/1.1\x00\x16\x00\x00\x00\x17\x00\x00\x001\x00\x00\x00\r\x00*\x00(\x04\x03\x05\x03\x06\x03\x08\x07\x08\x08\x08\t\x08\n\x08\x0b\x08\x04\x08\x05\x08\x06\x04\x01\x05\x01\x06\x01\x03\x03\x03\x01\x03\x02\x04\x02\x05\x02\x06\x02\x00+\x00\t\x08\x03\x04\x03\x03\x03\x02\x03\x01\x00-\x00\x02\x01\x01\x003\x00&\x00$\x00\x1d\x00 \xdd\xabO\xa1\xfd\xf2"\xb7\xaa\xaf|\xdf\xfd\xbc\xa3\xec\x96n\x1f\x1d\xd1\x05\x95\xc1\xfa\xe9\xa4\x00kG\xcdH\x00\x15\x00\xb7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

None

Expected result

The TLS layer is printed

Related resources

pcaps.zip

[gpotter2]

Did you bind?

bind_layers(TCP, TLS, dport=8443)
bind_layers(TCP, TLS, dport=8443)

Also use sniff(offline="...", session=TCPSession) instead of rdpcap. with the PR you mentionned

[marcindulak]

So the issue is due to non-standard port, like in tintinweb/scapy-ssl_tls#52.

If it's not already there, maybe it's good to add this hint to the docs.

The script below reads the TLS layer, also with scapy 2.5.0 (with some minor differences in printing the raw b prefix for strings in sid and padding in gpotter2@e28c0a2).

from scapy.all import *

print(scapy.__version__)

load_layer('tls')

conf.tls_session_enable = True
conf.tls_nss_filename = 'SSLKEYLOGFILE'

bind_layers(TCP, TLS, dport=8443)
# packets = sniff(offline='server.pcap', session=TCPSession)
packets = rdpcap('server.pcap')

print(packets[3].show())

If I packets = sniff(offline='server.pcap', session=TCPSession) then

diff sniff.txt rdpcap.txt
10c10
<      len       = None
---
>      len       = 569
16c16
<      chksum    = None
---
>      chksum    = 0x4699
35c35
<            len       = 512
---
>            len       = 512    [deciphered_len= 512]

This is the output from gpotter2@e28c0a2

2.3.3.dev4234
###[ Ethernet ]### 
  dst       = 02:42:c0:a8:30:02
  src       = 02:42:c0:a8:30:03
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = None
     id        = 207
     flags     = DF
     frag      = 0
     ttl       = 64
     proto     = tcp
     chksum    = None
     src       = 192.168.48.3
     dst       = 192.168.64.3
     \options   \
###[ TCP ]### 
        sport     = 57146
        dport     = 8443
        seq       = 1342615260
        ack       = 3137762623
        dataofs   = 8
        reserved  = 0
        flags     = PA
        window    = 502
        chksum    = 0xf382
        urgptr    = 0
        options   = [('NOP', None), ('NOP', None), ('Timestamp', (4110525849, 588790177))]
###[ TLS ]### 
           type      = handshake
           version   = TLS 1.0
           len       = 512
           iv        = b''
           \msg       \
            |###[ TLS Handshake - Client Hello ]### 
            |  msgtype   = client_hello
            |  msglen    = 508
            |  version   = TLS 1.2
            |  gmt_unix_time= Thu, 26 Jul 2029 18:51:48 +0000 (1879786308)
            |  random_bytes= ec05ef5d24d4359e1dfda8cb8b606bb9e36690be4032005583648cfd
            |  sidlen    = 32
            |  sid       = b';\xb6\xbd\xd9\xd8\xb10Ycn\xf3Y\xcc\x11\xa8\xd4t\x8b\x8a\xb1\xfb\x015\xf6\xdbg\x9c\xfaF\xf9vR'
            |  cipherslen= 62
            |  ciphers   = [TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
            |  complen   = 1
            |  comp      = null
            |  extlen    = 373
            |  \ext       \
            |   |###[ TLS Extension - Server Name ]### 
            |   |  type      = server_name
            |   |  len       = 11
            |   |  servernameslen= 9
            |   |  servernames= [b'server']
            |   |###[ TLS Extension - Supported Point Format ]### 
            |   |  type      = ec_point_formats
            |   |  len       = 4
            |   |  ecpllen   = 3
            |   |  ecpl      = [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
            |   |###[ TLS Extension - Supported Groups ]### 
            |   |  type      = supported_groups
            |   |  len       = 22
            |   |  groupslen = 20
            |   |  groups    = [x25519, secp256r1, x448, secp521r1, secp384r1, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
            |   |###[ TLS Extension - Application Layer Protocol Negotiation ]### 
            |   |  type      = alpn
            |   |  len       = 14
            |   |  protocolslen= 12
            |   |  protocols = [b'h2', b'http/1.1']
            |   |###[ TLS Extension - Encrypt-then-MAC ]### 
            |   |  type      = encrypt_then_mac
            |   |  len       = 0
            |   |###[ TLS Extension - Extended Master Secret ]### 
            |   |  type      = extended_master_secret
            |   |  len       = 0
            |   |###[ TLS Extension - Post Handshake Auth ]### 
            |   |  type      = post_handshake_auth
            |   |  len       = 0
            |   |###[ TLS Extension - Signature Algorithms ]### 
            |   |  type      = signature_algorithms
            |   |  len       = 42
            |   |  sig_algs_len= 40
            |   |  sig_algs  = [sha256+ecdsa, sha384+ecdsa, sha512+ecdsa, ed25519, ed448, sha256+rsapss, sha384+rsapss, sha512+rsapss, sha256+rsaepss, sha384+rsaepss, sha512+rsaepss, sha256+rsa, sha384+rsa, sha512+rsa, sha224+ecdsa, sha224+rsa, sha224+dsa, sha256+dsa, sha384+dsa, sha512+dsa]
            |   |###[ TLS Extension - Supported Versions (for ClientHello) ]### 
            |   |  type      = supported_versions
            |   |  len       = 9
            |   |  versionslen= 8
            |   |  versions  = [TLS 1.3, TLS 1.2, TLS 1.1, TLS 1.0]
            |   |###[ TLS Extension - PSK Key Exchange Modes ]### 
            |   |  type      = psk_key_exchange_modes
            |   |  len       = 2
            |   |  kxmodeslen= 1
            |   |  kxmodes   = [psk_dhe_ke]
            |   |###[ TLS Extension - Key Share (for ClientHello) ]### 
            |   |  type      = key_share
            |   |  len       = 38
            |   |  client_shares_len= 36
            |   |  \client_shares\
            |   |   |###[ Key Share Entry ]### 
            |   |   |  group     = x25519
            |   |   |  kxlen     = 32
            |   |   |  key_exchange= ddab4fa1fdf222b7aaaf7cdffdbca3ec966e1f1dd10595c1fae9a4006b47cd48
            |   |###[ TLS Extension - Padding ]### 
            |   |  type      = padding
            |   |  len       = 183
            |   |  padding   = b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
           mac       = b''
           pad       = b''
           padlen    = None

None