`sniff` seems to crash with "ValueError: memoryview assignment: lvalue and rvalue have different structures" occasionally

[evverx]

Brief description

From time to time sniff(session=TCPSession) throws

...
File scapy/scapy/sessions.py:193, in StringBuffer.append(self, data, seq)
    188     assert len(self.content) == self.content_len
    189 # XXX removes empty space marker.
    190 # for ifrag in self.incomplete:
    191 #     if [???]:
    192 #         self.incomplete.remove([???])
--> 193 memoryview(self.content)[seq:seq + data_len] = data

ValueError: memoryview assignment: lvalue and rvalue have different structures

(the full backtrace is in the "Actual result" section)

Scapy version

577ab1f

Python version

Python 3.11.3

Operating system

Linux version 6.3.7-100.fc37.x86_64

Additional environment information

No response

How to reproduce

I haven't figured out how to reproduce it reliably. Sorry!

It seems it happens only when session=TCPSession is passed to sniff and load_layer('tls') is called before that somewhere. I'll try to capture packets triggering it and attach a pcap.

Actual result

ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 sniff(store=False, prn=lambda p: repr(p), session=TCPSession)

File scapy/scapy/sendrecv.py:1311, in sniff(*args, **kwargs)
   1307 @conf.commands.register
   1308 def sniff(*args, **kwargs):
   1309     # type: (*Any, **Any) -> PacketList
   1310     sniffer = AsyncSniffer()
-> 1311     sniffer._run(*args, **kwargs)
   1312     return cast(PacketList, sniffer.results)

File scapy/scapy/sendrecv.py:1254, in AsyncSniffer._run(self, count, store, offline, quiet, prn, lfilter, L2socket, timeout, opened_socket, stop_filter, iface, started_callback, session, session_kwargs, **karg)
   1252 p.sniffed_on = sniff_sockets[s]
   1253 # on_packet_received handles the prn/storage
-> 1254 session.on_packet_received(p)
   1255 # check
   1256 if (stop_filter and stop_filter(p)) or \
   1257         (0 < count <= session.count):

File scapy/scapy/sessions.py:395, in TCPSession.on_packet_received(self, pkt)
    393     return None
    394 # Now handle TCP reassembly
--> 395 pkt = self._process_packet(pkt)
    396 DefaultSession.on_packet_received(self, pkt)

File scapy/scapy/sessions.py:344, in TCPSession._process_packet(self, pkt)
    341 seq = seq - relative_seq
    342 # Add the data to the buffer
    343 # Note that this take care of retransmission packets.
--> 344 data.append(new_data, seq)
    345 # Check TCP FIN or TCP RESET
    346 if pkt[TCP].flags.F or pkt[TCP].flags.R:

File scapy/scapy/sessions.py:193, in StringBuffer.append(self, data, seq)
    188     assert len(self.content) == self.content_len
    189 # XXX removes empty space marker.
    190 # for ifrag in self.incomplete:
    191 #     if [???]:
    192 #         self.incomplete.remove([???])
--> 193 memoryview(self.content)[seq:seq + data_len] = data

ValueError: memoryview assignment: lvalue and rvalue have different structures

Expected result

sniff shouldn't crash probably.

Related resources

No response

[evverx]

I think I got a pcap where it can be triggered reliably. It's huge though so I'll try to trim it.

load_layer('tls')
sniff(offline=..., session=TCPSession)

throws the exception while sniff(offline=..., session=TLSSession) seems to work. TCPSession also works without tls).

The following packet

... TCP 1514 [TCP Out-Of-Order] 443 → 56477 [ACK] ...

stands out but it's just a guess.

[gpotter2]

Cool, would you mind sharing the pcap?

[evverx]

I somehow keep forgetting attaching pcaps. Sorry!

I reduced that pcap to about 10 packets. I'll attach it as soon as I turn on the laptop where I left it. It should take about an hour.

[evverx]

Here you go: gh-4030.pcap.zip

The issue can be reproduced by running something like

load_layer('tls')
sniff(offline='gh-4030.pcap.zip', session=TCPSession)

Below is the output of tshark just in case

    1   0.000000 88.221.16.141 → 192.168.1.6  TCP 1514 [TCP segment of a reassembled PDU]
    2   0.000005 88.221.16.141 → 192.168.1.6  TCP 1514 443 → 56477 [PSH, ACK] Seq=1449 Ack=1 Win=501 Len=1448 TSval=780777959 TSecr=3692086846 [TCP segment of a reassembled PDU]
    3   0.000007 88.221.16.141 → 192.168.1.6  TCP 1514 443 → 56477 [ACK] Seq=2897 Ack=1 Win=501 Len=1448 TSval=780777959 TSecr=3692086846 [TCP segment of a reassembled PDU]
    4   0.000010 88.221.16.141 → 192.168.1.6  TCP 1514 443 → 56477 [PSH, ACK] Seq=4345 Ack=1 Win=501 Len=1448 TSval=780777959 TSecr=3692086846 [TCP segment of a reassembled PDU]
    5   0.000014 88.221.16.141 → 192.168.1.6  TCP 1514 443 → 56477 [ACK] Seq=5793 Ack=1 Win=501 Len=1448 TSval=780778023 TSecr=3692086846 [TCP segment of a reassembled PDU]
    6   0.000016 88.221.16.141 → 192.168.1.6  TCP 78 [TCP Dup ACK 1#1] 443 → 56477 [ACK] Seq=7241 Ack=1 Win=501 Len=0 TSval=780778108 TSecr=3692087000 SLE=4294966547 SRE=1
    7   0.000018 88.221.16.141 → 192.168.1.6  TCP 1514 [TCP Out-Of-Order] 443 → 56477 [ACK] Seq=4294950884 Ack=1 Win=501 Len=1448 TSval=780778256 TSecr=3692087000

[gpotter2]

So, I think I've got this debugged in #4082, but the pcap isn't super conclusive considering the TLS packet is cropped. Would you by any chance have the rest of the TCP segments so that the packet is full? Thanks

[evverx]

I'll try to get hold of it this week. If it's gone I'll get another session triggering the exception and attach it hopefully.

[evverx]

I extracted all the packets where that IP was involved just in case and put them in gh-4030-full.pcap.zip:
gh-4030-full.pcap.zip

With #4082 applied sniff doesn't throw the exception anymore but it seems it gets stuck until it's interrupted with Ctrl-C explicitly.