Greengrass IPC pubsub seg fault

[lanceds15]

Hi, I have built and installed the sdk and am trying to test greengrass ipc pubsub. I have used the example application here: https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-iot-core-mqtt.html When deployed, the component seg faults at ipcClient.Connect(ipcLifecycleHandler).get()

[lanceds15]

Valgrind output:

Running sample.ipciot-cpp component
==25663== Memcheck, a memory error detector
==25663== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==25663== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==25663== Command: /var/lib/greengrass/packages/artifacts/sample.ipciot-cpp/1.0.5//build/greengrassv2_pubsub_subscriber
==25663==
Connecting to Greengrass Core IPC...
==25663== Thread 2 AwsEventLoop 1:
==25663== Invalid read of size 2
==25663== at 0x48900C8: __GI_memcpy (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x9877C7: aws_byte_buf_write (byte_buf.c:1476)
==25663== by 0x9878BF: aws_byte_buf_write_from_whole_buffer (byte_buf.c:1493)
==25663== by 0x5D7C6B: aws_event_stream_message_init (event_stream.c:370)
==25663== by 0x5DE1AF: s_send_protocol_message (event_stream_rpc_client.c:606)
==25663== by 0x5DE3B3: aws_event_stream_rpc_client_connection_send_protocol_message (event_stream_rpc_client.c:651)
==25663== by 0x5B9153: Aws::Eventstreamrpc::ClientConnection::s_sendProtocolMessage(Aws::Eventstreamrpc::ClientConnection*, std::__cxx11::list<Aws::Eventstreamrpc::EventStreamHeader, Aws::Crt::StlAllocatorAws::Eventstreamrpc::EventStreamHeader > const&, Aws::Crt::Optional<aws_byte_buf> const&, aws_event_stream_rpc_message_type, unsigned int, std::function<void (int)>) (EventStreamClient.cpp:535)
==25663== by 0x5B9D57: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:730)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== Address 0x4e300d0 is 0 bytes inside a block of size 32 free'd
==25663== at 0x48882A8: free (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F32B: s_non_aligned_free (allocator.c:118)
==25663== by 0x97FACB: aws_mem_release (allocator.c:264)
==25663== by 0x982527: aws_byte_buf_clean_up (byte_buf.c:79)
==25663== by 0x5CEF3F: Aws::Crt::ByteBufDelete(aws_byte_buf&) (Types.cpp:45)
==25663== by 0x5B7A7F: Aws::Eventstreamrpc::MessageAmendment::~MessageAmendment() (EventStreamClient.cpp:163)
==25663== by 0x5B9D07: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:727)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663== Block was alloc'd at
==25663== at 0x4885250: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F2CF: s_non_aligned_malloc (allocator.c:111)
==25663== by 0x97F513: aws_mem_acquire (allocator.c:177)
==25663== by 0x9822A7: aws_byte_buf_init_copy (byte_buf.c:48)
==25663== by 0x5CEEC7: Aws::Crt::ByteBufNewCopy(aws_allocator*, unsigned char const*, unsigned long) (Types.cpp:32)
==25663== by 0x5B740F: Aws::Eventstreamrpc::MessageAmendment::MessageAmendment(Aws::Eventstreamrpc::MessageAmendment const&) (EventStreamClient.cpp:76)
==25663== by 0x5B9C3B: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:719)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663==
==25663== Invalid read of size 2
==25663== at 0x48900D8: __GI_memcpy (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x9877C7: aws_byte_buf_write (byte_buf.c:1476)
==25663== by 0x9878BF: aws_byte_buf_write_from_whole_buffer (byte_buf.c:1493)
==25663== by 0x5D7C6B: aws_event_stream_message_init (event_stream.c:370)
==25663== by 0x5DE1AF: s_send_protocol_message (event_stream_rpc_client.c:606)
==25663== by 0x5DE3B3: aws_event_stream_rpc_client_connection_send_protocol_message (event_stream_rpc_client.c:651)
==25663== by 0x5B9153: Aws::Eventstreamrpc::ClientConnection::s_sendProtocolMessage(Aws::Eventstreamrpc::ClientConnection*, std::__cxx11::list<Aws::Eventstreamrpc::EventStreamHeader, Aws::Crt::StlAllocatorAws::Eventstreamrpc::EventStreamHeader > const&, Aws::Crt::Optional<aws_byte_buf> const&, aws_event_stream_rpc_message_type, unsigned int, std::function<void (int)>) (EventStreamClient.cpp:535)
==25663== by 0x5B9D57: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:730)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== Address 0x4e300d4 is 4 bytes inside a block of size 32 free'd
==25663== at 0x48882A8: free (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F32B: s_non_aligned_free (allocator.c:118)
==25663== by 0x97FACB: aws_mem_release (allocator.c:264)
==25663== by 0x982527: aws_byte_buf_clean_up (byte_buf.c:79)
==25663== by 0x5CEF3F: Aws::Crt::ByteBufDelete(aws_byte_buf&) (Types.cpp:45)
==25663== by 0x5B7A7F: Aws::Eventstreamrpc::MessageAmendment::~MessageAmendment() (EventStreamClient.cpp:163)
==25663== by 0x5B9D07: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:727)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663== Block was alloc'd at
==25663== at 0x4885250: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F2CF: s_non_aligned_malloc (allocator.c:111)
==25663== by 0x97F513: aws_mem_acquire (allocator.c:177)
==25663== by 0x9822A7: aws_byte_buf_init_copy (byte_buf.c:48)
==25663== by 0x5CEEC7: Aws::Crt::ByteBufNewCopy(aws_allocator*, unsigned char const*, unsigned long) (Types.cpp:32)
==25663== by 0x5B740F: Aws::Eventstreamrpc::MessageAmendment::MessageAmendment(Aws::Eventstreamrpc::MessageAmendment const&) (EventStreamClient.cpp:76)
==25663== by 0x5B9C3B: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:719)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663==
==25663== Invalid free() / delete / delete[] / realloc()
==25663== at 0x48882A8: free (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F32B: s_non_aligned_free (allocator.c:118)
==25663== by 0x97FACB: aws_mem_release (allocator.c:264)
==25663== by 0x982527: aws_byte_buf_clean_up (byte_buf.c:79)
==25663== by 0x5CEF3F: Aws::Crt::ByteBufDelete(aws_byte_buf&) (Types.cpp:45)
==25663== by 0x5B7A7F: Aws::Eventstreamrpc::MessageAmendment::~MessageAmendment() (EventStreamClient.cpp:163)
==25663== by 0x5B9D6F: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:737)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663== Address 0x4e300d0 is 0 bytes inside a block of size 32 free'd
==25663== at 0x48882A8: free (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F32B: s_non_aligned_free (allocator.c:118)
==25663== by 0x97FACB: aws_mem_release (allocator.c:264)
==25663== by 0x982527: aws_byte_buf_clean_up (byte_buf.c:79)
==25663== by 0x5CEF3F: Aws::Crt::ByteBufDelete(aws_byte_buf&) (Types.cpp:45)
==25663== by 0x5B7A7F: Aws::Eventstreamrpc::MessageAmendment::~MessageAmendment() (EventStreamClient.cpp:163)
==25663== by 0x5B9D07: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:727)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663== Block was alloc'd at
==25663== at 0x4885250: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-arm64-linux.so)
==25663== by 0x97F2CF: s_non_aligned_malloc (allocator.c:111)
==25663== by 0x97F513: aws_mem_acquire (allocator.c:177)
==25663== by 0x9822A7: aws_byte_buf_init_copy (byte_buf.c:48)
==25663== by 0x5CEEC7: Aws::Crt::ByteBufNewCopy(aws_allocator*, unsigned char const*, unsigned long) (Types.cpp:32)
==25663== by 0x5B740F: Aws::Eventstreamrpc::MessageAmendment::MessageAmendment(Aws::Eventstreamrpc::MessageAmendment const&) (EventStreamClient.cpp:76)
==25663== by 0x5B9C3B: Aws::Eventstreamrpc::ClientConnection::s_onConnectionSetup(aws_event_stream_rpc_client_connection*, int, void*) (EventStreamClient.cpp:719)
==25663== by 0x5DCBEF: s_on_channel_setup_fn (event_stream_rpc_client.c:154)
==25663== by 0x644DFF: s_connect_args_setup_callback_safe (channel_bootstrap.c:211)
==25663== by 0x644FEB: s_connection_args_setup_callback (channel_bootstrap.c:281)
==25663== by 0x64579F: s_on_client_channel_on_setup_completed (channel_bootstrap.c:510)
==25663== by 0x6411CB: s_on_channel_setup_complete (channel.c:171)
==25663==
OnConnectCallback
Successfully published to topic: test/topic/cpp

[lanceds15]

Seems like a double free issue, modified the MessageAmendment deconstructor in EventStreamClient.cpp. Just replaced the ByetBufDelete line with m_payload->buffer = nullptr and rebuilt and the issue appears to be resolved.

[bretambrose]

FWIW, we are still in the middle of a large rewrite of the event stream bindings and greengrass ipc client that should address this and other issues.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.