Fuzz targets seem to have started failing under MSan on CIFuzz

[evverx]

It should probably reach OSS-Fuzz a bit later.

From systemd/systemd#32609 (comment)

/github/workspace/build-out/fuzz-journald-native -timeout=25 -rss_limit_mb=2560 -len_control=0 -seed=1337 -artifact_prefix=/tmp/tmprl9fex94/ -max_total_time=25 -print_final_stats=1 /github/workspace/cifuzz-corpus/fuzz-journald-native >fuzz-0.log 2>&1
================== Job 1 exited with exit code 0 ============
Uninitialized bytes in fputs at offset 22 inside [0x71e000000000, 2774)
==37==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x564d5b17a1cf in Puts /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:155:3
    #1 0x564d5b17a1cf in fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:68:3
    #2 0x564d5b16fbaf in fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:249:5
    #3 0x564d5b16ff31 in __invoke<void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__type_traits/invoke.h:340:25
    #4 0x564d5b16ff31 in __thread_execute<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct> >, void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *, 2UL, 3UL, 4UL, 5UL> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:221:5
    #5 0x564d5b16ff31 in void* std::__Fuzzer::__thread_proxy[abi:v180000]<std::__Fuzzer::tuple<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct>>, void (*)(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*>>(void*) /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:232:5
    #6 0x7ff843509608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: c6d0d79d906d62bb768421fc6dada0d5e729f177)
    #7 0x7ff8425de352 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f352) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)

DEDUP_TOKEN: Puts--fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&)--fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was stored to memory at
    #0 0x564d5b0a2dd2 in __msan_memmove /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1752:3
    #1 0x564d5b1acffd in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char) cxa_noexception.cpp

DEDUP_TOKEN: __msan_memmove--std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>>::push_back(char)
  Uninitialized value was created by a heap allocation
    #0 0x564d5b0abd72 in malloc /src/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1007:3
    #1 0x564d5b19c6e6 in operator new(unsigned long) cxa_noexception.cpp
    #2 0x564d5b17a1b3 in fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:68:8
    #3 0x564d5b16fbaf in fuzzer::WorkerThread(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:249:5
    #4 0x564d5b16ff31 in __invoke<void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__type_traits/invoke.h:340:25
    #5 0x564d5b16ff31 in __thread_execute<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct> >, void (*)(const fuzzer::Command &, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int> *, unsigned int, std::__Fuzzer::atomic<bool> *, 2UL, 3UL, 4UL, 5UL> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:221:5
    #6 0x564d5b16ff31 in void* std::__Fuzzer::__thread_proxy[abi:v180000]<std::__Fuzzer::tuple<std::__Fuzzer::unique_ptr<std::__Fuzzer::__thread_struct, std::__Fuzzer::default_delete<std::__Fuzzer::__thread_struct>>, void (*)(fuzzer::Command const&, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*), std::__Fuzzer::reference_wrapper<fuzzer::Command>, std::__Fuzzer::atomic<unsigned int>*, unsigned int, std::__Fuzzer::atomic<bool>*>>(void*) /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/__thread/thread.h:232:5
    #7 0x7ff843509608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: c6d0d79d906d62bb768421fc6dada0d5e729f177)

DEDUP_TOKEN: __interceptor_malloc--operator new(unsigned long)--fuzzer::CopyFileToErr(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char>> const&)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerIO.cpp:155:3 in Puts
Exiting

[maflcko]

This should be fixed by #11887

[evverx]

Looking at #11714 it seems it was a bumpy ride :-) It was prudent to turn off the "function" sanitizer there too: systemd/systemd#29972, avahi/avahi#584

[maflcko]

Yes, the bump was more involved, because it skipped over several clang releases. Be aware that the function sanitizer may in the future be enabled. See #11778 for the tracking issue for that.

[evverx]

To judge from https://github.com/systemd/systemd/actions/runs/8972741048/job/24641404945 the fuzz targets keep failing on CIFuzz. Could it be that the base images haven't been updated?

[jonathanmetzman]

Are we sure this is a CIFuzz issue? I think CIFuzz says this happens with the oss-fuzz builds too. So it seems to be an issue with oss-fuzz. Do you agree?

[evverx]

I'm not sure. I haven't seen those backtraces there. They don't seem to have been reported on Monorail either. Let me double-check.

[evverx]

Looks like there are no those backtraces on OSS-Fuzz. I can't reproduce it locally with helper.py with the latest images either so it seems CIFuzz is the only place where they keep popping up for some reason.