Skip to content

Commit 77077f8

Browse files
Nanda Kishore Salemfacebook-github-bot
authored andcommitted
Increase RSS memory watchdog limit (crashing)
Summary: We are crashing due to memory watchdog with spr. Increased RSS memory to 800MB. In general BGP attributes, fibers etc will need more memory. This is the starting number, will increase or decrease depending on canary observations and future crashes. ``` Jun 11 21:14:15 rsw1lx.08.ftw1.facebook.com openr[3676396]: I0611 21:14:15.756893 3676786 Fib.cpp:398] Processing route add/update for 0 unicast, 0 mpls, and route delete for 0-unicast, 0-mpls, Jun 11 21:14:15 rsw1lx.08.ftw1.facebook.com openr[3676396]: I0611 21:14:15.756944 3676786 Fib.cpp:693] OpenR convergence performance. Duration=115 Jun 11 21:14:15 rsw1lx.08.ftw1.facebook.com openr[3676396]: I0611 21:14:15.757004 3676786 Fib.cpp:483] Done processing route add/update Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: W0611 21:14:16.675659 3676567 Watchdog.cpp:64] Memory usage critical:308940800 bytes, Memory limit:300 MB Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: W0611 21:14:16.676319 3676567 Watchdog.cpp:130] Memory limit exceeded the permitted limit. Mem used:308940800. Mem Limit:300 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: openr[3676520]: Memory limit exceeded the permitted limit. Mem used:308940800. Mem Limit:300 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: *** Aborted at 1560312856 (Unix time, try 'date -d 1560312856') *** Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: *** Signal 6 (SIGABRT) (0x381968) received by PID 3676520 (pthread TID 0x7f605777c700) (linux TID 3676567) (maybe from PID 3676520, UID 0) (code: -6), stack trace: *** Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676520]: Memory limit exceeded the permitted limit. Mem used:308940800. Mem Limit:300 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00000000010c3ca1 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*) Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./folly/experimental/symbolizer/SignalHandler.cpp:431 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a7ccb0f (unknown) Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a00014f gsignal Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f6059fe9934 abort Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00000000006acfd0 openr::Watchdog::monitorMemory() Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./openr/watchdog/Watchdog.cpp:133 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00000000006ada58 void folly::detail::function::FunctionTraits<void () noexcept>::callSmall<openr::Watchdog::Watchdog(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1l> >, unsigned int)::$_0>(folly::detail::function::Data&) Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./openr/watchdog/Watchdog.cpp:31 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000f32e92 void folly::detail::function::FunctionTraits<void () noexcept>::callSmall<fbzmq::ZmqTimeout::timeoutExpiredHelper()::$_1>(folly::detail::function::Data&) Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./fbzmq/async/ZmqTimeout.cpp:84 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000f2de77 fbzmq::ZmqEventLoop::loopForever() Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./folly/Function.h:451 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: -> ./fbzmq/async/ZmqEventLoop.cpp Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000f2da09 fbzmq::ZmqEventLoop::run() Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./fbzmq/async/ZmqEventLoop.cpp:92 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000465e19 std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_2> > >::_M_run() Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./openr/Main.cpp:270 Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605b2cf67f execute_native_thread_routine Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a7c16b5 start_thread Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a0daebe clone ``` Reviewed By: jstrizich Differential Revision: D15777716 fbshipit-source-id: e3d9a3561ef17b46ec1144d36286e14207d9cf1e
1 parent 06fa2d5 commit 77077f8

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

openr/scripts/run_openr.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ LINK_FLAP_MAX_BACKOFF_MS=60000
8686
LOGGING=""
8787
LINK_MONITOR_CMD_PORT=60006
8888
LOOPBACK_IFACE="lo"
89-
MEMORY_LIMIT_MB=300
89+
MEMORY_LIMIT_MB=800
9090
MIN_LOG_LEVEL=0
9191
OVERRIDE_LOOPBACK_ADDR=false
9292
PREFIX_MANAGER_CMD_PORT=60011

0 commit comments

Comments
 (0)