-
Notifications
You must be signed in to change notification settings - Fork 245
Commit 77077f8
Increase RSS memory watchdog limit (crashing)
Summary:
We are crashing due to memory watchdog with spr.
Increased RSS memory to 800MB. In general BGP attributes, fibers etc will need more memory.
This is the starting number, will increase or decrease depending on canary observations and future crashes.
```
Jun 11 21:14:15 rsw1lx.08.ftw1.facebook.com openr[3676396]: I0611 21:14:15.756893 3676786 Fib.cpp:398] Processing route add/update for 0 unicast, 0 mpls, and route delete for 0-unicast, 0-mpls,
Jun 11 21:14:15 rsw1lx.08.ftw1.facebook.com openr[3676396]: I0611 21:14:15.756944 3676786 Fib.cpp:693] OpenR convergence performance. Duration=115
Jun 11 21:14:15 rsw1lx.08.ftw1.facebook.com openr[3676396]: I0611 21:14:15.757004 3676786 Fib.cpp:483] Done processing route add/update
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: W0611 21:14:16.675659 3676567 Watchdog.cpp:64] Memory usage critical:308940800 bytes, Memory limit:300 MB
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: W0611 21:14:16.676319 3676567 Watchdog.cpp:130] Memory limit exceeded the permitted limit. Mem used:308940800. Mem Limit:300
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: openr[3676520]: Memory limit exceeded the permitted limit. Mem used:308940800. Mem Limit:300
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: *** Aborted at 1560312856 (Unix time, try 'date -d 1560312856') ***
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: *** Signal 6 (SIGABRT) (0x381968) received by PID 3676520 (pthread TID 0x7f605777c700) (linux TID 3676567) (maybe from PID 3676520, UID 0) (code: -6), stack trace: ***
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676520]: Memory limit exceeded the permitted limit. Mem used:308940800. Mem Limit:300
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00000000010c3ca1 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./folly/experimental/symbolizer/SignalHandler.cpp:431
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a7ccb0f (unknown)
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a00014f gsignal
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f6059fe9934 abort
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00000000006acfd0 openr::Watchdog::monitorMemory()
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./openr/watchdog/Watchdog.cpp:133
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00000000006ada58 void folly::detail::function::FunctionTraits<void () noexcept>::callSmall<openr::Watchdog::Watchdog(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1l> >, unsigned int)::$_0>(folly::detail::function::Data&)
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./openr/watchdog/Watchdog.cpp:31
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000f32e92 void folly::detail::function::FunctionTraits<void () noexcept>::callSmall<fbzmq::ZmqTimeout::timeoutExpiredHelper()::$_1>(folly::detail::function::Data&)
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./fbzmq/async/ZmqTimeout.cpp:84
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000f2de77 fbzmq::ZmqEventLoop::loopForever()
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./folly/Function.h:451
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: -> ./fbzmq/async/ZmqEventLoop.cpp
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000f2da09 fbzmq::ZmqEventLoop::run()
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./fbzmq/async/ZmqEventLoop.cpp:92
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 0000000000465e19 std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_2> > >::_M_run()
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: ./openr/Main.cpp:270
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605b2cf67f execute_native_thread_routine
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a7c16b5 start_thread
Jun 11 21:14:16 rsw1lx.08.ftw1.facebook.com openr[3676396]: @ 00007f605a0daebe clone
```
Reviewed By: jstrizich
Differential Revision: D15777716
fbshipit-source-id: e3d9a3561ef17b46ec1144d36286e14207d9cf1e1 parent 06fa2d5 commit 77077f8Copy full SHA for 77077f8
File tree
Expand file treeCollapse file tree
1 file changed
+1
-1
lines changedFilter options
- openr/scripts
Expand file treeCollapse file tree
1 file changed
+1
-1
lines changed+1-1Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
86 | 86 |
| |
87 | 87 |
| |
88 | 88 |
| |
89 |
| - | |
| 89 | + | |
90 | 90 |
| |
91 | 91 |
| |
92 | 92 |
| |
|
0 commit comments