Skip to content

YARN-9201.md

Latest commit

 

History

History
47 lines (35 loc) · 1.61 KB

YARN-9201.md

File metadata and controls

47 lines (35 loc) · 1.61 KB
Raw

YARN-9201

Scenario

YARN-9201
  1. Launcher is launching the AppAttempt, (line#2 below code)
  try {
       launch();
       handler.handle(new RMAppAttemptEvent(application.getAppAttemptId(),
              RMAppAttemptEventType.LAUNCHED, System.currentTimeMillis()));
       } catch(Exception ie) {
         onAMLaunchFailed(masterContainer.getId(), ie);
       }
  }
  1. Shutdown command is sent to NM who holds the container, so launch get exception.
  2. Recovery process will send KILL event to AppAttempt, and the state change to FAILED
  3. Then Launcher will send LAUNCH_FAILED event to RMContainer( in code#6)(PS, onAMLaunchFailed is written by me,:joy:)
  4. but it can't handle this event at KILLED state.

Trigger analysis

private void launch() throws IOException, YarnException {
    connect();
    ContainerId masterContainerID = masterContainer.getId();
}
  1. Crash timing is very important to trigger this bug.
    1. As above code show, connect() will check whether the node is live, if not , exception will be thrown and bug won't be triggered.
    2. line 3 is read operation, so CrashTuner will inject shutdown command between line#2 and line#3.
    3. Hence the bug is triggered.
  2. The sleep after shutdown is very important to flip the KILL event and LAUNCH_FAILED event.

Crash point

pre-read

org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher launch 132