HBASE-22050.md

HBASE-22050

Scenario

1. HMaster is opening RS, shutdown comes, hence open fails and RS start to abort.

2. 2428.if (rssStub != null && this.serverName != null) {
   2429   ReportRSFatalErrorRequest.Builder builder =
   2430.  ReportRSFatalErrorRequest.newBuilder();
   2431.  builder.setServer(ProtobufUtil.toServerName(this.serverName));
   2432   builder.setErrorMessage(msg);
   2433   rssStub.reportRSFatalError(null, builder.build());
   2434 }

3. 2428-2434 are assumed to be atomic, but if it step in the 2429-2433, meanwhile RS#run:

4. 1149 // Make sure the proxy is down.
   1150 if (this.rssStub != null) {
   1151    this.rssStub = null;
   1152 }

5. So the rssStub == null and NPE happens

Trigger analysis

1. Actually, this is a atomic violation bugs, CrashTuner can trigger it has three reasons:
2. First, CrashTuner Identify the RegionInfo as meta-info, and RS open the region based on the RegionInfo, so it can inject shutdown before open region and cause abort.
3. Second, CrashTuner inject sleep after shutdown, so it can make the above code snippet execute concurrently.
4. Third, We are so lucky to make the code snippet 2 comes in the middle of code snippet1.