Skip to content

HBASE-22041.md

Latest commit

 

History

History
38 lines (31 loc) · 1.36 KB

HBASE-22041.md

File metadata and controls

38 lines (31 loc) · 1.36 KB
Raw

HBASE-22041

Scenario

Figure1:HBASE-22041
  1. HMaster is starting ,RS who holds the meta table reports in
  2. HMaster adds the RS to onlineServers.
  3. RS crash but it has't register to ZK yet.
  4. In normal case, ZK can detect the RS crash.
  5. And remove the RS from onlineServers.
  6. HMaster will connect the meta table, but fails, then it checks RS whether exists in onlineServers, yes, hence HMaster will retry (forever).

Trigger analysis

  1. The crash time is very important to trigger this bug.
    1. if the node crash before report, then the onlineServers will not have RS
    2. if RS crash after register, then the recovery will remove it from onlineServers
  2. The origin developer have given the TODO fixing:
1 try {
2   sendRequest(getServerName(), request.build());
3 } catch (IOException e) {
4   e = unwrapException(e);
5   // TODO: In the future some operation may want to bail out early.
6   // TODO: How many times should we retry (use numberOfAttemptsSoFar)
7   if (!scheduleForRetry(e)) {
8     remoteCallFailed(procedureEnv, e);
9   }
10 }

Crash point

post-write

org.apache.hadoop.hbase.regionserver.HRegionServer handleReportForDutyResponse 1516