Recover from a checkpoint if training failed with an exception (#3265)

Recover from a checkpoint if training failed with an exception (#3265) #7692