Optimize event subscribe exception handling

[lurais]

Rationale

Why should this feature exist?
I notice that the codes of the event subscribe function are quite close to the applyBlock process codes in the pushBlock function, and any exception thrown by the event subscribe function will lead to normal blocks being removed from KhaosDB, which has been committed successfully.

In this case, the next block will be processed failed when putting into the KhaosDB since no parent block could be found in KhaosDB.

The detail code is here:

try (ISession tmpSession = revokingStore.buildSession()) {
    long oldSolidNum =
            chainBaseManager.getDynamicPropertiesStore().getLatestSolidifiedBlockNum();
    applyBlock(newBlock, txs);
    tmpSession.commit();
    // if event subscribe is enabled, post block trigger to queue
    postBlockTrigger(newBlock);
    // if event subscribe is enabled, post solidity trigger to queue
    postSolidityTrigger(oldSolidNum,
            getDynamicPropertiesStore().getLatestSolidifiedBlockNum());
  } catch (Throwable throwable) {
    logger.error(throwable.getMessage(), throwable);
    khaosDb.removeBlk(block.getBlockId());
    throw throwable;
  }

The block synchronization should work as we expected when any exception is thrown in the event subscribe function, in detail, it should work well if the event subscribe function is allowed to miss some data.

Implementation

Do you have ideas regarding the implementation of this feature?

Solution: The event subscribe logic is the subsequent processing logic of block processing, and the event subscribe logic is independent from the existing block processing logic. Therefore, exceptions related to the event subscribe logic should be handled separately. In addition, due to some users' high dependence on the stability of event subscribe functions, while others require synchronization services to remain stable and allow occasional exceptions to occur in the event subscribe logic, a new switch is added to control whether synchronization functions should be suspended when event subscribe services are abnormal.

The new added switch is located in config file path: event.subscribe.ignoreSubscribeError, and the node will stop when any exception is thrown in event subscribe services if the config is not set or set to false; and the node will ignore the exceptions thrown by the event subscribe services and continue to synchronize blocks only if it has set to true .

The related code will be like this:

    try (ISession tmpSession = revokingStore.buildSession()) {
      applyBlock(newBlock, txs);
      tmpSession.commit();
    } catch (Throwable throwable) {
      logger.error(throwable.getMessage(), throwable);
      khaosDb.removeBlk(block.getBlockId());
      throw throwable;
    }
    try {
      long oldSolidNum =
              chainBaseManager.getDynamicPropertiesStore().getLatestSolidifiedBlockNum();
      // if event subscribe is enabled, post block trigger to queue
      postBlockTrigger(newBlock);
      // if event subscribe is enabled, post solidity trigger to queue
      postSolidityTrigger(oldSolidNum,
              getDynamicPropertiesStore().getLatestSolidifiedBlockNum());
    } catch (Exception e) {
      logger.error("Post trigger error.", e);
      // as disscussed below, this switch will not be added 
      if (!ignoreSubscribeErrorSwitchOn()) {
        System.exit(1);
      }
    }

As discussed below, the config ignoreSubscribeError is unnecessary and will not be added. The next question is, if the event subscription function throws an exception, does it need to be retried? Or just stop the node if any exception is found in the event subscription:

     try (ISession tmpSession = revokingStore.buildSession()) {

        long oldSolidNum =
                chainBaseManager.getDynamicPropertiesStore().getLatestSolidifiedBlockNum();

        applyBlock(newBlock, txs);
        // if event subscribe is enabled, post block trigger to queue
        postBlockTrigger(newBlock);
        // if event subscribe is enabled, post solidity trigger to queue
        postSolidityTrigger(oldSolidNum,
                getDynamicPropertiesStore().getLatestSolidifiedBlockNum());
        tmpSession.commit();
      } catch (Throwable throwable) {
        logger.error(throwable.getMessage(), throwable);
        khaosDb.removeBlk(block.getBlockId());
        throw throwable;
      }

As discussed in the comments, the node will stop if an exception is found in the event subscription.

Are you willing to implement this feature?
Yes.

[halibobo1205]

why need ignoreSubscribeErrorSwitchOn ?

[lurais]

while others require synchronization services to remain stable and allow occasional exceptions to occur in the event subscribe logic

To offer users a choice, synchronization services can remain stable while allowing occasional exceptions to occur during subscribe logic.

[lurais]

@jwrct @317787106 @ss3344520 Do you have any ideas about this issue?

[xxo1shine]

@lurais From your description, there are more than two exception handling strategies. It is recommended to use enumeration types for configuration parameters instead of boolean types.

[halibobo1205]

I don't recommend adding more configurations for user purposes.

[lurais]

@lurais From your description, there are more than two exception handling strategies. It is recommended to use enumeration types for configuration parameters instead of boolean types.

@ss3344520 In my opinion, is there any situation in which it needs to use another strategy besides stopping or continuing to synchronize blocks?

[317787106]

@lurais is System.exit(1) has any potential problems, such as resource release ?

[lurais]

@lurais is System.exit(1) has any potential problems, such as resource release ?

@317787106 When you use the command "System.exit", it will take effect as same as "kill -15" actually.

[lurais]

@halibobo1205 Why do you not recommend adding more configurations for user purposes? Do you have any more detail reasons?
@317787106 @ss3344520 @jwrct The issue that needs to be discussed now is whether to add more configurations to continue to synchronize blocks for user purposes when there is a problem with the event subscription service. Do you have any ideas about this?

[jwrct]

Based on your plan and default configuration, if an exception is thrown causing the system to exit when publishing an event, how should the node user handle it to recover? Will there be any data loss issue after the node restarts? Will there be any impact on the event subscribers?