Dafda currently supports two modes:
Manual commit: Dafda will commit offset manually after processing each message. This is slow, but guarantees message delivery and decreases risk of message re-delivery
Auto commit: Offset is committed automatically by Kafka. This enables high throughput but at the risk of "message loss" if a handler fails to process a message. This is fast and great for scenarios where message loss is not a problem, such as processing sensor data.
A third option would be to do periodic manual commit where offset is committed at a certain interval, if no errors has occured. This enables high throughput but there is a risk of many messages being re-delivered, requiring handlers to be idempotent. For an implementation reference, have a look at how the Kafka "rider" for MassTransit is implemented.
Dafda currently supports two modes:
Manual commit: Dafda will commit offset manually after processing each message. This is slow, but guarantees message delivery and decreases risk of message re-delivery
Auto commit: Offset is committed automatically by Kafka. This enables high throughput but at the risk of "message loss" if a handler fails to process a message. This is fast and great for scenarios where message loss is not a problem, such as processing sensor data.
A third option would be to do periodic manual commit where offset is committed at a certain interval, if no errors has occured. This enables high throughput but there is a risk of many messages being re-delivered, requiring handlers to be idempotent. For an implementation reference, have a look at how the Kafka "rider" for MassTransit is implemented.