Skip to content

KAFKA-19011: Improve EndToEndLatency Tool with argument parser and message key/header support #20301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: trunk
Choose a base branch
from

Conversation

Rancho-7
Copy link
Collaborator

@Rancho-7 Rancho-7 commented Aug 4, 2025

jira: https://issues.apache.org/jira/browse/KAFKA-19011 kip:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1172%3A+Improve+EndToEndLatency+tool

This PR improves the usability and maintainability of the
kafka-e2e-latency.sh tool:

  • Replaces fixed-index argument parsing with a proper argument parser
    (joptsimple)
  • Adds support for configuring:
    • -record-key-size: size of the message key
    • -num-headers: number of headers per message
    • -record-header-key-size: size of each header key
    • -record-header-size: size of each header value
  • Renames existing arguments to align with Kafka CLI conventions:
    • broker_list → bootstrap-server
    • num_messages → num-records
    • message_size_bytes → record-size
    • properties_file → command-config

@github-actions github-actions bot added triage PRs from the community tools labels Aug 4, 2025
@Rancho-7
Copy link
Collaborator Author

Rancho-7 commented Aug 4, 2025

The updated usage of the tool with argument parser:

(base) lansg:bin/ (KAFKA-19011✗) $ ./kafka-e2e-latency.sh  --bootstrap-server localhost:9092 --topic test-topic --num-records 1000 --producer-acks 1 --record-size 512 --record-key-size 64 --num-headers 5 --record-header-key-size 64 --record-header-size 128
0       26.187541
Avg latency: 1.2853 ms
Percentiles: 50th = 1, 99th = 3, 99.9th = 26

It's compatible with the legacy version:

(base) lansg:bin/ (KAFKA-19011✗) $ ./kafka-e2e-latency.sh localhost:9092 test-topic 1000 1 512                                                       [21:21:28]
WARNING: Positional argument usage is deprecated and will be removed in Apache Kafka 5.0. Please use named arguments instead: --bootstrap-server, --topic, --num-records, --producer-acks, --record-size, --command-config
0       45.907666
Avg latency: 1.5202 ms
Percentiles: 50th = 1, 99th = 6, 99.9th = 88

Copy link
Collaborator

@Yunyung Yunyung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Partial review, some comments left. Also, could you add the KIP to the title or description?


// validate 'producer-acks'
String acksValue = options.valueOf(acksOpt);
if (!List.of("1", "all").contains(acksValue)) {
Copy link
Collaborator

@Yunyung Yunyung Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if -1?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The legacy version only had these two options, so I keep it the same. Since -1 has the same effect as all, I think one is enough?

@@ -221,4 +303,156 @@ private static KafkaProducer<byte[], byte[]> createKafkaProducer(Optional<String
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer");
return new KafkaProducer<>(producerProps);
}

// Visible for testing
static String[] convertLegacyArgsIfNeeded(String[] args) throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add Deprecated tag?

Copy link
Collaborator Author

@Rancho-7 Rancho-7 Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out! Added it.

@github-actions github-actions bot removed the triage PRs from the community label Aug 5, 2025
@chia7712
Copy link
Member

chia7712 commented Aug 7, 2025

@Rancho-7 could you please run e2e on you local?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants