Feature/openj9 integration #20197
Open
Feature/openj9 integration #20197
+35,147
−57,075
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pache#18497) Kafka 4.0 will remove support for zk mode and will require conversion to kraft before upgrading to 4.0. The minimum kraft version is 3.0 (aka 3.0-IV1). This provides an opportunity to remove exclusively server side protocols versions that only exist to allow direct upgrades from versions older than 3.0 or that are used only by zk mode. Since KRaft became production ready in 3.3, we should consider setting the baseline to 3.3. But that requires more discussion and it can be done via a separate change (KAFKA-18601). Protocol changes: * Remove RequestHeader v0 (only used by ControlledShutdown v0) * Remove WriteTxnMarkers v0 * Remove all versions of ControlledShutdown, LeaderAndIsr, StopReplica, UpdateMetadata In order to remove all versions safely, extend generator to support setting "versions" to "none". In this case, we no longer generate the `*Data` classes, but we still reserve the id for the relevant protocol api (so it doesn't get accidentally used for something else). The protocol documentation is correct after these changes. We kept a simplified version of `LeaderAndIsr{Request|Response}` because it's used by many tests that are still relevant in kraft mode. Once KAFKA-18486 is done, it may be possible to remove it (I left a comment on the ticket). Similarly, KAFKA-18487 may make it possible to remove the introduced `StopReplicaPartitionState` (left a comment on that ticket too). There are a number of places that were adjusted to include an `ApiKeys.hasValidVersion` check. Reviewers: Chia-Ping Tsai <[email protected]>
Although `MetadataCache`'s `getPartitionReplicaEndpoints` takes a single topic-partition, the `KRaftMetadataCache` implementation iterates over all partitions of the matching topic. This is not necessary and can cause significant performance degradation when the topic has a relatively high number of partitions. Note that this is not a recent regression - it has been a part of `KRaftMetadataCache` since its creation. Reviewers: Ismael Juma <[email protected]>, David Jacot <[email protected]>
This patch removes `ReplicaManager#stopReplicas`. I have ensured that removed unit tests are covered by other existing tests or are updated to use kraft. Reviewers: Ismael Juma <[email protected]>, Chia-Ping Tsai <[email protected]>
Part of KIP-1091. Reviewers: Bill Bejeck <[email protected]>
…apache#18587) We should provide the same informative error message for both timeout cases. Reviewers: Kirk True <[email protected]>, Andrew Schofield <[email protected]>, Ismael Juma <[email protected]>
…che#18607) Reviewers: Bruno Cadonna <[email protected]>, Lucas Brutschy <[email protected]>
…tor (apache#18661) Guard against the coordinator getting stuck due to deferred events throwing exceptions. Reviewers: David Jacot <[email protected]>
…8631) Reviewers: Christo Lolov <[email protected]>
…che#18629) Reviewers: Christo Lolov <[email protected]>
Reviewers: Mickael Maison <[email protected]>, Ismael Juma <[email protected]>, José Armando García Sancio <[email protected]>
Reviewers: Mickael Maison <[email protected]>, Ismael Juma <[email protected]>, Chia-Ping Tsai <[email protected]>
…pache#18532) Reviewers: Andrew Schofield <[email protected]>, Kirk True <[email protected]>
…dition with completing and new transaction(apache#18604) There is a subtle race condition with transactions V2 if a transaction is still completing when checking if we need to add a partition, but it completes when the request reaches the coordinator. One approach was to remove the verification for TV2 and just check the epoch on write, but a simpler one is to simply return concurrent transactions from the partition leader (before attempting to add the partition). I've done this and added a test for this behavior. Locally, I reproduced the race but adding a 1 second sleep when handling the WriteTxnMarkersRequest and a 2 second delay before adding the partition to the AddPartitionsToTxnManager. Without this change, the race happened on every second transaction as the first one completed. With this change, the error went away. As a followup, we may want to clean up some of the code and comments with respect to verification as the code is used by both TV0 + verification and TV2. But that doesn't need to complete for 4.0. This does :) Reviewers: Jeff Kim <[email protected]>, Artem Livshits <[email protected]>, Calvin Liu <[email protected]>
Reviewers: Christo Lolov <[email protected]>, Chia-Ping Tsai <[email protected]>
This patch is a first step towards removing `ReplicaManager#becomeLeaderOrFollower`. It updates the `LocalLeaderEndPointTest` tests. Reviewers: Christo Lolov <[email protected]>, Ismael Juma <[email protected]>
…e#18667) Ensure that unloading a coordinator always succeeds. Previously, we have guarded against exceptions from DeferredEvent completions. All that remains is handling exceptions from the onUnloaded() method of the coordinator state machine. Reviewers: David Jacot <[email protected]>
…sionManager (apache#18630) `forwardingManager` is always present now. Reviewers: Ismael Juma <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Chia-Ping Tsai <[email protected]>, PoAn Yang <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Christo Lolov <[email protected]>, Chia-Ping Tsai <[email protected]>
Reviewers: Christo Lolov <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Chia-Ping Tsai <[email protected]>
Reviewers: Chia-Ping Tsai <[email protected]>
This patch includes some maintenance updates for Develocity. * Publish build scans to develocity.apache.org * Update Develocity Gradle plugin to to 3.19 * Use `DEVELOCITY_ACCESS_KEY` to authenticate to `develocity.apache.org` Reviewers: Chia-Ping Tsai <[email protected]>, David Arthur <[email protected]>
…ct Values (apache#17510) Reviewers: Greg Harris <[email protected]>, Mickael Maison <[email protected]>
This patch reorganizes our test infrastructure into three Gradle modules: ":test-common:test-common-internal-api" is now a minimal dependency which exposes interfaces and annotations only. It has one project dependency on server-common to expose commonly used data classes (MetadataVersion, Feature, etc). Since this pulls in server-common, this module is Java 17+. It cannot be used by ":clients" or other Java 11 modules. ":test-common:test-common-util" includes the auto-quarantined JUnit extension. The @flaky annotation has been moved here. Since this module has no project dependencies, we can add it to the Java 11 list so that ":clients" and others can utilize the @flaky annotation ":test-common:test-common-runtime" now includes all of the test infrastructure code (TestKitNodes, etc). This module carries heavy dependencies (core, etc) and so it should not normally be included as a compile-time dependency. In addition to this reorganization, this patch leverages JUnit SPI service discovery so that modules can utilize the integration test framework without depending on ":core". This will allow us to start moving integration tests out of core and into the appropriate sub-module. This is done by adding ":test-common:test-common-runtime" as a testRuntimeOnly dependency rather than as a testImplementation dependency. A trivial example was added to QuorumControllerTest to illustrate this. Reviewers: Ismael Juma <[email protected]>, Chia-Ping Tsai <[email protected]>
…sapiAuthenticationTest should run for async consumer (apache#18555) Reviewers: Kirk True <[email protected]>, Lianet Magrans <[email protected]>
During testing, we identified that kafka-python (and aiokafka) relies on metadata request v0 and hence we need to add these back to comply with the premise of KIP-896 - i.e. it should not break the clients listed within it. I reverted the changes from apache#18218 related to the removal of metadata versions 0-3. I will submit a separate PR to undeprecate these API versions on the relevant 3.x branches. kafka-python (and aiokafka) work correctly (produce & consume) with this change on top of the 4.0 branch. Reviewers: David Arthur <[email protected]>
…adata flag (apache#18668) Reviewers: Lianet Magrans <[email protected]>
…#18726) Ensure we always return empty records (including cases where an error is returned). We also remove `nullable` from `records` since it is effectively expected to be non-null by a large percentage of clients in the wild. This behavior regressed in fe56fc9 (KAFKA-18269). Empty records were previously set via `FetchResponse.recordsOrFail(partitionData)` in the now-removed `maybeConvertFetchedData` method. Added an integration test that fails without this fix and also update many tests to set `records` to `empty` instead of leaving them as `null`. Reviewers: Chia-Ping Tsai <[email protected]>, David Arthur <[email protected]>
…apache#18733) While testing, it was found that the not_enough_replicas error was super common and could be easily confused. Since we are already bumping the request, we can signify that the produce request may return this error and new clients can handle it (Note, the java client should be able to handle this already as a retriable error, but other client libraries may need to implement this change) Reviewers: Justine Olshan <[email protected]>
… ScalaCompile and JavaCompile blocks in build.gradle
…ault args for OpenJ9 in Jenkinsfile
…bproject sections
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
…ks in subprojects
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
…adle restored to near original version with replacement of JVM arguments for IBM JDK
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this project is to re-platform Apache Kafka 4.0.0 to use IBM Semeru OpenJ9