Make utility methods in `IndexLifecycleTransition` project-aware #128930

nielsbauman · 2025-06-04T20:31:17Z

Modifies the methods to work with a project scope rather than a cluster scope.
This is part of an iterative process to make ILM project-aware.

Modifies the methods to work with a project scope rather than a cluster scope. This is part of an iterative process to make ILM project-aware.

elasticsearchmachine · 2025-06-04T20:31:42Z

Pinging @elastic/es-data-management (Team:Data Management)

PeteGillinElastic

LGTM, but can you have a look over each of the helper method names before you merge this?

PeteGillinElastic · 2025-06-05T10:17:25Z

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/ExecuteStepsUpdateTask.java

@@ -293,7 +297,12 @@ private ClusterState moveToErrorStep(final ClusterState state, Step.StepKey curr
            ),
            cause
        );
-        return IndexLifecycleTransition.moveClusterStateToErrorStep(index, state, cause, nowSupplier, policyStepsRegistry::getStep);
+        final var project = state.metadata().getProject();


It's the tiniest thing in the world, but: Is there a reason you pulled this out into a variable here and not above?

Yeah, I did it this way because I'll be making changes to this class in the future anyway, when I make it project-aware. Then I'll have to change these lines anyway, so I opted to go for the least amount of changes.

PeteGillinElastic · 2025-06-05T10:18:55Z

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleRunner.java

@@ -657,13 +657,14 @@ private final class MoveToRetryFailedStepUpdateTask extends IndexLifecycleCluste

        @Override
        protected ClusterState doExecute(ClusterState currentState) {
-            return IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(
-                currentState,
+            final var updatedProject = IndexLifecycleTransition.moveClusterStateToPreviouslyFailedStep(


And here you've pulled a different bit of the equivalent expression out?

In this class, I'll be passing a ProjectId to the task class, which I'll use to obtain the project from the state. That'll only be a one-line change then. Also, I try to minimize the total number of lines in the end result.

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleService.java

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleTransition.java

commit e6b8a64 Author: Lorenzo Dematte <[email protected]> Date: Tue Jun 10 12:02:16 2025 +0200 PR comments commit ad0902e Author: Moritz Mack <[email protected]> Date: Fri Jun 6 13:33:37 2025 +0200 Update ReproduceInfoPrinter to correctly print a reproduction for lucene / BC upgrade tests. Relates to ES-12005 commit 78b4168 Author: Alexander Spies <[email protected]> Date: Fri Jun 6 11:37:53 2025 +0200 ESQL: Throw ISE instead of IAE for illegal block in page (elastic#128960) IAE gets reported as a 400 status code, but that's inappropriate when inconsistent pages are always bugs, and should be reported with a 500. Throw ISE instead. commit 29e68bd Author: Aurélien FOUCRET <[email protected]> Date: Fri Jun 6 11:24:23 2025 +0200 [ES|QL] Fix test releases for telemetry. (elastic#128971) commit 1a76bc2 Author: Bogdan Pintea <[email protected]> Date: Fri Jun 6 11:01:49 2025 +0200 ESQL: Workaround for RLike handling of empty lang pattern (elastic#128895) Lucene's `org.apache.lucene.util.automaton.Operations#getSingleton` fails with an Automaton for a `REGEXP_EMPTY` `RegExp`. This adds a workaround for that, to check the type of automaton before calling into that failing method. Closes elastic#128813 commit e24fd32 Author: Aurélien FOUCRET <[email protected]> Date: Fri Jun 6 10:25:28 2025 +0200 [ES|QL] Enable the completion command as a tech preview feature (elastic#128948) commit 3f03775 Author: Niels Bauman <[email protected]> Date: Fri Jun 6 09:00:24 2025 +0200 Remove non-test usages of `Metadata.Builder#putCustom` (elastic#128801) This removes all non-test usages of ``` Metadata.Builder.putCustom(String type, ProjectCustom custom) ``` And replaces it with appropriate calls to the equivalent method on `ProjectMetadata.Builder`. In most cases this _does not_ make the code project aware, but does reduce the number of deprecated methods in use. commit 330d127 Author: Niels Bauman <[email protected]> Date: Fri Jun 6 08:07:59 2025 +0200 Make utility methods in `IndexLifecycleTransition` project-aware (elastic#128930) Modifies the methods to work with a project scope rather than a cluster scope. This is part of an iterative process to make ILM project-aware. commit 1b5720d Author: Aurélien FOUCRET <[email protected]> Date: Fri Jun 6 07:50:52 2025 +0200 [ES|QL] Fix test releases for LookupJoinTypesIT. (elastic#128985) commit 40cf2d3 Author: Tim Vernum <[email protected]> Date: Fri Jun 6 13:09:31 2025 +1000 Add "extension" attribute validation to IdP SPs (elastic#128805) This extends the change from elastic#128176 to validate the "custom attributes" on a per Service Provider basis. Each Service Provider (whether registered or wildcard based) has a field "attributes.extensions" which is a list of attribute names that may be provided by the caller of "/_idp/saml/init". Service Providers that have not be configured with extension attributes will reject any custom attributes in SAML init. This necessitates a new field in the service provider index (but only if the new `extensions` attribute is set). The template has been updated, but there is no data migration because the `saml-service-provider` index does not exist in any of the environments into which we wish to deploy this change. commit 496fb2d Author: Jordan Powers <[email protected]> Date: Thu Jun 5 19:50:09 2025 -0700 Skip UTF8 to UTF16 conversion during document indexing (elastic#126492) When parsing documents, we receive the document as UTF-8 encoded data which we then parse and convert the fields to java-native UTF-16 encoded Strings. We then convert these strings back to UTF-8 for storage in lucene. This patch skips the redundant conversion, instead passing lucene a direct reference to the received UTF-8 bytes when possible. commit c34f8b6 Author: Tim Vernum <[email protected]> Date: Fri Jun 6 12:03:25 2025 +1000 Improve cache invalidation in IdP SP cache (elastic#128890) The Identity Provider's Service Provider cache had two issues: 1. It checked for identity based on sequence numbers, but didn't include the `seq_no_primary_term` parameter on searches, which means the sequence would always by `-2` 2. It didn't track whether the index was deleted, which means it could be caching values from an old version of the index This commit fixes both of these issues. In practice neither issue was a problem because there are no deployments that use index-based service providers, however the 2nd issue did cause some challenges for testing. commit 923f029 Author: Nhat Nguyen <[email protected]> Date: Thu Jun 5 18:09:58 2025 -0700 Fix block loader with missing ignored source (elastic#129006) We miss appending null when ignored_source is not available. Our randomized tests already cover this case, but we do not check it when loading fields. I labelled this non-issue for an unreleased bug. Closes elastic#128959 Relates elastic#119546 commit 0f8178a Author: Bogdan Pintea <[email protected]> Date: Fri Jun 6 02:02:24 2025 +0200 ESQL: Forward port 8.19 RegexMatch serialization change version (elastic#128979) Fwd port ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY_8_19. Related: elastic#128919. commit ee716f1 Author: Simon Chase <[email protected]> Date: Thu Jun 5 15:20:08 2025 -0700 transport: edit TransportConnectionListener for close exceptions (elastic#129015) The TransportConnectionListener interface has previously included the Transport.Connection being closed and unregistered in its onNodeDisconnected callback. This is not in use, and can be removed as it is also available in the onConnectionClosed callback. It is being replaced with a Nullable exception that caused the close. This is being used in pending work (ES-11448) to differentiate network issues from node restarts. Closes ES-12007 commit aceaf23 Merge: f18f4ee 159c57f Author: elasticsearchmachine <[email protected]> Date: Thu Jun 5 19:27:54 2025 +0000 Merge patch/serverless-fix into main commit 159c57f Author: Rene Groeschke <[email protected]> Date: Tue Jun 3 11:56:27 2025 +0200 Prepare serverless patch including elastic#128784 elastic#128740 (elastic#128807) * Change default for vector.rescoring.directio to false (elastic#128784) On serverless (and potentially elsewhere), direct IO is not available, which can cause BBQ shards to fail to read with org.apache.lucene.CorruptIndexException when this setting is true. * Optimize sparse vector stats collection (elastic#128740) This change improves the performance of sparse vector statistics gathering by using the document count of terms directly, rather than relying on the field name field to compute stats. By avoiding per-term disk/network reads and instead leveraging statistics already loaded into leaf readers at index opening, we expect to significantly reduce overhead. Relates to elastic#128583 --------- Co-authored-by: Dave Pifke <[email protected]> Co-authored-by: Jim Ferenczi <[email protected]>

…stic#128930) Modifies the methods to work with a project scope rather than a cluster scope. This is part of an iterative process to make ILM project-aware.

Copilot

Pull Request Overview

This PR refactors ILM utility methods to be project-aware rather than cluster-aware as part of making ILM project‐aware. The changes include renaming methods, updating assertions and error messages in tests, and replacing ClusterState manipulations with ProjectMetadata updates.

Updated method names and signatures in both production and test code to reflect project-awareness.
Refactored utility methods and assertions to operate on ProjectMetadata instead of ClusterState.
Adjusted service and action layer implementations to build updated ClusterState using the new project metadata.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
x-pack/plugin/ilm/src/test/java/org/elasticsearch/xpack/ilm/IndexLifecycleTransitionTests.java	Updated test method names and assertions to use project-based methods.
x-pack/plugin/ilm/src/test/java/org/elasticsearch/xpack/ilm/IndexLifecycleRunnerTests.java	Removed duplicate cluster state assertions now handled via project metadata.
x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleTransition.java	Updated ILM transition methods to work on ProjectMetadata and updated helper assertions.
x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleService.java	Updated service methods to return ProjectMetadata and build ClusterState from project metadata.
Other files (e.g. TransportRetryAction, TransportRemoveIndexLifecyclePolicyAction, SetStepInfoUpdateTask, MoveToErrorStepUpdateTask, ExecuteStepsUpdateTask, IndexLifecycleRunner.java)	Minor refactorings to integrate the project-aware changes consistently.

Comments suppressed due to low confidence (2)

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleTransition.java:453

[nitpick] Consider renaming the variable 'newProject' to something more descriptive (e.g., 'updatedProjectBuilder') to improve clarity in this conditional return.

        return newProject == null ? currentProject : newProject.build();

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/IndexLifecycleService.java:176

[nitpick] Consider renaming the iterative variable 'newProject' (for example, to 'updatedProject') within the loop to better indicate that it accumulates updates.

            newProject = IndexLifecycleTransition.moveIndexToPreviouslyFailedStep(newProject, index, nowSupplier, policyRegistry, false);

The change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

A change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

A change in #128930 introduced a typo which could result in erroneous cluster state updates.

A change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

A change in #128930 introduced a typo which could result in erroneous cluster state updates.

A change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

Make utility methods in IndexLifecycleTransition project-aware

77f3667

Modifies the methods to work with a project scope rather than a cluster scope. This is part of an iterative process to make ILM project-aware.

nielsbauman requested a review from PeteGillinElastic June 4, 2025 20:31

nielsbauman added >non-issue :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team v9.1.0 labels Jun 4, 2025

PeteGillinElastic approved these changes Jun 5, 2025

View reviewed changes

nielsbauman added 2 commits June 5, 2025 16:56

Merge branch 'main' into ilm-transition-mp

9bdf7a1

Rename methods

18e962d

nielsbauman enabled auto-merge (squash) June 5, 2025 15:05

Merge branch 'main' into ilm-transition-mp

db3832f

nielsbauman merged commit 330d127 into elastic:main Jun 6, 2025
17 of 18 checks passed

nielsbauman deleted the ilm-transition-mp branch June 6, 2025 06:08

nielsbauman requested a review from Copilot June 27, 2025 12:06

Copilot AI reviewed Jun 27, 2025

View reviewed changes

nielsbauman added a commit to nielsbauman/elasticsearch that referenced this pull request Jun 27, 2025

Fix typo in ILM retry action

b30d15d

The change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

nielsbauman added a commit to nielsbauman/elasticsearch that referenced this pull request Jun 27, 2025

Fix typo in ILM retry action

009dbb4

A change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

This was referenced Jun 27, 2025

Fix typo in ILM retry action #130255

Merged

[CI] UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} failing #129097

Closed

nielsbauman added a commit that referenced this pull request Jun 30, 2025

Fix typo in ILM retry action (#130255)

5b9b033

A change in #128930 introduced a typo which could result in erroneous cluster state updates.

nielsbauman added a commit to nielsbauman/elasticsearch that referenced this pull request Jun 30, 2025

Fix typo in ILM retry action (elastic#130255)

1940d7f

A change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

elasticsearchmachine pushed a commit that referenced this pull request Jun 30, 2025

Fix typo in ILM retry action (#130255) (#130326)

40e5303

A change in #128930 introduced a typo which could result in erroneous cluster state updates.

mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jul 3, 2025

Fix typo in ILM retry action (elastic#130255)

859a3b7

A change in elastic#128930 introduced a typo which could result in erroneous cluster state updates.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make utility methods in `IndexLifecycleTransition` project-aware #128930

Make utility methods in `IndexLifecycleTransition` project-aware #128930

Uh oh!

nielsbauman commented Jun 4, 2025

Uh oh!

elasticsearchmachine commented Jun 4, 2025

Uh oh!

PeteGillinElastic left a comment

Uh oh!

PeteGillinElastic Jun 5, 2025

Uh oh!

nielsbauman Jun 5, 2025

Uh oh!

PeteGillinElastic Jun 5, 2025

Uh oh!

nielsbauman Jun 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Make utility methods in IndexLifecycleTransition project-aware #128930

Make utility methods in IndexLifecycleTransition project-aware #128930

Uh oh!

Conversation

nielsbauman commented Jun 4, 2025

Uh oh!

elasticsearchmachine commented Jun 4, 2025

Uh oh!

PeteGillinElastic left a comment

Choose a reason for hiding this comment

Uh oh!

PeteGillinElastic Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

PeteGillinElastic Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

nielsbauman Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Make utility methods in `IndexLifecycleTransition` project-aware #128930

Make utility methods in `IndexLifecycleTransition` project-aware #128930