[CI] MultiClusterSpecIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #9304 / 8.14.3_bwc
• elasticsearch-periodic #9290 / 8.14.3_bwc
• elasticsearch-periodic #9283 / 8.13.4_bwc
• elasticsearch-periodic #9276 / 8.13.4_bwc
• elasticsearch-periodic #9269 / 8.14.3_bwc
• elasticsearch-periodic #9269 / 8.13.4_bwc
• elasticsearch-pull-request #79913 / 8.14.3_bwc
• elasticsearch-pull-request #79913 / 8.13.4_bwc
• elasticsearch-periodic #9262 / 8.13.4_bwc
• elasticsearch-periodic #9255 / 8.14.3_bwc

Reproduction Line:

undefined

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [8.19] 10 failures in class org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT (1.0% fail rate in 992 executions)
• [8.19] 5 failures in step 8.14.3_bwc (55.6% fail rate in 9 executions)
• [8.19] 5 failures in step 8.13.4_bwc (62.5% fail rate in 8 executions)
• [8.19] 7 failures in pipeline elasticsearch-periodic (87.5% fail rate in 8 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[luigidellaquila]

Looks like a bwc problem.
Labeling as low risk

[alex-spies]

I can see issues with the inference endpoint during setup:

org.elasticsearch.client.ResponseException: method [DELETE], host [http://[::1]:39915], URI [_inference/completion/test_completion], status line [HTTP/1.1 400 Bad Request]	
{"error":{"root_cause":[{"type":"status_exception","reason":"Unknown task_type [completion]"}],"type":"status_exception","reason":"Unknown task_type [completion]"},"status":400}	

org.elasticsearch.client.ResponseException: method [DELETE], host [http://[::1]:37061], URI [_inference/rerank/test_reranker], status line [HTTP/1.1 400 Bad Request]	
{"error":{"root_cause":[{"type":"status_exception","reason":"Unknown task_type [rerank]"}],"type":"status_exception","reason":"Unknown task_type [rerank]"},"status":400}	

java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()" because "endpoints" is null	
at __randomizedtesting.SeedInfo.seed([4B1A80EA5C4F5C22]:0)	
at org.elasticsearch.xpack.esql.CsvTestsDataLoader.deleteInferenceEndpoints(CsvTestsDataLoader.java:353)

[alex-spies]

@afoucret , this appears to be related to #128907. Could you please have a look? The bwc tests seem to be failing consistently when run against older nodes. This is causing noise and affects all of the multi-/mixed-cluster tests against 8.14/8.13 as we fail during setup.

[alex-spies]

I was unable to reproduce the 8.14.3 bwc tests (:x-pack:plugin:esql:qa:server:multi-clusters:v8.14.3#oldToNew --tests "org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT").

Scratch that, I was probably on the wrong commit.

The 8.13.4 bwc tests definitely repro on current 8.19, e.g.

./gradlew ":x-pack:plugin:esql:qa:server:multi-clusters:v8.13.4#newToOld" -Dtests.class="org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT" -Dtests.seed=67572E68722A3BC9 -Dtests.bwc=true -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=24