Allow deleting a backing index via modify data streams API

[jbaiera]

Adds a new operation type to the modify data stream API that deletes a backing index from a data stream. PR includes a new node feature to ensure master node supports the operation before accepting it, as well as the requisite wire changes. Added new unit tests and a new rest test case. Deleting an index requires a ProjectState instance instead of a ProjectMetadata instance, so there's some light cluster state juggling in here to make that work too.

A quick thought: Instead of calling to MetadataIndexDeleteService directly for deleting the backing index (which would remove it from the data stream if safe to do so as well) this PR ensures that the index being deleted does actually belong to the data stream before deleting it. Not sure how helpful this invariant actually is, especially since in most cases that this feature would be useful, we may have more than one index to delete along with the backing index as part of a cleanup process. Indices not actively part of the data stream would still need to be deleted separately (see the DLM frozen transition maybeCleanup method which potentially deletes the force merge index if it is different from the original index).

Remains to do: Updating the rest spec with the new operation type

robots.txt

I asked claude to do the unit test callsite refactorings in 271a528

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Hi @jbaiera, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel