Implement SAML custom attributes support for Identity Provider (#128176)

* Implement SAML custom attributes support for Identity Provider

This commit adds support for custom attributes in SAML single sign-on requests
in the Elasticsearch X-Pack Identity Provider plugin. This feature allows
passage of custom key-value attributes in SAML requests and responses.

Key components:
- Added SamlInitiateSingleSignOnAttributes class for holding attributes
- Added validation for null and empty attribute keys
- Updated request and response objects to handle attributes
- Modified authentication flow to process attributes
- Added test coverage to validate attributes functionality

The implementation follows Elasticsearch patterns with robust validation
and serialization mechanisms, while maintaining backward compatibility.

* Add test for SAML custom attributes in authentication response

This commit adds a comprehensive test that verifies SAML custom attributes
are correctly handled in the authentication response builder. The test ensures:

1. Custom attributes with single and multiple values are properly included
2. The response with custom attributes is still correctly signed
3. The XML schema validation still passes with custom attributes
4. We can locate and verify individual attribute values in the response

This provides critical test coverage for the SAML custom attributes
feature implementation.

* Add backward compatibility overload for SuccessfulAuthenticationResponseMessageBuilder.build

This commit adds an overloaded build method that accepts only two parameters
(user and authenticationState) and forwards the call to the three-parameter
version with null for the customAttributes parameter. This maintains backward
compatibility with existing code that doesn't use custom attributes.

This fixes a compilation error in ServerlessSsoIT.java which was still using
the two-parameter method signature.

Signed-off-by: lloydmeta <[email protected]>

* Add validation for duplicate SAML attribute keys

This commit enhances the SAML attributes implementation by adding validation
for duplicate attribute keys. When the same attribute key appears multiple
times in a request, the validation will now fail with a clear error message.

Signed-off-by: lloydmeta <[email protected]>

* Refactor SAML attributes validation to follow standard patterns

This commit improves the SAML attributes validation by:

1. Adding a dedicated validate() method to SamlInitiateSingleSignOnAttributes
   that centralizes validation logic in one place
2. Moving validation from constructor to dedicated method for better error reporting
3. Checking both for null/empty keys and duplicate keys in the validate() method
4. Updating SamlInitiateSingleSignOnRequest to use the new validation method
5. Adding comprehensive tests for the new validation approach

These changes follow standard Elasticsearch validation patterns, making the
code more maintainable and consistent with the rest of the codebase.

* Update docs/changelog/128176.yaml

* Improve SAML response validation in identity provider tests

Enhanced the testCustomAttributesInIdpInitiatedSso test to properly validate
both SAML response structure and custom attributes using DOM parsing and XPath.

Key improvements:
- Validate SAML Response/Assertion elements exist
- Precisely validate custom attributes (department, region) and their values
- Use namespace-aware XML parsing for resilience to format changes

Signed-off-by: lloydmeta <[email protected]>

* Simplify SAML attributes representation using JSON object/Map structure

Also, replace internal Attribute class list with a simpler Map<String, List<String>>
structure

This change:

- Removes the redundant Attribute class and replaces it with a direct Map
  implementation for storing attribute key-value pairs
- Eliminates the duplicate "attributes" nesting in the JSON structure
- Simplifies attribute validation without needing duplicate key checking

- Updates all related tests and integration points to work with the new structure

Before:

```js
{
  // others
  "attributes": {
    "attributes": [
      {
        "key": "department",
        "values": ["engineering", "product"]
      }
    ]
  }
}

After:

```js
{
  // other
  "attributes": {
    "department": ["engineering", "product"]
  }
}
```

(Verified by spitting out JSON entity in IdentityProviderAuthenticationIT.generateSamlResponseWithAttributes
... saw `{"entity_id":"ec:123456:abcdefg","acs":"https://sp1.test.es.elasticsearch.org/saml/acs","attributes":{"department":["engineering","product"],"region":["APJ"]}}`)

Signed-off-by: lloydmeta <[email protected]>

* * Fix up toString dangling quote.

Signed-off-by: lloydmeta <[email protected]>

* * Remove attributes from Response object.

Signed-off-by: lloydmeta <[email protected]>

* * Remove friendly name.
* Make attributes map final in SamlInitiateSingleSignOnAttributes

Signed-off-by: lloydmeta <[email protected]>

* * Cleanup serdes by using existing utils in the ES codebase

Signed-off-by: lloydmeta <[email protected]>

* Touchup comment

Signed-off-by: lloydmeta <[email protected]>

* Update x-pack/plugin/identity-provider/src/test/java/org/elasticsearch/xpack/idp/action/SamlInitiateSingleSignOnRequestTests.java

Co-authored-by: Tim Vernum <[email protected]>

* Add transport-version checks

---------

Signed-off-by: lloydmeta <[email protected]>
Co-authored-by: Tim Vernum <[email protected]>

Author: lloydmeta

docs/changelog/128176.yaml

@@ -0,0 +1,5 @@
+pr: 128176
+summary: Implement SAML custom attributes support for Identity Provider
+area: Authentication
+type: enhancement
+issues: []

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -273,6 +273,7 @@ static TransportVersion def(int id) {
     public static final TransportVersion INFERENCE_CUSTOM_SERVICE_ADDED = def(9_084_0_00);
     public static final TransportVersion ESQL_LIMIT_ROW_SIZE = def(9_085_0_00);
     public static final TransportVersion ESQL_REGEX_MATCH_WITH_CASE_INSENSITIVITY = def(9_086_0_00);
+    public static final TransportVersion IDP_CUSTOM_SAML_ATTRIBUTES = def(9_087_0_00);
 
     /*
      * STOP! READ THIS FIRST! No, really,

x-pack/plugin/identity-provider/qa/idp-rest-tests/src/javaRestTest/java/org/elasticsearch/xpack/idp/IdentityProviderAuthenticationIT.java

@@ -18,23 +18,38 @@
 import org.elasticsearch.common.util.concurrent.ThreadContext;
 import org.elasticsearch.core.Nullable;
 import org.elasticsearch.xcontent.ObjectPath;
+import org.elasticsearch.xcontent.XContentBuilder;
 import org.elasticsearch.xcontent.json.JsonXContent;
 import org.elasticsearch.xpack.core.security.action.saml.SamlPrepareAuthenticationResponse;
 import org.elasticsearch.xpack.idp.saml.sp.SamlServiceProviderIndex;
 import org.junit.Before;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+import org.xml.sax.InputSource;
 
 import java.io.IOException;
+import java.io.StringReader;
 import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
 import java.util.Base64;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.xpath.XPath;
+import javax.xml.xpath.XPathConstants;
+import javax.xml.xpath.XPathFactory;
+
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.hasSize;
 import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.notNullValue;
 
 public class IdentityProviderAuthenticationIT extends IdpRestTestCase {
 
@@ -74,6 +89,81 @@ public void testRegistrationAndIdpInitiatedSso() throws Exception {
         authenticateWithSamlResponse(samlResponse, null);
     }
 
+    public void testCustomAttributesInIdpInitiatedSso() throws Exception {
+        final Map<String, Object> request = Map.ofEntries(
+            Map.entry("name", "Test SP With Custom Attributes"),
+            Map.entry("acs", SP_ACS),
+            Map.entry("privileges", Map.ofEntries(Map.entry("resource", SP_ENTITY_ID), Map.entry("roles", List.of("sso:(\\w+)")))),
+            Map.entry(
+                "attributes",
+                Map.ofEntries(
+                    Map.entry("principal", "https://idp.test.es.elasticsearch.org/attribute/principal"),
+                    Map.entry("name", "https://idp.test.es.elasticsearch.org/attribute/name"),
+                    Map.entry("email", "https://idp.test.es.elasticsearch.org/attribute/email"),
+                    Map.entry("roles", "https://idp.test.es.elasticsearch.org/attribute/roles")
+                )
+            )
+        );
+        final SamlServiceProviderIndex.DocumentVersion docVersion = createServiceProvider(SP_ENTITY_ID, request);
+        checkIndexDoc(docVersion);
+        ensureGreen(SamlServiceProviderIndex.INDEX_NAME);
+
+        // Create custom attributes
+        Map<String, List<String>> attributesMap = Map.of("department", List.of("engineering", "product"), "region", List.of("APJ"));
+
+        // Generate SAML response with custom attributes
+        final String samlResponse = generateSamlResponseWithAttributes(SP_ENTITY_ID, SP_ACS, null, attributesMap);
+
+        // Parse XML directly from samlResponse (it's not base64 encoded at this point)
+        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
+        factory.setNamespaceAware(true); // Required for XPath
+        DocumentBuilder builder = factory.newDocumentBuilder();
+        Document document = builder.parse(new InputSource(new StringReader(samlResponse)));
+
+        // Create XPath evaluator
+        XPathFactory xPathFactory = XPathFactory.newInstance();
+        XPath xpath = xPathFactory.newXPath();
+
+        // Validate SAML Response structure
+        Element responseElement = (Element) xpath.evaluate("//*[local-name()='Response']", document, XPathConstants.NODE);
+        assertThat("SAML Response element should exist", responseElement, notNullValue());
+
+        Element assertionElement = (Element) xpath.evaluate("//*[local-name()='Assertion']", document, XPathConstants.NODE);
+        assertThat("SAML Assertion element should exist", assertionElement, notNullValue());
+
+        // Validate department attribute
+        NodeList departmentAttributes = (NodeList) xpath.evaluate(
+            "//*[local-name()='Attribute' and @Name='department']/*[local-name()='AttributeValue']",
+            document,
+            XPathConstants.NODESET
+        );
+
+        assertThat("Should have two values for department attribute", departmentAttributes.getLength(), is(2));
+
+        // Verify department values
+        List<String> departmentValues = new ArrayList<>();
+        for (int i = 0; i < departmentAttributes.getLength(); i++) {
+            departmentValues.add(departmentAttributes.item(i).getTextContent());
+        }
+        assertThat(
+            "Department attribute should contain 'engineering' and 'product'",
+            departmentValues,
+            containsInAnyOrder("engineering", "product")
+        );
+
+        // Validate region attribute
+        NodeList regionAttributes = (NodeList) xpath.evaluate(
+            "//*[local-name()='Attribute' and @Name='region']/*[local-name()='AttributeValue']",
+            document,
+            XPathConstants.NODESET
+        );
+
+        assertThat("Should have one value for region attribute", regionAttributes.getLength(), is(1));
+        assertThat("Region attribute should contain 'APJ'", regionAttributes.item(0).getTextContent(), equalTo("APJ"));
+
+        authenticateWithSamlResponse(samlResponse, null);
+    }
+
     public void testRegistrationAndSpInitiatedSso() throws Exception {
         final Map<String, Object> request = Map.ofEntries(
             Map.entry("name", "Test SP"),
@@ -125,17 +215,37 @@ private SamlPrepareAuthenticationResponse generateSamlAuthnRequest(String realmN
         }
     }
 
-    private String generateSamlResponse(String entityId, String acs, @Nullable Map<String, Object> authnState) throws Exception {
+    private String generateSamlResponse(String entityId, String acs, @Nullable Map<String, Object> authnState) throws IOException {
+        return generateSamlResponseWithAttributes(entityId, acs, authnState, null);
+    }
+
+    private String generateSamlResponseWithAttributes(
+        String entityId,
+        String acs,
+        @Nullable Map<String, Object> authnState,
+        @Nullable Map<String, List<String>> attributes
+    ) throws IOException {
         final Request request = new Request("POST", "/_idp/saml/init");
-        if (authnState != null && authnState.isEmpty() == false) {
-            request.setJsonEntity(Strings.format("""
-                {"entity_id":"%s", "acs":"%s","authn_state":%s}
-                """, entityId, acs, Strings.toString(JsonXContent.contentBuilder().map(authnState))));
-        } else {
-            request.setJsonEntity(Strings.format("""
-                {"entity_id":"%s", "acs":"%s"}
-                """, entityId, acs));
+
+        XContentBuilder builder = JsonXContent.contentBuilder();
+        builder.startObject();
+        builder.field("entity_id", entityId);
+        builder.field("acs", acs);
+
+        if (authnState != null) {
+            builder.field("authn_state");
+            builder.map(authnState);
+        }
+
+        if (attributes != null) {
+            builder.field("attributes");
+            builder.map(attributes);
         }
+
+        builder.endObject();
+        String jsonEntity = Strings.toString(builder);
+
+        request.setJsonEntity(jsonEntity);
         request.setOptions(
             RequestOptions.DEFAULT.toBuilder()
                 .addHeader("es-secondary-authorization", basicAuthHeaderValue("idp_user", new SecureString("idp-password".toCharArray())))

x-pack/plugin/identity-provider/src/main/java/org/elasticsearch/xpack/idp/action/SamlInitiateSingleSignOnRequest.java

@@ -6,12 +6,14 @@
  */
 package org.elasticsearch.xpack.idp.action;
 
+import org.elasticsearch.TransportVersions;
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.LegacyActionRequest;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.io.stream.StreamInput;
 import org.elasticsearch.common.io.stream.StreamOutput;
 import org.elasticsearch.xpack.idp.saml.support.SamlAuthenticationState;
+import org.elasticsearch.xpack.idp.saml.support.SamlInitiateSingleSignOnAttributes;
 
 import java.io.IOException;
 
@@ -22,12 +24,16 @@ public class SamlInitiateSingleSignOnRequest extends LegacyActionRequest {
     private String spEntityId;
     private String assertionConsumerService;
     private SamlAuthenticationState samlAuthenticationState;
+    private SamlInitiateSingleSignOnAttributes attributes;
 
     public SamlInitiateSingleSignOnRequest(StreamInput in) throws IOException {
         super(in);
         spEntityId = in.readString();
         assertionConsumerService = in.readString();
         samlAuthenticationState = in.readOptionalWriteable(SamlAuthenticationState::new);
+        if (in.getTransportVersion().onOrAfter(TransportVersions.IDP_CUSTOM_SAML_ATTRIBUTES)) {
+            attributes = in.readOptionalWriteable(SamlInitiateSingleSignOnAttributes::new);
+        }
     }
 
     public SamlInitiateSingleSignOnRequest() {}
@@ -41,6 +47,17 @@ public ActionRequestValidationException validate() {
         if (Strings.isNullOrEmpty(assertionConsumerService)) {
             validationException = addValidationError("acs is missing", validationException);
         }
+
+        // Validate attributes if present
+        if (attributes != null) {
+            ActionRequestValidationException attributesValidationException = attributes.validate();
+            if (attributesValidationException != null) {
+                for (String error : attributesValidationException.validationErrors()) {
+                    validationException = addValidationError(error, validationException);
+                }
+            }
+        }
+
         return validationException;
     }
 
@@ -68,17 +85,38 @@ public void setSamlAuthenticationState(SamlAuthenticationState samlAuthenticatio
         this.samlAuthenticationState = samlAuthenticationState;
     }
 
+    public SamlInitiateSingleSignOnAttributes getAttributes() {
+        return attributes;
+    }
+
+    public void setAttributes(SamlInitiateSingleSignOnAttributes attributes) {
+        this.attributes = attributes;
+    }
+
     @Override
     public void writeTo(StreamOutput out) throws IOException {
         super.writeTo(out);
         out.writeString(spEntityId);
         out.writeString(assertionConsumerService);
         out.writeOptionalWriteable(samlAuthenticationState);
+        if (out.getTransportVersion().onOrAfter(TransportVersions.IDP_CUSTOM_SAML_ATTRIBUTES)) {
+            out.writeOptionalWriteable(attributes);
+        }
     }
 
     @Override
     public String toString() {
-        return getClass().getSimpleName() + "{spEntityId='" + spEntityId + "', acs='" + assertionConsumerService + "'}";
+        return getClass().getSimpleName()
+            + "{"
+            + "spEntityId='"
+            + spEntityId
+            + "', "
+            + "acs='"
+            + assertionConsumerService
+            + "', "
+            + "attributes='"
+            + attributes
+            + "'}";
     }
 
 }

x-pack/plugin/identity-provider/src/main/java/org/elasticsearch/xpack/idp/action/TransportSamlInitiateSingleSignOnAction.java

@@ -139,7 +139,7 @@ protected void doExecute(
                         identityProvider
                     );
                     try {
-                        final Response response = builder.build(user, authenticationState);
+                        final Response response = builder.build(user, authenticationState, request.getAttributes());
                         listener.onResponse(
                             new SamlInitiateSingleSignOnResponse(
                                 user.getServiceProvider().getEntityId(),