Hazelcast startup issues caused by default clustering configuration for AWS ECS

[jcgarciaa]

Description

As part of its member discovery strategy, Hazelcast requires a "tag" to filter the ECS tasks that are candidates to join the cluster. However, the IS documentation [1] does not recommend adding tagKey and tagValue properties to the clustering configuration (as it does for the EC2 membership scheme).

On the other hand, the IS is actually setting a default tagKey and tagValue properties for the ECS if none are defined [2]:

tagKey = "aws:cloudformation:stack-name"
tagValue = "EC2ContainerService-$ref{clustering.properties.clusterName}"

The above approach works only for ECS tasks created before January 12, 2023 when a CloudFormation stack was created automatically for the task definition upon creation [3].

Solution

• Remove the default tagKey and tagValue properties from the ECS membership scheme.
• Update the documentation to include tagKey and tagValue properties as part of the recommended configuration.

Steps to Reproduce

1. Configure a cluster on AWS with awsvpc network mode.
2. Configure an IS cluster with ECS membership scheme as instructed at [1].
3. Enable debug traces for hazelcast components in the log4j2.properties file:

logger.com-hazelcast.level = DEBUG

4. Start the IS instances.
5. Notice from the logs that the DescribeTasks response returns an empty array of IP addresses, hence both servers started in standalone mode:

DEBUG {com.hazelcast.aws.AwsEcsApi} - AWS ECS ListTasks found the following tasks: [<task-1>, <task-2>]
DEBUG {com.hazelcast.aws.AwsClient} - AWS ECS DescribeTasks found the following addresses: []
WARN {com.hazelcast.aws.AwsDiscoveryStrategy} - No IP addresses found! Starting standalone.

Version

WSO2IS 5.10.0 and above

References

[1] https://is.docs.wso2.com/en/5.10.0/setup/deployment-guide/#clustering-related-configurations:~:text=Click%20to%20see%20the%20instructions%20for%20AWS%20ECS%20membership%20scheme
[2] https://github.com/wso2/carbon-kernel/blob/v4.6.0/distribution/kernel/carbon-home/repository/resources/conf/infer.json#L307-L313
[3] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html#cloudformation-stack

[inthirakumaaran]

Doc is updated for IS 6.0.0[1] and above[2]

[1] https://is.docs.wso2.com/en/6.0.0/deploy/deployment-guide/#!
[2] https://is.docs.wso2.com/en/latest/deploy/deployment-guide/#clustering-related-configurations

[inthirakumaaran]

Docs are update for IS 5.11.0[1] and IS 5.10.0[2]

[1] https://is.docs.wso2.com/en/5.11.0/setup/deployment-guide/#!:~:text=Click%20to%20see%20the%20instructions%20for%20AWS%20ECS%20membership%20scheme
[2] https://is.docs.wso2.com/en/5.10.0/setup/deployment-guide/#!:~:text=Click%20to%20see%20the%20instructions%20for%20AWS%20ECS%20membership%20scheme