aws-ecs-patterns (QueueProcessingFargateService): non-editable Scaling Policy causes race conditions & dropped tasks

[AnuragMohapatra]

Describe the bug

Current Scenario

For the scaling policy of Queue processing fargate service, 2 parts are added -

1. Queue length-based scaling - In this scenario, if the user has not provided a step-down counter the system auto-calculates that it does not need a scale-in and does not create a scale-in alarm.
2. Cpu base scaling - In this scenario the scale-in and scale-out are now dependent on the avg CPU utilisation of the system.

This can be found here -

aws-cdk/packages/@aws-cdk/aws-ecs-patterns/lib/base/queue-processing-service-base.ts

Line 344 in fd5808f

protected configureAutoscalingForService(service: BaseService) {

Issue

The CPU base scaling does not seem appropriate in a Queue processing fargate service, the fargate service should only scale out or in depending on the number of messages are there in the queue, not the CPU utilization of the system.

Because of the CPU-based scaling, the auto-scaling group may start a new instance that will process the same message again if there is a CPU-intensive process triggered by the message and is not completed within the scaling alarm trigger.

Also, if the process is memory intensive then the CPU-based scaling will always be in alarm causing the auto-scaling group to remove a task till it reaches the desired capacity.

These scenarios are also relevant for the memory utilization metric but the running task is actually CPU intensive.

Since there is no task-level termination protection, and disable scale-in feature is missing from the patterns this can cause the ASG to terminate a task that is mid-execution.

Expected Behavior

When a Queue processing fargate service has been set up to only scale-out on an approximate number of messages in the queue and the scale-in has been disabled it should not terminate the tasks.

Current Behavior

The ASG on Queue Processing fargate service starts terminating the task if the task is memory intensive and has a long processing time, because of a CW scale in alarm triggered from the CPUUtilizationMetric Scaling policy, thus terminating a random task mid-execution.

Reproduction Steps

Following CDK -

  import { Stack, StackProps } from 'aws-cdk-lib';
  import { Construct } from 'constructs';
  import { QueueProcessingFargateService }  from 'aws-cdk-lib/aws-ecs-patterns'
  import { ContainerImage } from 'aws-cdk-lib/aws-ecs';
  
  export class QueueProcessingFargateServiceAutoscaleTestStack extends Stack {
    constructor(scope: Construct, id: string, props?: StackProps) {
      super(scope, id, props);  
      var containerImage = ContainerImage.fromAsset('test')  
      var service = new QueueProcessingFargateService(this, 'test', {
        image: containerImage,
        scalingSteps : [
          { upper: 0, change: 0 },{ lower: 1, change: +1 }
        ]
     })
    }
  }

will create a new QueueProcessingFargateService with following type of scaling policy -

which causes conflicting alarms to be always trigger -

Possible Solution

The issue is with this method in the Queue processing fargate service pattern base -

aws-cdk/packages/@aws-cdk/aws-ecs-patterns/lib/base/queue-processing-service-base.ts

Line 344 in fd5808f

protected configureAutoscalingForService(service: BaseService) {

It is adding a default CPUUtilizationScalingPolicy that cannot be removed, edited nor disabled.

Solution 1

Remove the CPU Utilization scaling factor if not necessarily required.

Solution 2

Add optional properties and let the user modify the value to disable scale-in on CPU utilization metric or let the user modify the values as per the user's will.

Additional Information/Context

No response

CDK CLI Version

2.27

Framework Version

No response

Node.js Version

16.14.2

OS

Linux

Language

Typescript

Language Version

3.9.7

Other information

No response

[indrora]

It is the responsibility of the queue worker when it pulls the message from the queue to hide it from the rest of the workers (with the maximum being 12 hours) upon pulling it from the message queue.

However, what I'm hearing from this ticket is that a sufficiently complex task (e.g. video de/encode) which pegs the CPU usage or memory usage for a prolonged time will be culled early because of the scaling configuration, is that correct?

[AnuragMohapatra]

Yes, kind of like -

1. a. There are 5 messages in queue and Queue based scaling says to increase if there are more than 2 messages.
   b. Fargate service scales out, and creates a 2 new container one after another.
   c. But none of the container are performing a CPU intensive activity just IO based activity (may be they need to wait for a
   third party doc generate service to complete before they can move ahead with it), so the avg CPU utilisation is low.
   d. So now the CPU base scale in alarm kicks in and removes a container which was processing a message mid way, you have
   lost valuable time and potentially some information that might have been time bound.
   e. You can only hide the messages from consumer not remove the message else queue message scale in will trigger and
   remove the container.

2. a. There is only 3 message in queue and Queue based scaling says to increase if there are more than 4 messages.
   b. Fargate service does nothing since there is no alarm yet.
   c. But the container is performing a CPU intensive activity so the avg CPU utilisation becomes high.
   d. So now the CPU base scale out alarm kicks in and adds a container which starts processing a message.
   e. Then the Queue based scale in kicks in because the messages are below the threshold thus removes a container that is
   processing a message mid way.
   e. Again, now you have lost the processing that was done by this container which may cause issues in downstream
   applications
   f. you can only hide the messages from consumer not remove the message else queue message scale in will trigger
   and remove the container.

An analogy will be SQS - lambda pattern you do not see any memory or CPU based scaling in that cause it have the same effect as above, scaling should be flexible and have a priority. Lambda just times out on high memory and cpu issues.

QueueProcessingFargateService is a solution for SQS-lambda type pattern where we know the task will take more than the 15min timeout of lambda to process and we do not want to maintain our own EC2 instances.

ASGs/ECS also have the same issue but EC2 instances have a feature called Termination Protection where you can disable the scale in of the EC2 instance until the process has completed and no more activity is required.

[danilvalov]

I have the same problem.

In my case I have a service using QueueProcessingFargateService for text parsing. Sometimes there are more than 1000 tasks in the AWS SQS. Each task has low CPU load. And when my SQS count scalling rule adds a new instance, CPU scalling rule stops it in 1 minute. After my SQS count scalling rule adds a new instance again, and CPU scalling rule stops it too.

To solve it I manually deleted the CPU scalling rule in the AWS console, but I think this is not a good solution to do it manually in the console