Scaling Kubernetes pods with KEDA based on Amazon SQS queue depth

In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in an Amazon SQS queue. In other cases, pods may continue running long after a traffic spike has passed.

For asynchronous, queue-based workloads, backlog is the true scaling signal – not infrastructure utilization.

In this article, you’ll learn:

How Amazon SQS queue depth can work as an autoscaling metric for event-driven workers

How to scale workloads on Amazon EKS using KEDA with AWS pod identity

How KEDA calculates desired replicas from queue depth

How to tune queueLength, activationQueueLength, cooldowns, and HPA behavior

How to validate scaling behavior and troubleshoot common issues

Why this matters:

In queue-driven systems, delayed message processing directly impacts users and downstream systems. Scaling based on SQS depth aligns autoscaling with actual demand, enabling faster burst handling and lower idle costs when queues are empty.

1. Prerequisites

Before implementing KEDA-based autoscaling with Amazon SQS, ensure you have:

An Amazon EKS cluster running a supported Kubernetes version

KEDA installed in the cluster (for example, via the kedacore Helm chart)

An AWS authentication method selected (IRSA or EKS Pod Identity)

An existing Amazon SQS queue

A worker deployment designed to consume messages from the queue

2. Architecture and Request Flow

This architecture relies on KEDA observing Amazon SQS queue metrics and managing a Kubernetes Horizontal Pod Autoscaler (HPA) based on backlog.

Request flow:

Producers send messages to the Amazon SQS queue

KEDA polls queue attributes to determine backlog

KEDA updates the target metrics in the HPA

The HPA scales the worker Deployment

Kubernetes schedules additional Pods to process messages

As the queue drains, replicas scale back down

This model keeps autoscaling decisions tied directly to outstanding work.

3. Implementation Steps

Install KEDA via Helm

The recommended way to install KEDA is via Helm:

# Add the KEDA Helm repository helm repo add kedacore https://kedacore.github.io/charts helm repo update # Install KEDA into a dedicated namespace helm install keda kedacore/keda \ --namespace keda \ --create-namespace

What this does:

Deploys the KEDA operator and metrics API server, along with CRDs such as ScaledObject and TriggerAuthentication.

Deploy the Worker with AWS Identity Attached

An SQS consumer typically requires permissions such as:

GetQueueAttributes

GetQueueUrl

ReceiveMessage

DeleteMessage

ChangeMessageVisibility

Example IAM policy:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "ConsumeFromQueue", "Effect": "Allow", "Action": [ "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:ChangeMessageVisibility" ], "Resource": "arn:aws:sqs:us-east-1:123456789012:orders-queue" } ] }

What this does:

Enforces least-privilege access by granting permissions only for the required SQS queue.

Configure KEDA Authentication and ScaledObject

KEDA connects the deployment to the SQS queue using TriggerAuthentication and ScaledObject. Using queueURLFromEnv avoids hardcoding the queue URL.

apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: sqs-processor-auth namespace: workers spec: podIdentity: provider: aws --- apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: sqs-processor namespace: workers spec: scaleTargetRef: name: sqs-processor pollingInterval: 10 cooldownPeriod: 120 minReplicaCount: 0 maxReplicaCount: 30 advanced: horizontalPodAutoscalerConfig: behavior: scaleDown: stabilizationWindowSeconds: 60 triggers: - type: aws-sqs-queue authenticationRef: name: sqs-processor-auth metadata: queueURLFromEnv: QUEUE_URL awsRegion: us-east-1 queueLength: "10" activationQueueLength: "1"

What this does:

Defines how KEDA authenticates with AWS and how replicas are calculated. Each pod targets 10 messages, and scale-to-zero is enabled when the queue is empty.

4. Replica Calculation Logic

KEDA calculates outstanding work as:

ApproximateNumberOfMessages

+ ApproximateNumberOfMessagesNotVisible

(+ delayed messages, if enabled)

Replica calculation:

desired replicas = ceil(outstanding messages / queueLength)

Example (queueLength: “10”):

0 messages → 0 pods

8 messages → 1 pod

25 messages → 3 pods

95 messages → 10 pods

Final replica counts are bounded by minReplicaCount and maxReplicaCount.

5. Verification

After applying the manifests:

1. Send 50 messages to the SQS queue

2. Inspect the ScaledObject:

kubectl get scaledobject sqs-processor -n workers

3. Watch pod scaling:

kubectl get pods -n workers -w

4. Confirm pods scale back to zero after processing completes

6. Troubleshooting Common Issues

No scale-out

Cause: Incorrect queue URL or IAM permissions

Check: kubectl describe scaledobject, KEDA operator logs

Too many pods

Cause: In-flight messages counted

Check: scaleOnInFlight, SQS visibility timeout

Never scales to zero

Cause: Delayed or unacknowledged messages

Check: Queue attributes and application behavior

HPA exists but no scaling

Cause: Authentication or metric fetch errors

Check: KEDA logs and fallback status

7. Production Tuning Tips

Choose queueLength based on throughput, not guesswork

Tune cooldown and HPA behavior separately – they control different scale-down paths

Enable fallback replicas for resilience if metrics become unavailable

Decide whether in-flight messages should count based on visibility timeout behavior

Conclusion

Queue-based workloads benefit most when autoscaling is driven by the amount of work waiting to be processed rather than traditional infrastructure metrics. Backlog-aware scaling enables applications to react more quickly to changing demand while reducing unnecessary resource consumption during quieter periods.

Although this article demonstrated the approach using Amazon SQS, Amazon EKS, and KEDA, the same design pattern applies across many event-driven architectures and messaging platforms. The key is selecting a scaling signal that accurately represents workload demand and tuning the autoscaling behaviour to match the characteristics of the application.

As organisations increasingly adopt asynchronous, event-driven systems, workload-aware autoscaling becomes an important part of building resilient, efficient, and cost-effective Kubernetes deployments.