unkey
  • BlogRead product updates and insights
  • Case studiesPractical results from real deployments
  • ChangelogTrack what shipped, improved, and fixed
  • GlossaryClear definitions for API terms
PricingDocs
Discord
LoginSign Up
  • BlogRead product updates and insights
  • Case studiesPractical results from real deployments
  • ChangelogTrack what shipped, improved, and fixed
  • GlossaryClear definitions for API terms
PricingDocsDiscord
LoginSign Up
  1. Glossary
  2. /Autoscaling

What Autoscaling Is

Explore with AI
Open in ChatGPTOpen in ClaudeOpen in Perplexity

TL;DR

Autoscaling is a loop that adds or removes capacity when a signal changes, inside a minimum and a maximum you set. The new capacity is not usable until it has started and passed a health check.

Loop

Measure then add or remove

Bounds

Minimum and maximum

Delay

Boot plus health check

Autoscaling is how a system adds or removes capacity on its own as traffic changes. A controller reads a signal, compares it with a target, and changes capacity inside a minimum and a maximum you set. The decision is the autoscaler. The extra machine, process, or function is just capacity.

Horizontal scaling is the act of adding instances. Vertical scaling is the act of making one instance larger. Autoscaling is the loop that decides to do one of those without someone typing the new size.

What usually changes

Most HTTP APIs scale horizontally: the controller changes the replica count. A new replica can start while the old ones keep serving. Making one machine larger often requires a restart, and the machine has a ceiling. Vertical autoscaling exists. It is a different loop, with a disruption the horizontal loop can avoid.

The new replica is not useful at the moment the controller creates it. It has to boot, pass a health check, and join the load balancer. Until then the previous replicas still carry the traffic that triggered the change.

Which signal

CPU and memory are easy to collect and late. By the time CPU is high, the instances are already busy. The extra replica arrives after that.

Request rate and the number of requests in flight track the work more closely. Queue depth is the right signal for a worker, because the backlog is work that has not started. Latency is a noisy reason to add replicas. A slow database looks like the API needs more copies, and more copies can open more database connections and make the database slower.

Match the signal to the thing that runs out. If the scarce resource is database connections, scaling the API on CPU can push that database over. If the scarce resource is CPU on the API itself, a CPU target is reasonable.

Bounds, delay, and flapping

Set a minimum so a quiet hour does not remove the last instances unless you mean to scale to zero. A minimum of two survives one instance dying. Set a maximum so a bug, a retry storm, or an attack cannot grow the fleet without a bound. The maximum is also a cost ceiling. Past it, the service should reject or queue work instead of starting another copy. A rate limit is one way to shed that extra load.

If you add a replica when CPU crosses 70 percent and remove it when CPU falls to 69 percent, the count flaps. Scale-in should wait longer than scale-out. A stabilization window holds the higher count through a short dip.

The Kubernetes HorizontalPodAutoscaler is one implementation. This manifest keeps a deployment between 2 and 10 pods and targets an average CPU utilization near 70 percent:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

averageUtilization is a percentage of the CPU request on each pod. If the container does not set a CPU request, utilization has no denominator and this metric will not scale the deployment. On the way down, Kubernetes holds the higher replica count for a stabilization window. The default is 300 seconds. The Horizontal Pod Autoscaler documentation describes that default and the other metrics you can target. A cloud autoscaling group does the same kind of job for virtual machines. Amazon EC2 Auto Scaling, launched in 2009, is one of those groups: it adjusts the number of instances from a signal you configure.

A serverless platform hides the loop and often allows the minimum to be zero. That is still autoscaling. The cold start is the cost of arriving from zero. API hosting is where the replicas run. Autoscaling only changes how many are running.

What another replica does not fix

The work has to be splittable. A lock that only one process can hold will not go faster because ten processes are waiting on it. A job that cannot be divided stays the same speed.

Shared dependencies have their own ceilings. Ten app replicas with a pool of 20 database connections each want 200 connections. If the database allows 100, autoscaling the app makes the errors worse. Raise the useful limit, or cap the app, before you raise maxReplicas.

Removing a replica can drop calls. The load balancer should stop sending it new requests, and the process should finish work it already accepted. A container that exits on SIGTERM gives the platform a chance to do that. Connections that outlive the drain window are still cut. Drain time reduces dropped calls. It is not evidence that the count of dropped calls was zero.

Autoscaling follows traffic inside the bounds you wrote down. It does not see a spike that has not hit the metric yet, and it does not create capacity that boots instantly. Size the minimum for the lag you can accept, and size the maximum for the dependency that fails first.

Questions & Answers about Autoscaling

Autoscaling adds or removes capacity on its own as a signal changes. You set the signal, the target, a minimum, and a maximum. The controller compares the signal with the target and changes the replica count or the size of an instance. New capacity serves traffic only after it has started and a health check has passed.

On this page

  1. What usually changes
  2. Which signal
  3. Bounds, delay, and flapping
  4. What another replica does not fix

Turn your API stack into one workflow. Start for free, integrate in minutes, and scale when you need to.

Start for free
unkey

Build better APIs faster

© 2026 Unkey Inc. All rights reserved.

SOC 2 Type II Certified

Company

  • About
  • Careers
  • Status Page
  • Roadmap

Resources

  • Blog
  • Case Studies
  • Changelog
  • Docs
  • Glossary
  • Feature Request
  • Archive

Connect

  • X (Twitter)
  • Discord
  • Book a Call

Legal

  • Terms of Service
  • Privacy Policy
  • Trust Center