What Autoscaling Is
TL;DR
Autoscaling is a loop that adds or removes capacity when a signal changes, inside a minimum and a maximum you set. The new capacity is not usable until it has started and passed a health check.
Loop
Bounds
Delay
Autoscaling is how a system adds or removes capacity on its own as traffic changes. A controller reads a signal, compares it with a target, and changes capacity inside a minimum and a maximum you set. The decision is the autoscaler. The extra machine, process, or function is just capacity.
Horizontal scaling is the act of adding instances. Vertical scaling is the act of making one instance larger. Autoscaling is the loop that decides to do one of those without someone typing the new size.
What usually changes
Most HTTP APIs scale horizontally: the controller changes the replica count. A new replica can start while the old ones keep serving. Making one machine larger often requires a restart, and the machine has a ceiling. Vertical autoscaling exists. It is a different loop, with a disruption the horizontal loop can avoid.
The new replica is not useful at the moment the controller creates it. It has to boot, pass a health check, and join the load balancer. Until then the previous replicas still carry the traffic that triggered the change.
Which signal
CPU and memory are easy to collect and late. By the time CPU is high, the instances are already busy. The extra replica arrives after that.
Request rate and the number of requests in flight track the work more closely. Queue depth is the right signal for a worker, because the backlog is work that has not started. Latency is a noisy reason to add replicas. A slow database looks like the API needs more copies, and more copies can open more database connections and make the database slower.
Match the signal to the thing that runs out. If the scarce resource is database connections, scaling the API on CPU can push that database over. If the scarce resource is CPU on the API itself, a CPU target is reasonable.
Bounds, delay, and flapping
Set a minimum so a quiet hour does not remove the last instances unless you mean to scale to zero. A minimum of two survives one instance dying. Set a maximum so a bug, a retry storm, or an attack cannot grow the fleet without a bound. The maximum is also a cost ceiling. Past it, the service should reject or queue work instead of starting another copy. A rate limit is one way to shed that extra load.
If you add a replica when CPU crosses 70 percent and remove it when CPU falls to 69 percent, the count flaps. Scale-in should wait longer than scale-out. A stabilization window holds the higher count through a short dip.
The Kubernetes HorizontalPodAutoscaler is one implementation. This manifest keeps a deployment between 2 and 10 pods and targets an average CPU utilization near 70 percent:
averageUtilization is a percentage of the CPU request on each pod. If the container does not set a CPU request, utilization has no denominator and this metric will not scale the deployment. On the way down, Kubernetes holds the higher replica count for a stabilization window. The default is 300 seconds. The Horizontal Pod Autoscaler documentation describes that default and the other metrics you can target. A cloud autoscaling group does the same kind of job for virtual machines. Amazon EC2 Auto Scaling, launched in 2009, is one of those groups: it adjusts the number of instances from a signal you configure.
A serverless platform hides the loop and often allows the minimum to be zero. That is still autoscaling. The cold start is the cost of arriving from zero. API hosting is where the replicas run. Autoscaling only changes how many are running.
What another replica does not fix
The work has to be splittable. A lock that only one process can hold will not go faster because ten processes are waiting on it. A job that cannot be divided stays the same speed.
Shared dependencies have their own ceilings. Ten app replicas with a pool of 20 database connections each want 200 connections. If the database allows 100, autoscaling the app makes the errors worse. Raise the useful limit, or cap the app, before you raise maxReplicas.
Removing a replica can drop calls. The load balancer should stop sending it new requests, and the process should finish work it already accepted. A container that exits on SIGTERM gives the platform a chance to do that. Connections that outlive the drain window are still cut. Drain time reduces dropped calls. It is not evidence that the count of dropped calls was zero.
Autoscaling follows traffic inside the bounds you wrote down. It does not see a spike that has not hit the metric yet, and it does not create capacity that boots instantly. Size the minimum for the lag you can accept, and size the maximum for the dependency that fails first.
Questions & Answers about Autoscaling
Autoscaling adds or removes capacity on its own as a signal changes. You set the signal, the target, a minimum, and a maximum. The controller compares the signal with the target and changes the replica count or the size of an instance. New capacity serves traffic only after it has started and a health check has passed.
