Module F-9·18 min read

Sticky sessions, the vertical ceiling, and why state — not traffic — is what actually makes scaling hard.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module F-9 — Scaling Up vs Scaling Out; Stateless vs Stateful

Q: After scaling from one instance to four, a team gets reports that users are "randomly logged out," a nightly email is sent four times, and a per-user rate limit no longer seems to apply. What is the common cause?

All three are in-process state that assumed a single instance: sessions in memory (only one instance has them), an in-process scheduler (each instance fires it), and per-instance rate-limit counters (the effective limit becomes 4× and a client rotating across instances sees none). Each must move to a shared store, with the scheduled job gated by a lock or leader election. — These are three of the module's catalogued hiding places for state, and they present exactly as described: in-memory sessions produce intermittent logouts because only one instance holds them, `setInterval`-style schedulers run once per instance, and in-memory counters both multiply the effective limit by N and fail entirely for a client whose requests land on different instances. The module also explains why this class of bug is expensive — the failures are probabilistic, so QA can't reproduce them. Option A is the sticky-session shortcut, which the module analyses as borrowing against the future: it papers over the session symptom, does nothing for the duplicated job or the rate limiter, and adds uneven load, session loss on instance death, and broken draining. Option B is a real hazard at larger fan-out (Module P-4) but wouldn't produce these three specific symptoms. Option C invents a cause the evidence doesn't support.

Q: A team is at 60% CPU on a single large instance and debating whether to invest six weeks in making the application horizontally scalable now. What does the module advise?

Scale up first — it's fast, effective, and buys years — while keeping the application *able* to scale out. But treat the ceiling as real: on the largest instance there is no next step, and one machine gives you no availability story for patching or redeploys. — The module's stated position is "scale up first, but design so you can scale out," and it gives both halves their due: vertical scaling is a card playable in an afternoon and avoids all of Phase 3's distributed-systems complexity, while the ceiling arrives suddenly, a single machine is a single failure domain that can't be patched without downtime, and top-end instances turn super-linear on cost. Option A discards a genuinely effective and cheap lever on ideological grounds. Option C ignores that availability — not just capacity — is a reason to run more than one instance, and waiting until 95% means doing the work under pressure. Option D takes the scale-up advice and drops the design-for-scale-out clause, which is precisely the position that produces an urgent migration later. Note also Module F-4's queueing result: 60% utilisation is already well into the region where latency is several times service time.

Q: A team makes its app tier fully stateless and scales from 2 to 20 instances. Total throughput drops. What does the module say is happening?

Scaling one tier relocates the bottleneck rather than removing it: the app tier stopped being the constraint and the shared dependency became it — 20 instances of connection pools overwhelming one Postgres. Any shared resource bounds total throughput regardless of parallel capacity added. — This is the module's closing point, given with the same numbers: 2 instances at 40% database CPU becomes 20 instances at 100% database CPU with the connection limit exceeded, yielding 10× the app capacity and less total throughput. The general principle is that a shared resource is a serialisation point that bounds the whole system, so "can we handle 10×?" must be asked of every component in the path — and the specific case is the setup for Module P-4. Option A is not the constraint at this scale; load balancers handle far more than 20 backends. Option B describes a real cost of externalising state that the module acknowledges, but it's a modest per-request addition, not a throughput reversal. Option D would apply to a cold-start-dominated workload (Module F-10), not to twenty long-running instances.

What this module covers: What actually stops you from adding a second server. This module covers vertical vs horizontal scaling and why "scale up first" is better advice than it sounds, then the real subject: state. Every kind of state that hides in an application — sessions, in-memory caches, local files, open sockets, in-process schedulers, rate-limit counters — and where each one has to move before a second instance is safe. Plus sticky sessions and the four things they cost, the services that are irreducibly stateful, and the reason scaling out the app tier so often just relocates the bottleneck.

Two Ways to Get Bigger

Scaling up (vertically) means a bigger machine: more cores, more RAM, faster disks. Scaling out (horizontally) means more machines.

	Scale up	Scale out
Code changes	Usually none	Statelessness required
Ceiling	Hard — the largest instance available	Effectively none
Cost curve	Super-linear at the top end	Roughly linear
Failure domain	One machine — a single point of failure	Many; one loss is survivable
Resize	Often needs a restart	Add or remove instances live
Operational surface	One thing to run	Load balancing, discovery, config, deploys across N

The conventional advice is "scale out, it's the modern way." The better advice is: scale up first, but design so you can scale out.

Why scaling up first is usually right:

It's astonishingly effective. Cloud instances go to hundreds of vCPUs and terabytes of RAM. A dataset that fits in RAM does not need a distributed cache, and a Postgres with 512 GB of shared_buffers handles workloads people assume require sharding.
It's a card you can play in an afternoon. Resizing an instance is minutes of work. Making a stateful application horizontally scalable is weeks, and introduces new failure modes you'll be debugging for months.
The complexity you avoid is the expensive part. Every module in Phase 3 exists because distribution creates problems that a single machine doesn't have.

Why designing for scale-out anyway matters:

The ceiling is real and it arrives suddenly. When you're on the largest available instance, there is no next step — and by then the migration is urgent, which is the worst time to attempt it.
One machine is one failure domain. Vertical scaling gives you no availability story. A single instance cannot be patched, redeployed, or rebooted without downtime.
Cost turns super-linear. The top-end instance is disproportionately expensive per unit of capacity, so at some point three medium instances genuinely beat one large one on price (Module O-8).

So: run on one box for as long as it's honestly working, and keep the application able to run as many. That second clause is the entire content of the rest of this module.

Analogy: scaling up is hiring a stronger single worker; scaling out is hiring a team. The team is unboundedly more capable, but only if the work can be divided — and work can only be divided if nobody is holding something the others need. State is that something.

State Is the Only Thing That Makes This Hard

An instance is stateless if any request can be served by any instance, and losing an instance loses nothing but the requests currently in flight. Statelessness isn't "no data" — it's "no data here." The data lives in a shared store; the instance is a replaceable worker.

Here is every place state hides in a typical application, and where it has to go:

In-memory sessions. The default in many frameworks. Instance A holds the session; the load balancer sends the next request to instance B; the user is logged out. → Move to Redis or a signed stateless token (Module A-13 covers the trade: revocation is easy with server-side sessions and awkward with JWTs).

In-process caches. A Map of computed results. With N instances you get N independent caches: N× the memory, a fraction of the hit rate, and N different answers to the same question after an update. → Shared cache (Module P-12). Worth noting the honest exception: a small, short-TTL in-process cache in front of a shared cache is a legitimate optimisation, as long as you've accepted that instances can briefly disagree.

Local file storage. Uploads written to ./uploads. Instance B can't serve a file instance A received, and a container restart deletes everything. → Object storage with pre-signed uploads, which also keeps the bytes out of your app tier entirely (Module A-11).

Long-lived connections. WebSockets and SSE bind a client to a specific instance for the connection's life. Publishing an event on instance A doesn't reach a subscriber on instance B. → A pub/sub backplane so any instance can deliver to any connection (Module A-10).

In-process schedulers. setInterval or an in-app cron. With N instances the job runs N times — which for a billing or email job is a customer-visible incident. → An external scheduler, or a distributed lock/leader election so exactly one instance acts (Modules A-2, P-19).

In-memory counters. Rate limiters, feature-flag caches, circuit-breaker state. A per-instance limit of 100 req/min becomes an effective limit of N × 100, and the client hitting a different instance each time sees no limit at all. → Redis-backed counters (Module P-18).

Uploaded-file temp state and multi-step wizards. Anything held between two requests belonging to the same user, in memory. → Externalise or make it client-carried.

Why this matters in production: these fail intermittently, which is what makes them expensive. With two instances, half the requests work. Users report "sometimes it logs me out," QA can't reproduce it, and the bug survives for weeks because the failure is a coin flip rather than a crash.

Sticky Sessions: The Tempting Shortcut

Sticky sessions (session affinity) make the load balancer route a given client to the same instance every time, usually via a cookie or a hash of the source IP. It lets in-memory state keep working with multiple instances, and it is genuinely useful for connection-oriented workloads.

What it costs — four things, all of which show up eventually:

Load becomes uneven. Assignment is by client, not by work. One instance can end up with the heavy users while another idles, and the balancer can't fix it because it isn't allowed to move anyone.
Instance loss is user-visible data loss. When an instance dies, everyone pinned to it loses their session, cart, or wizard progress. You've turned a routine event into an incident.
Deploys get worse. Draining an instance means either waiting for every pinned session to expire or breaking them. This directly conflicts with the rolling deploys of Module O-4.
Autoscaling is undermined. A new instance receives only new clients, so scaling out during a spike doesn't relieve the instances that are already overloaded. (Exactly the HTTP/2 connection-pinning problem from Module F-6, arriving by a different route.)

The defensible use is where affinity is inherent — WebSocket connections, or an in-memory working set expensive enough to rebuild that you accept the costs deliberately. As a workaround for not having externalised session state, it's borrowing against the future at a bad rate.

What Externalising State Actually Looks Like

typescript

// ── Before: works on exactly one instance ────────────────────────────────────
const sessions = new Map<string, Session>();        // lost on restart, not shared
const rateLimits = new Map<string, number>();       // per-instance ⇒ N× the limit
setInterval(sendDailyDigest, 24 * 60 * 60 * 1000);  // runs N times with N instances
app.post('/upload', (req, res) => saveTo('./uploads', req.file));  // local disk

// ── After: any instance can serve any request ────────────────────────────────
// Sessions: shared store, TTL handled by the store.
async function getSession(id: string) {
  const raw = await redis.get(`session:${id}`);
  return raw ? (JSON.parse(raw) as Session) : null;
}

// Rate limiting: one authoritative counter, atomic increment.
async function allow(key: string, limit: number, windowSec: number) {
  const n = await redis.incr(`rl:${key}`);
  if (n === 1) await redis.expire(`rl:${key}`, windowSec);
  return n <= limit;
}

// Scheduled work: exactly one instance acts, and the lock expires so a crashed
// holder doesn't block the job forever. (A-2 covers why a lease is not a lock,
// and why the work itself still has to be idempotent.)
async function runDailyDigestOnce() {
  const acquired = await redis.set('lock:daily-digest', instanceId, {
    NX: true,
    EX: 3600,
  });
  if (acquired) await sendDailyDigest();
}

// Uploads: bytes never touch the app tier. The client PUTs straight to storage.
app.post('/uploads/sign', async (req, res) => {
  const key = `uploads/${req.auth.userId}/${crypto.randomUUID()}`;
  res.json({ url: await s3.getSignedUrl('putObject', { Key: key, Expires: 300 }) });
});

Once every one of those is externalised, an instance holds nothing that matters. You can kill it, replace it, or run forty of them. That property — instances are disposable — is what horizontal scaling, rolling deploys, autoscaling, and chaos testing all depend on.

Note what happened to the state, though: it didn't vanish, it moved. Redis and the object store are now on the critical path for every request, which is why Module P-12's cache failure modes and Module A-7's degradation strategies exist. Externalising state converts a scaling problem into a dependency problem, and that's a trade rather than a free win.

The Services That Can't Be Stateless

Some components are state, definitionally: databases, Kafka, Redis, Elasticsearch. You cannot make Postgres stateless — its entire job is holding state durably.

So the techniques change. Instead of "move the state elsewhere," you get:

Replication — more copies, which scales reads and provides failover (Module P-5).
Partitioning/sharding — splitting the data so each node owns a slice, which is the only thing that scales writes (Modules P-6, P-7).
Consensus — coordinating replicas so they agree on what's true (Module A-1).

These are the hard parts of distributed systems, and it's not a coincidence that they cluster around stateful components. It's also why the practical advice is to keep the number of stateful components in your architecture small. Every additional one is another thing to back up, restore, patch, monitor, and fail over — and Module O-7 will ask you to prove you can restore each of them.

Kubernetes encodes exactly this distinction: a Deployment assumes interchangeable pods, while a StatefulSet gives stable identities and stable per-pod storage because the workload genuinely can't be shuffled (Module O-3).

Scaling Out Usually Just Moves the Bottleneck

The last thing worth internalising: making the app tier horizontally scalable doesn't make the system scalable. It makes the app tier stop being the constraint, and the constraint then reappears wherever the shared dependency is.

text

Before:  2 app instances → Postgres at 40% CPU
Scale to: 20 app instances → Postgres at 100% CPU, connection limit exceeded
Result:  10× the app capacity, less total throughput than before

That's not a hypothetical; it's the standard shape of the incident, and it's the direct setup for Module P-4 — 500 app instances each holding a connection pool will destroy one Postgres, and the app tier's elasticity is what makes it possible.

The general form: any shared resource is a serialisation point, and the fraction of work that must go through it bounds your total speedup no matter how much parallel capacity you add. Practically, this means the question "can we handle 10× traffic?" has to be asked of every component in the request path, not just the one you're currently scaling. When you scale one tier, always ask what it's now going to overwhelm.

Where This Shows Up in Your Stack

Node.js: cluster and PM2 fork multiple workers to use multiple cores — which makes a single machine multi-instance, so every state problem in this module appears at N=4 on one box, long before you add a second server. In-memory caches and setInterval schedulers break here first.
PostgreSQL: the archetypal scale-up component. Vertical scaling takes it remarkably far; then read replicas take reads; then sharding takes writes, at a large step up in complexity. Postponing the last step by scaling up is a legitimate strategy, not procrastination.
Serverless: stateless by force. Every constraint in this module is enforced by the platform rather than chosen — no in-memory sessions, no local disk you can rely on, no background timers after the response. Module F-10 covers exactly what that takes away, and it's the reason serverless is the harshest test of whether your application was really stateless.
WebSockets on autoscaling infrastructure: the connection-affinity case where stickiness is inherent. You still need a pub/sub backplane for cross-instance delivery, and scaling down is disruptive because it terminates live connections (Module A-10).

Summary

Scale up first; design so you can scale out. Vertical scaling is fast, effective, and buys years; horizontal scaling removes the ceiling and gives you an availability story.
Statelessness means "no data here," not "no data." Any request served by any instance; losing an instance loses nothing durable.
Find the state: in-memory sessions, in-process caches, local files, long-lived sockets, in-process schedulers (which run N times), in-memory rate-limit counters, and cross-request temp state.
These failures are intermittent, which makes them expensive. "Sometimes it logs me out" survives for weeks because it's a coin flip, not a crash.
Sticky sessions cost four things: uneven load, user-visible data loss on instance death, harder draining and deploys, and undermined autoscaling. Use them where affinity is inherent, not as a substitute for externalising state.
Externalising state moves it rather than removing it — the shared store is now on the critical path, which converts a scaling problem into a dependency problem.
Some components are irreducibly stateful. For those the tools are replication, partitioning, and consensus — and you should keep their number small.
Scaling one tier relocates the bottleneck. Ask "what will this now overwhelm?" every time, because a shared resource bounds the whole system's throughput.

The next module, Compute Models: VMs, Containers, Serverless, and the Edge, takes the disposable-instance idea to its conclusion — and catalogues exactly what each abstraction takes away in exchange.

Knowledge Check

After scaling from one instance to four, a team gets reports that users are "randomly logged out," a nightly email is sent four times, and a per-user rate limit no longer seems to apply. What is the common cause?

A team is at 60% CPU on a single large instance and debating whether to invest six weeks in making the application horizontally scalable now. What does the module advise?

A team makes its app tier fully stateless and scales from 2 to 20 instances. Total throughput drops. What does the module say is happening?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule F-8: API Design: REST, GraphQL, and gRPC Next Module F-10: Compute Models: VMs, Containers, Serverless, and the Edge

Discussion

Join the discussion

Loading comments...