Module F-10·20 min read

What each abstraction takes away — cold starts, execution ceilings, no work after the response returns, ephemeral disk, and why an edge runtime cannot hold a WebSocket or a TCP pool.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module F-10 — Compute Models: VMs, Containers, Serverless, and the Edge

Q: A team moves an Express API to serverless functions. Order confirmation emails now arrive for most orders but not all, with no errors logged. The code responds to the client and then calls `void sendConfirmationEmail(order)`. What is happening, and what's the fix?

The execution environment may be frozen or destroyed once the response is returned, so post-response work runs unreliably. Anything that must happen has to be enqueued durably *before* responding — which is also the more robust design on a long-running server. — The module identifies "no work after the response returns" as the constraint that breaks the most existing code, and describes exactly this signature: work that might complete, might partially complete, or might never start, varying per invocation — silent, partial, non-deterministic loss with nothing in the logs. The remedy is to make the queue the durability boundary and enqueue before responding, which also protects a container-based server against a crash between response and side effect. Option C identifies a genuine code smell, but awaiting the send merely trades dropped emails for slower responses and still loses the work if the send fails — it doesn't make the side effect durable. Option A invents a cause inconsistent with the absence of errors. Option B misapplies cold starts, which affect the latency of the invocation rather than whether post-response work executes.

Q: A Next.js app moves its API routes to the edge runtime to reduce latency for international users. The primary Postgres stays in `us-east-1`. Users in Asia report the app got *slower*. Why?

Edge compute reduced client-to-compute latency but increased compute-to-data latency: each database query now crosses an ocean, so a handler making three sequential queries pays roughly 3 × 180 ms instead of 3 × 2 ms. Latency is governed by where code sits relative to data. — This is the module's "edge is fast" fallacy, with the worked arithmetic: single-region compute totals ~366 ms while edge compute against a distant database totals ~550 ms, and every additional sequential query widens the gap. The stated law is that latency is governed by code-to-data distance, not by how modern the runtime is — so edge compute pays off for work that doesn't need origin data, or when the data is genuinely replicated too. Option D describes a real and compounding cost that the module also raises, but it's an additional penalty on top of the round trips rather than the sole cause; the ocean crossing dominates. Option B is a genuine edge constraint that doesn't match the symptom, which is latency proportional to query count rather than CPU-limit failures. Option C describes propagation behaviour that isn't how these platforms work and wouldn't produce a sustained regression.

Q: A serverless API scales to about 1,000 concurrent invocations during a spike. Postgres begins refusing connections, and unrelated services that share the database fail too. Which mitigation does the module describe, and with what stated cost?

A transaction-mode connection pooler multiplexes thousands of client connections onto a few dozen server connections by holding one only for the duration of a transaction — at the cost of session-scoped features such as prepared statements, `SET` variables, `LISTEN/NOTIFY`, and session advisory locks. — The module lists three real answers — a transaction-mode pooler, an HTTP-based data API, or keeping the pool in a container — and names the pooler's cost precisely: session-scoped features stop working, which is why some ORM configurations need changing. Option A treats a connection as free; each Postgres connection is a forked backend process with its own memory, so raising the limit to 2,000 trades connection refusals for memory pressure and context-switching. Option C adds load during the exact window the database is out of slots, which is Module A-6's retry storm. Option D misunderstands the local temp directory: it is per-instance and ephemeral, so with 1,000 isolated invocations it provides almost no cache hits — and the module explicitly rules it out as a shared cache. Note also that the harm here landed on *other* services sharing the database, which is Module F-9's point that scaling one tier relocates the bottleneck.

What this module covers: Each step up the compute abstraction ladder hands you something and quietly takes something away, and the things taken away are what cause the outages. This module covers VMs, containers, serverless containers, functions, and edge runtimes — with a specific inventory of what each one removes: cold starts, execution-time and memory ceilings, the rule that no work happens after the response returns, ephemeral local disk, and the inability of an edge runtime to hold a WebSocket or a database connection pool. It ends with the constraint that decides more architectures than any other: where the code runs relative to where the data lives.

The Ladder, and the Trade at Each Rung

	Unit	Start-up	You manage	Scales to zero	Holds state between requests
Bare metal	Machine	Minutes	Everything	No	Yes
VM	Guest OS	30s–2min	OS, runtime, app	No	Yes
Container	Process + namespaces	1–10s	Image, app	Rarely	Within the container's life
Serverless container	Container, managed	1–10s cold	Image, app	Yes	Only while an instance is warm
Function (FaaS)	Handler invocation	50ms–2s cold	App code	Yes	Unreliably
Edge runtime	V8 isolate	~0–5ms	App code	Yes	No

Read the last two columns together. As start-up time and operational burden go down, the ability to hold anything goes down with them — and "anything" includes connections, caches, sockets, timers, and in-flight background work. That's not incidental; it's the mechanism. A platform can only start you in five milliseconds and scale you to zero if it's free to discard you at any moment.

Analogy: the ladder runs from owning a house, to renting a flat, to a hotel room, to a hot-desk, to a phone booth. Each step is cheaper, faster to get into, and less of your responsibility. Each step also gives you less room to leave anything lying around — and by the phone booth, you can't even keep a phone line open between calls.

Why this matters in production: every one of these platforms runs a "hello world" identically well. The differences only appear when your code wants to keep a database connection, do work after responding, write a temp file, or hold a socket open. Those are precisely the things a real application does, and discovering the constraint after committing to the platform is the expensive path.

VMs and Containers: The Familiar End

Virtual machines give you a full guest OS with its own kernel. Strong isolation, any workload, complete control — and you own patching, configuration drift, and multi-minute boot times. Autoscaling on a two-minute boot means you're always reacting late.

Containers share the host kernel and isolate with namespaces and cgroups. Start-up drops to seconds, images are immutable and reproducible, and density goes up substantially. Two properties matter for design:

The filesystem is ephemeral. Anything written inside a container disappears on restart unless it's on a mounted volume. This is the same "local files are state" problem as Module F-9, now enforced by the platform.
Resource limits are enforced by cgroups, and runtimes often don't notice. A JVM or Node process that reads the host's memory rather than its cgroup limit will happily grow past the container's ceiling and be OOM-killed — a restart loop with no application error to explain it.

Containers keep the crucial property the higher rungs give up: a container is a long-running process. It can hold a connection pool, an in-memory cache, a WebSocket, a background timer. Everything below this line loses some of that.

Functions: Four Constraints That Reshape Designs

FaaS (AWS Lambda, Vercel Functions, Cloud Functions) runs a handler per request and charges per invocation-millisecond. The operational win is real — no servers, scale to zero, scale to thousands. Four constraints come with it.

1. Cold starts

When no warm instance exists, the platform provisions one: fetch the code, initialise the runtime, run module-level code, then invoke the handler. Roughly:

Runtime / situation	Typical cold start
Node.js / Python, small bundle	100–400 ms
Node.js with a large dependency tree	500 ms – 2 s
JVM / .NET	1–5 s
Inside a VPC (historically)	+seconds
Edge / V8 isolate	~0–5 ms

Cold starts are a tail-latency phenomenon by construction (Module F-4). Your p50 can be excellent while p99 includes a full initialisation. And the shape is spiky: a traffic surge triggers many simultaneous cold starts precisely when latency matters most.

What actually helps: smaller bundles and fewer dependencies (the single biggest lever), moving expensive setup out of the hot path, provisioned concurrency where the platform offers it — with the cost named, since provisioned concurrency is paying for idle capacity, which is most of what serverless promised to eliminate.

2. No work after the response returns

This is the constraint that breaks the most existing code. In a long-running server, this is idiomatic:

typescript

// Perfectly fine on a container. Broken on FaaS.
app.post('/orders', async (req, res) => {
  const order = await createOrder(req.body);
  res.json(order);                    // respond immediately

  void sendConfirmationEmail(order);  // fire and forget
  void trackAnalytics('order.created', order);
  void warmRecommendationCache(order.customerId);
});

On FaaS the execution environment may be frozen or destroyed the moment the response is returned. Those three background tasks might complete, might partially complete, or might never start — and it varies invocation to invocation, so it "works in staging" and drops a fraction of emails in production. Silent, partial, non-deterministic data loss is the worst failure category to debug.

The fix is architectural, not a workaround: anything that must happen goes onto a durable queue before you respond.

typescript

app.post('/orders', async (req, res) => {
  const order = await createOrder(req.body);

  // Enqueue durably *before* responding. The queue is the durability boundary.
  await queue.sendBatch([
    { type: 'email.order_confirmation', orderId: order.id },
    { type: 'analytics.order_created', orderId: order.id },
  ]);

  res.json(order);
});

That's Modules P-14 and P-19, and it's a better design on a container too — it survives a crash between response and side effect, which the fire-and-forget version never did. The serverless platform didn't create the bug; it removed the ambiguity that was hiding it.

3. Execution and memory ceilings

Hard walls: a maximum execution duration (commonly 10–15 minutes, far less for HTTP-triggered edge functions), a maximum memory allocation, and on most platforms CPU allocated in proportion to memory — so a CPU-bound function may need to be configured with excess memory purely to get more CPU, and the cheapest setting is frequently not the cheapest total cost, because doubling memory can more than halve duration.

Design consequences: no long-running jobs (chunk them, or run them on a container), no unbounded in-memory processing (stream instead), and no assuming a large upload fits in memory.

4. Ephemeral, per-instance local disk

There's usually a writable temp directory, but it's per-instance, small, and vanishes. It cannot be a cache other invocations will read, and it cannot be where uploads accumulate. Pre-signed direct-to-object-storage uploads (Module A-11) are the answer, and they're better anyway — the bytes never pass through your compute at all.

The Connection Problem

This one deserves its own section because it's the most common way serverless takes down a database, and it's the payoff Module P-4 builds on.

A container holds a connection pool: 10 connections serving thousands of requests over hours. A function invocation can't share a pool with other invocations — each execution environment is isolated. So:

text

Container model:    20 containers × pool of 10 = 200 connections, stable
Function model:   1,000 concurrent invocations × 1 connection = 1,000 connections

Postgres's default max_connections is 100. Each connection is a forked backend process with its own memory. A traffic spike that scales your functions to 1,000 concurrent executions doesn't degrade the database gracefully — it exhausts connection slots, and then every client fails, including the ones that were working fine. The elasticity you bought is the thing that breaks the dependency.

The three real answers, with what each actually fixes:

A transaction-mode connection pooler (pgBouncer, Supabase's pooler, RDS Proxy). Thousands of client connections multiplex onto a few dozen server connections, because a connection is only held for the duration of a transaction rather than a session. The cost, named: session-scoped features stop working — prepared statements, SET session variables, LISTEN/NOTIFY, and session-level advisory locks — so some ORM configurations need changing.
An HTTP-based data API (Neon's serverless driver, PlanetScale's HTTP driver, Data API-style endpoints). Each query is an independent HTTP request, so there is no connection to hold at all. Cost: no transactions spanning multiple round trips in the usual sense, and per-query HTTP overhead.
Keep the connection-holding tier a container. Functions call an API that owns the pool. This is often the least-clever and most robust option.

Why this matters in production: notice the failure is not in your function. It's in a shared dependency, triggered by your function's ability to scale. This is Module F-9's "scaling one tier relocates the bottleneck," in its sharpest form.

Edge Runtimes: Fast, Close, and Very Constrained

Edge runtimes (Cloudflare Workers, Vercel Edge Functions, Deno Deploy) run V8 isolates rather than containers — no OS process per tenant, so start-up is effectively free and code runs in hundreds of locations near users.

What they take away, and each of these is a hard architectural fact:

No Node.js runtime surface. No fs, no native modules, no arbitrary TCP sockets. You get a web-standard API set (fetch, Request, Response, Web Crypto). A dependency reaching for net or a native binding simply won't run.
Very small CPU budget per request — often single-digit to low-tens of milliseconds. Fine for routing, auth checks, header rewriting, personalisation, A/B assignment. Not for image processing or heavy computation.
No long-lived connections. No WebSocket server in the conventional sense, no database connection pool. This is definitional rather than a limitation to work around, and it's why Module A-10's real-time designs assume a stateful tier somewhere.
No shared memory between isolates. Any cross-request state needs an external store — which at the edge means a globally-distributed one (a KV store, a durable-object-style primitive), because a single-region store defeats the purpose.

The "edge is fast" fallacy

This is the most valuable thing in the module, and it catches good teams.

Moving compute to the edge reduces the client-to-compute latency. It does nothing about compute-to-data latency — and if your database is in one region, you've just moved the code away from the data.

text

Origin in us-east-1, user in Mumbai, 3 sequential DB queries.

Compute in us-east-1 (single region):
  user → origin           180 ms
  3 queries × 2 ms local     6 ms
  origin → user           180 ms
  Total                  ≈ 366 ms

Compute at the Mumbai edge, database still in us-east-1:
  user → edge                5 ms
  3 queries × 180 ms       540 ms   ← each query crosses the ocean
  edge → user                5 ms
  Total                  ≈ 550 ms   ← slower, by a lot

The edge made it worse, and every additional sequential query makes it worse still. Edge compute pays off when it can answer without the origin data — cached content, a token verified with a local key, a header rewrite, a redirect, a geo decision — or when the data is genuinely replicated to the edge too.

Which is the general law, and it applies to every rung of the ladder: latency is governed by where the code is relative to the data, not by how modern the runtime is. Module F-2's distance-dominates conclusion, arriving as a platform decision.

Choosing

Workload	Fits best	Because
Standard web API, steady traffic	Container	Holds pools and caches; predictable cost
Spiky or unpredictable traffic	Functions	Scale to zero, scale out instantly
Long-running jobs, batch, transcoding	Container / VM	No execution ceiling
WebSockets, SSE, streaming state	Container (or a managed real-time service)	Long-lived connections required
Auth checks, redirects, personalisation, A/B	Edge	Data-independent, latency-sensitive
Anything needing native modules or `fs`	Container / VM	Edge and some FaaS runtimes exclude them
Cron and scheduled work	Platform scheduler + queue + worker	Avoids the N-instances-fire-N-times problem (F-9)

Mixed is the norm, and mixed is fine: an edge middleware doing auth in front of container-based APIs, with functions handling bursty webhook ingestion and a container fleet running the queue workers. What isn't fine is choosing a rung and discovering its constraints afterwards.

Where This Shows Up in Your Stack

Next.js: the App Router lets you choose per route — export const runtime = 'edge' or the default Node.js runtime. Middleware runs at the edge, so it can check a cookie but shouldn't query your primary database. The failure mode is subtle: it works locally, where everything is 1ms away.
PostgreSQL: the connection-exhaustion story above is the single most common serverless-plus-Postgres incident, and Module P-4 is largely about it.
Redis: a serverless function reconnecting to Redis per invocation adds a TCP round trip to every call, undoing most of Redis's speed advantage (Module F-5). HTTP-based Redis clients exist precisely for this shape.
Background work: if your platform can't run work after a response, you need a queue. That's not a serverless tax — it's the durable design you should have had, now mandatory.
Cost modelling: invocations × duration × memory is the whole billing formula, and CPU scaling with memory means the cheapest memory setting often isn't the cheapest bill. Module O-8 treats this as unit economics.

Summary

Every rung up the ladder trades the ability to hold things for lower start-up and less operational burden. Connections, caches, sockets, timers, and post-response work are what get traded.
Containers are the last rung that is definitely a long-running process — and their filesystem is still ephemeral, and cgroup limits are still invisible to some runtimes.
Functions impose four constraints: cold starts (a p99 problem, worst during surges), no work after the response returns, hard execution and memory ceilings with CPU tied to memory, and ephemeral per-instance disk.
"No work after the response" causes silent, partial, non-deterministic loss. Enqueue durably before responding — a better design on any platform.
Function concurrency exhausts database connections. Fix it with a transaction-mode pooler (losing session-scoped features), an HTTP data API (losing multi-round-trip transactions), or by keeping the pool in a container.
Edge runtimes give near-zero cold start and global placement, and take away Node APIs, TCP sockets, long-lived connections, meaningful CPU budget, and shared memory.
The "edge is fast" fallacy: moving compute closer to users moves it further from a single-region database, and sequential queries then cross the ocean each time. Latency is governed by code-to-data distance.
Mix models deliberately — edge for data-independent work, containers for stateful and long-running work, functions for bursts.

The next module, Load Balancing, goes back to the front door: how requests get distributed across whichever of these you chose, why health checks lie, and what happens to in-flight requests during a deploy.

Knowledge Check

A team moves an Express API to serverless functions. Order confirmation emails now arrive for most orders but not all, with no errors logged. The code responds to the client and then calls void sendConfirmationEmail(order). What is happening, and what's the fix?

A Next.js app moves its API routes to the edge runtime to reduce latency for international users. The primary Postgres stays in us-east-1. Users in Asia report the app got *slower*. Why?

A serverless API scales to about 1,000 concurrent invocations during a spike. Postgres begins refusing connections, and unrelated services that share the database fail too. Which mitigation does the module describe, and with what stated cost?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule F-9: Scaling Up vs Scaling Out; Stateless vs Stateful Next Module F-11: Load Balancing

Discussion

Join the discussion

Loading comments...