AI Integration and Streaming Route Handlers26 min read
Module P-18·26 min read
ReadableStream and TransformStream in Route Handlers, the Vercel AI SDK (streamText, useChat, useCompletion), token-by-token streaming to the browser, abort signal propagation for cancelled requests, rate limiting streaming endpoints, streaming error handling constraints, and cost control via token budgets.
P-18 — AI Integration and Streaming Route Handlers
Who this is for: Engineers who've built Route Handlers before (F-7) and now need to wire an LLM into one — a chat endpoint, a completion box, a "summarize this" button — and want the response to appear token-by-token instead of as one long spinner. This module covers the raw streaming primitives underneath, the Vercel AI SDK that wraps them, and the operational concerns that only show up once real users start hammering an endpoint that costs money per request: cancellation, rate limiting, in-stream error handling, and token budgets.
Why Streaming Matters Here More Than Anywhere Else
You've already seen Route Handlers return a ReadableStream for Server-Sent Events in F-7. AI text generation is the other major reason to reach for streaming, and it's a more urgent one. A non-streamed call to a large language model can easily take 5–20 seconds to produce a few paragraphs — the model is generating output token by token internally regardless, you're just choosing whether to buffer all of it before sending anything back, or to forward each piece as it's produced.
For a chat interface, buffering means the user stares at a blank screen for the full duration, then the whole answer appears at once. Streaming means the first words appear within a second and the rest arrives progressively, which is a dramatically different perceived-performance experience for identical total latency. This module is entirely about the mechanics of getting tokens from a model provider's response to the browser as they're generated, and about not blowing up your AI provider bill or your server while doing it.
One thing to be clear about up front, because it resurfaces later: streaming changes when the user sees output, not how much output costs. The model still generates and bills for the same number of tokens whether you stream them or send them all at once. Keep that distinction in mind — it matters a lot for the cost-control section near the end.
The Raw Primitive: Returning a ReadableStream
Before reaching for a library, it's worth seeing the raw mechanism, because the library is just a well-built wrapper around it. A Route Handler can return a ReadableStream directly as the body of a Response:
ts
This is the entire mechanism. ReadableStream is a standard Web API, not a Next.js-specific construct — it works because Route Handlers speak the same Request/Response interfaces as the rest of the web platform. Every chunk you enqueue() is flushed to the client as soon as it's written, rather than waiting for controller.close().
When you swap setTimeout and a hardcoded word list for an actual model provider call, this is conceptually what token-by-token streaming is: a loop that receives tokens from the provider's stream and re-enqueues them onto the response stream as they arrive. In practice you won't write that loop by hand for a production chat feature — the AI SDK does it — but understanding that it's "just" a ReadableStream demystifies a lot of what the library does under the hood, and it's useful when you need to debug a stream that isn't behaving, or build something the SDK doesn't directly support.
TransformStream — Reshaping Chunks in Flight
Sometimes you don't want to just pass raw provider chunks straight through — you want to annotate them, filter them, or reformat them before they reach the client. TransformStream, also a standard Web API, sits between a readable source and your response and lets you rewrite each chunk as it passes:
ts
pipeThrough() chains the source stream into the transform, and the result is itself a ReadableStream you hand to Response exactly as before. This pattern is what you'd use if, say, you wanted to inject a [DONE] sentinel, wrap raw text chunks into a JSON envelope for the client to parse consistently, or count tokens as they pass through for logging without buffering the whole response. The AI SDK's response helpers do something structurally similar internally — this is what's happening one layer down.
You won't usually need to write this by hand for a standard chat endpoint. Reach for it when you need custom framing around a provider's raw stream that the SDK's built-in response helpers don't already give you.
The Vercel AI SDK — Server Side: streamText
Writing the token relay loop by hand for every provider (OpenAI, Anthropic, and others each have their own streaming response shape) gets old fast, and it's exactly the kind of undifferentiated plumbing a library should own. The ai package (the Vercel AI SDK) provides streamText() as the core server-side primitive: you give it a model and a prompt or message history, and it returns a stream you can turn directly into a Response.
ts
A few things worth being precise about here, because the AI SDK has moved fast across major versions and this is exactly the kind of detail that goes stale:
The provider packages (@ai-sdk/openai, @ai-sdk/anthropic, and so on) are separate installs from the core ai package — the SDK deliberately splits "the streaming/orchestration layer" from "the model provider adapters."
The exact response-conversion method name (toDataStreamResponse() here) and the exact shape of the streamed protocol have changed between AI SDK major versions. Treat the method name above as illustrative of the pattern — a server function that turns a streamText() result into a Response your Route Handler can return — and verify the current method name against the version of ai in your package.json before shipping. This is not a detail worth memorizing across versions; it's a detail worth looking up each time you upgrade.
streamText() itself does the actual work of calling the provider's streaming API and re-exposing the output as an async-iterable/stream you can consume multiple ways (as a Response, as an async iterator of text deltas, or as a raw ReadableStream) depending on what the current SDK version exposes on the result object.
The practical takeaway: streamText() is doing exactly what the raw ReadableStream example did — receiving tokens from a provider and forwarding them — but it also normalizes the wire format across providers, handles the provider-specific streaming protocol parsing for you, and gives you hooks for the things covered later in this module (abort signals, token limits, error events).
The Vercel AI SDK — Client Side: useChat and useCompletion
On the client, the AI SDK's React bindings give you hooks that manage the entire request/response/streaming lifecycle so you're not hand-rolling fetch plus manual stream parsing in a useEffect.
tsx
useChat manages message history, the in-flight request, streamed token accumulation into messages, and a stop() function that aborts the current request client-side. useCompletion is the sibling hook for single-turn (non-chat, no message history) text generation — a "summarize this" or "rewrite this" button rather than a back-and-forth conversation — with a similar shape (completion, input, handleSubmit, isLoading, stop).
The import path for these hooks is genuinely version-sensitive and worth flagging explicitly rather than glossing over: earlier AI SDK major versions exported them from ai/react, and later versions moved React-specific bindings into a dedicated @ai-sdk/react package as part of splitting framework-specific code out of the core ai package. Both paths have existed at different points in the SDK's history. Check the AI SDK's changelog or your installed package.json for which one is current for your version rather than trusting either path as a permanent fact — this is precisely the kind of detail that silently breaks a build after a routine dependency bump.
Token-by-Token Streaming to the Browser, End to End
Putting the last two sections together, the full path a token takes is:
The model provider generates a token and sends it over its own streaming API (SSE or a provider-specific chunked protocol) to your server.
streamText() on your Route Handler receives that chunk, normalizes it, and re-enqueues it onto the stream backing your Response.
The Response — a real ReadableStream under the hood, same as the raw example earlier in this module — flushes that chunk to the network as soon as it's enqueued, rather than buffering.
useChat or useCompletion on the client reads the incoming stream, decodes each chunk, and appends it to the relevant message's content in React state.
React re-renders with the updated content, and the user sees the new text appear.
None of these five steps is exotic on its own — it's the same ReadableStream mechanics as any other streaming Route Handler, just with a model provider as the ultimate source and a purpose-built React hook as the ultimate sink. What's specific to AI streaming is everything covered in the remaining sections: what happens when the user leaves mid-stream, what happens when the provider errors mid-stream, and how you stop a single user from generating an unbounded number of tokens at your expense.
Abort Signal Propagation — Stopping Generation When the Client Leaves
This is the part of AI streaming that's easy to get wrong in a way that costs real money. If a user closes the tab, navigates away, or clicks "stop" mid-response, the HTTP connection between browser and server closes — but unless you explicitly wire it up, your server keeps calling the model provider and paying for tokens that no one will ever see.
The incoming Request in a Route Handler carries a standard AbortSignal at request.signal, which fires when the client disconnects. The fix is to forward that signal into the provider call so the upstream generation stops the moment the downstream connection does:
ts
streamText() accepts an abortSignal option for exactly this purpose — forwarding it down to the underlying provider request so that when the signal fires, the provider connection is torn down rather than left running to completion in the background. Without this wired up, a user who bails out of a slow response early still burns the full generation cost server-side; the browser tab closing doesn't do anything to stop the model on its own; only forwarding the signal does.
The same principle applies if you're working closer to the metal with a raw ReadableStream: listen for request.signal's abort event and use it to cancel whatever loop or upstream fetch is producing your chunks, mirroring the request.signal.addEventListener('abort', ...) cleanup pattern from the SSE example in F-7.
One caveat worth being upfront about: whether the underlying provider SDK actually respects an aborted signal and stops billing immediately, versus finishing the in-flight token generation it already started, is a provider-and-SDK-version detail you should verify against your specific model provider's documentation rather than assume. Propagating the signal is the correct and necessary first step; it's what makes stopping possible on the provider's end, even if the provider's own cutoff behavior varies.
Rate Limiting Streaming Endpoints — Before the Stream Starts
The config/security module covered a standard rate-limiting pattern for Route Handlers — check a request against a limit (by IP, by user ID, by API key) before doing expensive work, and return a 429 if the caller is over budget. AI streaming endpoints need that same check, but the ordering matters more here than it does for a typical JSON endpoint.
Once you've called new Response(stream, ...) and started sending bytes, you cannot go back and change the response's status code — that decision is made the instant the first bytes go out. So the rate-limit check has to happen before you call streamText(), not somewhere inside the streaming logic:
ts
This is the same rate-limiting primitive you already have from the security module — a check-and-reject step against a store like Upstash Redis or an in-memory sliding window — just applied at a specific point in the handler: before the (expensive, per-token-billed) call to the model provider, and before any bytes of the response have been sent. Get this ordering backwards — say, checking the limit inside the stream's start() callback after the response has already begun — and you've already committed to a 200 and can't cleanly reject the request anymore, which leads directly into the next section.
Streaming Error Handling — You Can't Change the Status Code Mid-Stream
This is the single most important operational constraint to internalize about streaming responses, AI or otherwise: HTTP status codes are set once, in the response headers, before the body starts sending. Once your Route Handler has returned new Response(stream, { status: 200, ... }) and the first chunk has gone out over the wire, there is no mechanism to retroactively turn that into a 500. The status line is already gone.
That means if the model provider errors partway through generation — a rate limit on the provider's side, a content policy rejection, a network blip — you cannot express that failure as an HTTP error status the way you would for a normal JSON endpoint. The error has to be communicated in-band, as part of the stream's content, and the client has to know to look for it there.
The AI SDK's stream protocol includes a mechanism for this — an error part or error event type distinct from ordinary text chunks, which useChat/useCompletion surface through an error value in the hook's return so your UI can render a "Something went wrong" state instead of silently truncating the message. The exact wire format for how an error is encoded in the stream (a specific prefix, a specific JSON envelope) is, again, an SDK-version detail worth checking against your installed version rather than hardcoding from memory — but the shape of the solution is stable and worth understanding regardless of version: catch the error server-side inside the stream-producing logic, enqueue an in-band error signal instead of throwing past the point where headers are already committed, and close the stream cleanly rather than leaving the client hanging on a connection that silently died.
If you're writing this by hand against a raw ReadableStream rather than relying entirely on the SDK's built-in handling, the shape looks like this:
ts
The client-side contract that follows from this: any UI consuming a streamed AI response needs to actively check for an in-band error marker (whatever shape your protocol uses) as it reads chunks, rather than relying on response.ok or a caught fetch rejection the way it would for a normal request — by the time an error occurs, response.ok was already true.
Cost Control via Token Budgets
The point raised at the top of this module is worth restating now that you've seen the full mechanism: streaming improves perceived latency, and does nothing on its own to reduce total token cost. A user who streams a 4,000-token response and a user who waits for the same 4,000-token response non-streamed are billed identically by the model provider — same input tokens, same output tokens. Streaming is a UX improvement layered on top of a cost structure it doesn't change. Real cost control needs its own, separate mechanisms.
Per-request output caps. Every mainstream model provider's API accepts a maximum-output-tokens parameter (commonly maxTokens in the AI SDK's own option naming, or max_tokens in several providers' raw REST APIs — check the exact key name/casing against the provider adapter package you're using, since this is another spot where naming has drifted across SDK versions). Setting this puts a hard ceiling on how much a single request can generate regardless of what the prompt asks for:
ts
Per-user rate limits. The same rate-limiting pattern used to protect the endpoint from abuse (covered above) also functions as a cost control — capping requests per user per time window bounds the worst case regardless of how large any individual response is allowed to get.
System-prompt and context-length discipline. Every message in a chat history gets re-sent as input tokens on every subsequent turn. A long-running conversation with a large system prompt re-bills that system prompt's tokens on every single request in the conversation. Trimming history, summarizing older turns, or capping how many prior messages get included in the context sent to the provider are all real levers, not micro-optimizations — for a chat feature with any meaningful usage, context growth is usually a bigger cost driver over time than any single response's length.
Monitoring, not just limits. Static caps catch the obvious cases (a runaway loop, a malicious prompt trying to extract a maximal response) but won't tell you that your average request crept up over a month as users found longer and longer prompts to send. Logging token usage per request — most provider responses include usage metadata (input/output token counts) alongside the generated content — and feeding it into whatever metrics/alerting you already run for the rest of the application is what actually catches a slow cost creep before it shows up as a surprise on next month's provider invoice.
None of these four levers is exotic, and none of them require guessing at unstable SDK details beyond the two option-naming caveats already flagged above. What matters is treating them as required infrastructure for any AI feature that's reachable by real, untrusted traffic — not as an optimization to add later once the bill is already a problem.
Next: RSC Internals — The React Flight Protocol and React 19 →
Knowledge Check
In a Route Handler, what is the correct way to ensure that a client disconnecting from a streaming AI response actually stops the underlying model provider from continuing to generate (and bill for) tokens?
Why must rate limiting for a streaming AI endpoint be checked *before* calling the model provider, rather than inside the stream's generation logic?
According to this module, why does streaming a response NOT reduce the total cost of an AI request compared to a non-streamed response of the same length?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
// app/api/raw-stream/route.tsexportasyncfunctionGET(){const encoder =newTextEncoder();const stream =newReadableStream({asyncstart(controller){const words =['Streaming','from','a','raw','ReadableStream.'];for(const word of words){ controller.enqueue(encoder.encode(word +' '));awaitnewPromise((resolve)=>setTimeout(resolve,200));} controller.close();},});returnnewResponse(stream,{ headers:{'Content-Type':'text/plain; charset=utf-8'},});}
// app/api/annotated-stream/route.tsexportasyncfunctionGET(){const encoder =newTextEncoder();const decoder =newTextDecoder();const sourceStream =newReadableStream({asyncstart(controller){const chunks =['Hello',', ','this ','is ','streamed ','text.'];for(const chunk of chunks){ controller.enqueue(encoder.encode(chunk));awaitnewPromise((r)=>setTimeout(r,150));} controller.close();},});// Wraps every chunk as a Server-Sent-Events-style "data:" lineconst sseTransform =newTransformStream({transform(chunk, controller){const text = decoder.decode(chunk); controller.enqueue( encoder.encode(`data: ${JSON.stringify({ text })}\n\n`));},});const transformedStream = sourceStream.pipeThrough(sseTransform);returnnewResponse(transformedStream,{ headers:{'Content-Type':'text/event-stream','Cache-Control':'no-cache', Connection:'keep-alive',},});}
// app/api/chat/route.tsimport{ streamText }from'ai';import{ openai }from'@ai-sdk/openai';// or @ai-sdk/anthropic, etc.exportasyncfunctionPOST(request: Request){const{ messages }=await request.json();const result =streamText({ model:openai('gpt-4o'), messages,});return result.toDataStreamResponse();}
// app/chat/page.tsx'use client';import{ useChat }from'@ai-sdk/react';// import path has moved across SDK versions — see note belowexportdefaultfunctionChatPage(){const{ messages, input, handleInputChange, handleSubmit, isLoading, stop }=useChat({ api:'/api/chat',});return(<div><div>{messages.map((message)=>(<divkey={message.id}><strong>{message.role}:</strong>{message.content}</div>))}</div><formonSubmit={handleSubmit}><inputvalue={input}onChange={handleInputChange}placeholder="Ask something…"/><buttontype="submit"disabled={isLoading}>Send</button>{isLoading &&(<buttontype="button"onClick={stop}> Stop
</button>)}</form></div>);}
// app/api/chat/route.tsimport{ streamText }from'ai';import{ openai }from'@ai-sdk/openai';exportasyncfunctionPOST(request: Request){const{ messages }=await request.json();const result =streamText({ model:openai('gpt-4o'), messages, abortSignal: request.signal,// propagate client disconnect to the provider call});return result.toDataStreamResponse();}
// app/api/chat/route.tsimport{ streamText }from'ai';import{ openai }from'@ai-sdk/openai';import{ checkRateLimit }from'@/lib/rate-limit';// the pattern from the security moduleexportasyncfunctionPOST(request: Request){const identifier = request.headers.get('x-forwarded-for')??'anonymous';const{ success, limit, remaining }=awaitcheckRateLimit(identifier);if(!success){returnnewResponse(JSON.stringify({ error:'Rate limit exceeded. Try again later.'}),{ status:429, headers:{'Content-Type':'application/json','X-RateLimit-Limit':String(limit),'X-RateLimit-Remaining':String(remaining),},});}const{ messages }=await request.json();const result =streamText({ model:openai('gpt-4o'), messages, abortSignal: request.signal,});return result.toDataStreamResponse();}
// app/api/raw-chat/route.tsexportasyncfunctionPOST(request: Request){const encoder =newTextEncoder();const stream =newReadableStream({asyncstart(controller){try{forawait(const chunk ofcallModelProvider(request)){ controller.enqueue(encoder.encode(chunk));}}catch(err){// The response is already a 200 — the error has to travel inside the stream controller.enqueue( encoder.encode(`\n[ERROR] Generation failed: ${(err as Error).message}\n`));}finally{ controller.close();}},});returnnewResponse(stream,{ status:200});}
const result =streamText({ model:openai('gpt-4o'), messages, maxTokens:1000,// hard ceiling per request — verify the exact option name for your SDK version abortSignal: request.signal,});