Module P-7 — Connecting External Services and Caching

What this module covers: Production APIs don't live in isolation. They send email, upload files to object storage, call third-party APIs, and cache expensive queries to keep response times fast. This module covers Redis with ioredis and the cache-aside pattern, sending transactional email with Nodemailer, uploading files to S3-compatible object storage, making outbound HTTP requests with the native fetch API, and processing webhooks idempotently. Each section gives you production-ready patterns, not just the happy path.

Redis and Caching with ioredis

Redis is an in-memory data store. The two primary use cases in a Node.js API are caching (store the result of an expensive query, serve it for subsequent requests) and session/token storage (store refresh tokens so they can be revoked).

Redis client singleton

The Cache-Aside Pattern

The most common caching strategy: check the cache first, hit the database only on a miss, then populate the cache for next time.

Cache invalidation

The cache must be invalidated when data changes — stale cache is worse than no cache:

Caching list queries

Lists are harder — when a post is created, which lists are stale? The simplest approach: use short TTLs for lists and longer TTLs for individual records.

Redis for refresh token storage

Using Redis instead of a Postgres table for refresh tokens — faster lookups, automatic expiry:

Sending Email with Nodemailer

Nodemailer is the standard Node.js library for sending email. In production you connect it to a transactional email provider (SendGrid, Resend, Postmark, SES) — never your own SMTP server.

Email transport singleton

Sending emails

Don't block the request on email

Email sending should never make the response wait:

For reliable email delivery in high-volume apps, put emails on a job queue (covered in P-12, BullMQ) instead of sending inline.

File Uploads to Object Storage (S3/R2)

Never store uploaded files on the API server's disk. Servers are ephemeral in cloud deployments, and local disk doesn't scale horizontally. Store files in object storage: AWS S3, Cloudflare R2, DigitalOcean Spaces (all S3-compatible).

Multer for parsing multipart uploads

S3 client singleton

Upload service

Upload route

Pre-signed URLs for client-side uploads

For large files, upload directly from the client to S3 — skip your API server entirely:

The client receives uploadUrl, PUTs the file directly to S3, then sends just the key to your API to record the uploaded file.

Outbound HTTP with fetch

Node.js 18+ ships a global fetch — no extra libraries needed for most use cases.

Wrapper with timeout and error handling

Idempotent Webhook Processing

Webhooks are HTTP requests sent by third-party services (Stripe, GitHub, Twilio) to notify your API of events. Two rules for production webhook handling:

1. Verify the signature — don't trust the payload without checking it came from the real sender.
2. Process idempotently — webhook delivery is at-least-once. The same event may arrive twice.

The webhook route needs raw body access — register it before express.json():

Summary

• Redis caching: cache-aside pattern — check cache, query DB on miss, populate cache, invalidate on write. Use short TTLs (60s) for lists, longer (5 min) for individual records.
• Refresh token storage in Redis: setex with matching TTL — tokens auto-expire without cleanup jobs.
• Email with Nodemailer: connect to a transactional provider (Resend, SendGrid, SES). Fire-and-forget with .catch() for non-critical emails; use a job queue for critical ones.
• File uploads: multer parses multipart form data, stream to S3 from memory. Pre-signed URLs let large files bypass your server entirely.
• Outbound HTTP: native fetch with AbortController for timeouts. Centralise in a wrapper that handles non-OK responses uniformly.
• Webhook idempotency: verify signature + deduplicate with Redis before processing. Acknowledge with 200 immediately to prevent provider retries.

Next: structured logging with Pino, correlation IDs that make logs searchable, multi-stage Docker builds, PM2 for production process management, and deploying via GitHub Actions CI/CD.