Backend Developer Interview Prep Guide

This is the dual-write problem, and naming it by name is the senior signal. The two systems are not in one transaction: if the process dies after the DB commit but before kafka.send(), the database moved and the event never fired — and the inverse (event sent, transaction rolled back) corrupts downstream state. The textbook fix is the Transactional Outbox pattern: in the SAME local DB transaction, insert a row into an outbox table; a separate relay (a poller, or change-data-capture on the DB log) reads committed outbox rows and publishes them. The pattern's own framing is literally "How to atomically update the database and send messages to a message broker?" (microservices.io); AWS describes it as resolving "the dual write operations issue ... when a single operation involves both a database write operation and a message or event notification." Always add the next sentence: outbox guarantees at-least-once delivery, so the consumer MUST be idempotent. Candidates who reach for a distributed two-phase commit instead of outbox/CDC usually get probed on why 2PC is operationally avoided at scale.

The honest framing first: true exactly-once DELIVERY across a network is not achievable; what you build is exactly-once EFFECT via an idempotency key. The client generates a unique key per logical operation and sends it on the request (and on every retry of that operation). The server records the key with the result; a second request bearing a seen key returns the stored result instead of charging again. Stripe's own docs describe idempotency keys as letting you "safely retrying requests without accidentally performing the same operation twice." Cover the sharp edges interviewers push on: the key must be stored atomically with the side effect (an INSERT ... ON CONFLICT, or the charge and the key row in one transaction — this is where outbox thinking returns), key scope and TTL, and what you return for an in-flight duplicate (409 / "request already being processed"). This is THE backend payments question at Stripe and Amazon.

The trap is reciting two definitions. The signal is treating consistency as a DECISION grounded in the CAP theorem — a distributed system "can simultaneously provide only two of three guarantees: consistency, availability, and partition tolerance" (ScyllaDB), so when a partition happens you are choosing whether to sacrifice consistency or availability. Inventory on the last unit needs strong consistency (or an explicit atomic conditional/compare-and-set), because two eventually-consistent reads both seeing "1 left" oversells — a real money/UX bug. A follower count tolerates eventual consistency: a few seconds of staleness is invisible and the availability/latency win is large. Name the concrete mechanism, not just the label: a single-row conditional update or SELECT ... FOR UPDATE for inventory; an async-incremented counter (or Cassandra/DynamoDB tunable consistency, e.g. QUORUM) for the count. Stating the failure mode you are buying down ("eventual here would oversell the last unit") is what separates a senior answer.

This question grades whether you understand that partial failure is the steady state of a distributed system, not an exception. The non-obvious senior answer is that a naive distributed fallback can make things WORSE — AWS's Builders' Library is blunt: "Distributed fallback strategies often make the outage worse" and they "increase the scope of impact of failures." The failure mode: when the dependency degrades under load, every caller simultaneously hits the fallback path; if that path is itself under-provisioned or shares a resource (the same DB, the same cache cluster), you convert a single degraded feature into a full outage. Safer primitives to name: a circuit breaker (stop hammering the dead dependency), a timeout + bounded retry with jittered backoff (un-jittered retries cause thundering herds), load shedding, and serving a STATIC, already-warm default rather than a fallback that does new work. The strongest candidates say "fallbacks must be tested under the same load as the primary, because the fallback path is the one you exercise least and trust most."

This is the N+1 query problem — the single most common backend p99 bug, and the canonical "did you build real systems" tell. PlanetScale's definition: it happens "when you ... first do a query to get a list of records, then subsequently do another query for each of those records." One query fetches 50 orders, then the ORM lazily fires 50 more to load each customer — 51 round-trips, each fine in a dev DB with 10 rows, lethal in prod with network latency × 50. Fixes, in order: eager-load with a JOIN or a single batched WHERE customer_id IN (...) (ORM "includes"/DataLoader), confirm with EXPLAIN ANALYZE that you collapsed the round-trips, and verify the join column is indexed. The senior add-on is HOW you would have caught it before prod: a per-request query counter or a distributed trace (OpenTelemetry) showing the 51-span fan-out. Tracing-first debugging is a strong signal for backend roles.

Compare token bucket vs sliding-window-log vs sliding-window-counter and pick one with a reason (token bucket is the common default — smooth, allows controlled bursts, O(1) state). The distributed crux is that stateless servers cannot each keep a local count, so state lives in a shared store, typically Redis with atomic INCR + EXPIRE (or a Lua script for an atomic check-and-decrement). The differentiator is the failure question: if Redis is unavailable, do you fail OPEN (allow traffic, protect availability, risk overload) or fail CLOSED (block, protect the backend, risk a self-inflicted outage)? State the choice and the trade-off explicitly. Finish with correct semantics: HTTP 429 plus a Retry-After header, and separate tiers for free vs paid users. Vague "we use Redis" without naming the atomic operation and the partition behavior underperforms.

The senior pattern is expand-and-contract (a.k.a. parallel change), and the failure you are avoiding is a long-held table lock or a rewrite that stalls writers. Sequence: (1) EXPAND — add the column nullable / with no default rewrite (on modern Postgres a plain nullable add is metadata-only; avoid a volatile DEFAULT that forces a full rewrite); (2) deploy code that writes BOTH old and new paths; (3) BACKFILL existing rows in small batched chunks to avoid one giant locking transaction and replication lag; (4) add the NOT NULL constraint as NOT VALID then VALIDATE separately (Postgres) to dodge a full-table lock; (5) flip reads to the new column; (6) CONTRACT — drop the old column in a later deploy. Mention online-DDL tooling (pg_repack, gh-ost / pt-online-schema-change for MySQL), testing the migration on a production-sized replica, and a rollback plan for every step. Backwards-compatibility of each intermediate state is the whole point — every deploy must be safe to roll back.

Async via a queue (SQS, Kafka, RabbitMQ) buys decoupling, load leveling (absorb spikes), and retriable durability — choose it when the caller does not need the result in its own response path (sending email, processing an upload, fanning out events). But the queue is not free: you now own at-least-once delivery (so consumers must be idempotent — tie this back to the idempotency answer), ordering (per-partition/FIFO vs global), poison messages and dead-letter queues, consumer lag and backpressure, and the fact that "it succeeded" now means "it was accepted," not "it completed." The senior framing is the sync-vs-async trade as a product decision: synchronous gives the user an immediate definitive answer at the cost of coupling and tail latency; asynchronous gives resilience and throughput at the cost of eventual confirmation and more moving parts you must observe.

Show you choose by constraint, not by fashion. REST: ubiquitous, cache-friendly via HTTP semantics, great for a public API with diverse unknown clients — usually the right call for the third-party surface. gRPC: binary Protobuf over HTTP/2, strict schemas, low latency, streaming — the common pick for INTERNAL east-west microservice traffic where you control both ends and want speed plus contract enforcement. GraphQL: solves client over/under-fetching for rich, varied front-end data needs, but you take on caching complexity (no free HTTP caching), field-level authorization, and the N+1 resolver problem (mitigated with DataLoader). Name the trade you accept with each. Mentioning that large companies run more than one (a public REST/GraphQL edge with gRPC internally) signals real-world exposure rather than a single-tool worldview.

Separate the three concerns cleanly — interviewers notice when a candidate conflates authentication with authorization. AuthN (who you are): OAuth 2.0 / OpenID Connect, short-lived JWT access tokens with refresh-token rotation, validated signature + expiry on every request. AuthZ (what you may do): RBAC or ABAC, enforced server-side per request — never trust a client-supplied role claim without verifying it. Transport + storage: TLS 1.3 in transit, encryption at rest, secrets in a manager (not in env files committed to the repo). Then proactively name input-boundary defenses without being asked — parameterized queries / prepared statements (SQL injection), output encoding (XSS), and the relevant OWASP Top 10 items. Raising security before the interviewer prompts you is a recognized senior tell on backend loops.

Backend behavioral rounds reward data-integrity stories specifically, because correctness incidents are harder and scarier than latency ones. Pick a real example with shape: the symptom (duplicate charges, a counter that drifted, orders stuck between two services), how you detected it (an alert, a reconciliation job, a customer report), how you triaged WITHOUT making it worse, the true root cause (a missing idempotency key, a dual-write with no outbox, a consistency assumption that broke under a partition), and — the part seniors are graded on — the structural fix you drove afterward: the idempotency key you added, the outbox you introduced, the reconciliation check, the alert on drift. "I stayed up and manually patched the data" is a junior narrative; "I added a uniqueness constraint + idempotency key and a nightly reconciliation so the class of bug is now impossible" is the senior one.