Site Reliability Engineer Interview Prep Guide

Define SLIs: availability (successful requests / total requests), latency (p99 < 200ms), and correctness (accurate payment processing). Set SLOs: 99.99% availability, 99.9% latency compliance. Error budget = 1 - SLO target. Policy: when error budget is exhausted, freeze feature releases and focus on reliability work. Discuss burn rate alerts that predict error budget exhaustion before it happens.

Start with symptoms: which services are affected, what metrics are anomalous. Check the dependency graph to identify the root cause service. Investigate: resource exhaustion, retry storms, connection pool exhaustion, or a slow dependency causing upstream timeouts. Immediate mitigation: circuit breakers, load shedding, emergency scaling, or traffic shifting. Long-term: add backpressure mechanisms, retry budgets, and bulkhead isolation.

Monitoring is about known unknowns (predefined metrics and alerts). Observability is about unknown unknowns (ability to ask arbitrary questions about system state using metrics, logs, and traces). AIOps applies ML to reduce alert noise and correlate incidents. Build the stack: Prometheus/Thanos for metrics, OpenTelemetry for distributed tracing, structured logging with correlation IDs, and Grafana for dashboards. Discuss high-cardinality challenges and sampling strategies.

Follow the incident response process: acknowledge the page, assess impact and severity, check recent deployments (potential rollback candidate), examine error logs and metrics dashboards, identify the failing component, implement mitigation (rollback, failover, or circuit break), communicate status to stakeholders, and after resolution, write a blameless postmortem. Time-box investigation steps: if you cannot identify root cause in 15 minutes, escalate.

Design health checks at multiple levels: shallow (process is running), deep (can reach database, cache, and external APIs), and dependency-aware (aggregate downstream health). Implement circuit breaker with three states (closed, open, half-open) and configurable thresholds. The health endpoint should return structured JSON with per-dependency status, latency, and degradation level. Use goroutines or async for parallel dependency checks with timeouts.

Use error budgets: when the error budget is healthy, prioritize features; when it is being consumed, prioritize reliability. Quantify toil as a percentage of engineering time and target reducing it below 50%. Discuss how to communicate reliability costs to product teams using business metrics: downtime cost per minute, customer churn from outages, and SLA penalty risks. Frame reliability as a feature enabler, not a feature blocker.

Describe a specific incident: root cause, timeline, impact metrics, and resolution. Explain the blameless postmortem culture: focus on systemic causes, not individual blame. Show that the postmortem led to concrete action items with owners and deadlines. Mention which action items prevented future similar incidents and how you tracked completion. The best postmortems change systems and processes, not just fix the immediate bug.

Start with game days in staging: inject known failures (kill a pod, add latency, block a dependency) and observe system behavior. Graduate to production chaos with blast radius controls: target non-critical services first, run during business hours, have a kill switch, and monitor closely. Use tools like Chaos Monkey, Litmus, or Gremlin. Build a hypothesis for each experiment: "We believe the system will failover in under 30 seconds when database primary fails." Track reliability improvements over time.