Last Minute Notes
  • Github
  • Docs
  • Learn
    @sayanstwt
Lessons
0/37 done
  • Timeouts, Retries & Breakers
  • Multi-Region Design
  • Cells & Shuffle Sharding
  • SLIs, SLOs & Error Budgets
  • Observability
  • Deployment Strategies
  • Backups & Disaster Recovery
Learn/System Design/Reliability & Operations

Timeouts, Retries & Breakers

8 min Intermediate 8 steps

Stop one slow dependency from taking the whole system down.

Timeout
nonelibrary default: infinite
Threads busy
3 / 10
Callers
waiting
ClientServicecalls a dependencythreads 3/10Dependencyslow todayslowreqreqreqreq
request response data
In this lessonStep 1 of 8
  1. The problem
  2. Deadlines
  3. Retries
  4. Amplification
  5. Breaker
  6. Bulkheads
  7. Metastable
  8. Recap
Step 1 / 8problem

One slow dependency, one dead service

Assume every dependency will be slow or broken. Here the dependency is only slow, yet the service calling it has no timeout. The default in many libraries is infinite.

Each request parks a thread on the slow call until none are left.

Key idea

Resilience patterns exist to keep one failure from becoming an outage.

←→ step·R replay·Space pause

You'll learn

  • Why every network call needs a timeout and a propagated deadline
  • Retry safely: idempotent calls, backoff with jitter, one layer, a budget
  • How a circuit breaker moves between closed, open and half-open
  • Contain damage with bulkheads and load shedding
  • Recognise a metastable failure and how to escape it
Read the full notesDistributed Systems, 10