Last Minute Notes
  • Github
  • Docs
  • Learn
    @sayanstwt
Lessons
0/37 done
  • SQL vs NoSQL
  • Indexes & Storage Engines
  • Replication
  • Sharding
  • Consistent Hashing
  • Object Storage & Uploads
  • Search & Inverted Indexes
Learn/System Design/Data & Storage

Sharding

8 min Intermediate 8 steps

Split data across machines and choose a key that will not hot-spot.

Nodes
1
Holds
all data
Write rate
at the limit
AppPrimaryholds everythingsize + writes 96%u1u2u3u4+8writewritewritewritewriteshard when one primary cannot hold the data or the writes, not before
request response data
In this lessonStep 1 of 8
  1. The problem
  2. Split it
  3. Range
  4. Hash
  5. Shard key
  6. Cross-shard
  7. Hot keys
  8. Recap
Step 1 / 8problem

One primary is full

Replicas scale reads, but every write still lands on one primary. When the data no longer fits, or the write rate is above what one node can take, you are stuck.

Key idea

Sharding splits data across nodes so no node holds it all. Do it when you must, not before.

←→ step·R replay·Space pause

You'll learn

  • Know when to shard: data size or write rate beyond one primary
  • Compare range and hash sharding
  • Choose a shard key: cardinality, even load, matches the main query
  • Handle cross-shard queries and hot keys
Read the full notesHLD Building Blocks, 10