Blog 5 — Discord: How We Store Billions of Messages

12015: MongoDB
2  └── Hits scaling limits → too slow for message history
3         │
42017: Apache Cassandra
5  └── Works but operational complexity grows
6         │
72023: ScyllaDB (Cassandra-compatible, written in C++)
8  └── Current solution — 4x better performance

1MongoDB stores messages as documents:
2  {
3    channel_id: "123",
4    messages: [ msg1, msg2, msg3... ]
5  }
6
7Problem:
8  - Hot channels get huge documents
9  - Random access to old messages = full document scan
10  - No native time-series support
11  - Uneven data distribution across shards

1Partition Key: (channel_id, bucket)
2Clustering Key: message_id (snowflake timestamp-based)
3
4Example:
5  Partition: (channel_123, 2024-01)
6    Row: msg_id=1710000001 → "Hey!"
7    Row: msg_id=1710000002 → "How are you?"
8    Row: msg_id=1710000003 → "Great thanks!"
9
10Fetch last 50 messages:
11  SELECT * FROM messages
12  WHERE channel_id = 123 AND bucket = '2024-01'
13  ORDER BY message_id DESC LIMIT 50
14  → Single partition read → extremely fast

1Cassandra: Written in Java → JVM GC pauses → latency spikes
2ScyllaDB:  Written in C++ → no GC → predictable low latency
3
4Result at Discord:
5  - 4x better performance
6  - Same Cassandra query language (CQL)
7  - No application code changes needed
8  - Fewer nodes needed → cost reduction