Andrew’s Distributed Systems Blog

Introduction

I am a senior software engineer with an interested for large scale distributed systems in the platform/infrastructure space. My current interests really center around databases, streaming platforms, workflow engines, and other systems which at their core are responsible for data management.

I am starting this blog as a place to record my learnings about distributed systems. For the foreseeable future each blog post will center around some interesting technology in industry.

Upcoming Posts

• Spanner: Highly available, externally consistent, horizontally scalable database from Google.
• BigTable: This came before Spanner, and was one of the pieces of tech that lead Google to realize they needed Spanner.
• MegaStore: This was a database that came out at Google before Spanner. And it shared some vague similarities with Spanner.
• Colossus: This is the file system the backs Spanner.
• Google File System: This is the filesystem that came before Colossus.
• CockroachDB: This is a database that is open source that is based on Spanner. The goal is to provide similar abstractions without requiring use of atomic clocks.
• RocksDB: A key value store that CockroachDB uses.

Technologies to Write About

• Cadence
• Spanner
• Kafka
• ZooKeeper
• Chubby
• S3
• GFS
• MapReduce
• ElasticSearch
• Redis
• BigTable
• Rocks DB
• Dynamo DB
• Cassandra
• MemSQL
• Cockroach DB
• Megastore
• M3
• BigQuery
• Firestore
• MongoDB
• Kinesis
• QLDB
• AWS SQS
• Apache Kudu
• Apache HBase
• Voldemort DB
• Neo4j
• Memcached
• Facebook Gorilla Time Series DB
• Vitess
• Presto
• Apache Spark
• Apache Flink
• Apache Pinot
• Kubernetes

In order to find more things to write about and as a starting point to understand technologies I can use the following resources

• https://projects.apache.org/projects.html?category#database
• Youtube videos from Amazon ReInvent
• Youtube videos from Google Next
• https://aws.amazon.com/ pick a technology and click on resources
• https://cloud.google.com/products
• https://github.com/gregsramblings/google-cloud-4-words

For each one of these technologies fundamentally there are three questions I want to try to answer.

What problem does it solve?

1. What is the problem space?
2. What are sample use cases?
3. What interface does it expose to users?
4. What other projects are similar to this one?
5. What are the guarantees provided by the system?

How is it built?

1. What is the high level architecture?
2. How are failures handled?
3. How is replication handled?
4. How are the guarantees provided?

What can we learn from it?

1. What did this system do well?
2. What could have it done better?
3. What future features might be added to this system?
4. What learnings about industry/software can we take from this system?

Exploring

While right now I want this blog to focus on technologies in industry, there are several topics I am interested in that do not fit squarely into the above framework. The following is a brain dump of things for me to explore and potentially write posts about.

• SWIM protocol
• RAFT protocol
• Paxos protocol
• Distributed Priority Queuing
• Time Series Databases
• Consistent Hashing
• Graph Databases
• Location Databases
• Automatically consistent types like counters, or sets
• Blockchain