Building a Database: Intro [ISSUE #01]

The database series I wanted and couldn’t find. 

“Do NOT build a database!” is the internets advice. But what if everyone listened? There is still plenty of opportunity to improve the space and I don’t believe that the last database has been built yet. 

The developer experience around databases hasn’t evolved like the application side. There is a gap - an opportunity for more integrated and intuitive solutions and I have a wishlist full of interesting concepts waiting to be explored.

So here we are, I am going to build database and document the journey here.

Will this database ever become a mainstream data solution? Probably not. Can it become a production quality solution for niche use cases? I think it’s quite possible. Will we have some fun and learn something as we go? Absolutely. 

What We're Trying to Achieve

With such an audacious plan (at least according to the internet) like this one, here is what I hope we can accomplish: 

1. A Great Educational Experience: There is a lot to learn here for everyone. In my experience, developers often view the database as a black box. There are relatively few who have the experience; and the ‘magic’ that makes them work seems elusive to many. One of our goals is to open that box to expose the foundational concepts and explore what it takes to implement an efficient and performant database engine.
2. Implement a Relational Database System with New Ideas: We will aim to implement a working relational database that introduces concepts like evolving schemas, new ways to interact with data, scalable + embedded operation and more. Even if it never sees a production workload maybe it can inspire changes in other systems.
3. A Better Developer Experience: Rethink ORMs, administration, interactions, schemas and more. Most available options are often costly, complex or both. 

The Vision at a Glance

To give a slightly better idea of what we are building, below are some high-level concepts that this project will aim to explore and implement. 

1. A scalable and embeddable relational database system with MVCC and common relational database features such as indexes and a query language.
2. Support for schema evolution with richer semantics and flexibility.  
3. A database API that improves how developers interact with the database. 
4. Built-in ability to stream data change events in real time for integration, analytics, and reactive applications.
5. Understandability as a feature to go along with our learning. Full visibility into queries, transactions, and system state through rich logging, metrics, and traceable execution paths.

This series will document the full process of building, testing and optimizing a database build from scratch. In the next post we will dive into the specifics of an initial design and plan our first steps. Along the way we will take some detours to learn from other solutions and discover how they work.

One last thought: I don't expect this series to attract a huge audience, but I do hope to learn and evolve alongside everyone who follows along. Feel free to join the conversation in the comments throughout the series.