Version Controlled SQL Database Dolt Releases 2.0 with Automatic Storage Cleanup and Compression

DoltHub has recently released Dolt 2.0, a major update to the open source version-controlled SQL database. The latest major version adds automatic storage optimization, including garbage collection and compression, along with improved support for large and vector data types.

Dolt is a MySQL-compatible SQL database with built-in Git-style version control, supporting operations such as branching, merging, cloning, and diffing. It stores data in content-addressed Prolly Trees, enabling row-level versioning, efficient structural sharing across versions, and fast diffs and merges.

Following the first major version released three years ago, Version 2.0 adds automated garbage collection and archive compression enabled by default, adaptive storage, vector support in beta, and improved performance, with the team claiming the database is now faster than MySQL on sysbench. Tim Sehn, founder and CEO of DoltHub, explains why garbage collection matters:

Dolt makes a lot of disk garbage, especially during import. Dolt is copy-on-write so all intermediate committed transaction state is preserved to disk. Any intermediate state that is not in a Dolt commit is garbage and can be collected (…) Dolt already must preserve all history in the commit graph on disk. Adding extra garbage can eat through your disk very quickly.

Source: Dolt blog

The database adds a new on-disk format called “archives” that is designed to reduce the storage footprint by 30-50% through dictionary compression to deduplicate storage. Discussing how the team used sysbench to measure and benchmark the latency of SQL queries across versions, Sehn claims:

We started at about 10X slower on reads and 20X slower on writes than MySQL. We’ve worked tirelessly to improve Dolt’s performance and we are now 13% faster than MySQL on writes and 5% faster on reads.

Dolt 2.0 also introduces beta support for version-controlled vector indexes, using MariaDB’s Vector type. According to the announcement, Dolt is the only database that version-controls vectors, and beta status will be removed once the remaining read-path gaps are fixed.

Dolt is not the only Git-like tool for data versioning, with different projects taking different approaches to versioning data without duplicating large datasets; popular alternatives include LakeFS, a data version control solution for data lakes, and Nessie, a transactional catalog for data lakes with Git-like semantics. While Dolt is generally available and provides a MySQL query interface, DoltgreSQL, the Postgres-compatible version of Dolt, shares the same storage engine and implements the same version control interfaces, but is currently in beta.

Simon Späti wrote a series of articles explaining what a Git-inspired approach to data is and comparing the different available options. Späti writes:

Git-like workflows are becoming table stakes. Maybe not today or tomorrow, but with the right tools and changes in workflow we can achieve significantly better change management, testing on production data, fast rollbacks, isolated experiments, and most importantly, peace of mind when deploying changes.

Dolt is available on GitHub under an Apache 2.0 license.