[{"content":"FoundationDB is a transactional database with an unusual shape: its architecture separates transaction processing from storage, allowing the two to scale independently. There is another database, LeanXcale, that took the same route. The two share enough that they look like twins who agreed on the overall approach, were then separated, and made slightly different decisions because they were solving different problems. That story is for another post. Here the goal is to introduce FoundationDB\u0026rsquo;s components by following a transaction through them, from the moment it starts to the moment its writes become readable. The next post will explain FoundationDB\u0026rsquo;s five-second read-version window and how it constrains transactions.\nThe first architectural point is that FoundationDB is a transactional key-value store. There is no SQL engine inside it. A SQL interface exists — the Record Layer\u0026rsquo;s relational layer — but it is a library that runs inside the application, not a server-side component.\nThe second distinctive point is that there is no single server side that performs operations on the client\u0026rsquo;s behalf. The client library talks to the storage servers directly, and to two components that orchestrate transactions: GRV proxies, which provide read versions, and commit proxies, which commit transactions.\nFigure 1. FoundationDB's components and the messages exchanged during a transaction, from read version to readable.\nThe figure names the actual requests the components exchange. You don\u0026rsquo;t need any of them to follow what comes next — they are there for a second reading, and for the next post.\nVersions FoundationDB retains multiple versions of key-value data so that transactions can read consistent snapshots. Versions are tagged with monotonically increasing commit versions that fix the order in which transactions serialize. The component that hands out those versions is the master, and there is one active master at a time.\nTo keep client applications from overwhelming a singleton, they never talk to the master directly — the proxies do it for them.\nThis gives each transaction a snapshot of the database as of its start. A client starting a transaction asks a GRV proxy for a read version. The proxy batches these requests and fetches one version from the master for the whole batch, which is what keeps traffic to the singleton bounded.\nExecution Reads go straight to the storage servers, carrying the read version. A storage server returns, for each key, the value with the highest commit version less than or equal to that read version.\nExample. Suppose key k5 has three versions:\nVersion Commit version k5 v1 10 k5 v2 15 k5 v3 20 A transaction with read version 16 reading k5 gets v2: 15 is the highest commit version for k5 that is less than or equal to 16. Version 3 committed at 20, after the transaction\u0026rsquo;s snapshot, so the transaction cannot see it.\nWrites are different: they are buffered in the client. To give read-your-own-writes semantics, reads have to be merged with whatever is sitting in that write buffer.\nExample. Suppose storage holds k0, k1, k8 and k9, and the transaction writes k5.\nA point read of k5 returns the value the transaction just wrote. k5 does not exist in storage at all, so without the merge the read would come back empty.\nA range read over [k1, k9) returns k1, k5 and k8. Two of them, k1 and k8, come from the storage servers; k5 comes from the write buffer and has to be spliced into the result in key order. k0 falls before the start of the range and k9 is its exclusive end.\nCommit A read-only transaction is finished at this point — there is nothing left to do.\nIf the transaction wrote, the client asks a commit proxy to do the rest.\nThe commit proxy first obtains a commit version from the master. It then sends the transaction\u0026rsquo;s conflict ranges to the resolvers, which check whether any write committed after the transaction\u0026rsquo;s read version intersects its read conflict ranges. That check is what enforces isolation. The guarantee is strict serializability: read versions come from a single master and reflect everything committed before them, so a transaction that begins after another has committed is guaranteed to observe it.\nIf no conflict is found, the write set — mutations, in FoundationDB\u0026rsquo;s vocabulary — is made durable in the transaction logs. That is the point of no return: once it completes, the transaction is committed.\nStorage servers continuously pull mutations from the transaction logs and apply them asynchronously. Transactions whose read version is at or beyond a mutation\u0026rsquo;s commit version observe that change; if a storage server has not yet reached the requested read version, the read waits until it catches up.\n","permalink":"https://ricardojimenezperis.com/systems/foundationdb-architecture/","summary":"FoundationDB separates transaction processing from storage. Rather than list its components, this post follows a single transaction through them, from the moment it starts to the moment its writes become readable.","title":"FoundationDB: the architecture, seen through a transaction"},{"content":"I\u0026rsquo;m Ricardo Jiménez-Peris — Ric. I\u0026rsquo;ve spent most of my career researching and building distributed database systems.\nFor 25 years I was an academic researcher at the Universidad Politécnica de Madrid, working on distributed databases, especially replication and MVCC. I also taught distributed systems and spent a postdoc at ETH Zürich with Gustavo Alonso. During the last five of those years, I built the prototype that eventually became LeanXcale.\nI then spent 11 years building LeanXcale as a company, leading it as CEO and CTO. LeanXcale is a distributed HTAP SQL database, and taking a database from research prototype to a system running real workloads changes the questions you ask: not only whether an idea is correct, but how it behaves under load, how it fails, how it scales, and which architectural decisions are difficult to undo.\nI\u0026rsquo;m currently working on Beyond 5 Seconds, a public open-source project around FoundationDB exploring how to extend MVCC beyond its current five-second transaction window.\nThis site is where I take database systems apart and look at the ideas behind them. The first series focuses on FoundationDB — following its architecture and transaction path in detail, and comparing its design choices with those of other databases.\nIf you want to talk about any of this, LinkedIn is the easiest way to reach me.\n","permalink":"https://ricardojimenezperis.com/about/","summary":"\u003cp\u003eI\u0026rsquo;m \u003ca href=\"https://www.linkedin.com/in/ricardojimenezperis/\"\u003eRicardo Jiménez-Peris\u003c/a\u003e — Ric. I\u0026rsquo;ve spent most of my career researching and building distributed database systems.\u003c/p\u003e\n\u003cp\u003eFor 25 years I was an academic researcher at the Universidad Politécnica de Madrid, working on distributed databases, especially replication and MVCC. I also taught distributed systems and spent a postdoc at ETH Zürich with Gustavo Alonso. During the last five of those years, I built the prototype that eventually became \u003ca href=\"https://www.leanxcale.com/\"\u003eLeanXcale\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eI then spent 11 years building LeanXcale as a company, leading it as CEO and CTO. LeanXcale is a distributed HTAP SQL database, and taking a database from research prototype to a system running real workloads changes the questions you ask: not only whether an idea is correct, but how it behaves under load, how it fails, how it scales, and which architectural decisions are difficult to undo.\u003c/p\u003e","title":"About"}]