Git at any scale

Hosting Git repositories at scale is a nightmare. When Linus Torvalds designed the first version of the information manager from hell (that’s actually the tagline for Git, look it up), he had a very specific use case in mind: his own. He wanted to replace BitKeeper, the distributed version control system that was being used to develop the Linux Kernel. Of course, the replacement had to be distributed too. The Kernel is an unusual software project; it is extremely decentralized, with many different maintainers for its many different subsystems. A distributed version control system is a natural fit for this workflow.

大规模托管 Git 存储库是一场噩梦。Linus Torvalds 设计第一个版本的 the information manager from hell（这确实是 Git 的自我调侃口号，查一下吧）时，心里想的是一个非常具体的用例：他自己的用例。他想替换掉 BitKeeper——当时用于开发 Linux 内核的分布式版本控制系统。当然，替代品也必须是分布式的。内核是一个不寻常的软件项目：它极度去中心化，众多子系统各有各的维护者。分布式版本控制系统天然适合这种工作流。

Twenty years later, Git has become an industry standard, but the truth is that its distributed nature is more of a hindrance than an advantage. The average open-source software project doesn’t operate with a decentralized workflow. The average company definitely doesn’t. They use the many advantages of the distributed model (such as being able to work offline, delay pushes, etc) but they very much rely on a centralized host. And hosting a Git repository, it turns out, is an incredibly hard thing to do.

二十年后，Git 已经成为行业标准，但事实是，它的分布式特性更多是一种阻碍而非优势。普通的开源软件项目并不会以去中心化的方式运作，普通公司更不会。它们享受着分布式模型的诸多好处（例如可以离线工作、稍后再推送等），但本质上仍然依赖一个集中式托管方。而事实证明，托管一个 Git 存储库是一件极其困难的事。

What’s hard about Git?

Git 有什么难的？

The challenge in hosting Git repositories at scale is inherent in the design of Git itself: a distributed version control system means that all instances of a repository are identical. There’s nothing special about the repository on a Git server that doesn’t apply to a repository on a developer’s laptop. Although at first it may appear that this makes hosting Git repositories straightforward (simply put an HTTP daemon in front of an on-disk copy of a repository and you’ve got a Git server going!), there are many hard scalability and reliability challenges that make this quite the opposite.

大规模托管 Git 存储库的挑战根植于 Git 自身的设计：分布式版本控制系统意味着存储库的所有实例都是相同的——Git 服务器上的存储库与开发者笔记本上的存储库没有任何本质区别。乍看之下，这似乎让托管 Git 存储库变得简单（只需在磁盘上的存储库副本前挂一个 HTTP 守护进程，一个 Git 服务器就诞生了！），但大量棘手的可扩展性与可靠性挑战让事情恰恰相反。

In a normal Git repository, your code and metadata (files, commits, trees) are compressed and stored in packfiles — a simple binary serialization format which is convenient to deal with on a local machine, but not ideal to manage at scale on a server. Packfiles are the fundamental building block of Git storage and Git networking. When you push or fetch data from a repository, it’s transferred as a packfile.

在普通 Git 存储库中，您的代码与元数据（文件、提交、树）被压缩进 packfile——这是一种简单的二进制序列化格式，在本地计算机上处理很方便，但不适合在服务器上进行大规模管理。Packfile 是 Git 存储和 Git 网络传输的基本构建块：当您从存储库推送或获取数据时，数据就是以 packfile 形式传输的。

This is how Git works by design, but it would be fair to think that it needn’t be that way. After all, you do not control the Git client (at least not without annoying your users and adding a lot of friction), but within the walls of your own server, you can do anything you want. Nothing ties you to using packfiles — Linus is not going to come over and check. The only restriction is that you do need to receive and send packfiles over the network for all Git operations.

这就是 Git 按设计工作的方式，但说句公道话，它完全可以不必如此。毕竟，您无法控制 Git 客户端（至少在不惹恼用户、不增加大量摩擦的前提下），但在您自己的服务器内部，您可以做任何想做的事。没有任何东西强迫您使用 packfile——Linus 不会跑来检查。唯一的限制是：所有 Git 操作都必须通过网络接收和发送 packfile。

Over the years, companies that tried hosting Git repositories at scale noticed that this packfile -based design was a major limitation on both availability and scalability. Packfiles are large binary files that must exist on a filesystem for Git to access them. The simple approach of having an HTTP server in front of a repository on disk has a very low ceiling. Ideally you’d want the repository to exist on many disks and many machines (this lets you run many Git operations in parallel, and keeps your repository available when a server crashes). But how do you do that?

多年来，尝试大规模托管 Git 存储库的公司都注意到，这种基于 packfile 的设计在可用性和可扩展性上是重大瓶颈。Packfile 是大型二进制文件，必须存在于文件系统上，Git 才能访问。在磁盘上的存储库前面放一个 HTTP 服务器，这种简单方案的上限非常低。理想情况下，您希望存储库同时存在于许多磁盘和许多机器上（这样既能并行执行大量 Git 操作，也能在服务器崩溃时保持存储库可用）。但怎么做得到呢？

There are broadly three possible approaches to accomplish this, in increasing order of complexity: distribute the filesystem, distribute the packfiles, or distribute Git itself.

实现这一点大体上有三种方法，按复杂度递增排列：分发文件系统、分发 packfile，或者分发 Git 本身。

Git without packfiles

不使用 packfile 的 Git

Git is a content-addressable data store. All objects in a Git repository (blobs, trees, commits, etc) are keyed by the SHA-1 of their contents. This is something that intuitively maps very well to a distributed key-value store (the key is the SHA-1; the value is the actual object), and could provide a clean way to scale out the storage of a repository. But this actually doesn’t work

Git 是一个内容可寻址的数据存储。Git 存储库中的所有对象（blob、树、提交等）都以内容的 SHA-1 作为键。这直观上非常适合映射到分布式键值存储（键是 SHA-1，值是实际对象），本可以为存储库的水平扩展提供一条干净路径，但实际上行不通。

Here’s the issue: the actual layout of a Git repository is a directed acyclic graph (DAG for short). You can look up any object via its SHA, but to perform even the most trivial operation in the repo, you must actually walk the DAG step by step.

问题在于：Git 存储库的实际结构是有向无环图（directed acyclic graph，简称 DAG）。您可以通过 SHA 查找任何对象，但要在存储库中执行哪怕最简单的操作，都必须一步步遍历 DAG。

If you want to do an operation like listing the recent changes in a repository, you must process its commits. When you process a commit, you get a pointer to the root of its tree. From that tree, you get pointers to each file and each subtree. From the original commit, you get a pointer to its parent (the one that comes before it in the history). Crucially, at every step of this walk, you don’t know the value of the next pointer until you fetch the previous one. If every fetch requires a round trip to a distributed store, things become very expensive very fast.

要执行「列出存储库最近更改」这样的操作，您必须处理一系列提交：处理一个提交时，会得到指向其树根的指针；从树中得到指向每个文件和每个子树的指针；从原始提交中得到指向其父提交（历史中前一个提交）的指针。关键在于，遍历的每一步，在取到前一个指针之前，您都不知道下一个指针的值。如果每次取指针都要与分布式存储往返一次，成本很快就会高得离谱。

This approach to distributing Git at the object level has been tried before, many times, and it often fails at scale. The most promising implementation was attempted by my former mentor Shawn Pearce when he was working on the version control systems team at Google. His approach was storing the objects in a distributed hash table. This was only possible thanks to JGit, a custom Git implementation in Java. Like any good ol’ Java library, JGit provides enough interfaces and factories and interface factories to abstract all the details of a normal Git repository, including replacing its on-disk packfiles with a DHT. Although the system worked and results were good enough for normal Git operations, the limitations of the Git protocol (which again, require packfiles to be sent over the network regardless of how you store data on the server) made the git clone performance bad enough to discard the design altogether.

在对象级别分发 Git 的做法以前被尝试过很多次，而且往往在大规模下失败。最有前景的实现来自我的前导师 Shawn Pearce，他在 Google 的版本控制系统团队工作时，尝试把对象存进分布式哈希表。这之所以可行，全靠 JGit——一个用 Java 实现的自定义 Git。和任何老牌 Java 库一样，JGit 提供了足够的接口、工厂和接口工厂，把普通 Git 存储库的所有细节都抽象掉，包括用 DHT 替换磁盘上的 packfile。这套系统能跑，普通 Git 操作的效果也够好，但 Git 协议的限制（无论服务器端怎么存数据，网络传输仍要求 packfile）让 git clone 的性能差到足以让整个设计被放弃。

GitHub and filesystems

GitHub 与文件系统

A couple years after Git started to escape its Linux Kernel bubble, a scrappy startup was born in San Francisco. GitHub was founded in 2008 as a social coding platform with a very prescient tagline, “Git repository hosting: no longer a pain in the ass.” I’m not joking here either, look it up. There was, all the way back in 2008, a broad consensus that despite (or perhaps because of) Git’s distributed design, you actually needed a centralized way to host Git repositories to make them user-friendly, and doing this was very painful. GitHub was set on changing that

Git 开始走出 Linux 内核的小圈子几年后，一家干劲十足的初创公司在旧金山诞生了。GitHub 成立于 2008 年，是一个社交编码平台，口号极具先见之明：「Git 存储库托管：不再是一件麻烦事」。我可不是在开玩笑，去查一下吧。早在 2008 年，人们就普遍认同：尽管（或许恰恰因为）Git 采用分布式设计，要让大家用得舒服，Git 存储库实际上仍需要一个集中式托管方式——而这件事做起来非常痛苦。GitHub 立志改变这一切。

Its platform started as (and mostly still is) a Rails monolith. The very first versions were running off a single, albeit beefy, machine, with a Ruby server and copies of the repositories on disk next to it. Scaling a Rails app is easy: deploy more instances of it. But in this particular case, since Git is involved, they quickly ran into the recurring question we’re trying to solve here: If the Rails app needs to access the Git repositories on disk, how do you deploy more copies of them?

它的平台最初是（很大程度上现在仍是）一个 Rails 单体应用（monolith）。最早的版本跑在一台（虽然结实但只有一台）机器上：一个 Ruby 服务器，旁边磁盘上放着存储库副本。扩展 Rails 应用很容易：多部署几个实例就行。但在 Git 这个特殊场景下，他们很快撞上了我们这篇文章反复要解决的老问题：如果 Rails 应用需要访问磁盘上的 Git 存储库，怎么把存储库也部署出更多副本？

Being a thrifty bunch of misfits, the early systems engineers at GitHub tried the simplest approach that could possibly fix their scaling problems. The thinking was that, if they focused on distributing the filesystem (instead of packfiles, or Git itself), they could keep the Rails app unchanged and spend their time shipping more features for the ever-growing user base, instead of doing weird stuff with Git. Very pragmatic. It didn’t work.

作为一群精打细算的怪咖，GitHub 的早期系统工程师尝试了可能解决扩展问题的最简单方案。思路是：如果专注分发文件系统（而不是 packfile 或 Git 本身），就能保持 Rails 应用不变，把时间花在为不断增长的用户群交付更多功能上，而不是跟 Git 较劲。非常务实。但行不通。

The team attempted many approaches to a distributed filesystem for Git data: the most obvious one, using NFS to store all repositories on a centralized server, was quickly discarded. The default implementation of Git makes a lot of assumptions about filesystem semantics (locking, tearing, reading, syncing…) that ensure decent performance on the local filesystem of a slow developer laptop, but pay no attention to how they behave over a networked filesystem. It was slow, and it was buggy.

团队尝试了多种为 Git 数据构建分布式文件系统的方案：最直接的一种是用 NFS 把所有存储库放在集中式服务器上，很快就被放弃了。Git 的默认实现对文件系统语义（锁、写入撕裂、读取、同步……）做了大量假设，这些假设保证了慢速开发者笔记本电脑上的本地文件系统有不错的性能，却完全不管它在网络文件系统上的行为。这套方案又慢又有 bug。

Further attempts were made with (frankly, in retrospect, horrific) technologies that replicated the filesystem at the block level. A short-lived deployment with GFS. A longer-lived deployment based on DRBD. They all hit a wall. They were terrible to operate day to day, and they didn’t make up for it with good performance. It all boils down to the design of packfiles on disk.

他们还尝试过在块级别复制文件系统的技术（坦率地说，事后回想，这些技术堪称恐怖）：一个短命的 GFS 部署，一个活得更久的基于 DRBD 的部署。全都撞了墙。它们日常运维糟糕透顶，性能也弥补不了。一切都要归结到磁盘上 packfile 的设计。

We’ve already seen how Git’s graph-like data structures make round-trips prohibitively expensive. Unfortunately, a very similar principle also applies to the underlying data on-disk. There is no correlation between the layout of objects in the DAG and the way they’re placed in a packfile. The key heuristic used when generating packfiles is minimizing their size; objects are placed randomly throughout the pack, they are compressed, and crucially they’re rarely stored whole. Most objects are stored as a delta on top of another object in the same packfile. Reading an individual object, after following the many logical hops in the graph data structure, also involves following physical hops in the on-disk format.

我们已经看到，Git 图状的数据结构如何让往返访问变得极其昂贵。不幸的是，一个非常相似的原则也适用于磁盘上的底层数据：DAG 中对象的布局与它们在 packfile 中的排列方式毫无关联。生成 packfile 时最关键的经验法则是最小化体积——对象在包内随机散布、被压缩，而且关键是很少被完整存储：大多数对象都以 delta 形式叠放在同一 packfile 中另一个对象之上。因此，读取单个对象时，除了要跟随图数据结构里的许多逻辑跳转，还得跟随磁盘格式里的物理跳转。

This kind of random walk across gigabytes of data, which must happen for every single Git operation performed on a repository, just doesn’t play well with a networked filesystem (whether it replicates at the file or at the block level). The only way this works without slowing down to a crawl is if you can cache the whole file locally. But with hundreds of thousands of repositories in the same filesystem, caching is not an option.

这种跨越 GB 级数据的随机访问，是存储库上每一次 Git 操作都不可避免的，但它与网络文件系统（无论是文件级还是块级复制）天生不合。要想不慢到爬行，唯一可行的办法是在本地缓存整个文件；但当同一个文件系统里躺着数十万个存储库时，缓存根本无从谈起。

Eventually, the systems engineers at GitHub bit the bullet and gave up distributing the filesystem. They started developing an RPC system so that repositories could live on dedicated fileservers, and updated the Rails app to do all operations remotely. This provided a good chunk of horizontal scalability, but didn’t fix their availability, nor the performance for the busiest repositories. After all, every repository was still stored only on a single machine.

最终，GitHub 的系统工程师咬咬牙放弃了分发文件系统。他们转而开发 RPC 系统，让存储库驻留在专用文件服务器上，并改造 Rails 应用，使所有操作都在远端执行。这带来了可观的水平可扩展性，却没有解决可用性问题，也没有解决最繁忙存储库的性能问题——毕竟，每个存储库仍然只存在一台机器上。

Spokes and Consistency

Spokes 与一致性

Spokes was originally developed at GitHub around 2013, and it has since become an industry standard. Most Git hosting services use a variant of the Spokes approach (application-level replication for Git repositories) in their architecture. The main reason Spokes has worked well for many years is that it made three fundamental choices that, over time, have been proven to be optimal

Spokes 大约于 2013 年在 GitHub 诞生，此后成为行业标准——大多数 Git 托管服务都在架构中采用了 Spokes 方法的变体（即对 Git 存储库做应用级复制）。Spokes 多年运转良好，主要归功于三个根本性选择，而时间证明了这些选择是近乎最优的：

1. It doesn’t distribute Git itself; it works at the packfile level.

   它不去分发 Git 本身，而是在 packfile 层面工作。

2. It stores all data as actual Git repositories on local NVMe disks.

   它把所有数据都以真实 Git 存储库的形式存放在本地 NVMe 磁盘上。

3. It replicates the Git data, but keeps all copies consistently in sync.

   它复制 Git 数据，但始终保持所有副本一致同步。

Because of the random read patterns across packfiles we’ve just discussed, storing plain Git repositories on NVMe drives is basically a requirement to ensure all basic Git operations remain fast. They also keep clones efficient because you don’t have to transform the data into what the Git client expects. They also let you focus on building a product on top of Git, as opposed to maintaining a fork of Git yourself that can operate on your weird repositories.

正是由于我们刚才谈到的跨 packfile 随机读取模式，把普通 Git 存储库放在 NVMe 盘上基本是保证所有基本 Git 操作保持快速的前提。同时，这也让克隆保持高效，因为无需把数据转换成 Git 客户端期望的形态；还让团队能专注于在 Git 之上构建产品，而不是自己维护一个能处理各种古怪存储库的 Git 分支。

Keeping all the copies of the data consistently in sync is also, crucially, very good. This is something you find out the hard way, but the Git client really doesn’t play well with eventual consistency. If your local Git client pushes a commit and then fails to read it immediately after a fetch, that’s bad news. Git finds that very confusing. If you run your CI pipeline across a hundred runners and three of them don’t find the commit they’re supposed to test after cloning your repository, that’s bad news. It’s also a very poor user experience.

同样非常关键的是，让所有数据副本保持一致同步。这一点往往要吃了亏才明白：Git 客户端与最终一致性（eventual consistency）真的水火不容。如果您的本地 Git 客户端推送了一个提交，紧接着 fetch 却读不到它，那就是坏消息——Git 会觉得非常困惑。如果您在一百台 CI runner 上跑流水线，其中三台克隆存储库后找不到要测试的提交，同样是坏消息，还是极其糟糕的用户体验。

Working with an eventually consistent view of a Git repository has a lot of sharp edges, whether it’s on the client or in the backend. Hence, Spokes pays a very high complexity cost to ensure the system is always fully consistent. Let’s see exactly what this means.

无论客户端还是后端，面对一个最终一致的 Git 存储库视图，都会踩到许多尖锐的边角。因此，Spokes 为了确保系统始终完全一致，付出了极高的复杂度成本。下面来看看这具体意味着什么。

Spokes is a consensus-based distributed system. It works by storing several copies of your Git repository on different servers. Whenever you push new data, an orchestrator fans out your push so that every instance of your repository receives a copy. The “fan-out” is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it

Spokes 是一个基于共识的分布式系统：它在不同服务器上存有同一存储库的多个副本。每当您推送新数据，编排器（orchestrator）会把推送扇出（fan-out），让存储库的每个实例都收到一份副本。「扇出」由一个经典共识算法同步，即 3PC（三阶段提交，three-phase commit）：只有当大多数节点确认后，推送才会被接受。

1 · VOTING 2 · PRE-COMMIT 3 · DO COMMIT

1·投票 2·预提交 3·执行提交

Before we can talk more about the way Spokes uses 3PC, we need to understand how a Git push works. A Git push has two components: a packfile and a reference transaction. The packfile, which we’ve already talked about, contains the objects you’re pushing to the repository (blobs, trees, and commits with your changes). The transaction is what actually publishes your changes to the repository by updating one or more references (e.g. the branch you’re working on) to point to the commits you’ve just pushed.

在深入讨论 Spokes 如何使用 3PC 之前，我们需要先理解一次 Git 推送（push）的工作方式。一次推送包含两个部分：packfile 和引用事务（reference transaction）。packfile 我们已经讨论过，它包含您推送到存储库的对象（blob、tree，以及包含您改动的 commit）；事务则通过更新一个或多个引用（比如您正在操作的分支）指向刚推送的提交，真正把您的改动发布到存储库。

This separation is very convenient here, because a pushed commit is not visible (“reachable” in Git parlance) until the reference that points to it has been updated. This means we can implement consensus for our pushes by fanning out the packfiles to all hosts simultaneously (we don’t need to synchronize here) and then doing three-phase commit with the reference transaction, which is much smaller and faster to synchronize than the packfile. Git itself has support for preparing reference transactions: it can acquire a lock on the reference, verify that the existing value is what’s expected, and then hold the lock until it receives a commit or an abort command for the transaction.

这种分离非常方便：因为推送的提交在指向它的引用更新之前是不可见的（用 Git 术语说，即尚不可达（unreachable））。这意味着我们可以先把 packfile 同时扇出到所有主机（这一步不需要同步），然后对引用事务做三阶段提交——后者比 packfile 小得多，同步也快得多。Git 本身支持准备引用事务：它可以获取引用上的锁、校验现有值是否符合预期，然后一直持有锁，直到收到事务的提交或中止命令。

Playback speed 0.010x

播放速度 0.010 倍

Replicas 5

副本 5

One-way latency 20ms

单向延迟 20 毫秒

With this design, we ensure that every push is fully synchronized across all the replicas. Reads (fetches, clones) can then be safely routed to any single replica, because every replica is always up to date.

通过这种设计，我们保证每一次推送都在所有副本间完全同步。于是读操作（fetch、clone）可以安全地路由到任意一个副本，因为每个副本始终是最新的。

This is essentially how Spokes works, and it has been working quite well for the past 13 years. Of course, Spokes is not perfect — no system is. In 2026, the way people use Git repositories has changed drastically, and we have learned many important lessons about building distributed systems along the way. Time and experience have shown which of Spokes’s choices turned out to be optimal, and which did not.

这基本上就是 Spokes 的工作原理，而且过去 13 年它运行得相当好。当然，Spokes 并不完美——没有任何系统是完美的。到了 2026 年，人们使用 Git 存储库的方式已经发生巨变，我们也在这一路上学到了许多构建分布式系统的重要教训。时间和经验证明了 Spokes 的哪些选择是最优的，哪些不是。

One flaw that has turned out to be critical is the constrained horizontal scalability of 3PC. When Spokes was initially released, three replicas per repository was the sweet spot. You could serve your average repository from three copies with capacity to spare, with enough redundancy to keep accepting pushes even if one machine went down.

其中一个被证明致命的缺陷，是 3PC 的水平可扩展性受限。Spokes 刚发布时，每个存储库三个副本是最佳配置：三个副本服务一个普通存储库绰绰有余，即使一台机器宕机，冗余也足以继续接受推送。

In 2026, things look very different. The average repository for an enterprise company is now a massive monorepo. Three replicas are not enough to serve the traffic for such repos, particularly when it comes to CI. Of course, nothing stops Spokes from running with more than three replicas, except the dreaded tail at scale. Three-phase commit maps very elegantly to the Git transaction model, but as a consensus algorithm, it has fundamental limitations: the latency of every step is bound by the slowest of all the servers in the cluster. The more replicas you add to a cluster, the worse push throughput gets.

到 2026 年，情况已大不相同：企业级公司的「平均」存储库现在往往是庞大的 monorepo，三个副本根本扛不住这类仓库的流量，尤其是 CI。当然，Spokes 不是不能跑超过三个副本，除非你惧怕规模化尾部延迟（tail at scale）。三阶段提交与 Git 事务模型优雅契合，但作为共识算法它有根本性局限：每一步的延迟都被集群中最慢的服务器卡住。副本加得越多，推送吞吐量越差。

This scalability constraint also applies the other way. When agents work with Git repositories at scale, they often operate outside of a monorepo by creating vast numbers of small repositories, many of them throwaway, and most of them barely touched. Spokes struggles here because it still requires three replicas for every one of these repositories. Three mostly idle replicas, which cannot be trimmed down because then the system wouldn’t be fully consistent and data loss would be possible. With three-phase commit, the floor is always too high, and the ceiling too low.

这种可扩展性约束还有另一面。当 agent 大规模使用 Git 存储库时，它们往往跳出 monorepo，创建海量小仓库，其中很多是即用即弃的，大多数几乎没人碰。Spokes 在这类场景也很吃力：因为每个仓库仍然要有三个副本——三个大多闲置、又不能缩减的副本，一旦缩减，系统就不再完全一致，数据就可能丢失。三阶段提交让下限总是太高，而上限又太低。

Another flaw, impossible to see up front, but painfully obvious after having suffered through it, is that Spokes can be rough to operate at scale. Because the repositories on disk are always the source of truth for consensus, every copy of every repository is very important. You have to treat repositories as pets, not cattle.

另一个缺陷一开始看不出来，但吃尽苦头后才会痛彻心扉：Spokes 在大规模下很难运维。因为磁盘上的存储库始终是共识的事实来源，每个存储库的每一个副本都极其重要——您必须把存储库当*宠物（pets）养，而不是当牲口（cattle）*赶。

This means, for starters, that you need to know exactly where every repository is. This adds a dependency (and a potential availability issue) on an external database that must keep a very large routing table mapping every repository to every machine where it’s replicated. Every repository must also be checksummed, and its checksums constantly updated in that table, to ensure the repository remains valid on disk. As soon as something bad happens to the repository (and trust me, bad things happen all the time — Git can be very finicky in practice), you must detect it and schedule a repair job to bring it back to a healthy state. And you must do it very quickly! Because, again, the repositories on disk are the source of truth. A corrupted copy is as bad as a missing one. If two of the three copies are corrupt, the system can no longer accept pushes: there’s no quorum.

首先，这要求您确切知道每个存储库在哪里：一个外部数据库必须维护一张极其庞大的路由表，把每个存储库映射到它被复制到的每一台机器上——这既增加了依赖，也埋下了可用性隐患。每个存储库还必须做校验和，并在表里持续更新，以确保它在磁盘上保持有效。一旦存储库出了事（相信我，坏事源源不断——Git 在实践中非常挑剔），您必须立刻察觉并安排修复作业把它恢复到健康状态，而且动作要快！因为磁盘上的存储库就是事实来源：一个损坏的副本与一个丢失的副本同样糟糕。如果三个副本里坏了两个，系统就再也无法接受推送——没有法定多数（quorum）了。

Continuity

Continuity（Cnt）

Continuity (Cnt for short) is the Git storage system we’ve developed at Cursor, with a very clear approach: learning from everything that Spokes did well, and fixing the things that, after many years, we now know are problems

Continuity（简称 Cnt）是我们在 Cursor 开发的 Git 存储系统，思路非常明确：把 Spokes 做得好的地方全部继承，再修复那些多年后已经确认是问题的地方。

Cnt is a simple system (a system cannot be easy to operate if it is not simple). The core primitive behind it is a write-ahead log, which we store in S3-compatible object storage. In production, we run directly on S3, but we designed it so it can be deployed on any cloud

Cnt 是一个简单的系统（一个系统如果不够简单，就不可能容易运维）。它的核心原语是预写日志（write-ahead log），存储在兼容 S3 的对象存储里。生产环境中我们直接跑在 S3 上，但设计上它可以部署到任何云。

When a repository receives a push, we store the push as a WAL entry in S3. We never acknowledge a push until it has been fully persisted. Each push is stored as a separate object; we write the pushed packfile to disk and upload it to S3 simultaneously. Uploading a WAL entry, however, does not publish it. A push is only visible once we successfully prepare its reference transaction on a local copy of the repository and record a pointer to the WAL entry in the WAL index file, which is its own object in the store. This forces all pushes to be linearizable.

当存储库收到一次推送，我们会把这次推送作为一条 WAL 记录存进 S3。**在推送被完整持久化之前，我们绝不确认（acknowledge）它。**每条推送都是独立的对象：我们把推送的 packfile 写入磁盘，同时上传到 S3。不过，上传 WAL 记录并不会发布推送——只有当我们在存储库的本地副本上成功准备了引用事务，并在 WAL 索引文件（它在存储里也是独立对象）中记下指向该 WAL 记录的指针后，推送才可见。这让所有推送都被线性化（linearizable）。

We try not to do one single S3 write per push, because in busy repositories, this puts a hard cap on push throughput based on the latency of the S3 PUT operation. With a carefully tuned batching implementation, and with the only requirement of having to synchronize the reference transaction with a single local repository instead of a quorum of replicas, we have a system that can ingest pushes as fast as our disk allows.

我们尽量避免每笔推送都做一次 S3 写入，因为在繁忙的存储库中，S3 PUT 操作的延迟会给推送吞吐量设下一个硬性上限。通过精心调优的批处理实现，加上只需将引用事务与单个本地存储库同步（而不是与副本的法定多数同步）这一要求，我们得到了一套能以磁盘允许的速度吸收推送的系统。

The local copy of the repository is, of course, a normal Git repository stored on a very fast NVMe drive. We do the same thing that Spokes does because I think Spokes got that exactly right. It allows us to reuse all the amazing OSS work of the Git community, including the upstream Git client and its many performance optimizations. It lets us focus on shipping new features, instead of doing weird stuff with Git.

当然，存储库的本地副本就是一个普通 Git 存储库，放在非常快的 NVMe 盘上。我们和 Spokes 做一样的事，因为我认为 Spokes 这一点做对了：它让我们能复用 Git 社区全部出色的开源成果，包括上游 Git 客户端及其大量性能优化；也让我们能专注于交付新功能，而不是跟 Git 较劲。

Consensus

共识

We’ve seen that one thing that makes a Spokes cluster hard to operate is that it’s very important to keep track of the location of every repository on each server. Cnt does this very differently. Where does every repository live? The answer is “anywhere”. It doesn’t matter! We treat repositories like a warm cache on disk, but the source of truth is always the write-ahead log in S3. The system is stateless, and there are no routing tables (and no relational database to operate — hashtag blessed). If a repository is missing from the local disk when accessed on a host, we just materialize it from the WAL. We can do this very efficiently, but of course we don’t want to do this all the time, because it’d be wasteful. In production, we use rendezvous hashing to map a repository ID to the list of nodes where we expect it to be. All the state we require to route repositories is the repository ID and the current set of healthy nodes in a cluster. But if this state gets out of sync (e.g., a node becomes unhealthy), that’s perfectly fine too. We’ll just materialize the repository on whichever node comes next.

前面我们看到，Spokes 集群难以运维的一点，是要持续追踪每台服务器上每个存储库的位置。Cnt 的做法截然不同：每个存储库在哪里？答案是「随便哪里」，无所谓！我们把存储库当作磁盘上的一层热缓存，事实来源永远是 S3 里的预写日志。系统是无状态的，没有路由表（也不需要运维关系数据库——谢天谢地）。如果某个主机上访问的存储库不在本地磁盘，我们直接从 WAL 把它物化（materialize）出来。这可以做得相当高效，但当然不能老是这么干——那就太浪费了。生产中我们用会合哈希（rendezvous hashing）把存储库 ID 映射到预期所在的节点列表。路由存储库所需的全部状态，就是存储库 ID 加集群当前健康节点集合。即便这个状态失同步了（比如某个节点变得不健康），也完全没问题——我们就在下一个轮到的节点上把存储库物化出来。

What about consensus? Elections? Which server is the primary for a given repository? It also doesn’t matter! There’s no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it’s always safe for any instance of a repository to receive a push. Again, just like with routing, letting an arbitrary server act as the primary isn’t the most efficient thing (it leads to CAS retries, which can delay pushes), so in practice we always choose the same server as the primary, the first one in the ranked list from rendezvous hashing. But in the corner cases — when there’s a deploy, a failover, a network blip — we just don’t care exactly which server is the primary. The system is designed to always be correct when degraded, and always fast when healthy.

那共识呢？选举呢？谁是一个给定存储库的主服务器？同样不重要！这里既没有状态，也没有共识——任何服务器都可以当主服务器。对预写日志的所有更新都在 S3 上由一个原子的比较并交换（compare-and-swap，CAS）操作来同步，因此任何存储库实例接收推送都是安全的。当然，和路由一样，让任意服务器当主服务器并非最高效的做法（会导致 CAS 重试、延迟推送），所以实践中我们总是选同一个服务器当主节点，也就是会合哈希排名列表里的第一个。但在边界情况下——部署、故障转移、网络抖动——我们根本不在乎谁是主服务器。这套系统设计目标就是：降级时永远正确，健康时永远快速。

Replication

复制

Having a write-ahead log in S3 opens a world of possibilities when it comes to scale. We can have literally any number of replicas, because the scalability of S3 is unmatched and all the replicas catch up directly from there. We perform optimistic replication by sending gossip UDP packets around our cluster. The packets contain all the required metadata for each replica to catch up directly from S3 after every push. “That is insane,” I hear you mumble from behind your screen across time and space. “UDP is not a reliable transport.” Of course it isn’t. Nothing is reliable in a distributed system! The wire is not reliable, the routing is not reliable, and the topology is not reliable either. But it’s OK: it doesn’t matter. Each replica knows the ETag of the last version of the WAL index it’s caught up with. When you perform a read operation on a replica, we do a conditional GET to S3 with the ETag we expect. A 304 response with no body (conveniently, an almost instant operation — less than 10ms on average because it’s a metadata-only S3 operation) means we’re up to date and we can serve the fetch or the clone straight away. A 200 response comes with the newest version of the WAL index, which we use to catch up before serving the read.

把预写日志放在 S3，为扩展性打开了一个充满可能的世界：我们几乎可以拥有任意多的副本，因为 S3 的可扩展性无可匹敌，所有副本都直接从 S3 追赶进度。我们通过在集群中传播 gossip UDP 包来做乐观复制（optimistic replication），这些包携带每个副本在每次推送后直接从 S3 补齐进度所需的全部元数据。「太疯狂了，」我仿佛听到你在屏幕另一头隔着时空咕哝，「UDP 可不是可靠传输。」当然不是。分布式系统里没有什么是可靠的！线路不可靠，路由不可靠，拓扑也不可靠。但这没关系：因为无所谓。每个副本都知道自己追到的 WAL 索引最新版本的 ETag。当你在副本上执行读操作时，我们会带上预期的 ETag 对 S3 做条件 GET：返回无正文的 304 响应（恰好几乎是瞬时操作——平均不到 10 毫秒，因为这只是个元数据级 S3 操作）说明我们已是最新，可以直接服务 fetch 或 clone；返回 200 响应则携带最新版 WAL 索引，我们先补齐进度再服务读取。

It doesn’t matter if the replication UDP packet is lost, or if it arrives at the wrong server because the topology shifted. All reads on all replicas are fully consistent, because they’re verified against the source of truth, which is S3. The system is designed to always be correct when degraded, and always fast when healthy.

复制用的 UDP 包丢了，或者因拓扑变化送到了错误的服务器，都无关紧要：所有副本上的所有读取都完全一致，因为它们都对照事实来源（即 S3）做过校验。这套系统的设计目标就是：降级时永远正确，健康时永远快速。

The implications of this are twofold. First, because the system is always consistent, building infrastructure on top of it is trivial. We (our agents, our web interface, our clients) always see a globally consistent view of the repository. And because the system scales in both directions, every repository gets just the right number of replicas. A large monorepo can be deployed across hundreds of replicas to serve all the load from its CI jobs. Millions of tiny repositories created by agents can be served with one replica each; we don’t need more than one to ensure availability, because S3 is the source of truth. In fact, an idle repository doesn’t even need that: when a replica hasn’t received traffic for a while, we garbage collect it from the node’s disk, and simply materialize it again from the WAL the next time a fetch comes in.

这带来两方面的意义。首先，因为系统始终一致，在其之上构建基础设施就变得非常直接——我们（我们的 agent、Web 界面、客户端）看到的始终是存储库的全局一致视图。其次，由于系统向两个方向扩展，每个存储库都能得到恰好合适的副本数：大型 monorepo 可以铺到数百个副本上，顶住 CI 作业的全部负载；agent 创建的几百万个小存储库，每个只配一个副本就够了——我们不需要更多副本来保证可用性，因为 S3 才是事实来源。其实连这一个副本都不是必须的：当某个副本一段时间没有流量，我们就把存储库从节点磁盘上垃圾回收掉，下次 fetch 进来时再从 WAL 物化出来。

Compaction

压缩（Compaction）

Write-ahead logs require periodic compaction. You cannot let the log grow unbounded: a full restore replays every entry, so the more entries, the more expensive it becomes.

预写日志需要定期压缩（compaction）：不能让日志无限增长——完整恢复要重放每一条记录，记录越多，成本越高。

Coincidentally, a normal Git repository also requires periodic compaction, even though Git is not based on a WAL. We’ve seen that the fundamental unit of storage in a Git repository is the packfile. Each time you push to a remote copy of a repository, or fetch into your local copy, you create a new packfile. This doesn’t scale indefinitely: each packfile has its own attached index, which allows Git to efficiently look up the objects it contains, but this lookup is only efficient on a per- packfile basis. If you’re looking for a specific object, and your repository has 100 packfiles, you’ll need to open the index for each one of them and look up the object until you find it in one of the packfiles. An efficient operation is not efficient if it must be performed hundreds or thousands of times.

巧合的是，普通 Git 存储库虽然不基于 WAL，却也需要定期压缩。前面说过，Git 存储库的基本存储单元是 packfile：每次推送到远端副本、或拉取到本地副本，都会产生一个新的 packfile。这没法无限扩展——每个 packfile 都有自己的配套索引，让 Git 能高效查找其中的对象，但这种查找只在一个 packfile 的范围内高效。如果你要找某个对象，而存储库里有 100 个 packfile，就得逐个打开索引查一遍，直到在某个 packfile 里找到为止。一个高效的操作，如果要重复几百上千次，也就不再高效了。

Modern Git has gotten very good at working around this; it now supports multi-pack indexes and incremental geometric compaction. But eventually you must bite the bullet and repack your Git repository on disk. Historically, this has been a constant availability issue for systems like Spokes, because repacking is a very CPU-heavy operation, even when done incrementally, and it must be performed on all the replicas of the system. Accidentally triggering a maintenance operation on two or more Spokes nodes for the same repository will easily cause the repository to fail over.

现代 Git 已经非常善于缓解这个问题——现在它支持多包索引（multi-pack index）和增量几何压缩（incremental geometric compaction）。但最终你还是得硬着头皮把 Git 存储库在磁盘上重新打包（repack）。历史上，这对 Spokes 这类系统是一个持续的可用性隐患：重新打包极其消耗 CPU，即使增量执行也一样，而且必须在系统的所有副本上进行。一旦不小心在同一个存储库的两个或多个 Spokes 节点上触发了维护操作，很容易导致该存储库故障转移（fail over）。

Here, we amortize the cost of compaction. Only the primary does compactions, and the result of the compaction applies to both the on-disk repository and the WAL. Since all replicas follow the WAL, they also follow the compaction events. Replicas don’t repack; they simply download the already-compacted packs from S3, trading bandwidth for CPU.

在 Continuity 里，我们把压缩成本摊销掉：只有主节点执行压缩，压缩结果同时作用于磁盘存储库和 WAL。由于所有副本都跟随 WAL，它们也就跟随压缩事件。副本不重新打包，只从 S3 下载已经压缩好的 pack，用带宽换 CPU。

Scale

规模

Replication and compaction are the two key factors that determine how well a Git storage system behaves under load. As we’ve just seen, they’re intrinsically linked: the more pushes per second a repository ingests, the more read performance degrades, because the packfiles of every push must be compacted for Git operations to remain efficient. If you replicate these pushes, the compaction must be either replicated or performed independently on each replica.

复制与压缩是决定 Git 存储系统在负载下表现的两个关键因素。正如前面看到的，两者内在相连：存储库每秒吸收的推送越多，读性能退化得越厉害——因为每笔推送的 packfile 都必须被压缩，Git 操作才能保持高效。如果你把这些推送复制出去，压缩要么跟着复制，要么在每个副本上独立执行。

Continuity’s WAL-first design offers fully consistent horizontal scalability: you can deploy an arbitrary number of replicas, and the throughput for read-only Git operations grows linearly with them. Because all replicas in the cluster are fully consistent, this allows us to scale the Git protocol (clones, fetches) and all the RPC operations that Origin performs on top of repositories (web UI interactions, the REST API, all our agentic interfaces, etc.)

Continuity 的 WAL-first 设计提供了完全一致的水平可扩展性：你可以部署任意数量的副本，只读 Git 操作的吞吐量随之线性增长。因为集群中所有副本完全一致，我们既可以扩展 Git 协议本身（clone、fetch），也能扩展 Origin 在存储库之上执行的所有 RPC 操作（Web UI 交互、REST API、所有 agent 接口等）。

We have run synthetic stress tests with up to 100 replicas and seen consistent linear scaling for reads, without any regressions in push throughput.

我们已经用多达 100 个副本跑过合成压力测试，读到一致的线性扩展，且推送吞吐量没有任何回退。

The push throughput of a cluster depends on the latency at which we can update our WAL on S3. Using S3 Standard, we can sustain up to 120 pushes/s while compacting and replicating the compacted data to all other nodes. We have also deployed high-performance clusters on S3 Express One Zone, which has much lower latency for PUT operations. There, we can ingest more than 300 pushes/s, and we are effectively bottlenecked by the speed at which Git can compact the on-disk data. We’re working on innovative ways to lay out this data on disk to reduce the impact of compaction: our goal is to continue optimizing the speed at which a Git repository can ingest code without relaxing our hard durability and consistency guarantees.

集群的推送吞吐量取决于我们在 S3 上更新 WAL 的延迟。用 S3 Standard，我们可以在压缩并复制压缩结果到所有其他节点的同时，维持高达 120 pushes/s；我们也部署过跑在 S3 Express One Zone 上的高性能集群，那里的 PUT 延迟低得多——超过 300 pushes/s，实际瓶颈只剩 Git 压缩磁盘数据的速度。我们正在研究在磁盘上布局这些数据的创新方式以减轻压缩的影响：目标是在不放松持久性与一致性硬保证的前提下，继续优化 Git 存储库吸收代码的速度。

• S3 Standard

  S3 标准

• S3 Express One Zone

  S3 Express One Zone

Push/Clone throughput for everysphere, Cursor’s monorepo.

everysphere（Cursor 的 monorepo）的推送/克隆吞吐量。 All pushes are linearizable and persisted to external storage before acknowledgement.

所有推送都是线性化的，并在确认之前已持久化到外部存储中。 All clones are fully consistent.

所有克隆都完全一致。

WAL as truth

WAL 作为事实来源（WAL as truth）

S3 is a great piece of technology. The whole concept of blob storage that was pioneered with the S3 API has turned out to be a very powerful building block for large data storage systems, and this most definitely also applies to hosting Git repositories. The design presented here is novel on many ways, but it’s not the first one to store packfiles as blobs. Azure DevOps (Microsoft’s own competitor to Microsoft’s own GitHub) has a very successful Git storage system that stores packfiles in blob storage and their references in a relational database (MS SQL Server). There are many trade-offs to a system like this. A relational database scales well with large reference transactions. But then you have to operate a relational database. We have a strong belief that the consistency of Git data is more important than any other consideration. This is what really tipped the scales for us into designing a WAL-based system that doesn’t depend on external databases

S3 是一项伟大的技术。S3 API 开创的 blob 存储这一整个概念，已被证明是大型数据存储系统极其强大的积木，托管 Git 存储库当然也不例外。这里的设计在很多方面都是新颖的，但把 packfile 存成 blob 并不是首创：Azure DevOps（微软自家 GitHub 的竞争对手）的 Git 存储系统就非常成功，它把 packfile 放在 blob 存储里，把引用存在关系数据库（MS SQL Server）里。这样的系统有很多取舍：关系数据库处理大规模引用事务很在行，但你就得运维一个关系数据库。我们坚信，Git 数据的一致性比任何其他考量都重要——这正是我们设计一个不依赖外部数据库的 WAL 式系统的决定性理由。

There are many things that can go wrong with a Git repository in production. Data corruption at rest, bugs during repacking, races during pushes. It’s one big collection of corner cases. Most of these have been ironed out in Git upstream. But not all of them. No system is without bugs, not even those that are OSS and widely deployed. Our consistency model ensures that we keep track of every fundamental operation that happens to a repository. We never acknowledge a push until it has been fully persisted to the WAL. We linearize all pushes. Every view of every repository we access is always fully consistent. Since every push is in the WAL, we can look at every state a repository has ever been in. We have full provenance data for all pushes, and also for all repacks. We can rewind and fast-forward every replica. We don’t have to synchronize any state with any external database, whether it’s a database that only stores references, or a database that stores all object data. When (not if) we hit a bug in Git, we can pinpoint exactly what happened and revert it. And besides the bugs that already exist in Git, we introduce very few new ones, because throughout all this, all Git operations are performed on a normal Git repository on disk, using off-the-shelf tooling.

生产环境中的 Git 存储库可能出很多问题：静置时的数据损坏、重新打包期间的 bug、推送期间的竞争条件——这是一大堆边角情况的集合。其中大部分已被 Git 上游修平，但并非全部。没有系统没有 bug，哪怕是开源的、被广泛部署的也不例外。我们的一致性模型确保记录下存储库发生的每一个基本操作：**在推送完整持久化到 WAL 之前，我们绝不确认；我们线性化所有推送；我们访问的每个存储库的每个视图都始终完全一致。**由于每笔推送都在 WAL 里，我们可以查看存储库曾经处于的每一个状态——我们拥有所有推送、所有重打包的完整溯源（provenance）数据，可以把任意副本倒带和快进。我们不需要与任何外部数据库同步任何状态，无论是只存引用的数据库，还是存全部对象数据的数据库。当我们（不是「如果」）在 Git 里遇到 bug 时，可以精确定位发生了什么并回滚。而且除了 Git 里本来就存在的 bug，我们引入的新 bug 极少——因为整个过程里，所有 Git 操作都是在磁盘上的普通 Git 存储库上、用现成工具完成的。

Origin

Origin

We are acutely aware of how important it is to host somebody’s source code. I think everybody who reads and understands this blog post is just as aware of it. A company can grind to a halt if its developers cannot push or pull from its Git repositories. The productivity cost of five minutes of downtime in your CI system is hard to quantify in dollars, but it is, by any measure, a humongous amount.

我们非常清楚为他人托管源代码有多重要——我相信每个读完并理解这篇博文的人同样清楚。如果开发人员无法从 Git 存储库推送或拉取，一家公司就可能陷入停摆。CI 系统停机五分钟造成的生产力损失很难用美元量化，但无论怎么衡量，都是天文数字。

Agents have fundamentally changed the way we work with software, and in many ways they’ve made this situation worse. More code, more PRs, more CI runs. Version control is at the core of all of this, and it is possibly the hardest thing to change overnight.

Agent 从根本上改变了我们使用软件的方式，而且在很多方面让形势更严峻：更多代码、更多 PR、更多 CI 运行。版本控制是这一切的核心，也很可能是最难在一夜之间改变的东西。

We’ve faced these difficulties internally at Cursor for many months now, and we’ve put considerable thought and care into building a platform that solves them for us and that can hopefully solve them for our customers too. Our focus right now is on providing the smoothest possible off-ramp into more reliability, more performance and more scale, and making the migration as painless as possible.

几个月来，Cursor 内部一直在直面这些困难，我们投入了大量思考与心血，构建一个既能替我们解决问题、也希望能为客户解决问题的平台。我们目前的重点是提供尽可能顺畅的出口（off-ramp），通向更高的可靠性、更强的性能和更大的规模，让迁移过程尽可能无痛。

Origin is not an experiment; it is the result of many decades of experience building these same systems, from people who deeply understand the magnitude of the challenges involved. We have an engineering and operational philosophy that has been proven to work, and a strong commitment to continue evolving it as the landscape of version control evolves.

Origin 不是一个实验；它是数十年构建同类系统经验的产物，来自深刻理解这些挑战之艰巨的人们。我们秉持一套经过验证的工程与运维理念，并且郑重承诺，随着版本控制领域的演进不断迭代它。

We’re hoping you’ll place your trust in us and our platform.

我们希望你能信任我们，信任我们的平台。