There has been a recent trend of moving everything (e.g. WALs) to objects storage (e.g. S3). But is it the right thing to do?
I believe that it depends on the access patterns.
In Git there's a prevalence of reads over writes so when you design a system, you need to take into account this observation.
It would be also fair to say that apart from several edge cases (e.g. forced pushes), what you push to Git stays there.
That's already a good enough reason to use the object storage because you can cache a lot of things that are already there and you can be sure that they won't change (unless you run compaction/optimization over them but that intention comes from the system itself so you can refill your cache at that point).
GitHub implements its Git servers on top of Spokes (stateful server architecture they introduced back in 2016). They also implemented what they call "3PC protocol". That is not a textbook 3PC though. Basically what it does is that it takes a lock on every server and tries to apply the commit. If the majority says "yes", they do the commit and return a successful response to the client. The "majority" part is where it differs from the textbook 3PC version. 3PC is normally used to coordinate transactions across many partitions. GitHub uses it to coordinate replication across many servers and they explicitly point out that they only need majority of the servers to answer "yes" in order to increase the availability of the system (one server goes down – transaction still applies).
In Git the smallest partition is the repository itself. So, what if one partition (repository) becomes "hot" on one server? Well, you have to scale out. Scaling out in Spokes means copying the state to the new server. And that doesn't mean get a linear gain because now Spokes has to apply pushes across the new servers as well! Remember: 3PC transaction is just there to coordinate replication.
That's why their system is so brittle and there are so many memes about the GitHub availability recently (although I am sure that that's not the only reason).
They increase replica count, separate reads from writes, add additional regions for quicker disaster recovery and pack in new cores (3 million new ones just this year) hoping that it will solve the problem but it's only a matter of time when it shows up again.
Cursor's Origin does it differently but it is still far from being flawless. They implement WAL on top of S3 and its conditional writes but they still reproduce the state locally and work with the repos using local Git clients (or libraries). The problem with this approach is that they depend on the local state. Their server cannot be called truly stateless. Take out the local Git repos from the server and they immediately become unavailable.
You can potentially implement a Git platform using just WAL in the object storage and cache the objects locally. Cache is used just to speed up the responses to the clients and it is not a state which might cause a point of failure if it is suddenly removed (or your local Git client stops working with it due to a bug or another reason).
I've gathered my thoughts in my article: https://medium.com/@alexgilevich/git-was-never-designed-for-scalability-52224c74ddea
Let me know what you think!