О стратегиях репликации или о возвращении длинной статьи

Мультимастерные системы разрешают, принимают и поощряют приход и уход узлов, когда они хотят, они предполагают, что записи могут и будут конфликтовать, и необходимость решать это на постоянной основе. От других узлов нет никаких ожиданий относительно относительной синхронизации, и обычно «заново заполняют» новый узел, просто начав репликацию на него, а это означает, что вам нужно реплицировать все данные с начала времени до Это. Также распространено, чтобы узел всплывал один раз в синей луне, ожидал получить все изменения, которые произошли, когда он ушел, и затем снова выпал.

Давайте посмотрим на фактические детали реализации каждого, включая некоторые примеры, и, надеюсь, будет понятнее то, о чем я говорю.

Доставка журналов

Ведущий / ведомый обычно реализуется через доставку журналов. Самый простой способ думать о доставке журналов состоит в том, что основная база данных отправит (как ни странно, нас не особо волнует, как на этом этапе) подчиненные инструкции о том, как напрямую изменять файлы базы данных. Другими словами, концептуально он посылает им следующее:

   writep(fd1, 1024, new[]{ 17,85,124,13,86}, 5);
   writep(fd1, 18432, new[]{ 12,95,34,83,76,32,59}, 7);b

Это очень низкие модификации уровня, как вы можете себе представить. Преимущество здесь в том, что эти изменения очень легко записать и воспроизвести. Недостатком является то, что вы не можете сделать что-нибудь еще. Поскольку изменения происходят в самом низу стека, нет никакой возможности использовать какую-либо логику. Мы просто пишем в файл, так же как и главный сервер.

Это основная причина, по которой рабу так сложно разрешить запись. В тот момент, когда он делает какую-либо независимую запись, он подвергается риску того, что мастер также сделает запись, что приведет к повреждению данных. Вот почему вы должны исполнить главную песню и танец, если хотите сменить хозяина и раба. Вы должны пройти через все эти проблемы, чтобы гарантировать, что у вас никогда не будет сценария, когда у вас будет запись с обоих концов.

Как только это произойдет, вы никогда больше не сможете синхронизировать эти два. Это просто происходит на слишком низком уровне.

Однако создать новый узел очень просто. Обязательно ведите журнал, делайте полную резервную копию базы данных и перемещайте ее на другой узел. Затем начните пересылку журналов. Поскольку они начались в одной точке, их можно безопасно применять.

Обратите внимание, что эта версия очень чувствительна к проблемам с версиями. Вы не можете иметь даже незначительного изменения в версиях работы с хранилищем низкого уровня, потому что тогда весь ад может потерпеть неудачу. Этот метод очень хорош для генерации реплик чтения. Действительно, это то, что это используется в большинстве случаев.

Теоретически, вы можете даже заставить его выполнять аварийное переключение, потому что, пока мастер не работает, подчиненный может писать. Проблема заключается в том, как вы справляетесь со случаем, когда раб думает, что мастер не работает, а мастер думает, что все в порядке. В этот момент вы можете принять обе записи, что приведет к невозможности объединения.

Теоретически, поскольку они имеют общий корень, вы можете решить, что один из них является победителем, и пойти на это, но это приведет к потере данных с сервера-неудачника и, вероятно, данных, которые у вас нет реального способа получить обратно. , Изменения, которые мы отслеживаем здесь, очень малы и, вероятно, слишком гранулированы, чтобы позволить вам действительно сделать что-то значимое для извлечения измененной информации.

Oplog

На самом деле это очень похоже на метод доставки журналов, но вместо того, чтобы отправлять операции ввода-вывода файлов очень низкого уровня, мы фактически отправляем команды более высокого уровня. Это приводит к довольно большому количеству преимуществ для нас. Основной сервер может отправить свой журнал как:

   set("users/1", {"name": "oren" });
   set("users/2", {"name": "ayende" });
   del("users/1");

Executing this set of instruction on the secondary will result in identical state on the secondary.  Unlike Log Shipping option, this actually require the secondary server to perform work, so it is more expensive than just apply the already computed file updates.

However, the upside of this is that you can have a far more readable log. It is also much easier to turn a secondary into a primary. Mostly, this is silly. The actual operation is the exact same thing. But because you are working at the protocol level, rather than the file level. You can get some interesting benefits.

Let us assume that you have the same split brain issue, when both primary & secondary think that they are the primary. In the Log Shipping case, we had no way to reconcile the differences. In the case of Oplog, we can actually do this.  The key here is that we can:

• Dump one of the servers rejected operations into a recoverable state.
• Attempt to apply both severs logs, hoping that they didn’t both work on the same document.

This is the replication mode used by MongoDB. And it has chosen the first approach for handling such conflicts. Indeed, that is pretty much the only choice that it can safely make. Two servers making modifications to the same object is always going to require manual resolution, of course. And it is usually better to have to do this in advance and explicitly rather than “sometimes it works”.

You can see some discussion on how merging back divergent writes works in MongoDB here. In fact, continuing to use the same source, you can see the internal oplog in MongoDB here:

   // Operations
    
   > use test
   switched to db test
   > db.foo.insert({x:1})
   > db.foo.update({x:1}, {$set : {y:1}})
   > db.foo.update({x:2}, {$set : {y:1}}, true)
   > db.foo.remove({x:1})
    
   // Op log view
    
   > use local
   switched to db local
   > db.oplog.rs.find()
   { "ts" : { "t" : 1286821527000, "i" : 1 }, "h" : NumberLong(0), "op" : "n", "ns" : "", "o" : { "msg" : "initiating set" } }
   { "ts" : { "t" : 1286821977000, "i" : 1 }, "h" : NumberLong("1722870850266333201"), "op" : "i", "ns" : "test.foo", "o" : { "_id" : ObjectId("4cb35859007cc1f4f9f7f85d"), "x" : 1 } }
   { "ts" : { "t" : 1286821984000, "i" : 1 }, "h" : NumberLong("1633487572904743924"), "op" : "u", "ns" : "test.foo", "o2" : { "_id" : ObjectId("4cb35859007cc1f4f9f7f85d") }, "o" : { "$set" : { "y" : 1 } } }
   { "ts" : { "t" : 1286821993000, "i" : 1 }, "h" : NumberLong("5491114356580488109"), "op" : "i", "ns" : "test.foo", "o" : { "_id" : ObjectId("4cb3586928ce78a2245fbd57"), "x" : 2, "y" : 1 } }
   { "ts" : { "t" : 1286821996000, "i" : 1 }, "h" : NumberLong("243223472855067144"), "op" : "d", "ns" : "test.foo", "b" : true, "o" : { "_id" : ObjectId("4cb35859007cc1f4f9f7f85d") } }

You can actually see the chain on command to oplog entry. The upsert command in line 7 was turned into an insert in line 18, for example. There appears to also be a lot of work done to avoid having to do any sort of computable work, in favor of resolving things to a simple idempotent operation.

For example, if you have a doc that looks like {counter:1} and you do an update like {$inc:{counter:1}} on the primary, you’ll end up with {counter:2} and the oplog will store {$set:{counter:2}}. The secondaries will replicate that instead of the $inc.

That is pretty nice feature, since it mean that you can much apply changes multiple times and end with the same result. But it all leads to the end result, in which you can’t merge divergent writes.

You do get a much better approach for actually going over the data and doing the fixup yourself, but still.. I don’t really like it.

Multi write partners

In this mode, we have a set of servers, each of which is familiar with their partners. All the writes coming are accepted, and logged. Replication happen from the source server contacting all of the destination servers and asking them: What is the last you heard from me? Here are all of my changes since then. Critically, it is at this point that we can trim the log for all of the actions that were already replicated to all of the servers.

A server being down means that the log of changes to go there is going to increase in size until the partner is up again, or we remove the entry for that server from our replication destination.

So far, this is very similar to how you would structure an oplog. The major difference is how you structure the actual data you log. In the oplog scenario, you’re going to write the changes that happens to the system. And the only way to act on this is to actually apply the op log in the same sequence as it was generated. This leads to a system where you can always have just a single primary node. And that leads to situations when split brains will result in data loss or manual merge steps.

In MWP case, we are going to keep enough context (usually full objects) so that we can give the user a better option to resolve the conflict. This also gives us the option of replaying the log in non sequential manner.

Note, however, that you cannot just bring a new server online and expect it to start playing nicely. You have to start from a known state, usually a db backup of an existing node. Like the log shipping scenario, the process is essentially, start replicating (to the currently non existent server), that will ensure that the log will be there when we actually have the new server. Backup the database and restore on a secondary server. Configure to accept replication from the source server.

The complexities here are that you need to deal with operations that you might already have. That is why this is usually paired with vector clocks, so you can automatically resolve such conflicts. When you cannot resolve such conflicts, this falls down to manual user intervention.

Multi Master

Multi master systems are quite similar to multi write partners, but they are designed to operate independently. It is common for servers to be able communicate with one another only rarely. For example, a mobile system that is only able to get connected just a few hours a week. As such, we cannot just cap the size of the operations to replicate. In fact, the common way to bring a new server up to speed is just to replicate to it. That means that we need to be able to replicate, essentially from any point in the server history, to a new server.

That works great, as long as you don’t have deletes. Those do tend to make things harder, because you need to keep track of those, and replicate them. RavenDB and CouchDB are both multi master systems, for example. Conflicts works the same way, pretty much, and we use a vector clock to determine if a value is in conflict or not.

Divergent writes

I mentioned this a few times, but I didn’t fully explain. For my purposes, we assume that we are using 2 servers (and yes, I know all about quorums, etc. Not relevant for this discussion) running in master/slave mode.

At some point, the slave think that the master is down and takes over, and the master doesn’t notice this and still think it is the master. During this time, both server accept the following writes:

Server A	Server B
write users/1	wrier users/2
write users/3	write users/3
delete users/4	delete users/5
delete users/6	write users/6
write users/7	delete all users
set all users to active	write users/8

After those operation happen, we restore communication between the two servers and they need to decide how to resolve those changes

Getting down to business

Okay, that is enough talking about what those terms mean. Let us consider the implications of using them. Log shipping is by far the simplest method to use. Well, assuming that you actually have a log mechanism, but most dbs do. It is strictly one writer model, and there is absolutely no way to either resolve divergent writes or even to find out what they were. The good thing about log shipping is that it is quite easy to get this working without actually needing to care anything about the actual data involved. We work directly at the file level, we don’t care at all about what the data is. The problem is that we can’t even solve simple conflicts, like writes to the different objects. This is because we are actually working at the file level, and all the changes are there. Attempting to merge changes from multiple logs would likely result in file corruption. The up side is that it is probably the most efficient way to go about doing this.

Oplog is a step above log shipping, but not a big one. It doesn’t resolve the divergent writes issues. This is now an application level protocol. The log needs to contain information specific to the actual type of data that we store. And you need to write explicit code to handle this. That is nice, but it also require strict sequence of all operations. Now, you can try to merge things between different logs. However, you need to worry about conflicts, and more to the point, there is usually nothing in the data itself that will help you even detect conflicts.

Multi write partners are meant to take this up a notch. They do keep track of the version history (usually via vector clocks). Here, the situation is more complex, because we need to explicitly decide how to deal with conflicts (either resolve automatically or defer to user decision), but also how to handle distribution of updates. Usually they are paired with some form of logic that tells you how to direct your writes. So all writes for a particular piece of data would go to a preferred node, to avoid generating multiple versions. The data needs to contains some information about that, so we keep vector clock information around. Once we sent the changes to all our partners, we can drop them, saving in space.

Multi master is meant to ensure that you can have partners that might only see one another occasionally, and it makes no assumptions about the topology. It can handle a node that comes on, get some data replicated, and drop off for a while (or forever). Each node is fully independent, and while they would collaborate with others, they don’t need them. The downside here is that we need to keep track of some things forever. In particular, we need to keep track of deletes, to ensure that we can get them to the remote machines.

What about set operations?

Interesting enough, that is probably the hardest issue to resolve. Consider the case when you have the following operations happen:

Server A	Server B
write users/7	delete all users
set all users to active	write users/8 (active: false)

What should be the result of this? There isn’t a really good answer. Should users/8 be set to active: true? What about users/7, should it be deleted or kept?

It gets hard because you don’t have good choices. The hard part here is actually figuring out that you have a conflict. And there isn’t a really good way to handle set operations nicely with conflicts. The common solution is to translate this to the actual operations made (delete users/1,user/2, users/3 – writer users/8, users/5) and leave it at that. The set based operation is translated to the actual individual operations that actually happened. And on that we can detect conflicts much more easily.

Log shipping is easiest to work with, operationally speaking. You know what you get, and you deal with that. Oplog is also simple, you have a single master, and that works. Multi master and multi write partners requires you to take explicit notice of conflicts, selection of the appropriate node to reduce conflicts, etc.

In practice, at least in the scenarios relating to RavenDB, the ability to take a server offline for weeks or months doesn’t seem to be used that often. The common deployment model is of servers running as steady partners. There are some optimizations that you can do for multi write partners that are hard/impossible to do with multi master.

My current personal preference at this point would like to go with either log shipping or multi write master. I think that either one of them would be fairly simple to implement and support operationally. I think that I’ll discuss actual design for the time series topic using either option in my next posts.