[DRBD-user] How does protocol A work across a WAN?

Lars Ellenberg lars.ellenberg at linbit.com
Tue Jul 3 20:35:28 CEST 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Tue, Jul 03, 2007 at 10:36:49AM -0700, Chris de Vidal wrote:
> *LOVE* DRBD.  Whatta killer app!
> Suppose I'm syncing with protocol A across a slow WAN link and suppose the primary node has
> several minutes' worth of changes that have not yet been synced to the secondary node.

"several minutes worth" ... unlikely, unless you have very infrequent
changes: the tcp buffer size is maybe ~1MB send buffer, plus the same
amount receive buffer on the other side, plus some MB worth of data "in
flight". so at most a few seconds worth of bandwidth of any
off-the-shelf hard disk.

though on our todo list is a DRBD protocol variant that in fact could do
delayed synchronisation with "arbitrary" delays.

> Suppose then the primary node goes down before all of the recent changes could be copied to the
> secondary node.
> Would the only losses to the secondary node be those recent few minutes?  Or might there be other,
> out-of-step changes that are lost as well?
> Another way to ask the question: are changes run through a strictly first-in first-out (FIFO)
> process or could they be sent across the wire in random order for optimization?  Is the order of
> changes strictly in the order that they happened on the primary node?  Or is there some reordering
> for performance or some other reason?

the image on the secondary is always consistent, regardless of when the
connection is lost or the primary crashes (looks the same from this
perspective: no more requests from Primary).
(however, it is inconsistent during resynch, so keep that in mind.)

the only reodering taking place is the reordering in the local io stacks
below drbd. this is restricted to happen within reorder domains which are 
bounded by "drbd barriers". drbd barriers are issued whenever the
primary relays an io-completion event to the upper layers (file system).

rationale: if the "user" (file system) submits several write requests
while some other write request is not yet completed, these are obviously
independend and may be reordered. any write request submitted after some
other request has been completed however may well be dependend on that
very completion. follows: reordering may take place, but only between
completion events.

: Lars Ellenberg                            Tel +43-1-8178292-0  :
: LINBIT Information Technologies GmbH      Fax +43-1-8178292-82 :
: Vivenotgasse 48, A-1120 Vienna/Europe    http://www.linbit.com :
please use the "List-Reply" function of your email client.

More information about the drbd-user mailing list