[DRBD-user] Re: Effect of writing directly to DRBD backingvolume while offline?

Lars Ellenberg lars.ellenberg at linbit.com
Wed Jun 4 22:42:08 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Wed, Jun 04, 2008 at 08:42:50PM +0200, Simon Carey-Smith wrote:
> Does anyone have a simple explanation of the difference between these 3 
> commands, and in which situations to use them ?

a data set may be
 inconsistent (invalid, not recoverable to any sane state, not usable)
   - drbd refuses to let you access inconsistent data
 consistent, which may be
   - up to date, or
   - possibly outdated (we may or may not be up to date, we just don't
     know yet, because we cannot talk to the peer), or
   - outdated (we know we are out of date, because we know by other
     means that the other node has been changed without being able to
     replicate those changes to us)

when drbd creates new meta data,
it is initialized to "inconsistent, all dirty"

when drbd connects two nodes,
the state of the data sets is compared.

it is slightly more complicated than the following, but basically:
  if both nodes are inconsistent, nothing is synced.

  if one node is inconsistent, and the other is consistent,
  the inconsistent one becomes sync target.

  if both are consistent, and our algorithm detects that one is a direct
  successor of the other, the "newer", more up-to-date one becomes sync

  if both are consistent, but our algorithm detects that they have
  _both_ been modified in diverging, incompatible ways, we recognize
  that there must have been a period of "resource internal split brain".

  the default behaviour after split brain recovery is to disconnect, and
  refuse to connect again.  because you have two consistent data sets
  that are different, you have to chose which one you want to use from
  now on. this can be somewhat automated (dangerous!) by the "after
  split brain handlers".

  on resync, transfered are only blocks that have been changed
  (marked dirty) since the last time the nodes have been in sync.

> > "drbdadm invalidate $r"

forcefully sets the data set to "inconsistent",
and marks all block dirty - thus initiating a full sync,
making this node sync target.

use this if you have reason to believe that the information in the dirty
bitmap does not match the changed blocks on disk, e.g. when you
"accidentally" bypassed drbd (mounted directly), or when you replaced
the drbd data store device but kept the external drbd meta data device.

> > "drbdadm --over-write-data-of-peer $r"

overrides DRBDs notion of the state, forcefully sets this node to
"consistent". if it was already consistent, this becomes a no-op.

if the other node is (still) inconsistent, this node will become
sync source, effectively overwriting the data of the peer.

if the other node is consistent as well, nothing will change,
so you cannot use this to recover from diverging data sets.

use this once, right after you newly created the drbd meta data area,
on one node only, to pick the sync source of the initial sync.

> > "drbdadm -- --discard-my-data connect $r"

if "after split brain" was detected, and therefore the nodes refuse to
talk to each other, you can pick one to become sync target.

this forcefully marks this node as inconsistent, but in contrast to the
"invalidate" it does not mark any additional blocks as dirty, so we only
need to transfer the union of marked-as-dirty blocks.

use this to recover from diverging data sets.

: Lars Ellenberg                           http://www.linbit.com :
: DRBD/HA support and consulting             sales at linbit.com :
: LINBIT Information Technologies GmbH      Tel +43-1-8178292-0  :
: Vivenotgasse 48, A-1120 Vienna/Europe     Fax +43-1-8178292-82 :
please don't Cc me, but send to list -- I'm subscribed

More information about the drbd-user mailing list