[DRBD-user] secundary not finish synchronizing [actually: automatic data loss by dual-primary, no-fencing, no cluster manager, and automatic after-split-brain recovery policy]

[Lars Ellenberg]

On Thu, Feb 08, 2018 at 02:52:10PM -0600, Ricky Gutierrez wrote:
> 2018-02-08 7:28 GMT-06:00 Lars Ellenberg <lars.ellenberg at linbit.com>:
> > And your config is?
> 
> resource zimbradrbd {

>         allow-two-primaries;

Why dual primary?
I doubt you really need that.

>         after-sb-1pri discard-secondary;

Here you tell it that,
if during a handshake after a cluster split brain
DRBD notices data divergence,
you want it to automatically resolve the situation
and discard all changes of the node that is Secondary
during that handshake, and overwrite it with the data
of the node that is Primary during that handshake.

> become-primary-on both;

And not even a cluster manager :-(

> > And you logs say?
> Feb  5 13:45:29 node-01 kernel:
> drbd zimbradrbd: conn( Disconnecting -> StandAlone )

That's the uninteresting part (the disconnect).
The interesting part is the connection handshake.

> > As is, I can only take an (educated) wild guess:
> >
> > Do you have no (or improperly configured) fencing,
> 
> i don't have.

Too bad :-(

> > and funky auto-resolve policies configured,
> > after-split-brain ... discard-secondary maybe?
>
> > Configuring automatic after-split-brain recovery policies
> > is configuring automatic data loss.
> >
> > So if you don't mean that, don't do it.
> 
> I am not an expert in drbd, but with the configuration that I have
> according to the documentation,
> should not this situation happen, or am I wrong?

Unconditional dual primary directly by init script,
 no fencing,
 no cluster manager,
and policies to automatically chose a victim after data divergence.

You absolutely asked for that situation to happen.

You told DRBD to
go primary on startup
ignoring the peer (no cluster manager, no fencing policies)
and in case it detects data divergence, automatically throw away
changes on whatever node may be Secondary at that point in time.
Which is likely the node most recently rebooted.

And that is what happened:
network disconnected,
you had both nodes Primary, both could (and likely did)
keep changing their version of the data.

Some time later, you rebooted (or deconfigured and reconfigured)
DRBD on at least one node, and during the next DRBD handshake,
DRBD noticed that both datasets have been changed independently,
would have refused to connect by default, but you told it to
automatically resolve that data conflict and discard changes on
whatever node was Secondary during that handshake.

Apparently the node with the "more interesting" (to you)
data has been Secondary during that handshake,
while the other with the "less inetersting" (to you) data
already? still? has been Primary.

And since you have configured DRBD for automatic data loss,
you got automatic data loss.

Note: those *example* settings for the auto-resolve policies
may be ok *IF* you have proper fencing enabled.

With fencing, there won't be data divergence, really,
but DRBD would freeze IO when it first disconnects,
the (or both) Primaries (at that time) will call the fence handler,
the fence handler is expected to hard-reset the peer,
the "winner" of that shoot-out will resume IO,
the victim may or may not boot back up,
NOT promote before it can see the peer,
and during that next handshake obviously be Secondary.

On the off chance that DRBD still thinks there had been data divergence,
you may want to help it by automatically throwing away the (hopefully
useless) changes on the then Secondary in this scenario.
Because if the fencing did work, the shot node should just be
"linearly older" than the survivor, and DRBD should do a normal resync.
Which is why I would probably NOT want to do add those settings,
but in case there is a data divergence detected, rather
investigate what went wrong, and make a concious decision.

But it is an option DRBD offers to those that want to enable it,
and may make sense in some setups.  Mostly for those that value
"online" over "most recent data", and can not afford
"administrative intervention" for whatever reason.

Maybe we should add a big red stay-out-here-be-dragons sign
to both "allow-two-primaries" and the after-split-brain
auto-recovery policy settings in the guide.

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker

DRBD® and LINBIT® are registered trademarks of LINBIT
__
please don't Cc me, but send to list -- I'm subscribed