[DRBD-user] Split Brain due to 'diskless' state with pacemaker/heartbeat

Lars Ellenberg lars.ellenberg at linbit.com
Mon Jun 4 11:23:11 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Sat, Jun 02, 2012 at 12:20:14AM +0200, Florian Haas wrote:
> On 06/01/12 18:22, Lars Ellenberg wrote:
> > There is one improvement we could make in DRBD:
> > call the fence-peer handler not only for connection loss,
> > but also for peer disk failure.
> 
> That sounds like a good and simple idea to me.
> 
> >>> Alternitively, a constraint in pacemaker on diskless state until a
> >>> re-sync has been completed.
> >>
> >> You could actually do that with using the crm-fence-peer.sh handler as
> >> your local-io-error handler, albeit with two drawbacks:
> >>
> >> 1. The local-io-error has an exit code convention that is different from
> >> the fence-peer one (so you'd need to use a wrapper).
> > 
> > exit code of local-io-error handler is ignored
> 
> Makes sense. Good to know.
> 
> >> 2. In order to actually mask the I/O error from your upper layers, you'd
> >> now have to call "drbdadm detach" from the local-io-error handler, and
> >> iirc calling drbdadm from a drbdadm handler is a bad idea.
> > 
> > local-io-error handler is called after the device was detached already.
> > it is just an additional action.
> 
> Oh. That I didn't know, and the man page doesn't say so. Well then that
> approach is out anyway.

No. Why?
Just place a "fence-myself" constraint from local-io-error handler,
as I think you suggested earlier.

> 
> >>> Any Suggestions?
> >>
> >> Lars: would it make sense for a Secondary that detaches (either by user
> >> intervention or after an I/O error) to at least _try_ to outdate itself
> >> in the metadata?
> > 
> > I think it does.
> > 
> > There is a related scenario:
> 
> Related, yes. This one is about to dual node failure. But Philip's
> scenario is about 1 node, 1 disk failure. So even if the dual-node
> failure can't be helped, Philip's problem might be.

As I tried to explain, currently both situations look identical
to the node that is to be promoted, and both are solved in the same way:
by using the drbd fence-peer stuff.

> > In any case, if you configure fencing resource-and-stonith,
> > drbd comes up as "Consistent" only (not UpToDate),
> > so it needs to fence the peer, or promotion will fail.
> 
> resource-and-stonith only? Isn't this true for fencing resource-only as
> well?

By using "resource-only", you basically already told DRBD
to prefer being online over having the most recent transactions,
so the typical configuration and chosen (respectively: not chosen, thus
default) timeouts may cause the crm-fence-peer.sh to think "peer
reachable", even if it actually is not, then place the constraint and exit 4.

Not so much about the fencing policy setting itself,
more about the concept behind it.

> > If the peer is unreachable (exit code 5), and DRBD is only Consistent,
> > drbd counts that as fail, and will refuse promotion.

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed



More information about the drbd-user mailing list