Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, Feb 05, 2010 at 12:46:24PM -0500, Petrakis, Peter wrote:
> Hi All,
>
> We use resource-only fencing currently which is causing us a problem
> when we're bringing up a cluster for the first time. This all worked
> well with 8.2.7. We only have one node active at this time and the
> fencing handler is bailing with '5' which is correct. The problem is,
> this return failure is stopping us from becoming Primary.
>
>
> block drbd5: helper command: /usr/lib/spine/bin/avance_drbd_helper
> fence-peer minor-5 exit code 5 (0x500)
> block drbd5: State change failed: Refusing to be Primary without at
> least one UpToDate disk
> block drbd5: state = { cs:StandAlone ro:Secondary/Unknown
> ds:Consistent/DUnknown r--- }
> block drbd5: wanted = { cs:StandAlone ro:Primary/Unknown
> ds:Consistent/DUnknown r--- }
I'm quoting our internal bugzilla here:
scenario:
A (Primary) --- b (Secondary)
A (Primary) b (down, crashed; does not know that it is out of date)
a (down)
clean shutdown or crashed, does not matter, knows b is Outdated...
a (still down) b (reboot)
after init dead, heartbeat tries to promote
current behaviour:
b (Secondary, Consistent, pdsk DUnknown)
calls dopd, times out, considers other node as outdated,
and goes Primary.
proposal:
only try to outdate peer if local data is UpToDate.
- works for primary crash (secondary data is UpToDate)
- works for secondary crash (naturally)
- works for primary reboot while secondary offline,
because pdsk is marked Outdated in local meta data,
so we can become UpToDate on restart.
- additionally works for previously described scenario
because crashed secondary has pdsk Unknown,
thus comes up as Consistent (not UpToDate)
avoids one more way to create diverging data sets.
if you want to force Consistent to be UpToDate,
you'd then need either fiddle with drbdmeta,
or maybe just the good old "--overwrite-data-of-peer" does the trick?
does not work for cluster crash,
when only former secondary comes up.
------- Comment #1 From Florian Haas 2009-07-03 09:07:57 [reply] -------
Bumping severity to "normal" and setting target milestone to 8.3.3.
This is not an enhancement feature, it's a real (however subtle) bug. This
tends to break dual-Primary setups with fencing, such as with GFS:
- Dual Primary configuration, GFS and CMAN running.
- Replication link goes away.
- Both nodes are now "degraded clusters".
- GFS/CMAN initiates fencing, one node gets rebooted.
- Node reboots, link is still down. Since we were "degraded clusters" to begin
with, degr-wfc-timeout now applies, which is finite by default.
- After the timeout expires, the recovered node is now Consistent and attempts
to fence the peer, when it should not.
- Since the network link is still down, fencing fails, but we now assume the
peer is dead, and the node becomes Primary anyway.
- We have split brain, diverging datasets, all our fencing precautions are moot.
------- Comment #3 From Philipp Reisner 2009-08-25 14:42:58 [reply] -------
Proposed solution:
Just to recap, the expected exit codes of the fence-peer-hander:
3 Peer's disk state was already Inconsistent.
4 Peer's disk state was successfully set to Outdated (or was Outdated to begin with).
5 Connection to the peer node failed, peer could not be reached.
6 Peer refused to be outdated because the affected resource was in the primary role.
7 Peer node was successfully fenced off the cluster. This should never occur
unless fencing is set to resource-and-stonith for the affected resource.
Now, if we get a 5 (peer not reachable) and we are not UpToDate (that means we
are only Consistent) then refuse to become primary and do not consider the peer
as outdated.
The change in DRBD is minimal, but we need then to implement exit code 5 also
in crm-fence-peer.sh
------- Comment #4 From Philipp Reisner 2009-08-26 19:15:33 [reply] -------
commit 3a26bafa2e27892c9e157720525a094b55748f09
Author: Philipp Reisner <philipp.reisner at linbit.com>
Date: Tue Aug 25 14:49:19 2009 +0200
Do not consider peer dead when fence-peer handler can not reach it and we are < UpToDate
------- Comment #5 From Philipp Reisner 2009-10-21 13:26:58 [reply] -------
Released with 8.3.3
> We setup our metadata in advance before we use drbdsetup to attach the
> disk. Here's the
> metadata.
>
> version "v08";
> uuid {
> 0x0000000000000006; 0x0000000000000000; 0x0000000000000000;
> 0x0000000000000000;
> flags 0x00000011;
Right. I assume you do it this way to avoid the initial sync.
There is no need for that anymore, we now have the "new-uuid --clear-bitmap"
for that.
Anyways, it should work for you again if you initialize the flags to also
mark the peer as outdated, i.e. instead of 0x11, make that 0x31.
> The flags we set ought to be telling DRBD that our data is UpToDate
> enough to become
> Primary but it doesn't seem to matter while fencing is enabled. The
> result is
> the same regardless of whether I specify '-o' while calling drbdsetup
> primary.
>
> This is how the disk is being setup.
>
> /sbin/drbdsetup /dev/drbd5 disk /dev/disk-drbd5 /dev/disk-drbd5 internal
> --set-defaults --create-device
> --on-io-error=detach -f resource-only
>
> Thanks in advance.
>
> Peter
--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed