[DRBD-user] notify-split-brain.sh[153967]: Environment variable $DRBD_PEER not found (this is normally passed in by drbdadm).

[Lars Ellenberg]

On Wed, Sep 19, 2018 at 04:57:08PM -0400, Daniel Ragle wrote:
> On 9/18/2018 10:51 AM, Lars Ellenberg wrote:
> > On Thu, Sep 13, 2018 at 04:36:54PM -0400, Daniel Ragle wrote:
> > > Anybody know where I need to start looking to figure this one out:
> > > 
> > > notify-split-brain.sh[153967]: Environment variable $DRBD_PEER not found
> > > (this is normally passed in by drbdadm).
> > > 
> > > I have a split-brain situation and apparently this notification script tried
> > > to run but failed.
> > > 
> > > This is in a pacemaker setup with ocf:linbit:drbd controlling DRBD.
> > > 
> > > # drbdadm --version
> > > DRBDADM_BUILDTAG=GIT-hash:\ fed9a1df82015e52c14c912fa4b93336e2ab4fcc\ build\
> > > by\ mockbuild@\,\ 2018-04-20\ 19:30:46
> > > DRBDADM_API_VERSION=2
> > > DRBD_KERNEL_VERSION_CODE=0x09000e
> > > DRBD_KERNEL_VERSION=9.0.14
> > > DRBDADM_VERSION_CODE=0x090301
> > > DRBDADM_VERSION=9.3.1
> > 
> > Yup, sorry, even though apparently they are still documented in the UG,
> > those "notify*.sh" example scripts are not fit for DRBD 9 :-(
> > 
> > You want to configure proper fencing anyways,
> > to *avoid* split brain instead of being "maybe"
> > (or maybe not) notified by email.
> 
> Yah, in fact it was somewhere in the process of trying to get fencing
> configured (and testing) that I managed to end up with the split-brain.
> Unfortunately (see below) I don't know exactly what I did to trigger it.
> 
> > 
> > For now, if you feel comfortable patching the script,
> > you could try adding something like this:
> > if [[ $DRBD_PEER_NODE_ID ]] ; then
> > 	k=DRBD_NODE_ID_${DRBD_PEER_NODE_ID};
> > 	v=${!k};
> > 	[[ $v ]] && DRBD_PEER=$v;
> > fi

We may have to add back the equivalent setenv call
into the drbdadm tool itself.

> Looks like it should work, I tried adding it, but now I can't figure out how
> to test it. I tried:
> 
>     drbdadm split-brain r0
> 
> And that gives me the same problem, as evidently that test doesn't set any
> of the DRBD_PEER vars, only:
> 
>     DRBD_CONF
>     DRBD_LL_DISK
>     DRBD_MINOR
>     DRBD_MY_NODE_ID
>     DRBD_NODE_ID_0
>     DRBD_NODE_ID_1
>     DRBD_RESOURCE
>     DRBD_VOLUME

DRBD_PEER_NODE_ID is set by the module before calling the helper,
because obviously only the module knows which connection (peer node id)
it wants to call the helper for.

> And now of course I can't figure out how to manually create a split-brain
> for testing. Tried dropping the network interface while both nodes were up
> and primary, but that doesn't seem to be enough. Is there a simple way to
> test?

With DRBD 8, that would be sufficient.

With DRBD 9, we only bump the data gen uuid one the first change (write)

so: drbdadm disconnect; then write to both; then try to connect.

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker

DRBD® and LINBIT® are registered trademarks of LINBIT
__
please don't Cc me, but send to list -- I'm subscribed