[DRBD-user] notify-split-brain.sh[153967]: Environment variable $DRBD_PEER not found (this is normally passed in by drbdadm).

[Daniel Ragle]

On 9/18/2018 10:51 AM, Lars Ellenberg wrote:
> On Thu, Sep 13, 2018 at 04:36:54PM -0400, Daniel Ragle wrote:
>> Anybody know where I need to start looking to figure this one out:
>>
>> notify-split-brain.sh[153967]: Environment variable $DRBD_PEER not found
>> (this is normally passed in by drbdadm).
>>
>> I have a split-brain situation and apparently this notification script tried
>> to run but failed.
>>
>> This is in a pacemaker setup with ocf:linbit:drbd controlling DRBD.
>>
>> # drbdadm --version
>> DRBDADM_BUILDTAG=GIT-hash:\ fed9a1df82015e52c14c912fa4b93336e2ab4fcc\ build\
>> by\ mockbuild@\,\ 2018-04-20\ 19:30:46
>> DRBDADM_API_VERSION=2
>> DRBD_KERNEL_VERSION_CODE=0x09000e
>> DRBD_KERNEL_VERSION=9.0.14
>> DRBDADM_VERSION_CODE=0x090301
>> DRBDADM_VERSION=9.3.1
> 
> Yup, sorry, even though apparently they are still documented in the UG,
> those "notify*.sh" example scripts are not fit for DRBD 9 :-(
> 
> You want to configure proper fencing anyways,
> to *avoid* split brain instead of being "maybe"
> (or maybe not) notified by email.

Yah, in fact it was somewhere in the process of trying to get fencing 
configured (and testing) that I managed to end up with the split-brain. 
Unfortunately (see below) I don't know exactly what I did to trigger it.

> 
> For now, if you feel comfortable patching the script,
> you could try adding something like this:
> if [[ $DRBD_PEER_NODE_ID ]] ; then
> 	k=DRBD_NODE_ID_${DRBD_PEER_NODE_ID};
> 	v=${!k};
> 	[[ $v ]] && DRBD_PEER=$v;
> fi
> 
> 

Looks like it should work, I tried adding it, but now I can't figure out 
how to test it. I tried:

     drbdadm split-brain r0

And that gives me the same problem, as evidently that test doesn't set 
any of the DRBD_PEER vars, only:

     DRBD_CONF
     DRBD_LL_DISK
     DRBD_MINOR
     DRBD_MY_NODE_ID
     DRBD_NODE_ID_0
     DRBD_NODE_ID_1
     DRBD_RESOURCE
     DRBD_VOLUME

And now of course I can't figure out how to manually create a 
split-brain for testing. Tried dropping the network interface while both 
nodes were up and primary, but that doesn't seem to be enough. Is there 
a simple way to test?