Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi Digimer,
Thanks for help and explanation. I will try it out fencing option.
However, I would like to validate if what I am testing for split-brain is correct or not. Also what could be done for simple split-brain auto-recovery through configuration without fencing.
Regards,
Shailesh Vaidya
-----Original Message-----
From: Digimer [mailto:lists at alteeve.ca]
Sent: Wednesday, April 10, 2013 11:17 PM
To: Shailesh Vaidya
Cc: Dan Barker; drbd-user at lists.linbit.com
Subject: Re: [DRBD-user] Not able to test Automatic split brain recovery policies
I've not done fencing in DRBD alone, so I am unable to offer specific suggestions. I can speak to generally what you need though.
You can set DRBD's fencing policy to 'resource-and-stonith'. What this does is tell DRBD "When you lose your peer, block IO and call a fence against it". The fence action reaches out (usually via IPMI or managed
PDU) and forces the peer offline. After that, the surviving node will proceed. This way, at no time will both nodes be operating in StandAlone and Primary.
You will want to set a delay so that one of the nodes has a head start when trying to fence the other. This way, in your test, when the communication breaks but the nodes are still up, you remove the risk of both nodes being fenced. What this does is say "when you want to fence node 1, wait 15 seconds before doing so. when you want to fence node 2, don't wait and immediately fence". Thus, when it's a break in communications, you can predict which node will win the fence.
When a node really fails, it will obviously not try to fence, being dead, so the healthy node will always win the fence and then take over.
How you actually fence the peer will depends on what options you have available. Then you need a script that will actually do the work of reaching out and killing the peer. As I mentioned, this is usually done via IPMI (or branded out of band interfaces like iLO, DRAC, RSA, etc) or by using managed PDUs, like the APC AP7900. To do this, you need to have a scrip that reads certain environment variables set by DRBD, executes the request and then returns an appropriate exit code based on success or failure.
I wrote such a fence handler called "rhcs_fence" (based on
obliterate-peer.sh) which handles fencing by passing the request up to rhcs. You should be able to fairly easily adapt it to work with your setup.
https://github.com/digimer/rhcs_fence
Hope this helps clarify things.
digimer
On 04/10/2013 01:22 PM, Shailesh Vaidya wrote:
> Hi Don,
>
> Yup 8.3.8 is quit old but need to work with it for now.
>
> I am not using fencing and neither pacemaker or RHCS
>
> What I observed is that after split-brain its getting disconnected and dropped connection. both became unknown to each other. I am not sure is this issue with my test procedure itself.
>
> Do I need to make any additional configuration.
>
> Thanks,
> Shailesh Vaidya.
>
> ________________________________________
> From: Digimer [lists at alteeve.ca]
> Sent: Wednesday, April 10, 2013 8:11 PM
> To: Shailesh Vaidya
> Cc: Dan Barker; drbd-user at lists.linbit.com
> Subject: Re: [DRBD-user] Not able to test Automatic split brain recovery policies
>
> To your immediate problem;
>
> If you had configured fencing, drbd would not split-brain. Are you
> using pacemaker or RHCS?
>
> Secondly, 8.3.8 is very, very old. Upgrading to a newer 8.3.x version
> would be a good idea.
>
> Back to split-brain; DRBD declares a split-brain as soon as both nodes
> are StandAlone and Primary. To recover, you need to tell DRBD which
> node to consider "good" and then drop the changes on the peer and let
> the good node sync to the other node.
>
> On 04/10/2013 08:08 AM, Shailesh Vaidya wrote:
>> I have followed same procedure (disable Ethernet card) etc and after
>> that drbd status on both the nodes
>>
>> [root at drbd1 ~]# cat /proc/drbd
>>
>> version: 8.3.8 (api:88/proto:86-94)
>>
>> GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by
>> mockbuild at builder10.centos.org, 2010-06-04 08:04:09
>>
>> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
>>
>> ns:4 nr:0 dw:12 dr:82 al:1 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
>> oos:4
>>
>> [root at drbd1 ~]#
>>
>> [root at drbd2 ~]# cat /proc/drbd
>>
>> version: 8.3.8 (api:88/proto:86-94)
>>
>> GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by
>> mockbuild at builder10.centos.org, 2010-06-04 08:04:09
>>
>> 0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r----
>>
>> ns:0 nr:4 dw:56 dr:42 al:1 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
>> oos:4
>>
>> [root at drbd2 ~]#
>>
>> /var/log/messages shows
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm initial-split-brain minor-0
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: Split-Brain detected
>> but unresolved, dropping connection!
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm split-brain minor-0
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: conn( WFReportParams
>> -> Disconnecting )
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: error receiving
>> ReportState, l: 4!
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: asender terminated
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: Terminating asender
>> thread
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: Connection closed
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: conn( Disconnecting ->
>> StandAlone )
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: receiver terminated
>>
>> Apr 10 07:51:35 localhost kernel: block drbd0: Terminating receiver
>> thread
>>
>> Now if I do 'drbdadm connect r0' on both the machines then,
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: uuid_compare()=100 by
>> rule 90
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm initial-split-brain minor-0
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: Split-Brain detected,
>> 1 primaries, automatically solved. Sync from peer node
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: peer( Unknown ->
>> Primary
>> ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: conn( WFBitMapT ->
>> WFSyncUUID )
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm before-resync-target minor-0
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
>> /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: conn( WFSyncUUID ->
>> SyncTarget ) disk( UpToDate -> Inconsistent )
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: Began resync as
>> SyncTarget (will sync 4 KB [1 bits set]).
>>
>> Apr 10 07:56:37 localhost kernel: block drbd0: Resync done (total 1
>> sec; paused 0 sec; 4 K/sec)
>>
>> Regards,
>>
>> Shailesh Vaidya
>>
>> *From:*drbd-user-bounces at lists.linbit.com
>> [mailto:drbd-user-bounces at lists.linbit.com] *On Behalf Of *Dan Barker
>> *Sent:* Wednesday, April 10, 2013 5:16 PM
>> *To:* drbd-user at lists.linbit.com
>> *Subject:* Re: [DRBD-user] Not able to test Automatic split brain
>> recovery policies
>>
>> You don't show the status of the nodes, but I imagine you have two
>> primary nodes. There is no handler specified for two primary nodes.
>> Did you have two primary, disconnected nodes?
>>
>> It shouldn't be possible to create split brain without writing on
>> both nodes.
>>
>> Dan
>>
>> *From:*drbd-user-bounces at lists.linbit.com
>> <mailto:drbd-user-bounces at lists.linbit.com>
>> [mailto:drbd-user-bounces at lists.linbit.com] *On Behalf Of *Shailesh
>> Vaidya
>> *Sent:* Wednesday, April 10, 2013 1:58 AM
>> *To:* drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
>> *Subject:* [DRBD-user] Not able to test Automatic split brain
>> recovery policies
>>
>> Hello,
>>
>> I am using DRBD 8.3.8
>>
>> I have configured Automatic split brain recovery policies as below in
>> /etc/drbd.conf
>>
>> net {
>>
>> max-buffers 2048;
>>
>> ko-count 4;
>>
>> after-sb-0pri discard-zero-changes;
>>
>> after-sb-1pri discard-secondary;
>>
>> }
>>
>> My both machines are Virtual machines so not connected actual
>> back-to-back connection. To reproduce split-brain, I am using below
>> procedure,
>>
>> 1.On Primary disable Ethernet card from 'Virtual Machine properties'
>>
>> 2.Wait to Secondery to start switch over and again enable Ethernet
>> card on Primary
>>
>> Log shows mw that split-brain is occurred , however its shows
>> connection dropped.
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: uuid_compare()=100 by rule
>> 90
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command:
>> /sbin/drbdadm initial-split-brain minor-0
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command:
>> /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: Split-Brain detected but
>> unresolved, dropping connection!
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command:
>> /sbin/drbdadm split-brain minor-0
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command:
>> /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
>>
>> Apr 9 10:30:15 drbd1 kernel: block drbd0: conn( WFReportParams ->
>> Disconnecting )
>>
>> Full DRBD conf file
>>
>> [root at drbd1 ~]# cat /etc/drbd.conf
>>
>> global {
>>
>> usage-count no;
>>
>> }
>>
>> resource r0 {
>>
>> protocol C;
>>
>> #incon-degr-cmd "echo !DRBD! pri on incon-degr | wall ; sleep 60 ;
>> halt -f";
>>
>> on drbd1 {
>>
>> device /dev/drbd0;
>>
>> disk /dev/sda3;
>>
>> address 10.55.199.51:7789;
>>
>> meta-disk internal;
>>
>> }
>>
>> on drbd2 {
>>
>> device /dev/drbd0;
>>
>> disk /dev/sda3;
>>
>> address 10.55.199.52:7789;
>>
>> meta-disk internal;
>>
>> }
>>
>> disk {
>>
>> on-io-error detach;
>>
>> }
>>
>> net {
>>
>> max-buffers 2048;
>>
>> ko-count 4;
>>
>> after-sb-0pri discard-zero-changes;
>>
>> after-sb-1pri discard-secondary;
>>
>> }
>>
>> syncer {
>>
>> rate 25M;
>>
>> al-extents 257; # must be a prime number
>>
>> }
>>
>> startup {
>>
>> wfc-timeout 20;
>>
>> degr-wfc-timeout 120; # 2 minutes.
>>
>> }
>>
>> }
>>
>> [root at drbd1 ~]# vi /var/log/messages
>>
>> [root at drbd1 ~]#
>>
>> [root at drbd1 ~]# cat /etc/drbd.conf
>>
>> global {
>>
>> usage-count no;
>>
>> }
>>
>> resource r0 {
>>
>> protocol C;
>>
>> #incon-degr-cmd "echo !DRBD! pri on incon-degr | wall ; sleep 60 ;
>> halt -f";
>>
>> on drbd1 {
>>
>> device /dev/drbd0;
>>
>> disk /dev/sda3;
>>
>> address 10.55.199.51:7789;
>>
>> meta-disk internal;
>>
>> }
>>
>> on drbd2 {
>>
>> device /dev/drbd0;
>>
>> disk /dev/sda3;
>>
>> address 10.55.199.52:7789;
>>
>> meta-disk internal;
>>
>> }
>>
>> disk {
>>
>> on-io-error detach;
>>
>> }
>>
>> net {
>>
>> max-buffers 2048;
>>
>> ko-count 4;
>>
>> after-sb-0pri discard-zero-changes;
>>
>> after-sb-1pri discard-secondary;
>>
>> }
>>
>> syncer {
>>
>> rate 25M;
>>
>> al-extents 257; # must be a prime number
>>
>> }
>>
>> startup {
>>
>> wfc-timeout 20;
>>
>> degr-wfc-timeout 120; # 2 minutes.
>>
>> }
>>
>> }
>>
>> [root at drbd1 ~]#
>>
>> Is this configuration issue or my testing procedure is not proper?
>>
>> Regards,
>>
>> Shailesh Vaidya
>>
>> DISCLAIMER ========== This e-mail may contain privileged and
>> confidential information which is the property of Persistent Systems
>> Ltd. It is intended only for the use of the individual or entity to
>> which it is addressed. If you are not the intended recipient, you are
>> not authorized to read, retain, copy, print, distribute or use this
>> message. If you have received this communication in error, please
>> notify the sender and delete all copies of this message. Persistent
>> Systems Ltd. does not accept any liability for virus infected mails.
>>
>> DISCLAIMER ========== This e-mail may contain privileged and
>> confidential information which is the property of Persistent Systems
>> Ltd. It is intended only for the use of the individual or entity to
>> which it is addressed. If you are not the intended recipient, you are
>> not authorized to read, retain, copy, print, distribute or use this
>> message. If you have received this communication in error, please
>> notify the sender and delete all copies of this message. Persistent
>> Systems Ltd. does not accept any liability for virus infected mails.
>>
>>
>>
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user at lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>
>
> --
> Digimer
> Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer
> is trapped in the mind of a person without access to education?
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education?
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.