[DRBD-user] Secondary node io-error

Velayutham, Prakash Prakash.Velayutham at cchmc.org
Sun Oct 14 15:41:43 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Oct 10, 2012, at 10:01 PM, Velayutham, Prakash wrote:

> On Oct 10, 2012, at 5:01 AM, Lars Ellenberg wrote:
> 
>> On Wed, Oct 10, 2012 at 03:42:02AM +0000, Velayutham, Prakash wrote:
>>> 
>>> On Oct 8, 2012, at 9:19 AM, Velayutham, Prakash wrote:
>>> 
>>>> On Oct 8, 2012, at 4:55 AM, Lars Ellenberg wrote:
>>>> 
>>>>> On Sat, Oct 06, 2012 at 01:08:43PM +0000, Velayutham, Prakash wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> I recently got a DRBD (8.4.2-2) cluster up (still testing). It seems to work nicely with Pacemaker CRM in several scenarios I have tested. Here is my config.
>>>>>> 
>>>>>> global {
>>>>>>             usage-count     yes;
>>>>>> }
>>>>>> 
>>>>>> common {
>>>>>>     handlers {
>>>>>>             outdate-peer    /usr/lib/drbd/crm-fence-peer.sh;
>>>>>>             fence-peer      /usr/lib/drbd/crm-fence-peer.sh;
>>>>>>             after-resync-target     /usr/lib/drbd/crm-unfence-peer.sh;
>>>>>>             local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
>>>>>>             split-brain "/usr/lib/drbd/notify-split-brain.sh root";
>>>>>>     }
>>>>>> 
>>>>>>     startup {
>>>>>>             degr-wfc-timeout        0;
>>>>>>     }
>>>>>> 
>>>>>>     net {
>>>>>>             shared-secret   1QP69G4kWDslx2TMiaEStI6bwaGH5y8d;
>>>>>>             after-sb-0pri discard-zero-changes;
>>>>>>             after-sb-1pri discard-secondary;
>>>>>>             after-sb-2pri disconnect;
>>>>>>     }
>>>>>> 
>>>>>>     disk {
>>>>>>             on-io-error     call-local-io-error;
>>>>>>             fencing resource-and-stonith;
>>>>>>     }
>>>>>> 
>>>>>> }
>>>>>> 
>>>>>> The io-error handler only gets called when the primary node has a disk
>>>>>> issue. I have not seen the secondary node call the "local-io-error"
>>>>>> handler when it had disk access issues. Is this by design?
>>>>> 
>>>>> No.
>>>>> 
>>>>> "Works for me", though.
>>>>> 
>>>>> Can you please double check?
>>>>> And if in fact you can reproduce, tell us how, including logs?
>> 
>>>> If I disable all the FC ports in the fiber switch just for the
>>>> primary node, the node fences, reboots and comes up, as I would
>>>> expect. With the exact same config, if I disable the FC ports just
>>>> for the secondary node, the node just sits there and it even shows
>>>> up as Secondary in /proc/drbd.
>> 
>>>> That sounds odd and sounds like the
>>>> config should be "diskless", but it is "call-local-io-error".
>> 
>> Huh? What has "config" to do with things,
>> and what exactly is "config diskless"?
>> 
>> 
>>>> Which logs are you wanting me to share?
>> 
>> Those that show DRBD detecting an IO error,
>> but not calling the io-error handler.
>> 
>>>> Thanks,
>>>> Prakash
>>> 
>>> Just wanted to add this. I repeated my test again and get the exact
>>> same results again. Here is /proc/drbd of the primary (bmimysqlt3) and
>>> secondary (bmimysqlt4) before the secondary's disk is cut off
>>> (disabling the fiber switch port that the secondary is connected to)
>>> 
>>> [root at bmimysqlt3 ~]# cat /proc/drbd 
>>> version: 8.4.2 (api:1/proto:86-101)
>>> GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root at bmimysqlt3.chmcres.cchmc.org, 2012-10-02 00:02:32
>>> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
>>>   ns:184 nr:0 dw:160 dr:14317 al:6 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
>>> 
>>> [root at bmimysqlt4 ~]# cat /proc/drbd 
>>> version: 8.4.2 (api:1/proto:86-101)
>>> GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root at bmimysqlt3.chmcres.cchmc.org, 2012-10-02 00:02:32
>>> 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
>>>   ns:0 nr:184 dw:184 dr:0 al:0 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
>>> 
>>> Here is /proc/drbd of primary and secondary about 5 minutes after the disk is cut off.
>>> 
>>> [root at bmimysqlt3 ~]# cat /proc/drbd 
>>> version: 8.4.2 (api:1/proto:86-101)
>>> GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root at bmimysqlt3.chmcres.cchmc.org, 2012-10-02 00:02:32
>>> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
>>>   ns:184 nr:0 dw:160 dr:14317 al:6 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
>> 
>> No additional writes.
>> 
>>> [root at bmimysqlt4 ~]# cat /proc/drbd 
>>> version: 8.4.2 (api:1/proto:86-101)
>>> GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root at bmimysqlt3.chmcres.cchmc.org, 2012-10-02 00:02:32
>>> 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
>>>   ns:0 nr:184 dw:184 dr:0 al:0 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
>> 
>> Nothing transfered, nothing written, nothing changed.
>> 
>>> As you can see, there is absolutely nothing there to suggest that the
>>> secondary even noticed the io-error.
>>> 
>>> I can't understand what is going on.
>> 
>> Do you realize that you need to do IO to get (and then be able to notice) IO errors?
>> 
>> Cheers,
>> 
>> 	Lars
> 
> Wow, feeling like an idiot now. Sorry for the false alarm. I just got confused because the primary node got fenced right away without any sort of manual write operation from me, but the secondary did not exhibit that same behavior.
> 
> Thanks,
> Prakash

However, I have hit the snag again, in a different scenario.

/etc/drbd.d/global_common.conf:

global {
		usage-count	yes;
}

common {
	startup {
		degr-wfc-timeout	0;
	}

	net {
		cram-hmac-alg	sha1;
		shared-secret	xxxxxx;
	}

	disk {
		on-io-error	call-local-io-error;
	}

}

/etc/drbd.d/mysql1.res:

resource mysql1 {

	handlers {
		fence-peer	/usr/lib/drbd/crm-fence-peer.sh;
		local-io-error	"/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
		split-brain	"/usr/lib/drbd/notify-split-brain.sh root";
		after-resync-target	/usr/lib/drbd/crm-unfence-peer.sh;
	}

	net {
		after-sb-0pri	discard-zero-changes;
		after-sb-1pri	discard-secondary;
	}

	disk {
		fencing	resource-and-stonith;
	}

	on node1 {
		volume 0 {
			device		/dev/drbd1;
			disk		/dev/mapper/mysql_data1;
			flexible-meta-disk	internal;
		}
		address		x.x.x.x:7788;
	}
	on node2 {
		volume 0 {
			device		/dev/drbd1;
			disk		/dev/mapper/mysql_data1;
			flexible-meta-disk	internal;
		}
		address		x.x.x.x:7788;
	}
}


/etc/drbd.d/mysql2.res:

resource mysql2 {

	handlers {
		fence-peer	/usr/lib/drbd/crm-fence-peer.sh;
		local-io-error	"/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
		split-brain	"/usr/lib/drbd/notify-split-brain.sh root";
		after-resync-target	/usr/lib/drbd/crm-unfence-peer.sh;
	}

	net {
		after-sb-0pri	discard-zero-changes;
		after-sb-1pri	discard-secondary;
	}

	disk {
		fencing	resource-and-stonith;
	}

	on node1 {
		volume 0 {
			device		/dev/drbd2;
			disk		/dev/mapper/mysql_data2;
			flexible-meta-disk	internal;
		}
		address		x.x.x.x:7789;
	}
	on node2 {
		volume 0 {
			device		/dev/drbd2;
			disk		/dev/mapper/mysql_data2;
			flexible-meta-disk	internal;
		}
		address		x.x.x.x:7789;
	}
}

So there are 2 resources (mysql1, mysql2)

mysql1 is primary on node1, secondary on node2. An ext4 file system on this DRBD volume (/fs1).
mysql2 is primary on node2, secondary on node1. An ext4 file system on this DRBD volume (/fs2).

If I disable the FC ports on node1, it reboots instantaneously. No need to create any IO for this to happen. mysql1 gets promoted to primary on node2 and all is fine.

But, if I disable the FC ports on node2, nothing happens. Even if I do "mkdir /fs2/newdir", the command just hangs. I would expect that command to create the necessary IO error and reboot the node. Any thoughts?

Thanks,
Prakash


More information about the drbd-user mailing list