<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.30.3">
</HEAD>
<BODY>
Hi, been trying to google for this, but haven't found anything quite matching. Sorry if this is covered elsewhere.<BR>
<BR>
Using: RHEL 6 / DRBD 8.3.10-2 kmod from ElRepo / OCFS2 compiled from Redhat's kernel source.<BR>
<BR>
I have a setup with a Primary/Primary OCFS2 setup, which I set up using the following instructions:<BR>
<BR>
<A HREF="http://wiki.virtastic.com/display/howto/Clustered+Filesystem+with+DRBD+and+OCFS2+on+CentOS+5.5">http://wiki.virtastic.com/display/howto/Clustered+Filesystem+with+DRBD+and+OCFS2+on+CentOS+5.5</A><BR>
<BR>
Anyways, I have a pair of bonded GigE network ports hooked up to a switch to link up the two servers. I wanted it to be on a dedicated network, but networking just gave me regular IPs which are on the same subnet as the primary interfaces. SELinux is permissive. Everything was working, but then...<BR>
<BR>
I made a change to IPTables, and did a "service iptables restart", and next thing I knew, I had a split brain.<BR>
<BR>
And worse, somehow the OCFS2 filesystem started giving errors. I don't know if it really was a corruption, but the error messages came in pretty fast. I recovered from the split-brain manually, but that didn't stop the messages. It didn't clear up even with fsck.ocfs2. I finally had to find out what that inode was pointing to and remove it before the messages stopped.<BR>
<BR>
Testing later, it looks like sometimes when I restart IPtables, I get a split-brain. But I haven't replicated the OCFS2 corruption. I would have thought that short time that IPtables restarts in wouldn't cause a split-brain, but I guess it does sometimes. Not sure why it sometimes gets a split and sometimes not. Is this normal? Should I use "iptables -A" to add rules instead of doing a restart?<BR>
<BR>
Would posting the /etc/drbd.conf and /etc/sysconfig/iptables help? Any other info?<BR>
<BR>
I got the following /var/log/messages after restarting iptables:<BR>
<BR>
Apr 18 07:52:27 server-2 kernel: block drbd1: asender terminated<BR>
Apr 18 07:52:27 server-2 kernel: block drbd1: Terminating asender thread<BR>
Apr 18 07:52:27 server-2 kernel: block drbd1: sock_sendmsg returned -32<BR>
Apr 18 07:52:27 server-2 kernel: block drbd1: short sent ReportUUIDs size=56 sent=0<BR>
Apr 18 07:52:27 server-2 kernel: block drbd1: Connection closed<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: conn( NetworkFailure -> Unconnected )<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: receiver terminated<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: Restarting receiver thread<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: receiver (re)started<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: conn( Unconnected -> WFConnection )<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: Handshake successful: Agreed network protocol version 96<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: conn( WFConnection -> WFReportParams )<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: Starting asender thread (from drbd1_receiver [5944])<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: data-integrity-alg: <not-used><BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: drbd_sync_handshake:<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: self 7891B6FC1469AE31:F7F25E6B00607741:571973CB1489F5B9:571873CB1489F5B9 bits:1 flags:0<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: peer AA6330CFB23C2663:F7F25E6B00607741:571973CB1489F5B9:571873CB1489F5B9 bits:73 flags:0<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: uuid_compare()=100 by rule 90<BR>
Apr 18 07:52:28 server-2 kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: helper command: /sbin/drbdadm initial-split-brain minor-1 exit code 0 (0x0)<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: Split-Brain detected but unresolved, dropping connection!<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1<BR>
Apr 18 07:52:29 server-2 notify-split-brain.sh[8606]: invoked for res0<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: helper command: /sbin/drbdadm split-brain minor-1 exit code 0 (0x0)<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: conn( WFReportParams -> Disconnecting )<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: error receiving ReportState, l: 4!<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: meta connection shut down by peer.<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: asender terminated<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: Terminating asender thread<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: Connection closed<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: conn( Disconnecting -> StandAlone )<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: receiver terminated<BR>
Apr 18 07:52:29 server-2 kernel: block drbd1: Terminating receiver thread<BR>
<BR>
And after that, the following messages kept coming in really fast... rebooting, switching primary nodes, fsck, all didn't work. Only finding the actual owner of the inode and removing it worked:<BR>
<BR>
Apr 18 07:53:07 server-2 kernel: (8163,0):ocfs2_read_virt_blocks:853 ERROR: Inode #5377026 contains a hole at offset 466944<BR>
Apr 18 07:53:07 server-2 kernel: (8163,0):ocfs2_read_dir_block:533 ERROR: status = -5<BR>
Apr 18 07:53:08 server-2 kernel: (8163,12):ocfs2_read_virt_blocks:853 ERROR: Inode #5377026 contains a hole at offset 466944<BR>
Apr 18 07:53:08 server-2 kernel: (8163,12):ocfs2_read_dir_block:533 ERROR: status = -5<BR>
Apr 18 07:53:08 server-2 kernel: (8508,0):ocfs2_read_virt_blocks:853 ERROR: Inode #5377026 contains a hole at offset 466944<BR>
Apr 18 07:53:08 server-2 kernel: (8508,0):ocfs2_read_dir_block:533 ERROR: status = -5<BR>
Apr 18 07:53:08 server-2 kernel: (8508,0):ocfs2_read_virt_blocks:853 ERROR: Inode #5377026 contains a hole at offset 466944<BR>
Apr 18 07:53:08 server-2 kernel: (8508,0):ocfs2_read_dir_block:533 ERROR: status = -5<BR>
Apr 18 07:53:08 server-2 kernel: (8163,16):ocfs2_read_virt_blocks:853 ERROR: Inode #5377026 contains a hole at offset 466944<BR>
Apr 18 07:53:08 server-2 kernel: (8163,16):ocfs2_read_dir_block:533 ERROR: status = -5<BR>
<BR>
Thanks!<BR>
Herman<BR>
<BR>
</BODY>
</HTML>