<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.26.3">
</HEAD>
<BODY>
Hello all<BR>
<BR>
I have RHEL 5.3 (CentOS), drbd 8.3.1, 2.6.18-164.el5xen kernel.<BR>
<BR>
I have set up a two nodes cluster running among other services, BOINC on both nodes.<BR>
<BR>
For some reason BOINC always makes both nodes to crash / being fenced after a few hours of operation.<BR>
<BR>
However the issue is that ALWAYS after reboot, drbd (which runs the cluster storage) does not recover.<BR>
<BR>
This is the log from node 1:<BR>
...............<BR>
Oct 11 11:59:42 localhost kernel: drbd2: conn( WFBitMapT -> WFSyncUUID ) <BR>
Oct 11 11:59:42 localhost clurgmgrd: [4307]: <info> Executing /etc/init.d/drbd status <BR>
Oct 11 11:59:42 localhost kernel: drbd2: helper command: /sbin/drbdadm before-resync-target minor-2<BR>
Oct 11 11:59:42 localhost kernel: drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)<BR>
Oct 11 11:59:42 localhost kernel: drbd2: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) <BR>
Oct 11 11:59:42 localhost kernel: drbd2: Began resync as SyncTarget (will sync 2031616 KB [507904 bits set]).<BR>
Oct 11 11:59:43 localhost kernel: drbd1: peer( Secondary -> Primary ) <BR>
Oct 11 11:59:43 localhost kernel: drbd2: peer( Secondary -> Primary ) <BR>
Oct 11 11:59:43 localhost kernel: drbd0: role( Secondary -> Primary ) <BR>
Oct 11 11:59:43 localhost kernel: drbd1: role( Secondary -> Primary ) <BR>
Oct 11 11:59:44 localhost kernel: drbd2: role( Secondary -> Primary ) <BR>
Oct 11 11:59:44 localhost kernel: drbd2: Resync done (total 45 sec; paused 0 sec; 45144 K/sec)<BR>
Oct 11 11:59:44 localhost kernel: drbd2: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) <BR>
Oct 11 11:59:45 localhost kernel: drbd2: helper command: /sbin/drbdadm after-resync-target minor-2<BR>
Oct 11 11:59:46 localhost kernel: drbd2: helper command: /sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)<BR>
Oct 11 12:00:08 localhost kernel: drbd0: Resync done (total 84 sec; paused 0 sec; 12824 K/sec)<BR>
Oct 11 12:00:08 localhost kernel: drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) <BR>
Oct 11 12:00:08 localhost kernel: drbd0: helper command: /sbin/drbdadm after-resync-target minor-0<BR>
Oct 11 12:00:08 localhost kernel: drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)<BR>
Oct 11 12:00:13 localhost clurgmgrd: [4307]: <info> Executing /etc/init.d/drbd status <BR>
Oct 11 12:00:43 localhost clurgmgrd: [4307]: <info> Executing /etc/init.d/drbd status <BR>
Oct 11 12:01:13 localhost clurgmgrd: [4307]: <info> Executing /etc/init.d/drbd status <BR>
Oct 11 12:01:33 localhost kernel: drbd0: peer( Primary -> Secondary ) <BR>
Oct 11 12:01:33 localhost kernel: drbd0: State change failed: Refusing to be Primary while peer is not outdated<BR>
Oct 11 12:01:33 localhost kernel: drbd0: state = { cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate r--- }<BR>
Oct 11 12:01:33 localhost kernel: drbd0: wanted = { cs:TearDown ro:Primary/Unknown ds:UpToDate/DUnknown s--- }<BR>
Oct 11 12:01:33 localhost kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> Outdated ) <BR>
Oct 11 12:01:33 localhost kernel: drbd0: Creating new current UUID<BR>
Oct 11 12:01:33 localhost kernel: drbd0: meta connection shut down by peer.<BR>
Oct 11 12:01:33 localhost kernel: drbd0: asender terminated<BR>
Oct 11 12:01:33 localhost kernel: drbd0: Terminating asender thread<BR>
Oct 11 12:01:34 localhost kernel: drbd0: Connection closed<BR>
Oct 11 12:01:34 localhost kernel: drbd0: conn( TearDown -> Unconnected ) <BR>
Oct 11 12:01:34 localhost kernel: drbd0: receiver terminated<BR>
Oct 11 12:01:34 localhost kernel: drbd0: Restarting receiver thread<BR>
Oct 11 12:01:34 localhost kernel: drbd0: receiver (re)started<BR>
Oct 11 12:01:34 localhost kernel: drbd0: conn( Unconnected -> WFConnection ) <BR>
Oct 11 12:01:34 localhost kernel: drbd1: peer( Primary -> Secondary ) <BR>
Oct 11 12:01:34 localhost kernel: drbd1: State change failed: Refusing to be Primary without at least one UpToDate disk<BR>
Oct 11 12:01:34 localhost kernel: drbd1: state = { cs:SyncTarget ro:Primary/Secondary ds:Inconsistent/UpToDate r--- }<BR>
Oct 11 12:01:34 localhost kernel: drbd1: wanted = { cs:TearDown ro:Primary/Unknown ds:Inconsistent/DUnknown s--- }<BR>
Oct 11 12:01:34 localhost kernel: drbd1: State change failed: Refusing to be Primary without at least one UpToDate disk<BR>
Oct 11 12:01:34 localhost kernel: drbd1: state = { cs:SyncTarget ro:Primary/Secondary ds:Inconsistent/UpToDate r--- }<BR>
Oct 11 12:01:34 localhost kernel: drbd1: wanted = { cs:TearDown ro:Primary/Unknown ds:Inconsistent/Outdated r--- }<BR>
Oct 11 12:01:34 localhost kernel: drbd2: peer( Primary -> Secondary ) <BR>
Oct 11 12:01:34 localhost kernel: drbd2: State change failed: Refusing to be Primary while peer is not outdated<BR>
Oct 11 12:01:34 localhost kernel: drbd2: state = { cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate r--- }<BR>
Oct 11 12:01:34 localhost kernel: drbd2: wanted = { cs:TearDown ro:Primary/Unknown ds:UpToDate/DUnknown s--- }<BR>
Oct 11 12:01:34 localhost kernel: drbd2: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> Outdated ) <BR>
Oct 11 12:01:34 localhost kernel: drbd2: Creating new current UUID<BR>
Oct 11 12:01:34 localhost kernel: drbd2: meta connection shut down by peer.<BR>
Oct 11 12:01:34 localhost kernel: drbd2: asender terminated<BR>
Oct 11 12:01:34 localhost kernel: drbd2: Terminating asender thread<BR>
Oct 11 12:01:34 localhost kernel: drbd2: Connection closed<BR>
Oct 11 12:01:34 localhost kernel: drbd2: conn( TearDown -> Unconnected ) <BR>
Oct 11 12:01:34 localhost kernel: drbd2: receiver terminated<BR>
Oct 11 12:01:34 localhost kernel: drbd2: Restarting receiver thread<BR>
Oct 11 12:01:34 localhost kernel: drbd2: receiver (re)started<BR>
Oct 11 12:01:34 localhost kernel: drbd2: conn( Unconnected -> WFConnection ) <BR>
Oct 11 12:01:38 localhost kernel: drbd1: State change failed: Refusing to be Primary without at least one UpToDate disk<BR>
Oct 11 12:01:38 localhost kernel: drbd1: state = { cs:SyncTarget ro:Primary/Secondary ds:Inconsistent/UpToDate r--- }<BR>
Oct 11 12:01:38 localhost kernel: drbd1: wanted = { cs:TearDown ro:Primary/Unknown ds:Inconsistent/DUnknown s--- }<BR>
Oct 11 12:01:38 localhost kernel: drbd1: State change failed: Refusing to be Primary without at least one UpToDate disk<BR>
Oct 11 12:01:38 localhost kernel: drbd1: state = { cs:SyncTarget ro:Primary/Secondary ds:Inconsistent/UpToDate r--- }<BR>
Oct 11 12:01:38 localhost kernel: drbd1: wanted = { cs:TearDown ro:Primary/Unknown ds:Inconsistent/Outdated r--- }<BR>
<BR>
<BR>
And this is the log from node 2:<BR>
.........<BR>
drbd1: conn( WFBitMapS -> SyncSource ) <BR>
drbd1: Began resync as SyncSource (will sync 300165324 KB [75041331 bits set]).<BR>
drbd2: conn( WFBitMapS -> SyncSource ) pdsk( Outdated -> Inconsistent ) <BR>
drbd2: Began resync as SyncSource (will sync 2031616 KB [507904 bits set]).<BR>
drbd1: role( Secondary -> Primary ) <BR>
drbd2: role( Secondary -> Primary ) <BR>
drbd0: peer( Secondary -> Primary ) <BR>
drbd1: peer( Secondary -> Primary ) <BR>
drbd2: peer( Secondary -> Primary ) <BR>
virbr0: no IPv6 routers present<BR>
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs<BR>
drbd2: Resync done (total 45 sec; paused 0 sec; 45144 K/sec)<BR>
drbd2: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) <BR>
eth1: too many iterations (16) in nv_nic_irq.<BR>
drbd0: Resync done (total 84 sec; paused 0 sec; 12824 K/sec)<BR>
drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) <BR>
dlm: Using TCP for communications<BR>
dlm: got connection from 2<BR>
drbd0: role( Primary -> Secondary ) <BR>
drbd0: Requested state change failed by peer: Refusing to be Primary while peer is not outdated (-7)<BR>
drbd0: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) disk( UpToDate -> Outdated ) pdsk( UpToDate -> DUnknown ) <BR>
drbd0: short read expecting header on sock: r=-512<BR>
drbd0: asender terminated<BR>
drbd0: Terminating asender thread<BR>
drbd0: Connection closed<BR>
drbd0: conn( Disconnecting -> StandAlone ) <BR>
drbd0: receiver terminated<BR>
drbd0: Terminating receiver thread<BR>
drbd0: disk( Outdated -> Diskless ) <BR>
drbd0: drbd_bm_resize called with capacity == 0<BR>
drbd0: worker terminated<BR>
drbd0: Terminating worker thread<BR>
drbd1: role( Primary -> Secondary ) <BR>
drbd1: Requested state change failed by peer: Refusing to be Primary without at least one UpToDate disk (-2)<BR>
drbd1: Requested state change failed by peer: Refusing to be Primary without at least one UpToDate disk (-2)<BR>
drbd1: State change failed: Refusing to be inconsistent on both nodes<BR>
drbd1: state = { cs:SyncSource ro:Secondary/Primary ds:UpToDate/Inconsistent r--- }<BR>
drbd1: wanted = { cs:SyncSource ro:Secondary/Primary ds:Diskless/Inconsistent r--- }<BR>
drbd2: role( Primary -> Secondary ) <BR>
drbd2: Requested state change failed by peer: Refusing to be Primary while peer is not outdated (-7)<BR>
drbd2: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) disk( UpToDate -> Outdated ) pdsk( UpToDate -> DUnknown ) <BR>
drbd2: short read expecting header on sock: r=-512<BR>
drbd2: asender terminated<BR>
drbd2: Terminating asender thread<BR>
drbd2: Connection closed<BR>
drbd2: conn( Disconnecting -> StandAlone ) <BR>
drbd2: receiver terminated<BR>
drbd2: Terminating receiver thread<BR>
drbd2: disk( Outdated -> Diskless ) <BR>
drbd2: drbd_bm_resize called with capacity == 0<BR>
drbd2: worker terminated<BR>
drbd2: Terminating worker thread<BR>
drbd1: Requested state change failed by peer: Refusing to be Primary without at least one UpToDate disk (-2)<BR>
drbd1: Requested state change failed by peer: Refusing to be Primary without at least one UpToDate disk (-2)<BR>
drbd1: State change failed: Refusing to be inconsistent on both nodes<BR>
drbd1: state = { cs:SyncSource ro:Secondary/Primary ds:UpToDate/Inconsistent r--- }<BR>
drbd1: wanted = { cs:SyncSource ro:Secondary/Primary ds:Diskless/Inconsistent r--- }<BR>
<BR>
This is my config:<BR>
<BR>
global {<BR>
# minor-count 64;<BR>
<BR>
# dialog-refresh 5; # 5 seconds<BR>
<BR>
# disable-ip-verification;<BR>
<BR>
usage-count yes;<BR>
}<BR>
<BR>
<BR>
common {<BR>
<BR>
protocol C;<BR>
<BR>
syncer {<BR>
<BR>
rate 100M;<BR>
<BR>
al-extents 257;<BR>
}<BR>
<BR>
<BR>
handlers {<BR>
<BR>
pri-on-incon-degr "echo b > /proc/sysrq-trigger ; reboot -f";<BR>
<BR>
pri-lost-after-sb "echo b > /proc/sysrq-trigger ; reboot -f";<BR>
<BR>
local-io-error "echo o > /proc/sysrq-trigger ; halt -f";<BR>
<BR>
outdate-peer "/sbin/obliterate";<BR>
<BR>
pri-lost "echo pri-lost. Have a look at the log files. | mail -s 'DRBD Alert' root; echo b > /proc/sysrq-trigger ; reboot -f";<BR>
<BR>
split-brain "echo split-brain. drbdadm -- --discard-my-data connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' root";<BR>
<BR>
}<BR>
<BR>
startup {<BR>
<BR>
wfc-timeout 100;<BR>
<BR>
degr-wfc-timeout 60; # 1 minutes.<BR>
<BR>
become-primary-on both;<BR>
<BR>
}<BR>
<BR>
disk {<BR>
<BR>
fencing resource-and-stonith;<BR>
<BR>
}<BR>
<BR>
net {<BR>
<BR>
sndbuf-size 512k;<BR>
<BR>
timeout 60; # 6 seconds (unit = 0.1 seconds)<BR>
connect-int 10; # 10 seconds (unit = 1 second)<BR>
ping-int 10; # 10 seconds (unit = 1 second)<BR>
ping-timeout 50; # 500 ms (unit = 0.1 seconds)<BR>
<BR>
max-buffers 2048;<BR>
<BR>
max-epoch-size 2048;<BR>
<BR>
ko-count 10;<BR>
<BR>
allow-two-primaries;<BR>
<BR>
cram-hmac-alg "sha1";<BR>
shared-secret "*****";<BR>
<BR>
after-sb-0pri discard-least-changes;<BR>
<BR>
after-sb-1pri violently-as0p;<BR>
<BR>
after-sb-2pri violently-as0p;<BR>
<BR>
rr-conflict call-pri-lost;<BR>
<BR>
<BR>
<BR>
data-integrity-alg "crc32c";<BR>
<BR>
}<BR>
<BR>
<BR>
}<BR>
<BR>
<BR>
resource r0 {<BR>
<BR>
device /dev/drbd0;<BR>
disk /dev/hda4;<BR>
meta-disk internal;<BR>
<BR>
on tweety-1 { address 10.254.254.253:7788; }<BR>
<BR>
on tweety-2 { address 10.254.254.254:7788; }<BR>
<BR>
}<BR>
<BR>
resource r1 {<BR>
<BR>
device /dev/drbd1;<BR>
disk /dev/hdb4;<BR>
meta-disk internal;<BR>
<BR>
on tweety-1 { address 10.254.254.253:7789; }<BR>
<BR>
on tweety-2 { address 10.254.254.254:7789; }<BR>
}<BR>
<BR>
resource r2 {<BR>
<BR>
device /dev/drbd2;<BR>
disk /dev/sda1;<BR>
meta-disk internal;<BR>
<BR>
on tweety-1 { address 10.254.254.253:7790; }<BR>
<BR>
on tweety-2 { address 10.254.254.254:7790; }<BR>
}<BR>
<BR>
<BR>
This is the current status of drdb (after the above logged crash)<BR>
<BR>
version: 8.3.1 (api:88/proto:86-89)<BR>
GIT-hash: fd40f4a8f9104941537d1afc8521e584a6d3003c build by root@tweety-1, 2009-09-21 17:23:59<BR>
0: cs:Unconfigured<BR>
1: cs:SyncSource ro:Secondary/Primary ds:UpToDate/Inconsistent C r---<BR>
ns:64823832 nr:0 dw:0 dr:64831176 al:0 bm:4089 lo:35 pe:83 ua:256 ap:0 ep:1 wo:b oos:235344204<BR>
[===>................] sync'ed: 21.6% (229828/293128)M<BR>
finish: 14:51:27 speed: 4,064 (45,264) K/sec<BR>
2: cs:Unconfigured<BR>
<BR>
Can someone give some food for thoughts on I have done wrong?<BR>
<BR>
Thank you All for your time,<BR>
<BR>
Theophanis Kontogiannis<BR>
<BR>
</BODY>
</HTML>