<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'><font size="3">Hi Felix,</font><div style="color: rgb(0, 0, 0); font-size: 12pt; font-family: 'Times New Roman'; "><br></div><div style="color: rgb(0, 0, 0); font-size: 12pt; font-family: 'Times New Roman'; " id=""><span style="font-family: Helvetica, Arial, sans-serif; ">>> Jan 26 15:44:14 node1 kernel: [177694.517283] block drbd0: Requested</span><br style="font-family: Helvetica, Arial, sans-serif; "><span style="font-family: Helvetica, Arial, sans-serif; ">>> state change failed by peer : Refusing to be Primary while peer is not</span><br style="font-family: Helvetica, Arial, sans-serif; "><span style="font-family: Helvetica, Arial, sans-serif; ">>> outdated (-7)</span><br style="font-family: Helvetica, Arial, sans-serif; "><br style="font-family: Helvetica, Arial, sans-serif; "><span style="font-family: Helvetica, Arial, sans-serif; ">> This is odd. I don't think DRBD should attempt to become primary when</span><br style="font-family: Helvetica, Arial, sans-serif; "><span style="font-family: Helvetica, Arial, sans-serif; ">> you issue a stop command/</span></div><div style="color: rgb(0, 0, 0); font-size: 12pt; font-family: 'Times New Roman'; " id="">I agree. I don't understand why this happens when I attempt to stop it and remove the module. Does anyone know what this error means and why it would occur when attempting to stop DRBD?</div><div style="color: rgb(0, 0, 0); font-size: 12pt; font-family: 'Times New Roman'; " id=""><br></div><div style="color: rgb(0, 0, 0); font-size: 12pt; font-family: 'Times New Roman'; ">> <span style="font-family: Helvetica, Arial, sans-serif; ">Shouldn't pacemaker be stopping and starting this service for you?</span></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif">It is, however I discovered that entries for DRBD still existed in /etc/rc* - I removed those so now pacemaker is the only way to start/stop DRBD.</font></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif"><br></font></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif">> </font><span style="font-family: Helvetica, Arial, sans-serif; font-size: 12pt; ">I'm not sure it's normal for DRBD to outdate its disk on disconnection, </span><span style="font-family: Helvetica, Arial, sans-serif; font-size: 12pt; ">but it does seem to make sense.</span></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif">This is probably because I have configured resource-level fencing using dopd. I believe the peer would fence this node once it loses connection</font></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif"><br></font></div><div id="" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif" id="">After removing the /etc/rc*drbd entries and rebooting the nodes a couple of times, I am now able to stop heartbeat, which stops DRBD successfully. Now, however, one of my DRBD resources (ms_drbd_mount2) will not promote to master:</font></div><div id=""><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">Online: [ node1 ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">OFFLINE: [ node2 ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"><br></font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Master/Slave Set: ms_drbd_mount1 [p_drbd_mount1]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Masters: [ node1 ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Stopped: [ p_drbd_mount1:1 ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Master/Slave Set: ms_drbd_mount2 [p_drbd_mount2]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Slaves: [ node1 ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Stopped: [ p_drbd_mount2:1 ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Resource Group: g_apache</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> p_fs_varwww (ocf::heartbeat:Filesystem): Started node1</font></div><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> p_apache (ocf::heartbeat:apache): Started node1</font></div><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> Resource Group: g_mount1</font></div><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> p_fs_mount1 (ocf::heartbeat:Filesystem): Started node1</font></div><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="2"> p_ip_nfs (ocf::heartbeat:IPaddr2): Started node1</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; "><br></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; ">Any attempt to start it via crm resource [promote|start|stop|cleanup] does nothing. I am able to manually set the DRBD resource as primary. I took node2 offline in the hopes that it would start with just one node active, but it still remains slave. I see some error messages in the log about migrating the resource from node2:</div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; ">pengine: [30681]: WARN: common_apply_stickiness: Forcing ms_drbd_crm away from node after 1000000 failures (max=1000000)</div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; "><br></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; ">However, shouldn't it have migrated already when that node went offline? How can I what is preventing the DRBD resource from being promoted? The syslog contains</div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">crmd: [30323]: info: te_rsc_command: Initiating action 43: monitor p_drbd_mount2:0_monitor_30000 on node1 (local)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">crmd: [30323]: info: do_lrm_rsc_op: Performing key=43:111:0:f84ff0aa-9a17-4b66-954d-8c3011a3441e op=p_drbd_mount2:0_monitor_30000 )</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">lrmd: [30320]: info: rsc:p_drbd_mount2:0 monitor[192] (pid 14960)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">lrmd: [30320]: info: operation monitor[192] on p_drbd_mount2:0 for client 30323: pid 14960 exited with return code 0</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">crmd: [30323]: info: process_lrm_event: LRM operation p_drbd_mount2:0_monitor_30000 (call=192, rc=0, cib-update=619, confirmed=false) ok</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">crmd: [30323]: info: match_graph_event: Action p_drbd_mount2:0_monitor_30000 (43) confirmed on node1 (rc=0)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: unpack_rsc_op: Operation p_drbd_mount1:0_last_failure_0 found resource p_drbd_mount1:0 active in master mode on node1</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: unpack_rsc_op: Operation p_drbd_mount2:0_last_failure_0 found resource p_drbd_mount2:0 active on node1</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: common_apply_stickiness: ms_drbd_mount1 can fail 999998 more times on node2 before being forced off</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: common_apply_stickiness: ms_drbd_mount1 can fail 999998 more times on node2 before being forced off</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: WARN: common_apply_stickiness: Forcing ms_drbd_mount2 away from node2 after 1000000 failures (max=1000000)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: WARN: common_apply_stickiness: Forcing ms_drbd_mount2 away from node2 after 1000000 failures (max=1000000)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: LogActions: Leave p_drbd_mount1:0#011(Master node1)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: LogActions: Leave p_drbd_mount1:1#011(Stopped)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: LogActions: Leave p_drbd_mount2:0#011(Slave node1)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">pengine: [30681]: notice: LogActions: Leave p_drbd_mount2:1#011(Stopped)</font></div></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; "><br></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; ">I've attached my configuration (as outputted by crm configure show).</div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; "><br></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; ">Thanks,</div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; "><br></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="font-family: Helvetica, Arial, sans-serif; ">Andrew</div></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><font face="Helvetica, Arial, sans-serif"><br></font><br><hr id="zwchr" style="font-family: 'Times New Roman'; "><div style="color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt; " id=""><b>From: </b><span>"Felix Frank" <<a class="smarterwiki-linkify" href="mailto:ff@mpexnet.de" title="[GMCP] Compose a new mail to ff@mpexnet.de" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=ff@mpexnet.de','Compose new message','width=640,height=480');return false">ff@mpexnet.de</a>></span><br><b>To: </b><span>"Andrew Martin" <<a class="smarterwiki-linkify" href="mailto:amartin@xes-inc.com" title="[GMCP] Compose a new mail to amartin@xes-inc.com" rel="noreferrer" target="_blank" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=amartin@xes-inc.com','Compose new message','width=640,height=480');return false">amartin@xes-inc.com</a>></span><br><b>Sent: </b>Friday, January 27, 2012 2:52:05 AM<br><b>Subject: </b>Re: [DRBD-user] Removing DRBD Kernel Module Blocks<br><br>Hi,<br><br>On 01/26/2012 11:18 PM, Andrew Martin wrote:<br>> I am using DRBD with pacemaker+heartbeat for a HA cluster. There are no<br><br>fair choice.<br><br>> mounted filesystems at this time. Below is a copy of the kernel log<br><br>So the DRBDs are idle and managed by pacemaker, correct?<br><br>> after I attempted to stop the drbd service:<br><br>Shouldn't pacemaker be stopping and starting this service for you?<br><br>> Jan 26 15:44:14 node1 kernel: [177694.517283] block drbd0: Requested<br>> state change failed by peer : Refusing to be Primary while peer is not<br>> outdated (-7)<br><br>This is odd. I don't think DRBD should attempt to become primary when<br>you issue a stop command/<br><br>> Jan 26 15:44:14 node1 kernel: [177694.873466] block drbd0: peer( Primary<br>> -> Unknown ) conn( Connected -> Disconnecting ) disk( UpToDate -><br>> Outdated ) pdsk( UpToDate -> DUnknown )<br><br>I'm not sure it's normal for DRBD to outdate its disk on disconnection,<br>but it does seem to make sense.<br><br>> Jan 26 15:44:14 node1 kernel: [177695.209668] block drbd0: disk(<br>> Outdated -> Diskless )<br><br>This looks funny as well. But may just be correct.<br><br>Do you stop pacemaker before stopping DRBD?<br>What happens if you disable pacemaker, drbdadm down all and then stop DRBD?<br><br>Regards,<br>Felix<br></div><br></div></div></body></html>