<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 20, 2020 at 4:17 AM Jeremy Faith <<a href="mailto:jeremy.faith@jci.com">jeremy.faith@jci.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">Hi,</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<pre><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span>drbd</span>90 kernel module version:9.0.22-2(also 9.0.25-1 compiled from source)
<span>drbd</span>90-utils:9.12.2-1
kernel:3.10.0-1127.18.2.el7.x86_64</span></pre>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">4 nodes</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">n1 primary</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">n2,n3,n4 all secondary</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">If I run the folowing script then</span><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
sometimes, after the starts, some of the nodes get stuck in Outdated or Inconsistent state forever. The loop generally works correctly several times(max was about 14) before getting stuck.</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">I have NOT been able to replicate the spilt-brain state
in this way but I think it is related.</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</div>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">while true</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">do</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n4 'service corosync stop'</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n3 'service corosync stop'</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n2 'service corosync stop'</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n1 'service corosync stop'</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 5</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n1 'service pacemaker start'</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n2 'service pacemaker start'</span>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n3 'service pacemaker start'</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n4 'service pacemaker start'</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> #At this point r0 resource is mounted on /home</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> #and processes are writing to it on n1</span><br>
</div>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"></span></div>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> while true</span>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> do</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 1</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo "events2 `date`"</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> drbdsetup events2 --now -c</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> num_u2d=`drbdsetup events2 --now|grep -c disk:UpToDate`</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo "num UpToDate=$num_u2d"</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> [ "$num_u2d" = 4 ]&&break</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 4</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> done</span></div>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">done</span><br>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">If I change the stop order to</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> n1,n4,n3,n2</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">and the start order to</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> n2,n3,n4,n1</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">There is never a problem, I left this running over the weekend
and it worked over 5 thousand times.</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">I replicated the same issue using drbdadm commands directly
so this is not a corosync/pacemaker issue,</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">i.e. running following script on n1 also produces the problem:-</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">while true</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">do</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> for n in 4 3 2</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> do</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n$n 'drbdadm down r0'</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> done</span></div>
<div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> #stop process that write to /home</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> a2ksys stop >> a2ksys_stop.log 2>&1</span>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> while [ -d /home/cem ]</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> do</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> umount /home</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> [ ! -d /home/cem ]&&break</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo lsof output</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> lsof /home</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 5</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> done</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> drbdadm secondary r0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> drbdadm down r0</span></div>
</div>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> #END OF STOP</span><br>
</div>
<div><br>
</div>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> drbdadm up r0</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> while true</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> do</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> drbdadm primary r0&&break</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo "sleep 5 before retry primary"</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 5</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> done</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> while true</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> do</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> mount -orw /dev/drbd0 /home&&break</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo "sleep 5 before retry mount"</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 5</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> done</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> #start processes that do some writes HERE</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> a2ksys start >> a2ksys_start.log 2>&1</span></div>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n2 'drbdadm up r0'</span>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n3 'drbdadm up r0'</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> ssh n4 'drbdadm up r0'</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> while true</span>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> do</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 1</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo "events2 `date`"</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> drbdsetup events2 --now -c</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> num_u2d=`drbdsetup events2 --now|grep -c disk:UpToDate`</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> echo "num UpToDate=$num_u2d"</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> [ "$num_u2d" = 4 ]&&break</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> sleep 4</span></div>
<div><span style="font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> done</span></div>
</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">done</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
</span>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">r0.res:-</span><br>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">resource r0 {</span>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> handlers {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> fence-peer "/usr/lib/drbd/<a href="http://crm-fence-peer.9.sh" target="_blank">crm-fence-peer.9.sh</a>";</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> unfence-peer "/usr/lib/drbd/<a href="http://crm-unfence-peer.9.sh" target="_blank">crm-unfence-peer.9.sh</a>";</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> split-brain "/bin/touch /tmp/drbd_split_brain.flg";</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> startup {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> wfc-timeout 0; ## Infinite!</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> degr-wfc-timeout 120; ## 2 minutes.</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> outdated-wfc-timeout 120;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> disk {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> resync-rate 100M;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> on-io-error detach;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> disable-write-same;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> net {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> protocol C;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> device /dev/drbd0;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> disk /dev/VolGroup00/lv_home;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> meta-disk /dev/VolGroup00/lv_drbd_meta [0];</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> on n1 {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> address <a href="http://192.168.52.151:7789" target="_blank">192.168.52.151:7789</a>;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> node-id 1;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> on n2 {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> address <a href="http://192.168.52.152:7789" target="_blank">192.168.52.152:7789</a>;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> node-id 2;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> on n3 {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> address <a href="http://192.168.53.151:7789" target="_blank">192.168.53.151:7789</a>;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> node-id 3;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> on n4 {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> address <a href="http://192.168.53.152:7789" target="_blank">192.168.53.152:7789</a>;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> node-id 4;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> connection-mesh {</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> hosts n1 n2 n3 n4;</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"> }</span></div>
<div><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">}</span></div>
<br>
</div>
<span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">Is there any more information I can provide that would help to track down this issue?</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br>
</div>
</div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">Regards,</span></div>
<div style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">Jeremy Faith</span><span style="font-family:"Courier New",monospace;font-size:10pt;color:rgb(0,0,0);background-color:rgb(255,255,255)"></span><br>
</div>
</div>
_______________________________________________<br>
Star us on GITHUB: <a href="https://github.com/LINBIT" rel="noreferrer" target="_blank">https://github.com/LINBIT</a><br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="https://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">https://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div><br clear="all"><br><div style="font-size:small" class="gmail_default">You should definitely stop pacemaker before corosync and start in the opposite order.<br></div><br></div>
<br>
<p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><img src="https://c.ap4.content.force.com/servlet/servlet.ImageServer?id=0156F00000DRM7G&oid=00D90000000absk&lastMod=1526270984000"></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(68,189,216);vertical-align:baseline;white-space:pre-wrap"><span></span></span></p><p dir="ltr" style="line-height:1.2;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt"><span style="font-size:9pt;font-family:Arial;color:#2e4662;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap">Know Your Customer due diligence on demand, powered by intelligent process automation</span></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><a href="https://www.encompasscorporation.com/blog/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap">Blogs</span></a><span style="font-size:10pt;font-family:Arial;color:rgb(68,189,216);vertical-align:baseline;white-space:pre-wrap"> | </span><a href="https://www.linkedin.com/company/encompass-corporation/" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap">LinkedIn</span></a><span style="font-size:10pt;font-family:Arial;color:rgb(68,189,216);vertical-align:baseline;white-space:pre-wrap"> | </span><a href="https://twitter.com/EncompassCorp" target="_blank"><span style="font-size:10pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap">Twitter</span></a></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"> </p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="font-size:7pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="font-size:7pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap">Encompass Corporation UK Ltd | Company No. SC493055 | Address: Level 3, 33 Bothwell Street, Glasgow, UK, G2 6NL</span></p><p dir="ltr" style="line-height:1.44;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="font-size:7pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap">Encompass Corporation Pty Ltd | ACN 140 556 896 | Address: Level 10, 117 Clarence Street, Sydney, New South Wales, 2000</span></p><p dir="ltr" style="line-height:1.44;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="color:rgb(67,67,67);font-family:Arial;font-size:7pt;white-space:pre-wrap">This email and any attachments is intended only for the use of the individual or entity named above and may contain confidential information. </span></p><p dir="ltr" style="line-height:1.44;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="color:rgb(67,67,67);font-family:Arial;font-size:7pt;white-space:pre-wrap">If you are not the intended recipient, any dissemination, distribution or copying of this email is prohibited. </span></p><p dir="ltr" style="line-height:1.44;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="font-size:7pt;font-family:Arial;color:rgb(67,67,67);vertical-align:baseline;white-space:pre-wrap">I</span><span style="color:rgb(67,67,67);font-family:Arial;font-size:7pt;white-space:pre-wrap">f received in error, please notify us immediately by return email and destroy the original message.</span></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><span style="color:rgb(67,67,67);font-family:Arial;font-size:7pt;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.2;background-color:rgb(255,255,255);margin-top:0pt;margin-bottom:0pt"><br></p>