<div dir="ltr">Hello.<br><br>I have a 3 node drbd cluster, where 2 nodes (prod01, prod02) are managed by pacemaker and the third node (dr01) runs raw drbd.<br><br>The replication works well, but sometimes I have an intermittent problem with bringing the drbd down.<br>It happens regardless if the node is managed by a pacemaker or not.<br>The problem is that the drbdadm down &lt;resource&gt; command takes a very long time to complete (a couple of minutes).<br><br><br>I tried to debug this problem on dr01 and I found out that it only happens when drbd first attempts to disconnect prod02 (secondary) node and then prod01 (primary).<br>In that case the connection transitions from Connected to NetworkFailure after a while.<br>When the order is opposite: prod01 followed by prod02, the drbdadm down command returns immediately.<br>In that case I can see the connection goes from Connected -&gt; Disconnecting and the resource is brought down immediately.<br><br>I&#39;m pretty much stuck here and I would appreciate any help pointing out how to resolve, or further debug this problem.<div><br></div><div>DRBD_KERNEL_VERSION=9.1.4<br>DRBDADM_VERSION=9.17.0</div><div><br>Please find below my drbd config and two snippets of journalctl logs (one where the problem occurs and the other one where it does not).<br><br><br>Thanks in advance for your help.<br><br>Regards,<br>Janusz.<br><br>============================<br>DRBD config:<br><br>resource drbd-persistence {<br><br>  device    /dev/drbd1;<br>  disk      /dev/mapper/drbdpool-drbdata;<br>  meta-disk internal;<br><br>  on <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a> {<br>    node-id     0;<br>  }<br>  on <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a> {<br>    node-id     1;<br>  }<br>  on <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> {<br>    node-id     2;<br>  }<br><br>  connection {<br>    host <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a> address &lt;obfuscated&gt;:7789;<br>    host <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a> address &lt;obfuscated&gt;:7789;<br>    net {<br>      protocol C;<br>    }<br>  }<br>  connection {<br>    host <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a> address &lt;obfuscated&gt;:7789;<br>    host <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> address &lt;obfuscated&gt;:7789;<br>    net {<br>      protocol A;<br>      sndbuf-size 4M;<br>    }<br>  }<br>  connection {<br>    host <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a> address &lt;obfuscated&gt;:7789;<br>    host <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> address &lt;obfuscated&gt;:7789;<br>    net {<br>      protocol A;<br>      sndbuf-size 4M;<br>    }<br>  }<br>}<br><br>==============================<br><br>journalctl log when problem occurs:<br><br>Dec 23 10:30:31 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> sudo[64021]:      user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/sbin/drbdadm down drbd-persistence<br>Dec 23 10:30:31 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1840270664 (2-&gt;0 496/16)<br>Dec 23 10:31:01 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 239ms<br>Dec 23 10:31:01 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 2640669369 (2-&gt;0 496/16)<br>Dec 23 10:31:32 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 175ms<br>Dec 23 10:31:32 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 47999794 (2-&gt;0 496/16)<br>Dec 23 10:32:02 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 1110ms<br>Dec 23 10:32:03 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1719956804 (2-&gt;0 496/16)<br>Dec 23 10:32:33 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 405ms<br>Dec 23 10:32:34 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 4005278613 (2-&gt;0 496/16)<br>Dec 23 10:33:04 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 3378ms<br>Dec 23 10:33:07 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1216738202 (2-&gt;0 496/16)<br>Dec 23 10:33:38 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 5569ms<br>Dec 23 10:33:43 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 3440526386 (2-&gt;0 496/16)<br>Dec 23 10:34:14 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 6349ms<br>Dec 23 10:34:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 3514914987 (2-&gt;0 496/16)<br>Dec 23 10:34:51 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 4259ms<br>Dec 23 10:34:55 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1695181261 (2-&gt;0 496/16)<br>Dec 23 10:35:25 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 6294ms<br>Dec 23 10:35:32 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1507920047 (2-&gt;0 496/16)<br>Dec 23 10:36:02 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 767ms<br>Dec 23 10:36:03 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 438170821 (2-&gt;0 496/16)<br>Dec 23 10:36:33 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 3327ms<br>Dec 23 10:36:37 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 864136504 (2-&gt;0 496/16)<br>Dec 23 10:37:07 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 5029ms<br>Dec 23 10:37:12 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1182907387 (2-&gt;0 496/16)<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: meta connection shut down by peer.<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( Connected -&gt; NetworkFailure ) peer( Secondary -&gt; Unknown )<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1 <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: pdsk( UpToDate -&gt; DUnknown ) repl( Established -&gt; Off )<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: ack_receiver terminated<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating ack_recv thread<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Retrying cluster-wide state change after 921ms<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Starting sender thread (from drbd_r_service- [63376])<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Connection closed<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected exit code 0<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( NetworkFailure -&gt; Unconnected )<br>Dec 23 10:37:20 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Restarting receiver thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 510832083 (2-&gt;0 496/16)<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: State change 510832083: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Cluster is now split<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Committing cluster-wide state change 510832083 (16ms)<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: conn( Connected -&gt; Disconnecting ) peer( Primary -&gt; Unknown )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: disk( UpToDate -&gt; Outdated )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1 <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: pdsk( UpToDate -&gt; DUnknown ) repl( Established -&gt; Off )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: ack_receiver terminated<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating ack_recv thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( Unconnected -&gt; Connecting )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Starting sender thread (from drbd_r_service- [63375])<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Connection closed<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected exit code 0<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: conn( Disconnecting -&gt; StandAlone )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating receiver thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( Connecting -&gt; Disconnecting )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Starting sender thread (from drbd_r_service- [63376])<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Connection closed<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected exit code 0<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( Disconnecting -&gt; StandAlone )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating receiver thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: disk( Outdated -&gt; Detaching )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: disk( Detaching -&gt; Diskless )<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: drbd_bm_resize called with capacity == 0<br>Dec 23 10:37:21 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Terminating worker thread<br><br>================================<br>journalctl logs when problem doesn&#39;t occur:<br><br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> sudo[64116]:      user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/sbin/drbdadm down drbd-persistence<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 1232950875 (2-&gt;0 496/16)<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: State change 1232950875: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Committing cluster-wide state change 1232950875 (15ms)<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: conn( Connected -&gt; Disconnecting ) peer( Primary -&gt; Unknown )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: disk( UpToDate -&gt; Outdated )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1 <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: pdsk( UpToDate -&gt; DUnknown ) repl( Established -&gt; Off )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: ack_receiver terminated<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating ack_recv thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Starting sender thread (from drbd_r_service- [64099])<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Connection closed<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected exit code 0<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: conn( Disconnecting -&gt; StandAlone )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating receiver thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod01.ny5.dc.mycompany.com">prod01.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Preparing cluster-wide state change 3474969236 (2-&gt;1 496/16)<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: State change 3474969236: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Cluster is now split<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Committing cluster-wide state change 3474969236 (15ms)<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( Connected -&gt; Disconnecting ) peer( Secondary -&gt; Unknown )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1 <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: pdsk( UpToDate -&gt; DUnknown ) repl( Established -&gt; Off )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: ack_receiver terminated<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating ack_recv thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Starting sender thread (from drbd_r_service- [64101])<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Connection closed<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: helper command: /sbin/drbdadm disconnected exit code 0<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: conn( Disconnecting -&gt; StandAlone )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating receiver thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence <a href="http://prod02.ny5.dc.mycompany.com">prod02.ny5.dc.mycompany.com</a>: Terminating sender thread<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: disk( Outdated -&gt; Detaching )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: disk( Detaching -&gt; Diskless )<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence/0 drbd1: drbd_bm_resize called with capacity == 0<br>Dec 23 10:40:36 <a href="http://dr01.ch4.dc.mycompany.com">dr01.ch4.dc.mycompany.com</a> kernel: drbd drbd-persistence: Terminating worker thread<br><br><br><br><br> <br></div></div>