<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Hi,</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<pre><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span data-markjs="true" class="markyv228oi2c">drbd</span>90 kernel module version:9.0.22-2(also 9.0.25-1 compiled from source)
<span data-markjs="true" class="markyv228oi2c">drbd</span>90-utils:9.12.2-1
kernel:3.10.0-1127.18.2.el7.x86_64</span></pre>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">4 nodes</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">n1 primary</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">n2,n3,n4 all secondary</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">If I run the folowing script then</span><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
sometimes, after the starts, some of the nodes get stuck in Outdated or Inconsistent state forever. The loop generally works correctly several times(max was about 14) before getting stuck.</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">I have NOT been able to replicate the spilt-brain state
in this way but I think it is related.</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</div>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">while true</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">do</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n4 'service corosync stop'</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n3 'service corosync stop'</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n2 'service corosync stop'</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n1 'service corosync stop'</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 5</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n1 'service pacemaker start'</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n2 'service pacemaker start'</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n3 'service pacemaker start'</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n4 'service pacemaker start'</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> #At this point r0 resource is mounted on /home</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> #and processes are writing to it on n1</span><br>
</div>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"></span></div>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> while true</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> do</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 1</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo "events2 `date`"</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> drbdsetup events2 --now -c</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> num_u2d=`drbdsetup events2 --now|grep -c disk:UpToDate`</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo "num UpToDate=$num_u2d"</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> [ "$num_u2d" = 4 ]&&break</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 4</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> done</span></div>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">done</span><br>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">If I change the stop order to</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> n1,n4,n3,n2</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">and the start order to</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> n2,n3,n4,n1</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">There is never a problem, I left this running over the weekend
and it worked over 5 thousand times.</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">I replicated the same issue using drbdadm commands directly
so this is not a corosync/pacemaker issue,</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">i.e. running following script on n1 also produces the problem:-</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">while true</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">do</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> for n in 4 3 2</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> do</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n$n 'drbdadm down r0'</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> done</span></div>
<div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> #stop process that write to /home</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> a2ksys stop >> a2ksys_stop.log 2>&1</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> while [ -d /home/cem ]</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> do</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> umount /home</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> [ ! -d /home/cem ]&&break</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo lsof output</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> lsof /home</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 5</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> done</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> drbdadm secondary r0</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> drbdadm down r0</span></div>
</div>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> #END OF STOP</span><br>
</div>
<div><br>
</div>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> drbdadm up r0</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> while true</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> do</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> drbdadm primary r0&&break</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo "sleep 5 before retry primary"</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 5</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> done</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> while true</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> do</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> mount -orw /dev/drbd0 /home&&break</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo "sleep 5 before retry mount"</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 5</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> done</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> #start processes that do some writes HERE</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> a2ksys start >> a2ksys_start.log 2>&1</span></div>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n2 'drbdadm up r0'</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n3 'drbdadm up r0'</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> ssh n4 'drbdadm up r0'</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> while true</span>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> do</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 1</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo "events2 `date`"</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> drbdsetup events2 --now -c</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> num_u2d=`drbdsetup events2 --now|grep -c disk:UpToDate`</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> echo "num UpToDate=$num_u2d"</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> [ "$num_u2d" = 4 ]&&break</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> sleep 4</span></div>
<div><span style="font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> done</span></div>
</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">done</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
</span>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">r0.res:-</span><br>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">resource r0 {</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> handlers {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> split-brain "/bin/touch /tmp/drbd_split_brain.flg";</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> startup {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> wfc-timeout 0; ## Infinite!</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> degr-wfc-timeout 120; ## 2 minutes.</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> outdated-wfc-timeout 120;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> disk {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> resync-rate 100M;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> on-io-error detach;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> disable-write-same;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> net {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> protocol C;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> device /dev/drbd0;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> disk /dev/VolGroup00/lv_home;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> meta-disk /dev/VolGroup00/lv_drbd_meta [0];</span></div>
<div><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> on n1 {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> address 192.168.52.151:7789;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> node-id 1;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> on n2 {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> address 192.168.52.152:7789;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> node-id 2;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> on n3 {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> address 192.168.53.151:7789;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> node-id 3;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> on n4 {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> address 192.168.53.152:7789;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> node-id 4;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> connection-mesh {</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> hosts n1 n2 n3 n4;</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"> }</span></div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">}</span></div>
<br>
</div>
<span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Is there any more information I can provide that would help to track down this issue?</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</div>
</div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Regards,</span></div>
<div style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Jeremy Faith</span><span style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"></span><br>
</div>
</body>
</html>