<div dir="ltr">Hello Richard,<div><br></div><div>Not sure what caused Inconsistent state on your drbd resource on Node1, but my guess is that it experienced some kind of low level corruption on it's backing storage (hard disks) and auto sync was initiated from Node2.</div><div><br></div><div>Are you using linux raid? Then probably you don't have a battery backed raid controller, so it would be wise to remove the following entries since they could cause data loss in your case:</div><div><br></div><div><span style="font-size:13px"> no-disk-flushes;</span><br style="font-size:13px"><span style="font-size:13px"> no-md-flushes;</span><br style="font-size:13px"><span style="font-size:13px"> no-disk-barrier;</span><br></div><div><span style="font-size:13px"><br></span></div><div><span style="font-size:13px">Finally from my experince proxmox does not work well with HA enabled (it's not aware of underlying drbd resource) so it could cause frequent split-brains to occur.Use DRBD without HA enabled on Proxmox , in dual-primary mode (so you don't loose live migration capability on proxmox).</span></div><div><span style="font-size:13px"><br></span></div><div>You can also create separate drbd resources for each proxmox mode so you can better handle split brains.For example drbd0 -> drbdvg0 -> mounted always on Node1 and drbd1 -> drbdvg1 -> mounted on Node2.</div><div>This way you will always know that vms running on Node1 are located on drbd0 and vms running on Node2 are located on drbd1.</div><div><br></div><div>Regards</div><div>Yannis</div><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><div><br><div> <br><div><br></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sun, Jan 18, 2015 at 11:42 AM, Lechner Richard <span dir="ltr"><<a href="mailto:r.lechner@gmx.net" target="_blank">r.lechner@gmx.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Repost from 13ten Jan.<br>
<br>
Hello all,<br>
<br>
sorry this will be a longer post!<br>
<br>
I have some strange issues since a few weeks. Sometimes drbd running into a<br>
split brain but i not real understand why!<br>
I run a proxmox cluster with 2 nodes and only one VM is running on the first<br>
node (Node1), so the other node (Node2) is the HA-backupnode to switch the VM<br>
when something happen.<br>
<br>
The disc's are md's on both nodes:<br>
<br>
Personalities : [raid1]<br>
md2 : active raid1 sda3[0] sdb3[1]<br>
2930129536 blocks super 1.2 [2/2] [UU]<br>
<br>
<br>
Drbd-Config:<br>
resource r1 {<br>
protocol C;<br>
startup {<br>
wfc-timeout 0;<br>
degr-wfc-timeout 60;<br>
become-primary-on both;<br>
}<br>
net {<br>
sndbuf-size 10M;<br>
rcvbuf-size 10M;<br>
ping-int 2;<br>
ping-timeout 2;<br>
connect-int 2;<br>
timeout 5;<br>
ko-count 5;<br>
max-buffers 128k;<br>
max-epoch-size 8192;<br>
cram-hmac-alg sha1;<br>
shared-secret "XXXXXX";<br>
allow-two-primaries;<br>
after-sb-0pri discard-zero-changes;<br>
after-sb-1pri discard-secondary;<br>
after-sb-2pri disconnect;<br>
}<br>
on node1 {<br>
device /dev/drbd0;<br>
disk /dev/md2;<br>
address <a href="http://10.1.5.31:7788" target="_blank">10.1.5.31:7788</a>;<br>
meta-disk internal;<br>
}<br>
on node2 {<br>
device /dev/drbd0;<br>
disk /dev/md2;<br>
address <a href="http://10.1.5.32:7788" target="_blank">10.1.5.32:7788</a>;<br>
meta-disk internal;<br>
}<br>
disk {<br>
no-disk-flushes;<br>
no-md-flushes;<br>
no-disk-barrier;<br>
}<br>
}<br>
<br>
<br>
The disc for the VM are LV's and only mounted inside the VM, vm-101-disk-1<br>
is the VM-root-fs and disk-2 is the VM-mailstorage. There is/should no mount's<br>
or access from the nodes directly!<br>
<br>
--- Logical volume ---<br>
LV Path /dev/drbd0vg/vm-101-disk-1<br>
LV Name vm-101-disk-1<br>
VG Name drbd0vg<br>
LV Size 75,00 GiB<br>
<br>
--- Logical volume ---<br>
LV Path /dev/drbd0vg/vm-101-disk-2<br>
LV Name vm-101-disk-2<br>
VG Name drbd0vg<br>
LV Size 550,00 GiB<br>
<br>
<br>
The nodes don't use/mount any of /dev/drbd0vg/<br>
<br>
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf<br>
udev 10M 0 10M 0% /dev<br>
tmpfs 1,6G 504K 1,6G 1% /run<br>
/dev/mapper/pve-root 78G 3,0G 72G 4% /<br>
tmpfs 5,0M 4,0K 5,0M 1% /run/lock<br>
tmpfs 3,2G 50M 3,1G 2% /run/shm<br>
/dev/sdc1 232M 72M 148M 33% /boot<br>
/dev/fuse 30M 24K 30M 1% /etc/pve<br>
<br>
So DRBD runns Primary/Primary but how it can changed something on the<br>
second node if nothing is runnig there and the LV's are not mounted? It should<br>
not exist new data on the drbd-volume on Node2. But last time i did a resync<br>
(after i stoped the VM on Node1!) it sync's 15 GB from node2 to node1!<br>
Unbelievable!!<br>
I take a screenshot but i'm not sure i can attach it here?<br>
<br>
DRBD-Status on Node1 was:<br>
Primary/Primary ds:Inconsistent/UpToDate<br>
So i think the left is Node1 and the right is Node2? How Node2 can be<br>
UpToDate? I don't understand this because Node2 was running nothing with<br>
access to the LV's!<br>
Had some Filesystemerrors inside the VM when she was starting after the sync<br>
on Node1. :-(<br>
<br>
<br>
Before it was a crosslink cable and i want to make sure there is no problem,<br>
so sunday i installed a switch and normal cables only for the drbd-network!<br>
(But when the break was happen i not see a eth1 down or something like that)<br>
<br>
Jan 12 10:49:34 node1 kernel: block drbd0: Remote failed to finish a request<br>
within ko-count * timeout<br>
Jan 12 10:49:34 node1 kernel: block drbd0: peer( Primary -> Unknown ) conn(<br>
Connected -> Timeout ) pdsk( UpToDate -> DUnknown )<br>
Jan 12 10:49:34 node1 kernel: block drbd0: asender terminated<br>
Jan 12 10:49:34 node1 kernel: block drbd0: Terminating asender thread<br>
Jan 12 10:49:34 node1 kernel: block drbd0: new current UUID<br>
D4335C79AD0E0BC3:AE406068788B0F3B:95D06B8F4DD0CE03:95CF6B8F4DD0CE03<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Connection closed<br>
Jan 12 10:49:35 node1 kernel: block drbd0: conn( Timeout -> Unconnected )<br>
Jan 12 10:49:35 node1 kernel: block drbd0: receiver terminated<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Restarting receiver thread<br>
Jan 12 10:49:35 node1 kernel: block drbd0: receiver (re)started<br>
Jan 12 10:49:35 node1 kernel: block drbd0: conn( Unconnected -> WFConnection )<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Handshake successful: Agreed<br>
network protocol version 96<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Peer authenticated using 20 bytes<br>
of 'sha1' HMAC<br>
Jan 12 10:49:35 node1 kernel: block drbd0: conn( WFConnection -><br>
WFReportParams )<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Starting asender thread (from<br>
drbd0_receiver [2840])<br>
Jan 12 10:49:35 node1 kernel: block drbd0: data-integrity-alg: <not-used><br>
Jan 12 10:49:35 node1 kernel: block drbd0: drbd_sync_handshake:<br>
Jan 12 10:49:35 node1 kernel: block drbd0: self<br>
D4335C79AD0E0BC3:AE406068788B0F3B:95D06B8F4DD0CE03:95CF6B8F4DD0CE03 bits:42099<br>
flags:0<br>
Jan 12 10:49:35 node1 kernel: block drbd0: peer<br>
A5FBD7AF4A9FD583:AE406068788B0F3B:95D06B8F4DD0CE03:95CF6B8F4DD0CE03 bits:0<br>
flags:0<br>
Jan 12 10:49:35 node1 kernel: block drbd0: uuid_compare()=100 by rule 90<br>
Jan 12 10:49:35 node1 kernel: block drbd0: helper command: /sbin/drbdadm<br>
initial-split-brain minor-0<br>
Jan 12 10:49:35 node1 kernel: block drbd0: meta connection shut down by peer.<br>
Jan 12 10:49:35 node1 kernel: block drbd0: conn( WFReportParams -><br>
NetworkFailure )<br>
Jan 12 10:49:35 node1 kernel: block drbd0: asender terminated<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Terminating asender thread<br>
Jan 12 10:49:35 node1 kernel: block drbd0: helper command: /sbin/drbdadm<br>
initial-split-brain minor-0 exit code 0 (0x0)<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Split-Brain detected but<br>
unresolved, dropping connection!<br>
Jan 12 10:49:35 node1 kernel: block drbd0: helper command: /sbin/drbdadm<br>
split-brain minor-0<br>
Jan 12 10:49:35 node1 kernel: block drbd0: helper command: /sbin/drbdadm<br>
split-brain minor-0 exit code 0 (0x0)<br>
Jan 12 10:49:35 node1 kernel: block drbd0: conn( NetworkFailure -><br>
Disconnecting )<br>
Jan 12 10:49:35 node1 kernel: block drbd0: error receiving ReportState, l: 4!<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Connection closed<br>
Jan 12 10:49:35 node1 kernel: block drbd0: conn( Disconnecting -> StandAlone )<br>
Jan 12 10:49:35 node1 kernel: block drbd0: receiver terminated<br>
Jan 12 10:49:35 node1 kernel: block drbd0: Terminating receiver thread<br>
<br>
<br>
grep eth1 /var/log/kern.log<br>
Jan 11 15:10:42 node1 kernel: igb 0000:07:00.1: eth1: (PCIe:5.0GT/s:Width x2)<br>
Jan 11 15:10:42 node1 kernel: igb 0000:07:00.1: eth1: MAC: f8:0f:41:fb:32:21<br>
Jan 11 15:10:42 node1 kernel: igb 0000:07:00.1: eth1: PBA No: 106300-000<br>
Jan 11 15:10:42 node1 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready<br>
Jan 11 15:10:42 node1 kernel: igb 0000:07:00.1: eth1: igb: eth1 NIC Link is Up<br>
1000 Mbps Full Duplex, Flow Control: RX/TX<br>
Jan 11 15:10:42 node1 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes<br>
ready<br>
Jan 11 15:10:52 node1 kernel: eth1: no IPv6 routers present<br>
<br>
<br>
eth1 is still up, no errors, ping and ssh still working on that interface!<br>
<br>
eth1 Link encap:Ethernet Hardware Adresse f8:0f:41:fb:32:21<br>
inet Adresse:10.1.5.31 Bcast:10.1.5.255 Maske:255.255.255.0<br>
inet6-Adresse: fe80::fa0f:41ff:fefb:3221/64<br>
Gültigkeitsbereich:Verbindung<br>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metrik:1<br>
RX packets:13129653 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:13383845 errors:0 dropped:0 overruns:0 carrier:0<br>
Kollisionen:0 Sendewarteschlangenlänge:1000<br>
RX bytes:17271836211 (16.0 GiB) TX bytes:15506649645 (14.4 GiB)<br>
<br>
uname -a<br>
Linux node1 2.6.32-34-pve #1 SMP Fri Dec 19 07:42:04 CET 2014 x86_64 GNU/Linux<br>
<br>
<br>
On Node2 i don't find any disc error or something like this!<br>
So what can be the problem and how can i fix this? I read the docu and if i<br>
understand correctly a automatic split brain repair is not useable for my<br>
situation because i don't know where the VM is running last time.<br>
<br>
Try to attach the screen also.<br>
Any hints?<br>
<br>
Regards<br>
<br>
Richard<br>
<br>
PS: I get Eric's post where he mention: "The split brain would only happen on<br>
dual primary. "<br>
So i changed to Primary/Secondary and stoped the HA in Proxmox.<br>
Last few days no errors occur but i have to observe this the next weeks.<br>
<br>
<br>
<br>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br></div></div>