No subject
Mon Jul 20 09:52:07 CEST 2020
plit-brain-disconnect by rule 100
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n3: helper command: /s=
bin/drbdadm initial-split-brain
Then for n2:-
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: drbd_sync_handshak=
e:
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: self 30D0D1B4BD67B=
AEE:CE02E3A41E743EDA:1AB2F8FC4793AC46:95EE6B42F9156BF6 bits:786 flags:20
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: peer BC13E2E36CA8B=
2C6:CE02E3A41E743EDA:272E3DE9D9C74A66:001E2864952E2E96 bits:416 flags:20
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: uuid_compare()=3Ds=
plit-brain-auto-recover by rule 90
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /s=
bin/drbdadm initial-split-brain
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: meta connection shut down =
by peer.
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: conn( Connected -> Network=
Failure ) peer( Secondary -> Unknown )
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: ack_receiver terminated
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Terminating ack_recv threa=
d
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /s=
bin/drbdadm initial-split-brain exit code 0
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0: Split-Brain detected but u=
nresolved, dropping connection!
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /s=
bin/drbdadm split-brain
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /s=
bin/drbdadm split-brain exit code 0
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: conn( NetworkFailure -> Di=
sconnecting )
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: error receiving P_STATE, e=
: -5 l: 0!
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Restarting sender thread
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Connection closed
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: conn( Disconnecting -> Sta=
ndAlone )
Oct 12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Terminating receiver threa=
d
The logs also have FIXME messages(which may be unrelated) e.g:-
Oct 13 12:50:42 cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[97260] op =
clear, bitmap locked for 'send_bitmap (WFBitMapS)' by drbd_w_r0[1659]
Oct 13 12:50:42 cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[97260] op =
clear, bitmap locked for 'receive bitmap' by drbd_r_r0[95684]
Sep 23 12:41:34 cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[19978] op =
clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[176=
28]
Regards,
Jeremy Faith
--_000_CO1P132MB02414264745DD95C0E91693985040CO1P132MB0241NAMP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div>
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><span style=3D"font-family: "Cou=
rier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background=
-color: rgb(255, 255, 255);">Hi,</span></div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<pre><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">drbd=
90 kernel module version:9.0.22-2=0A=
drbd90-utils:9.12.2-1=0A=
kernel:3.10.0-1127.18.2.el7.x86_64=0A=
pacemaker:1.1.21-4=0A=
corosync-2.4.5-4=0A=
system is centos:7.6</span></pre>
<span style=3D"font-family: "Courier New", monospace; font-size: =
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">I have a =
4 node test system(only ever 1 active primary) which is going split-brain u=
nexpectedly.</span></div>
<div dir=3D"ltr">
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">n1 is the =
primary, n2/n3/n4 secondary.<br>
</span></div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">System is =
being shutdown every night and sometimes on restart(particularly
after weekend shutdown) some of the nodes are split-brain and require a fu=
ll resync to fix.</span></div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Logs seem =
to indicate a problem with uuid_compare.</span></div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">From the s=
ystem log on n1:-</span><br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct 12 07:=
58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n3:
drbd_sync_handshake: </span>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n3: self 30D0D1B4BD67BAEE:=
CE02E3A41E743EDA:1AB2F8FC4793AC46:95EE6B42F9156BF6
bits:786 flags:20</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n3: peer 554921683EF7CC82:=
0000000000000000:272E3DE9D9C74A66:04B370F60768109E
bits:0 flags:20</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n3: uuid_compare()=3Dsplit=
-brain-disconnect by rule 100</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n3: helper command: /sbin/=
drbdadm initial-split-brain</span></div>
<br>
<span style=3D"font-family: "Courier New", monospace; font-size: =
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Then for =
n2:-</span></div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct 12 07:=
58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2:
drbd_sync_handshake: </span>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: self 30D0D1B4BD67BAEE:=
CE02E3A41E743EDA:1AB2F8FC4793AC46:95EE6B42F9156BF6
bits:786 flags:20</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: peer BC13E2E36CA8B2C6:=
CE02E3A41E743EDA:272E3DE9D9C74A66:001E2864952E2E96
bits:416 flags:20</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: uuid_compare()=3Dsplit=
-brain-auto-recover by rule 90</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /sbin/=
drbdadm initial-split-brain</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: meta connection shut down by p=
eer.</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: conn( Connected -> NetworkF=
ailure ) peer( Secondary -> Unknown )</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: ack_receiver terminated</span>=
</div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Terminating ack_recv thread</s=
pan></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /sbin/=
drbdadm initial-split-brain exit code 0</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0: Split-Brain detected but unres=
olved, dropping connection!</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /sbin/=
drbdadm split-brain</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0/0 drbd0 cdc0-n2: helper command: /sbin/=
drbdadm split-brain exit code 0</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: conn( NetworkFailure -> Dis=
connecting )</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: error receiving P_STATE, e: -5=
l: 0!</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Restarting sender thread</span=
></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Connection closed</span></div>
<div><span style=3D"font-family: "Courier New", monospace; font-s=
ize: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct =
12 07:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: conn( Disconnecting -> Stan=
dAlone )</span></div>
<span style=3D"font-family: "Courier New", monospace; font-size: =
10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct 12 07=
:58:56 cdc0-n1 kernel: drbd r0 cdc0-n2: Terminating receiver thread</span><=
br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">The logs a=
lso have FIXME messages(which may be unrelated)
e.g:-</span><br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct 13 12:=
50:42 cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[97260]
op clear, bitmap locked for 'send_bitmap (WFBitMapS)' by drbd_w_r0[1659]</=
span></div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Oct 13 12:=
50:42 cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[97260]
op clear, bitmap locked for 'receive bitmap' by drbd_r_r0[95684]</span></d=
iv>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
br>
</div>
<div style=3D"color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><=
span style=3D"font-family: "Courier New", monospace; font-size: 1=
0pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Sep 23 12:=
41:34 cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[19978]
op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0=
[17628]</span></div>
<div style=3D"color: rgb(0, 0, 0);"><br>
</div>
<div style=3D"color: rgb(0, 0, 0);"><span style=3D"font-family: "Couri=
er New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-c=
olor: rgb(255, 255, 255);">Regards,</span></div>
<div>
<div style=3D"color: rgb(0, 0, 0);"><span style=3D"font-family: "Couri=
er New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-c=
olor: rgb(255, 255, 255);">Jeremy Faith</span></div>
</div>
</div>
</div>
</body>
</html>
--_000_CO1P132MB02414264745DD95C0E91693985040CO1P132MB0241NAMP_--
More information about the drbd-user
mailing list