[DRBD-user] Cluster split after short network outage

[Roman Makhov]

Hello,

I discovered the "Cluster is now split" message in log and moving to
StandAlone then after short (about 8 seconds) network failure between
cluster nodes.

Would you please to suggest something?

Thank you in advance!

The drbd log is:
=====================================================================
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: PingAck did not
arrive in time.
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: conn( Connected
-> NetworkFailure ) peer( Primary -> Unknown )
[Sat Jul  7 21:02:22 2018] drbd dhcp/0 drbd0: disk( UpToDate -> Consistent )
[Sat Jul  7 21:02:22 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: pdsk(
UpToDate -> DUnknown ) repl( Established -> Off )
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: ack_receiver terminated
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: Terminating
ack_recv thread
[Sat Jul  7 21:02:22 2018] drbd dhcp: Preparing cluster-wide state
change 1083152536 (1->-1 0/0)
[Sat Jul  7 21:02:22 2018] drbd dhcp: Committing cluster-wide state
change 1083152536 (2ms)
[Sat Jul  7 21:02:22 2018] drbd dhcp/0 drbd0: disk( Consistent -> UpToDate )
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: Connection closed
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: conn(
NetworkFailure -> Unconnected )
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: Restarting
receiver thread
[Sat Jul  7 21:02:22 2018] drbd dhcp dhcp-master.dhcp: conn(
Unconnected -> Connecting )
[Sat Jul  7 21:02:30 2018] drbd dhcp dhcp-master.dhcp: Handshake to
peer 0 successful: Agreed network protocol version 112
[Sat Jul  7 21:02:30 2018] drbd dhcp dhcp-master.dhcp: Feature flags
enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
[Sat Jul  7 21:02:30 2018] drbd dhcp dhcp-master.dhcp: Starting
ack_recv thread (from drbd_r_dhcp [28952])
[Sat Jul  7 21:02:30 2018] drbd dhcp dhcp-master.dhcp: Preparing
remote state change 1152846943 (primary_nodes=0, weak_nodes=0)
[Sat Jul  7 21:02:30 2018] drbd dhcp dhcp-master.dhcp: Committing
remote state change 1152846943
[Sat Jul  7 21:02:30 2018] drbd dhcp dhcp-master.dhcp: conn(
Connecting -> Connected ) peer( Unknown -> Primary )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0: disk( UpToDate -> Outdated )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp:
drbd_sync_handshake:
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: self
0967822D6718C8AC:0000000000000000:323BE7D71FABECCC:44CD99B02FF92950
bits:0 flags:120
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp:
uuid_compare()=-2 by rule 50
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: pdsk(
DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: receive
bitmap stats [Bytes(packets)]: plain 0(0), RLE 29(1), total 29;
compression: 100.0%
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: send
bitmap stats [Bytes(packets)]: plain 0(0), RLE 29(1), total 29;
compression: 100.0%
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: helper
command: /sbin/drbdadm before-resync-target
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: helper
command: /sbin/drbdadm before-resync-target exit code 0 (0x0)
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0: disk( Outdated -> Inconsistent )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: repl(
WFBitMapT -> SyncTarget )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: Began
resync as SyncTarget (will sync 12 KB [3 bits set]).
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: Resync
done (total 1 sec; paused 0 sec; 12 K/sec)
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: updated
UUIDs 64F87B33597A7F68:0000000000000000:0967822D6718C8AC:323BE7D71FABECCC
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0: disk( Inconsistent -> UpToDate )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: repl(
SyncTarget -> Established )
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: helper
command: /sbin/drbdadm after-resync-target
[Sat Jul  7 21:02:30 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: helper
command: /sbin/drbdadm after-resync-target exit code 0 (0x0)
[Sat Jul  7 21:02:38 2018] drbd dhcp: Preparing cluster-wide state
change 3007526931 (1->0 496/16)
[Sat Jul  7 21:02:38 2018] drbd dhcp: State change 3007526931:
primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFE
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: Cluster is now split
[Sat Jul  7 21:02:38 2018] drbd dhcp: Committing cluster-wide state
change 3007526931 (50ms)
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: conn( Connected
-> Disconnecting ) peer( Primary -> Unknown )
[Sat Jul  7 21:02:38 2018] drbd dhcp/0 drbd0: disk( UpToDate -> Outdated )
[Sat Jul  7 21:02:38 2018] drbd dhcp/0 drbd0 dhcp-master.dhcp: pdsk(
UpToDate -> DUnknown ) repl( Established -> Off )
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: ack_receiver terminated
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: Terminating
ack_recv thread
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: Connection closed
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: conn(
Disconnecting -> StandAlone )
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: Terminating
receiver thread
[Sat Jul  7 21:02:38 2018] drbd dhcp dhcp-master.dhcp: Terminating sender thread
[Sat Jul  7 21:02:38 2018] drbd dhcp/0 drbd0: disk( Outdated -> Detaching )
[Sat Jul  7 21:02:38 2018] drbd dhcp/0 drbd0: disk( Detaching -> Diskless )
[Sat Jul  7 21:02:38 2018] drbd dhcp/0 drbd0: drbd_bm_resize called
with capacity == 0
[Sat Jul  7 21:02:38 2018] drbd dhcp: Terminating worker thread
=====================================================================

The version is:
=====================================================================
[root at dhcp-slave ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 3fc9d321f9f60e84d7dcdaac545fe8be4b280a63\
build\ by\ mockbuild@\,\ 2017-09-14\ 17:52:30
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090009
DRBD_KERNEL_VERSION=9.0.9
DRBDADM_VERSION_CODE=0x090100
DRBDADM_VERSION=9.1.0
=====================================================================

Roman.