[DRBD-user] Closing unexpected connection from primary node

[kvaps]

Hi, today we've got an alert that some resources on three nodes become to
Outdated state.
Nothing was changed, issue occurred suddenly and is currently persisting.

drbd version: 9.0.19-1
kernel version: 4.15.18-12-pve

The weird thing is that some resources working fine on same nodes, but some
of them dont.

I've tried to run drbdadm disconnect && drbdadm connect for all failed
resources on all three nodes, but it didn't help much.
ifdown/ifup for data network interface restart didn't help too. TCP-ports
are open, but pve1 and pve3 resets the connection immediately.

What's happening and how can we resolve this?
Thank you!

I attach the logs for one resource from three nodes. It was Primary on pve2
and Secondary on pve1 amd pve3 nodes.

root at pve2:~# drbdadm status pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f role:Primary
  disk:UpToDate
  pve1 connection:Connecting
  pve3 connection:Connecting

root at pve2:~# dmesg -T | grep pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f:
Preparing cluster-wide state change 1999068598 (2->3 496/16)
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f:
State change 1999068598: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF9
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f:
Committing cluster-wide state change 1999068598 (0ms)
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069 pve1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: ack_receiver terminated
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: Terminating ack_recv thread
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: Connection closed
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: conn( Disconnecting -> StandAlone )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: Terminating receiver thread
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f:
Preparing cluster-wide state change 303080422 (2->1 496/16)
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f:
State change 303080422: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFFB
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Cluster is now split
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f:
Committing cluster-wide state change 303080422 (0ms)
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069 pve3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: ack_receiver terminated
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Terminating ack_recv thread
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Connection closed
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: conn( Disconnecting -> StandAlone )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Terminating receiver thread
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069: rs_discard_granularity feature disabled
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: conn( StandAlone -> Unconnected )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: Starting receiver thread (from drbd_w_pvc-0c26 [16956])
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve1: conn( Unconnected -> Connecting )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: conn( StandAlone -> Unconnected )
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Starting receiver thread (from drbd_w_pvc-0c26 [16956])
[Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: conn( Unconnected -> Connecting )
[Fri Oct 25 14:47:45 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069: new current UUID: 0407098B236D1403 weak: FFFFFFFFFFFFFFFB
...

root at pve1:~# drbdadm status pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f role:Secondary
  disk:Outdated
  pve2 connection:Connecting
  pve3 connection:Connecting

root at pve1:~# dmesg -T | grep pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Preparing remote state change 1999068598
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Committing remote state change 1999068598 (primary_nodes=4)
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: conn( Connected -> TearDown ) peer( Primary -> Unknown )
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069: disk( UpToDate -> Outdated )
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069 pve2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: ack_receiver terminated
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Terminating ack_recv thread
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Restarting sender thread
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Connection closed
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: conn( TearDown -> Unconnected )
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Restarting receiver thread
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: conn( Unconnected -> Connecting )
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Preparing remote state change 303080422
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: Committing remote state change 303080422 (primary_nodes=4)
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069 pve3: pdsk( UpToDate -> Outdated )
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve3: No reconciliation resync even though 'pve2' disappeared. (o=0)
[Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:21 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:30 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:38 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:50 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:02 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:11 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:23 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:32 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:43 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:56 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:54:07 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
...

root at pve3:~# drbdadm status pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f role:Secondary
  disk:Outdated
  pve1 connection:Connecting
  pve2 connection:Connecting

dmesg -T | grep pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Preparing remote state change 1999068598
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Committing remote state change 1999068598 (primary_nodes=4)
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069 pve1: pdsk( UpToDate -> Outdated )
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Preparing remote state change 303080422
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Committing remote state change 303080422 (primary_nodes=4)
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: conn( Connected -> TearDown ) peer( Primary -> Unknown )
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069: disk( UpToDate -> Outdated )
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0
drbd2069 pve2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: ack_receiver terminated
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Terminating ack_recv thread
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Restarting sender thread
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Connection closed
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: conn( TearDown -> Unconnected )
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: Restarting receiver thread
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
pve2: conn( Unconnected -> Connecting )
[Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:13 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:22 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:33 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:45 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:52:54 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:06 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:15 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:27 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:35 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:47 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:53:56 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
[Fri Oct 25 14:54:05 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f
tcp:pve2: Closing unexpected connection from 10.37.20.2
...

- kvaps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20191025/5a136565/attachment-0001.htm>