[DRBD-user] drbdmanage commands take a long time to complete on the leader if a node in the cluster is down

[Brady, Mike]

I am doing some testing with drbd9 and drbdmanage and am seeing some 
behaviour that I do not understand.

I have three nodes in a cluster.  Node names are kvm09, kvm10 and kvm11. 
  kvm09 is the leader.  All three systems are up to date Centos 7.3 with 
drbd 9.0.6, drbd-utils 8.9.10  and drbdmanage 0.98.2

If I shutdown a node, drbdmanage commands executed on the leader now 
take a "long time" to complete.

For instance if I shutdown kvm11 and run on the leader (kvm09) I see

[root at kvm09 ~]# time drbdmanage role
Operation completed successfully
leader

real	0m42.975s
user	0m0.084s
sys	0m0.019s

The same operation prior to shutting down kvm11 (times are similar on 
all three nodes)

[root at kvm09 ~]# time drbdmanage role
Operation completed successfully
leader

real	0m0.120s
user	0m0.088s
sys	0m0.031s

On the non-leader node (kvm10) in this scenario the commands always 
execute "quickly", but do not show up to date information.

For instance on the leader (which is correct and takes a long time)
[root at kvm09 ~]# time drbdmanage list-nodes
+---------------------------------------------------------------------------------------------------+
| Name                 | Pool Size | Pool Free |                         
                  |   State |
|---------------------------------------------------------------------------------------------------|
| kvm09.devnull.net.nz |   1668076 |   1114992 |                         
                  |      ok |
| kvm10.devnull.net.nz |   1668076 |   1069948 |                         
                  |      ok |
| kvm11.devnull.net.nz |   1668076 |   1069948 |                         
                  | OFFLINE |
+---------------------------------------------------------------------------------------------------+

real	0m37.045s
user	0m0.079s
sys	0m0.026s

But on kvm10 (which is incorrect and does NOT take a long time)
[[root at kvm10 ~]# time drbdmanage list-nodes
+------------------------------------------------------------------------------------------------------------+
| Name                 | Pool Size | Pool Free |                         
                             | State |
|------------------------------------------------------------------------------------------------------------|
| kvm09.devnull.net.nz |   1668076 |   1114992 |                         
                             |    ok |
| kvm10.devnull.net.nz |   1668076 |   1069948 |                         
                             |    ok |
| kvm11.devnull.net.nz |   1668076 |   1069948 |                         
                             |    ok |
+------------------------------------------------------------------------------------------------------------+

real	0m0.101s
user	0m0.071s
sys	0m0.025s

Is this expected behaviour in this type of situation?

Regards

Mike