[DRBD-user] rdma: setting resource 'down' fails

[Thomas Roth]

No use. All variants invariably lead to 'sticky' resources:
- Two-way setup
- Single disk instead of zvol to keep ZFS out of the picture
- protocol A/C
- various sndbuf-size, rcvbuf-size, max-buffers

And all parameter changes require a server reset - takes ages to test 
all this.

Only starting without the "transport rdma" leads to the expected 
behavior - "drbdadm down res" is instantaneous.



The sndbuf/rcvbuf/max-buffers variations seemed necessary because I also 
got many "Not sending flow_control mgs, no receive window!".

But with the tedious server-resets it is no fun trying to figure out 
what might be 'big enough to hold all rcvbuf-size' in our case (8000 
pages is roughly 32MB, isn't it? 1M rcvbuf should fit ...).


Cheers
Thomas

On 11/12/23 12:16, Thomas Roth wrote:
> Hi all,
> 
> we have a 3-node setup with drbd 9.2.5 on Infiniband-connected servers, 
> so I tried 'drbd_transport_rdma'.
> 
> Seemed to work, but when I want to take down the resource on one of the 
> servers, 'drbdadm down res' does not return for 10 minutes, when it runs 
> into a timeout with
>  > Command 'drbdsetup down zvol0' did not terminate within 600 seconds.
> 
> There are still two 'D' processes 'drbdsetup status res' and 'drbdsetup 
> down res'.
> 
> 
> As an additional goodie, the three resources are zvols - ZFS block 
> devices. Of course I have no idea how that might interfere with the 
> transport mode.
> 
> And of course, the same setup / command works when using the tcp module 
> instead.
> 
> Anything obvious to look for?
> 
> 
> Best regards
> Thomas
-- 
------------------------------------------------------------------------------------------------
Thomas Roth
HPC Department

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstr. 1, 64291 Darmstadt, http://www.gsi.de/

Gesellschaft mit beschraenkter Haftung

Sitz der Gesellschaft / Registered Office:                    Darmstadt
Handelsregister       / Commercial Register:
                                         Amtsgericht Darmstadt, HRB 1528

Geschaeftsfuehrung    / Managing Directors:
          Professor Dr. Paolo Giubellino, Ursula Weyrich, Jörg Blaurock

Vorsitzender des GSI-Aufsichtsrates /
   Chairman of the Supervisory Board:
    Staatssekretaer / State Secretary Dr. Georg Schütte