[DRBD-user] DRBD Synchronization Rate Questions

[Robert Altnoeder]

On 12/23/19 7:48 PM, cptype at yahoo.co.jp wrote:
> Hi developers!

Hello!

> Could you answer the following questions?

Yes

> 1. Can DRBD only provide up to 80 MB/s of network bandwidth per volume 
> during the initial full synchronization?

No

> We did an initial full synchronization in DRBD with the following settings:
---8X---snip--snap---
> Then, I increased the number of volumes from the above settings to 8 
> and performed an initial full synchronization again, 
> and the sync speed was about 640 MB/s.
> From these results, it seems that DRBD can only provide a sync speed of only 80 MB/s per volume.

The environment that it is running in can only provide an aggregate sync
speed of 640 MB/s. That seems quite normal, considering the fact that no
performance tuning was done to try increasing the performance any further.
Storage bandwidth, storage latency, network bandwidth, network latency
and various other parameters, such as e.g. scheduling characteristics,
influence the maximum sync performance that can be achieved. With
substantial performance tuning, this can often be close to the system's
theoretical maximum performance, but it usually isn't in the default
configuration.

The current method for setting resync speed paramters is by using the
dynamic resync speed controller and typically its parameters
c-plan-ahead, c-fill-target and c-max-rate. These allow better control
than the old "resync-rate" parameter.

Also, your configuration actually states "rsync-rate", which is not a
valid parameter for DRBD (but "resync-rate" is). So if you attempted to
apply this change using "drbdadm adjust", it would not have had any
effect (other than displaying an error message).

> 2. Related to the previous question, 
> can DRBD provide a fast sync rate (For example, about 1.25 GB/s) for a one volume?

If the system could achieve that performance without DRBD, then probably
yes.

> In our project, we want to achieve fast synchronization speed with less volume.
> For example, is it possible to achieve a sync rate of 1.25 GB/s with only 4 volumes?

Again, if the system can do it, then you could even have a sync rate of
1.25 GB/s with only a single volume.

> 3. Is the maximum number of peers in DRBD 32?

The maximum number of nodes per resource is 32. The maximum number of
peers as seen from each node of such a cluster would be 31.

> The drbdadm command with the following options completes successfully:
>   # drbdadm create-md --max-peers 32 r0
> ("--max-peers 33" results in an error.)

I was able to recreate this, and I suspect that this is a bug, and that
the range should actually be 1-31. Valid node IDs are 0-31, so the 32nd
peer slot would be unusable (since a node only keeps peer slots for
other nodes, but not for itself, so with 32 available node IDs, there
can be only 31 peers). It is possible that an additional peer slots was
added or will be added in the future on purpose (maybe for tracking
block status in thin provisioning scenarios), but I don't think this is
the case here. I'll investigate and ask my coworkers about recent
developments and future plans.

> Defined in the drbd-utils source code as follows:
> https://github.com/LINBIT/drbd-utils/blob/master/user/v84/linux/drbd.h
>   #define MAX_PEERS 32
> However, resynchronization does not work well with the metadata created by the above command.

I tested this, but could not reproduce any problem, it seemed to resync
normally on my test systems, despite the possibly incorrect peer slot
count of 32.
> Is the maximum number of peers in DRBD really 32?

Yes

> The DRBD version and the DRBD Utils version and OS version used are as follows:
> DRBD: 9.0.16 (kmod-drbd90-9.0.16-1.el7_6.elrepo.x86_64.rpm)
> DRBD Utils: 9.6.0 (drbd90-utils-9.6.0-1.el7.elrepo.x86_64.rpm)
> OS: Red Hat Enterprise Linux 7.6

DRBD 9.0.16 is quite outdated by now and has a number of known bugs. The
current version is 9.0.21. Compared to 9.0.16, this version also enables
significant improvements in performance.

The current drbd-utils version is 9.11.0.

br,
Robert