[DRBD-user] Bitmap for that device got too large.

[Robert Altnoeder]

Hello,

On 5/28/19 8:03 AM, Oleksiy Evin wrote:
> The linbit's article dated 2012 says the current maximum is 1PiB, but
> when I'm trying to create a 3 node replication with 527TiB each, it
> gets me to the bitmap size error. 
>
> I have an assumption that overall bitmap size limit is divided for 2
> replicas.

An investigation of the issue in the source code of the current
drbd-utils release 9.9.0 confirmed that you are right with your assumption.

> Is there any technical reason behind it?

We will have to discuss the issue internally, I believe that this is an
unintentional limitation, possibly a left-over from DRBD 8.x code, since
DRBD 8 was limited to a single peer, but please allow us some time to
double-check whether or not removing this limitation causes any problems.

> Is there any other way to create this volume without splitting it up
> and joining by LVM on the top (LVM-> DRBD -> LVM)?

Unfortunately, I do not think there is at this time. If we can determine
that this limitation is unnecessary, then we will follow up with a new
release of the drbd-utils that will fix this problem and allow the
creation of larger volumes.

> Can anyone please clarify what's the actual volume size limit for DRBD?

With the current version of the drbd-utils (drbdmeta program), the
actual limits I have calculated are:

 1 peers:   1099510448128 kiB (~ 1,048,574.87 GiB)
 2 peers:    549755224064 kiB (~  524,287.43 GiB)
 3 peers:    366503482624 kiB (~  349,524.95 GiB)
 4 peers:    274877612032 kiB (~  262,143.71 GiB)
 5 peers:    219902089472 kiB (~  209,714.97 GiB)
 6 peers:    183251741184 kiB (~  174,762.47 GiB)
 7 peers:    157072921088 kiB (~  149,796.41 GiB)
 8 peers:    137438806016 kiB (~  131,071.85 GiB)
 9 peers:    122167827456 kiB (~  116,508.31 GiB)
10 peers:    109951044608 kiB (~  104,857.48 GiB)
11 peers:     99955495168 kiB (~   95,324.98 GiB)
12 peers:     91625870592 kiB (~   87,381.23 GiB)
13 peers:     84577726720 kiB (~   80,659.60 GiB)
14 peers:     78536460544 kiB (~   74,898.20 GiB)
15 peers:     73300696320 kiB (~   69,904.99 GiB)
16 peers:     68719403008 kiB (~   65,535.92 GiB)
17 peers:     64677085184 kiB (~   61,680.87 GiB)
18 peers:     61083913728 kiB (~   58,254.15 GiB)
19 peers:     57868970752 kiB (~   55,188.15 GiB)
20 peers:     54975522304 kiB (~   52,428.74 GiB)
21 peers:     52357640192 kiB (~   49,932.13 GiB)
22 peers:     49977747456 kiB (~   47,662.49 GiB)
23 peers:     47804802048 kiB (~   45,590.21 GiB)
24 peers:     45812935168 kiB (~   43,690.61 GiB)
25 peers:     43980417792 kiB (~   41,942.99 GiB)
26 peers:     42288863232 kiB (~   40,329.80 GiB)
27 peers:     40722609152 kiB (~   38,836.10 GiB)
28 peers:     39268230144 kiB (~   37,499.10 GiB)
29 peers:     37914153216 kiB (~   36,157.75 GiB)
30 peers:     36650348032 kiB (~   34,952.49 GiB)
31 peers:     35468078848 kiB (~   33,824.99 GiB)

If in your case a volume size of 511 TiB would be sufficient, that
should work with 2 peers.

Best regards,
Robert