[DRBD-user] expand 3 -> 4 nodes

Thomas Roth t.roth at gsi.de
Sat Jan 14 18:55:22 CET 2017

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


I'm afraid this email is tl;dr, but I'd still like to document the steps.

There is, in https://www.drbd.org/en/doc/users-guide-90/ch-admin-manual, section 5.1.6, the paragraph

 > Please note that the number of bitmap slots that are allocated in the meta-data depends on the number of hosts for this resource; per default the
 > hosts in the resource configuration are counted. If all hosts are specified before creating the meta-data, this will "just work"; adding bitmap
 > slots for further nodes is possible later, but incurs some manual work.

To be taken seriously ;-)

Section 5.19.1 explains this manual work (Works only if the cluster is down):

1. dump the metadata,

Somewhere I got the impression that I'd have to use 'drbdmeta' - rather difficult to use if you don't know exactly the parameters, no examples to be 
found quickly - but of course this also works with 'drbdadm' :

drbdadm dump-md pool2 > pool2_md

Did not work at first
 > Found meta data is "unclean", please apply-al first

drbdadm apply-al pool2
drbdadm dump-md pool2 > pool2_md

2.enlarge the metadata space, 3. edit the dumpfile,

In the dump, I increased the number "max-peers" from 2 to 3.

The subsequent restore-md failed with "Parse error in line 236: keyword 'bitmap' expected, but end of file encountered",
so I also added at the end of the dump file

 > bitmap[2] {
 >    # at 0kB
 >     4055040 times 0x0000000000000000;
 > }

4. load the changed metadata.

It seems there is no 'drbdadm restore-md', but running the dry-run-dump command lets you figure out the parameters of drbdmeta:

drbdadm -d dump-md pool2
 > drbdmeta 2 v09 /dev/drbdpool2/drbd2lv internal dump-md

so it must be

drbdmeta 2 v09 /dev/drbdpool2/drbd2lv internal restore-md pool2_md

This did not work either, because the increased metadata size would mean that
 > Device size would be truncated,

Since this resource is a LV, I extended it by 5GB, ignored the xfs_growfs warnings and got the 'drbdmeta restore-md' to work.

Now 'drbdadm attach pool2' did not work

 > No usable activity log found. Do you need to create-md?
 > 2: Failure: (127) Device minor not allocated
 > additional info from kernel:
 > unknown minor

That this should be an issue with the minor-count seems rather improbable, but I still put the number to 100 in global_common.conf.

drbdadm up pool2

worked, worked also on the new, fourth host. Sync is running.
And the resource even has survived the ordeal: made it primary, mounted it, data are still there.


On 14.01.2017 12:57, Thomas Roth wrote:
> Hi all,
> I'm on my first steps with DRBD 9. Seems I did not understand (many things) how to add a new node to an existing cluster.
> Atm I have a resource "pool2" on three nodes, it is UpToDate on all three, primary on one node, mountable/usable there, all fine.
> Now I want to add node 4. I have added this node to the config file and created the resource on the fourth node.
> When trying to adjust the resource on e.g. node 1, it fails with exit code 10:
> node1:# drbdadm adjust  pool2
> pool2: Failure: (162) Invalid configuration request
> Command 'drbdsetup new-peer pool2 4 --_name=node4 --protocol=C --sndbuf-size=0 --rcvbuf-size=0' terminated with exit code 10
> drbdadm: new-path pool2: skipped due to earlier error
> I have since set pool2 to secondary also on node1, no change.
> And since this entire endeavor is a part of 'data rebalancing', and suspecting some magic in the labeling given in the user guide, renamed the node in
> the configuration from "node4" to "for-later-rebalancing" - no change either.
> So how do I tell DRBD that there is a new peer in the game?
> Regards,
> Thomas

Thomas Roth
Department: HPC
Location: SB3 1.262
Phone: +49-6159-71 1453  Fax: +49-6159-71 2986

GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt

Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528

Geschäftsführung: Professor Dr. Paolo Giubellino
Ursula Weyrich
Jörg Blaurock

Vorsitzender des Aufsichtsrates: St Dr. Georg Schütte
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt

More information about the drbd-user mailing list