[DRBD-user] linstor-proxmox-2.9.0

Roberto Resoli roberto at resolutions.it
Mon Aug 27 11:11:30 CEST 2018

Il 27/08/2018 09:33, Roland Kammerer ha scritto:
> On Mon, Aug 27, 2018 at 09:12:12AM +0200, Roberto Resoli wrote:
>> Il 24/08/2018 10:54, Roland Kammerer ha scritto:
>>> This feature is documented here:
>>> https://docs.linbit.com/docs/users-guide-9.0/#s-proxmox-ls-HA
>> Hello,
>> I read and tried the described procedure, but my findings were not positive.
>> In two cases the entire drbd storage freezed and all vm were stopped.
> Could you describe these scenarios in more detail? How many nodes, where
> was the controller VM started, what did you do exactly, autoboot of
> other VMs? Sorry, as always, "does not work" is not good enough to debug
> things.

Yes, of course. The scenario is:

=== general ===

3 pve nodes, each one with a 2TB disk dedicated to drbd; I recently 
migrated from drbdmanage, all 3 nodes are COMBINED type, each resource 
is replicated on each node.

=== networking ===

drbd storage uses a dedicated network mesh, with dedicated connections 
between nodes (no switch). There is a bridge used as drbd interface on 
each node, stp is disabled, and broadcasts storm are blocked using 
ebtable rules. Here's the relevant part of /etc/network/interfaces file 
for node pve3

auto vmbr2
iface vmbr2 inet static
         bridge_ports eth2 eth3
         bridge_stp off
	bridge_ageing 30
         bridge_fd 5
# Only with stp on
       # pve1 and pve2 are preferred
         #bridge_bridgeprio 32768
# Only with stp off
         pre-up ifconfig eth2 mtu 9000 && ifconfig eth3 mtu 9000
	up   ebtables -I FORWARD -i eth2 -j DROP
	up   ebtables -I FORWARD -i eth3 -j DROP
	up   ebtables -I FORWARD -o tap200i1 -j ACCEPT
         down ebtables -D FORWARD -i eth2 -j DROP
         down ebtables -D FORWARD -i eth3 -j DROP
	down ebtables -D FORWARD -o tap200i1 -j ACCEPT

The "tap200i1" interface corresponds to th controller vm, the resulting 
ebtables rules are:

# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 3, policy: ACCEPT
-o tap200i1 -j ACCEPT
-i eth3 -j DROP
-i eth2 -j DROP

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
root at pve3:~# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 3, policy: ACCEPT
-o tap200i1 -j ACCEPT
-i eth3 -j DROP
-i eth2 -j DROP

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

=== drbd resources ===

Resource definitions are as follows:

# linstor rd l -p
| ResourceName  | Port | State |
| vm-100-disk-1 | 7009 | ok    |
| vm-101-disk-1 | 7001 | ok    |
| vm-101-disk-2 | 7002 | ok    |
| vm-102-disk-1 | 7003 | ok    |
| vm-103-disk-1 | 7000 | ok    |
| vm-103-disk-2 | 7004 | ok    |
| vm-104-disk-1 | 7008 | ok    |
| vm-104-disk-2 | 7005 | ok    |
| vm-105-disk-1 | 7006 | ok    |
| vm-106-disk-1 | 7013 | ok    |
| vm-120-disk-1 | 7010 | ok    |
| vm-120-disk-2 | 7015 | ok    |
| vm-121-disk-1 | 7007 | ok    |
| vm-122-disk-1 | 7011 | ok    |
| vm-123-disk-1 | 7012 | ok    |
| vm-200-disk-1 | 7014 | ok    |
| vm-999-disk-1 | 7016 | ok    |
| vm-999-disk-2 | 7017 | ok    |

=== drbd nodes before controller virtualization ===

| Node  | NodeType   | IPs               | State   |
| pve1  | COMBINED   |   | Online  |
| pve2  | COMBINED   |   | Online  |
| pve3  | COMBINED   |   | Online  |

I installed the controller on and this basic setup is working 

I installed a debian stretch minimal vm with id 200, connected with 
distict interfaces to both the drbd storage and the regular network 
segments. I installed linbit controller on it, and gave hostname 
"drbdc". I stopped and disabled the controller on pve1 and migrated the 
service on drbdc. Then i added drbdc to the drbd cluster:

=== drbd nodes after controller virtualization ===

| Node  | NodeType   | IPs               | State   |
| drbdc | CONTROLLER | | OFFLINE |
| pve1  | COMBINED   |   | Online  |
| pve2  | COMBINED   |   | Online  |
| pve3  | COMBINED   |   | Online  |

After that I enabled and started drbd service on all three nodes.

Then i changed the drbd proxmox storage configuration as follows:

drbd: drbdthin
	content rootdir,images
	redundancy 3
	controllervm 200

Then i enabled HA for vm 200


At this point all worked quite well, but when i tried to shut down the 
node on which the controller vm resided, the resources on it did not 
came up, even that of controller vm. In one case HA did its job moving 
it on another node, but since in my setup many times the quorum is lost 
temporarily even restarting one node only, the controller restarted only 
after the quorum was established.

I have still to investigate the condition under that drbd storage became 
unavailable to pve, causing all vms to stop. Hopefully I will have a 
chance to give you some more details after examining the logs.

At the moment I can report only a bunch of these messages in syslog:

Aug 25 22:49:04 pve3 pvestatd[2598]: malformed JSON string, neither tag, 
array, object, number, string or atom, at character offset 0 (before 
"(end of string)") at 
/usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 321.

may be generated when i switched to the "controllervm" configuration.

>> The main problem, if I understand well, is that even if proxmox plugin does
>> not manage controller vm anymore, controller vm storage depends on itself.
> What do you mean by that? The DRBD resource (== storage for the
> controller VM) is brought up by the drbd.service and can then be
> auto-promoted. 

I noted that resources definition files inside /var/lib/linstor.d are 
recreated each time the node is started, so i guess drbd service cannot 
bring them up.

> The plugin code ignores that VM. The Proxmox HA service
> should do its job and start the VM on one node. So again, where exactly
> does that chain break in your setup?

See before.

>> If it is stopped, or cannot be contacted, no corresponding resource will be
>> created, resulting in a deadlock.
> That is true. But how would that happen? The Proxmox HA feature should
> make sure that the VM is always running (that is its job, isn't it). The
> storage should be accessible, because up'ed by drbd.service.

Is drbd service dependent on definitions files? If so, I guess that if 
controller is unavailable, definitions of resources linstor-managed are 
not provided to satellite that cannot create definitions. It's only a 
guess, i'm not inside linstor/drbd internals.

> Did Proxmox try to start another "autoboot" VM before the HA service
> kicked in to start the controller VM? That would at least explain it.

May be; i have another two vm with autoboot feature enabled, but I had 
set them in order after controller vm, and with minutes of delay.

> In
> that case we have to document that "autoboot" has to be disabled if one
> goes that route.

Ok, will retry and hopefully give you some more feedback.


> Thanks, rck
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

More information about the drbd-user mailing list