[DRBD-user] linstor-proxmox-2.9.0

[Roland Kammerer]

On Mon, Aug 27, 2018 at 09:53:36AM +0100, Yannis Milios wrote:
> Sorry for interrupting, but I wanted to share my experience with this as
> well (see below) ...
> 
> 
> > What do you mean by that? The DRBD resource (== storage for the
> > controller VM) is brought up by the drbd.service and can then be
> > auto-promoted. The plugin code ignores that VM. The Proxmox HA service
> > should do its job and start the VM on one node. So again, where exactly
> > does that chain break in your setup?
> >
> 
> In my case, I had success in making the LINSTOR controller HA in two
> different clusters, but by following a slightly different approach.
> 
> In documentation, it's stated the following ...
> 
> "The basic idea is to execute the LINSTOR controller within a VM that is
> controlled by Proxmox and
> its HA features, where the storage resides on DRBD managed by LINSTOR
> itself."
> 
> So by this, I understand that the LINSTOR controller VM will be managed by
> PVE by using its HA feature, all good so far.
> Then it's stated that, the storage that will be used to store this VM will
> be DRBD, which will be managed by LINSTOR itself.
> 
> 
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Question: Isn't the above a paradox? How can PVE HA start a machine which
> is stored in a shared storage, which the only way to make it available, is
> by starting this HA VM  in the first place?

No there is no contradiction, the last part of your sentence is wrong.
Let's try it this way:
What does Proxmox need to start the Controller VM?
- the actual storage has to be available. It is just a DRBD resource. A
  known one. So if "somebody" drbdadm ups it, it is there. That should
  be done by the drbd.service. Why this did not work, see below. But for
  now assume the DRBD resource where the controller VM is on is DRBD up.
- Proxmox will ask the drbd/linstor plugin "to make the storage for the
  controller VM available". The plugin is there. It knows which VM is
  the special one. The storage should be available. So the plugin just
  returns to Proxmox "hey, everything is good, do your qemu magic".

Then, and only then, after the controller VM is up, other "normal" VMs
can be started.

Do we agree on that?

> My tests showed that whenever PVE tried to start this HA VM, it failed,
> because the underlying storage (DRBD) was not available  (and yes,
> drbd.service was set to "enabled" and was "started" in all nodes).
> Checked inside /var/lib/linstor.d to see if the resource file for this VM
> was in there, but it wasn't. Actually, there were no resource files at all
> in there, apart from "linstor_common.conf".

And that is the problem we have to fix. The linstor satellite deletes
its resource files from /var/lib/linstor.d on startup. So
linstor-satellite.service and drbd.service started more or less at the
same time. The satellite deleted the res file, drbd.service could not
bring it up, the rest is obvious.

> Question: Where should drbd.service find the resource configuration file
> for the LINSTOR controller VM? Inside /var/lib/linstor.d or in /etc/drbd.d ?

/var/lib/linstor.d

But linstor-satellite should not delete it on startup. We will fix that
by introducing a parameter in the linstor-satellite.service file.

> For the former, LINSTOR should auto generate that resource file, but it
> doesn't, because the controller is not available (yet) ?.

The satellite just deleted it. If that would not have happened,
drbd.service would have brought it up, and the controller VM would have
started (assuming the rest works expected). The key thing here is that
we do *not* need to controller to start the controller VM.

> For the latter, someone can manually create a .res file (the classic
> DRBD approach), copy it to all nodes and do the required backed
> storage preparations before deploying the LINSTOR controller VM.
> 
> I used the latter method and so far seems to work properly (manually
> created .res files in /etc/drbd.d). 

That is good to know. Basically with that you avoided that the res file
got deleted, because it was at a different location ;-).

In general we do not want that. We want the resource to be under linstor
control. For example if you add a cluster nodes, it would be nice to
just add that node and assign the resource to it.

Thanks, rck