[DRBD-user] DRBD cluster and updating system

[Dušan Maček]

Dear All,

I need some advice regarding cluster system update. I've built a cluster 
in a hope of zero downtime, but unfortunately it doesn't work this way.

Configuration

- two physical machines running Centos 7 and KVM
- on each installed virtual machine with Ubuntu 16.04 (apache, mysql, 
postfix, ftp, dovecot, ispfonfig...) and drbd

The issue : Before each upgrade I really want to make a snapshot. But 
there's a problem.

In order to do this, I have to

- disconnect DRBD synchronization (using LCMC cluster management tool 
putting one node offline)
- at the same time make a snapshot on both nodes

This means during a snapshot a cluster is down for about 15-20 minutes. 
And of course if I want to revert to this snapshot I have to do it on 
both nodes at the same time (downtime again). If I revert to a snapshot 
only on one node sync is broken and have to reinitialize sync of 
everything. Which could take days when there's about 5TB of data.

For some time I was performing apt-get update&upgrade on inactive node, 
but on one occasion mariadb was upgraded and since there was no database 
(mounted on other node) the upgrade made a total mess. I was lucky I had 
a snapshot. Since then I always do upgrade on active node with sync 
disabled. After successful upgrade I put both nodes online, wait for 
sync to finish, change role, disconnect, upgrade another one, connect.

Now the main question : what are your experiences with system upgrades 
in cluster environment ? How to avoid downtime ?

Best regards,
Dušan Maček