[DRBD-user] DRBD cluster and updating system
dusan at 8000plus.si
Thu Aug 29 09:46:30 CEST 2019
I need some advice regarding cluster system update. I've built a cluster
in a hope of zero downtime, but unfortunately it doesn't work this way.
- two physical machines running Centos 7 and KVM
- on each installed virtual machine with Ubuntu 16.04 (apache, mysql,
postfix, ftp, dovecot, ispfonfig...) and drbd
The issue : Before each upgrade I really want to make a snapshot. But
there's a problem.
In order to do this, I have to
- disconnect DRBD synchronization (using LCMC cluster management tool
putting one node offline)
- at the same time make a snapshot on both nodes
This means during a snapshot a cluster is down for about 15-20 minutes.
And of course if I want to revert to this snapshot I have to do it on
both nodes at the same time (downtime again). If I revert to a snapshot
only on one node sync is broken and have to reinitialize sync of
everything. Which could take days when there's about 5TB of data.
For some time I was performing apt-get update&upgrade on inactive node,
but on one occasion mariadb was upgraded and since there was no database
(mounted on other node) the upgrade made a total mess. I was lucky I had
a snapshot. Since then I always do upgrade on active node with sync
disabled. After successful upgrade I put both nodes online, wait for
sync to finish, change role, disconnect, upgrade another one, connect.
Now the main question : what are your experiences with system upgrades
in cluster environment ? How to avoid downtime ?
More information about the drbd-user