[DRBD-user] High IOwait with rsync on DRBD-OCFS2-NFS node

[Dan Barker]

> -----Original Message-----
> From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user-
> bounces at lists.linbit.com] On Behalf Of Vasily S. Kostroma
> Sent: Friday, January 03, 2014 7:05 PM
> To: drbd-user at lists.linbit.com
> Subject: [DRBD-user] High IOwait with rsync on DRBD-OCFS2-NFS node
> 
> Happy holidays, friends.
> It's my first message in this list.
> 
> I have two 100% same servers with two logical drives - one for system
> and second one for DRBD primary-primary configuration. The servers
> connected via dedicated Gb NIC's via crossover cable. Operational system
> is Debian 7, file system for DRBD is OCFS2. Additionally I using NFS
> server to share a files from DRBD drive to other servers. The
> configuration is very simple. Just one resource with internal metadata.
> Nothing special.
> 
> But I have a problem and, unfortunately, I have no idea how to fix it.
> I using Node1 as primary and Node2 as secondary. I mean clients
> (web-servers) are connecting to Node1 via NFS, and backup's running on
> Node2. Example. I starting rsync on Node2 to sync a local folder with
> folder on the same server, but DRBD+OCFS2. It's working fine, but in
> backup time the NFS server on Node1 stop working till backup on Node2 is
> in progress. A NFS processes (threads) is on "D" state in top. After
> backups completed the NFS start working perfectly. Just for additional
> attention: I running rsync on Node2, but NFS stop working on Node1.
> 
> I will appreciate any idea what is wrong.. All settings is almost
> default, but problem is there :(
> 
> Sorry for my English.
> And thank you in advance.
> Happy holidays!
> 

Happy Holidays back at you! It's too cold for my liking, however.

I do not believe you can run rsync successfully against an OCFS2 file system in that way. Your experience appears to bear that out. You have several solutions that could allow this sort of hot backup, however.

1) disconnect the resource before taking the rsync backup. This will effectively "snapshot" the resource at the time of the disconnect. Node1 will continue serving the shares on NFS, but the Node2 disk version will slowly diverge from the "snapshot" on node2. Be sure to mount the resource Read Only on Node 2 or you will have a split brain for sure (noatime is probably not sufficient to protect you). As soon as the backup is complete, unmount and connect the resource, and the disks will rapidly re-synchronize. PROS: No interruption to NFS and you produce a "crash consistent" version of your filesystem for the rsync backup. CONS: No redundancy during the time the backup takes.

2) Run your rsync backup from Node1.

3) There may be a snapshot facility in OCFS2. I do not know - I've used OCFS2 primarily with xen and use reflink for snapshots. But, reflink snapshots a file but I believe you want to snapshot a file system. If there is an appropriate facility in OCFS2, produce the snapshot on Node1, and then backup the snapshot from Node2. That shouldn't lockout NFS on Node1 accessing the live disk - the snapshot is static.

hth

Dan in Atlanta