<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Lars,<div><br></div><div>Thank you for responding. It was quite helpful. I don't know why I thought I needed a temporary drbd resource to get at the snapshot. I do indeed see the VM img files just mounting the snapshot.</div><div><br></div><div>What can I do to reduce the effect of the backup on the primary node ? Would it make sense to do this before the copy starts:</div><div><br></div><div>/sbin/drbdadm net-options --protocol=A zapp</div><div><br></div><div>or perhaps increasing the sndbuf-size or al-extents ?</div><div><br></div><div>Irwin<br><br><div>> Date: Wed, 30 Apr 2014 22:31:17 +0200<br>> From: lars.ellenberg@linbit.com<br>> To: drbd-user@lists.linbit.com<br>> Subject: Re: [DRBD-user] Load high on primary node while doing backup on secondary<br>> <br>> On Wed, Apr 23, 2014 at 10:16:24AM -0700, Irwin Nemetz wrote:<br>> > I have a two node cluster. There are 3 mail nodes running as KVM virtual<br>> > machines on one node. The 3 VM's sit on top of a DRBD disk on a LVM volume<br>> > which replicates to the passive 2nd node.<br>> > <br>> > Hardware: 2x16 core AMD processors, 128gb memory, 5 3tb sas drives in a raid5<br>> <br>> You likely won't get "terrific" performance out of a few large drivers in raid5.<br>> <br>> > The drbd replication is over a crossover cable.<br>> > <br>> > version: 8.4.4 (api:1/proto:86-101)<br>> > GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06<br>> <br>> > resource zapp<br>> > {<br>> > startup {<br>> > wfc-timeout 10;<br>> > outdated-wfc-timeout 10;<br>> > degr-wfc-timeout 10;<br>> > }<br>> > disk {<br>> > on-io-error detach; <br>> > rate 40M;<br>> > al-extents 3389;<br>> > }<br>> > net {<br>> > verify-alg sha1;<br>> > max-buffers 8000;<br>> > max-epoch-size 8000;<br>> > sndbuf-size 512k;<br>> > cram-hmac-alg sha1;<br>> > shared-secret sync_disk;<br>> > data-integrity-alg sha1;<br>> <br>> Don't enable data-integrity in production. It will just burn your<br>> cycles and limit your throughput to however fast your core can crunch sha1.<br>> <br>> That's a *diagnostic* feature.<br>> It does not really do anything for the integrity of your data,<br>> it just happens to help to *detect* when integrity *may* be compromised.<br>> We should have named that<br>> "burn-cpu-cycles-and-calculate-extra-checksums-for-diagnostic-purposes".<br>> <br>> <br>> > }<br>> > on nodea.cluster.dns {<br>> > device /dev/drbd1;<br>> > disk /dev/virtimages/zapp;<br>> > address 10.88.88.171:7787;<br>> > meta-disk internal;<br>> > }<br>> > on nodeb.cluster.dns {<br>> > device /dev/drbd1;<br>> > disk /dev/virtimages/zapp;<br>> > address 10.88.88.172:7787;<br>> > meta-disk internal;<br>> > }<br>> > }<br>> <br>> You could probably do a lot of tuning ...<br>> <br>> > I am trying to do a backup of the VM's nightly. They are about 2.7TB each.<br>> > I create a snapshot on the backup node, mount it and then do a copy to a<br>> > NAS backup storage device. The NAS is on it's own network.<br>> > <br>> > Here's the script:<br>> > <br>> > [root@nodeb ~]# cat backup-zapp.sh<br>> > #!/bin/bash<br>> > <br>> > date<br>> > cat > /etc/drbd.d/snap.res <<EOF<br>> <br>> *OUTCH*<br>> <br>> What would you do that.<br>> <br>> There is no point to create a throw away DRBD resource automatically<br>> to access a snapshot from below an other DRBD.<br>> <br>> Just use that snapshot directly.<br>> <br>> Or am I missing something?<br>> <br>> > /sbin/lvcreate -L500G -s -n snap-zapp /dev/virtimages/zapp<br>> > <br>> > /sbin/drbdadm up snap<br>> <br>> No need. Really.<br>> <br>> > sleep 2<br>> > /sbin/drbdadm primary snap<br>> > mount -t ext4 /dev/drbd99 /mnt/zapp<br>> <br>> instead, this should do all you need:<br>> mount -t ext4 -o ro /dev/virtimages/snap-zapp /mnt/zapp<br>> <br>> > cd /rackstation/images<br>> > mv -vf zapp.img zapp.img.-1<br>> > mv -vf zapp-opt.img zapp-opt.img.-1<br>> > cp -av /mnt/zapp/*.img /rackstation/images<br>> > umount /mnt/zapp<br>> > /sbin/drbdadm down snap<br>> > rm -f /etc/drbd.d/snap.res<br>> > /sbin/lvremove -f /dev/virtimages/snap-zapp<br>> > date<br>> > <br>> > About half way thru the copy, the copy starts stuttering (network traffic<br>> > stops and starts) and the load on the primary machine and the virtual<br>> > machine being copied shoots thru the roof.<br>> <br>> Maybe snapshots filling up, disks being slow and all,<br>> and everything that is written on the primary needs to be written on the<br>> secondary *twice* now, while you are hammering that secondary IO<br>> subsystem with reads...<br>> of course that impacts the Primary, as soon as the secondary RAID 5<br>> can no longer keep up with the load you throw at it.<br>> <br>> > I am at lose to explain this since it's dealing with a snapshot of a<br>> > volume on a replicated node. The only reasonable explanation I can think<br>> > of is that the drbd replication is being blocked by something and this is<br>> > causing the disk on the primary node to become unresponsive.<br>> <br>> Yep.<br>> See above.<br>> <br>> You could try to use rsync --bwlimit instead of cp, that will reduce the<br>> read load, but it will also prolong the lifetime of the snapshot,<br>> so it may or may not actually help.<br>> <br>> Or maybe you just need to defrag your VM images...<br>> possibly they are fragmented, and what you see is the secondary IO<br>> subsystem hitting the IOPS limit while trying to seek through all the VM<br>> image fragments...<br>> <br>> Or use a "stacked" DBRD setup,<br>> and disconnect the third node.<br>> Or, if you can live with reduced redundancy during the backup,<br>> disconnect the secondary for that time.<br>> <br>> Or add a dedicated PV for the snapshot "exeption store",<br>> or add non-volatile cache to your RAID.<br>> or a number of other options.<br>> <br>> Thing is, if you stress the secondary IO subsystem enough,<br>> that *will* impact the (write performance on the) primary.<br>> <br>> -- <br>> : Lars Ellenberg<br>> : LINBIT | Your Way to High Availability<br>> : DRBD/HA support and consulting http://www.linbit.com<br>> <br>> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>> __<br>> please don't Cc me, but send to list -- I'm subscribed<br>> _______________________________________________<br>> drbd-user mailing list<br>> drbd-user@lists.linbit.com<br>> http://lists.linbit.com/mailman/listinfo/drbd-user<br></div></div> </div></body>
</html>