Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Thanks for your reply Tom, I assume it was meant to also go to the list? On Mon May 26, 2008 at 23:55:12 -0700, Tom Brown wrote: > >if you've got the option of going to either the absolutely most current >drbd version 8.0.12 or the 8.2.6rc1, try turning off the I'd like to do as little work as possible to fix the problem, so I'm trying to avoid upgrading to 8. If I can get concrete evidence that that will fix the problem then I will make preparations to do so though. > > no-disk-flushes; > no-md-flushes; > >options. Apparently that is the big difference between 0.7 and current 8.x >versions. > >(hhmm, I'm not sure what "upgrade" you're refering to, I don't know if the >write barrier stuff was added in the 0.7.x versions... I believe in the >8.0 line it was added between 8.0.8 and 8.0.9 ) The upgrade was hardware only - adding of a hardware RAID card and the 300GB disks to give battery-backed write cache and additional storage. >I haven't tried it with those disabled in standalone/disconnected mode, >but I have observed that there was a large performance hit even in >stand-alone mode when I tried going from .7.x to the current 8.x versions. > >I don't claim to understand barrier writes and what not... at least I >don't claim to understand why it's slower running through drbd than it is >running direct to disk... it seems that drbd is more than a bit paranoid >about making sure everything gets to disk in the right order. A good thing >in some circumstances I am sure. I know 0.7.x has no barrier support, and the last I checked barrier support was only in a half-done state in 8.x. You are right in that it would involve a performance hit - everything I have read and my own personal tests show that enabling write-caching and write barriers give you an aggregate loss in performance. But as I said, I'm using 0.7.x and battery-backed write caching on the controller (and NOT on the drives) so this isn't really applicable. >My benchmark scenario was similar to yours... things were fine, seemed >plenty fast, and then I had some stability issues with drbd showing up in >one of the kernel dumps... so I tried to move to the official centos >modules which were 8.0.x and 8.2.x and that's when I had performance >issues and started benchmarking... > >(and IMHO 53 MB/s seems slow for 10k or 15k drives?) Perhaps it is. My testing has shown when you disable write caching on the drives the performance tends to be a lot less than advertised (probably because they sneakily enable it by default and hope nobody would notice), but then it shouldn't matter anyway because it is sequential writes... All I know is that I'm taking a large hit just by going through DRBD from my baseline performance figures. > >-Tom > >On Tue, 27 May 2008, Oliver Hookins wrote: > >>Hi, >> >>I've recently upgraded a couple of machines we already had running DRBD >>0.7.22 and previously the performance we were seeing seemed ok so I didn't >>do comprehensive tests. Now that the machines have been upgraded and are >>doing a bit more we are starting to notice the performance isn't quite what >>it should be. >> >>Briefly the systems are: >>Supermicro X7DBP >>2 x Intel Xeon 5160 dual core @ 3GHz >>4GB ECC DDR400 Actually this is wrong, it is DDR2-667 >>Adaptec 2020ZCR RAID card >>2 x 147GB 15krpm SCSI drives >>2 x 300GB 10krpm SCSI drives >>RHEL4ES 64bit on 2.6.9-67.0.7.ELsmp kernel >>DRBD 0.7.22 (which is the latest stable version that was available at the >>time the cluster was built) >> >>Each of the drives is only set up as a logical drive in the RAID card, and >>RAID1 is done in the software for each pair of drives. We have >>write-caching >>turned on for the 147GB drives. >> >>If I test write speed with something like >>'time sync; time dd if=/dev/zero of=test bs=1G count=4; time sync' >>on a non-DRBD partition of the 147GB RAID, I see write speeds of about >>53MB/s. >> >>If I perform the same test on a DRBD partition on the 147GB RAID that is >>disconnected, I'm seeing write speeds of about 38MB/s. To go down to about >>70% of non-DRBD performance before we have even taken into account >>slowdowns >>due to the network is a very big performance hit. >> >>I'm also seeing similar results on the 300GB drives, but about 33% less >>performance overall again due to the drop in spindle speed. DRBD-connected >>speed is even worse but I'm holding off on persuing that until I have the >>disconnected speeds sorted out. >> >>Some other info: >>al-extents 601 >>meta-disk is internal >>(I know this represents a metadata write after only 2.3GB or so, but we see >>the same slow performance with smaller write sizes) >> >>comms over 1GbE NICs directly connected >>MTU of 9000 >>syncer rate 50M >>sndbuf-size 256K >>max-buffers 2048 >>max-epoch-size 2048 I bumped up sndbuf-size to 2M, max-buffers and max-epoch-size to 8192 but didn't see any appreciable difference when resyncing, so perhaps the problem lies elsewhere... >> >>As I said, the network side probably needs attention as well but for the >>moment I need to address the large discrepancy between the non-DRBD and >>disconnected-DRBD speeds first. Any suggestions would be appreciated. >> >>-- >>Regards, >>Oliver Hookins >>_______________________________________________ >>drbd-user mailing list >>drbd-user at lists.linbit.com >>http://lists.linbit.com/mailman/listinfo/drbd-user >> > >---------------------------------------------------------------------- >tbrown at BareMetal.com | Drive thy business, or it will drive thee. >http://BareMetal.com/ | - Benjamin Franklin >web hosting since '95 | > -- Regards, Oliver Hookins