[DRBD-user] The need for speed

Jeff Buck jeffb at umci.com
Fri Nov 19 19:30:37 CET 2004

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


I don't know if this will help you, but we get around 20MB/sec with our
3ware raid + lvm2 + drbd. Our version of drbd is fairly old, and I
haven't messed with it lately. I think there have been "speed ups" on
one of the releases I've seen since our version (We're on 0.7.1 I
think). We've got all 8 disks in use on it, one of which is a hot spare.

On Fri, 2004-11-19 at 05:42, Jan Bruvoll wrote:
> Hi there,
> 
> I find your comment about S-ATA RAID controllers highly interesting -
> I 
> am struggling bigtime with getting anywhere near reasonable speeds
> using 
> a NFS - DRDB - 3Ware RAID chain. What do you mean exactly with "slow",
> and would you be able to share any pointers on how to pinpoint why my 
> server pair is so painfully slow?
> 
> Best regards & TIA
> Jan
> 
> Philipp Reisner wrote:
> 
> >Hi,
> >
> >Again I had the chance to set up a DRBD Cluster for a LINBIT
> >Customer. It was the first time I a had one of these new SATA
> >devices really under my fingers [ withouh any of these enterprise
> >ready RAID controlers, which are in reality rather slow... ]
> >
> >The machines are some DELLs with a single P4 w 2.8 GHz and an
> >"Intel Corp. 6300ESB SATA Storage Controller" IDE Controller,
> >two "Intel Corp. 82547GI Gigabit Ethernet Controller" NICs
> >and 512 MB RAM, the disk calles itsef "ST380011AS" Seagate
> >Baracuda.
> >
> >At first the performance of the disk was miserable, in the
> >area of ~5 MB/sec, as it turned out the reason for this was
> >because we used the LINUX's common IDE (PATA).
> >
> >Then we tried the libata/ata_piix driver combination, and suddenly
> >we got a write performance in the area of 40MB/sec.
> >
> >BTW, with libata suddenly the disk appear as SCSI disk !
> >[ -> changing all config files from "hdc" to "sda" ]
> >
> >Networksetup:
> >e1000 driver, the machines connected with a straight
> >cat5E cable, forced the cards into "speed 1000" with
> >ethtool, and set the MTU to 9000 aka Jumboframes.
> >
> >I am interested in raw data throughput, so I did sequential
> >writes on an ext3 Filesystem. 
> >
> >Test1
> >I wrote a 1GB File (with sync) to the root partition
> >[Cyl: 250 to 1466] 3 times:
> >
> >43.35 MB/sec (1073741824 B / 00:23.621594)
> >40.43 MB/sec (1073741824 B / 00:25.328009)
> >40.78 MB/sec (1073741824 B / 00:25.112768)
> >avg: 41.52
> >
> >Test2
> >The I did the same on a connected DRBD device (protocol C), 
> >also ext3: [Cyl: 2747 to 6151]
> >
> >39.05 MB/sec (1073741824 B / 00:26.226047)
> >35.95 MB/sec (1073741824 B / 00:28.483070)
> >36.48 MB/sec (1073741824 B / 00:28.068153)
> >avg: 37.16
> >
> >At first I was satisfied with the outcome that DRBD [with protocol C]
> >costs you about 10% of your throughput with sequential writes.
> >
> >Test3
> >But the I did the same test with DRBD disconnect and got these
> numbers:
> >[Cyl: 2747 to 6151]
> >39.63 MB/sec (1073741824 B / 00:25.840004)
> >40.30 MB/sec (1073741824 B / 00:25.406312)
> >39.82 MB/sec (1073741824 B / 00:25.713998)
> >avg: 39.91
> >
> >I aked myself: Why is it 4% below the first test ?
> >
> >Assumption: Maybe because the mirrored partition is behind the
> >            root partition, and harddisk are slower on the outer 
> >            cylinders than on the inner cylinders.
> >
> >Test4:
> >So I unloaded the DRBD module and mounted the backing storage devices
> >on the mountpoints directly! [Cyl: 2747 to 6151]
> >39.65 MB/sec (1073741824 B / 00:25.823633)
> >38.54 MB/sec (1073741824 B / 00:26.570280)
> >37.26 MB/sec (1073741824 B / 00:27.479914)
> >avg: 38.48
> >
> >Test3 was 3.5% faster than Test4. This could be explained by the fact
> >that DRBD sometimes triggers the immediate write of buffers to disk.
> >
> >The DRBD mirroring overhead, thus Test4 to Test2 is 3.4% which is
> smaller
> >then the performance differences within the disk device Test1 to
> Test4 
> >is 7.3%
> >
> >CPU Usage:
> > I monitored CPU Usage on the secondary system using the "top"
> utitily
> > and the hightes value for the drbd_receiver was 7.7%.
> >
> >Resync performance:
> > For the Customer I configured the syncer to run with 10MB/sec, this
> > makes sure that the Customer's application will continue to run
> > during a resync operation. For testing purpose I set the 
> > resync rate to 40M and got a resync rate in the area of 33MByte/sec.
> >
> >Effect of JumboFrames / 9000 MTU
> > I repeated Test2 with an MTU of 1500 Byte and got these numbers:
> > 36.27 MB/sec (1073741824 B / 00:28.234703)
> > 36.22 MB/sec (1073741824 B / 00:28.270442)
> > 36.41 MB/sec (1073741824 B / 00:28.121841)
> > On the secondary system the CPU system time's highest point was 7%,
> > and the spotted maximum on the drbd_receiver thread was 9.7%
> > So it seems the the JumboFrames only ease the task of the secondary
> node,
> > but do not improve performance.
> >
> >Test Setup:
> > Linux 2.6.9
> > DRBD 0.7.5
> > Writes were this command: ./dm -a 0x00 -o /mnt/ha0/1Gfile -b1M -m -p
> -y -s1G
> > dm is from drbd-0.7.5.tar.gz in the benchmark directory.
> >
> >Conclusion:
> >===========
> > The Performance inhomogeneity within a single disk drive can be
> > bigger (measusred 7.3%) than the loss of performance caused by DRBD
> > mirroring (measured 3.4%).
> >
> > This only holds true it the limiting factor is the performance of
> > the disk. In other words: Your network link and your CPU needs to 
> > be strong enough.
> >
> >-phil
> >  
> >
> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
> 




More information about the drbd-user mailing list