[DRBD-user] The need for speed

Jan Bruvoll jan at bruvoll.com
Tue Nov 23 12:52:33 CET 2004

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi there,

my rather strange problem is that I get reasonable throughput when 
writing directly to the device as well as writing a file through DRBD. 
Even using a single NFS client is ok, but when I let my web server 
cluster loose on the DRBD-based NFS server pair, everything grinds to a 
halt. No errors, retransmissions, packet loss, nothing - just extremely 
slow, almost frozen-up servers.

The strange thing is that I have an old NFS server that does just fine 
under the exact same circumstances - however without the DRBD mirroring.

Since last time I posted here I've upgraded the pair to kernel 2.6.9 and 
drbd 0.7.5, but I have yet to dare switching the pair into action. (Last 
time I tried I had the first and worst outage on our web portal in three 
years - wouldn't like to have that happen again.)

Will post my progress on this.

Best regards
Jan

Jeff Buck wrote:

>I don't know if this will help you, but we get around 20MB/sec with our
>3ware raid + lvm2 + drbd. Our version of drbd is fairly old, and I
>haven't messed with it lately. I think there have been "speed ups" on
>one of the releases I've seen since our version (We're on 0.7.1 I
>think). We've got all 8 disks in use on it, one of which is a hot spare.
>
>On Fri, 2004-11-19 at 05:42, Jan Bruvoll wrote:
>  
>
>>Hi there,
>>
>>I find your comment about S-ATA RAID controllers highly interesting -
>>I 
>>am struggling bigtime with getting anywhere near reasonable speeds
>>using 
>>a NFS - DRDB - 3Ware RAID chain. What do you mean exactly with "slow",
>>and would you be able to share any pointers on how to pinpoint why my 
>>server pair is so painfully slow?
>>
>>Best regards & TIA
>>Jan
>>
>>Philipp Reisner wrote:
>>
>>    
>>
>>>Hi,
>>>
>>>Again I had the chance to set up a DRBD Cluster for a LINBIT
>>>Customer. It was the first time I a had one of these new SATA
>>>devices really under my fingers [ withouh any of these enterprise
>>>ready RAID controlers, which are in reality rather slow... ]
>>>
>>>The machines are some DELLs with a single P4 w 2.8 GHz and an
>>>"Intel Corp. 6300ESB SATA Storage Controller" IDE Controller,
>>>two "Intel Corp. 82547GI Gigabit Ethernet Controller" NICs
>>>and 512 MB RAM, the disk calles itsef "ST380011AS" Seagate
>>>Baracuda.
>>>
>>>At first the performance of the disk was miserable, in the
>>>area of ~5 MB/sec, as it turned out the reason for this was
>>>because we used the LINUX's common IDE (PATA).
>>>
>>>Then we tried the libata/ata_piix driver combination, and suddenly
>>>we got a write performance in the area of 40MB/sec.
>>>
>>>BTW, with libata suddenly the disk appear as SCSI disk !
>>>[ -> changing all config files from "hdc" to "sda" ]
>>>
>>>Networksetup:
>>>e1000 driver, the machines connected with a straight
>>>cat5E cable, forced the cards into "speed 1000" with
>>>ethtool, and set the MTU to 9000 aka Jumboframes.
>>>
>>>I am interested in raw data throughput, so I did sequential
>>>writes on an ext3 Filesystem. 
>>>
>>>Test1
>>>I wrote a 1GB File (with sync) to the root partition
>>>[Cyl: 250 to 1466] 3 times:
>>>
>>>43.35 MB/sec (1073741824 B / 00:23.621594)
>>>40.43 MB/sec (1073741824 B / 00:25.328009)
>>>40.78 MB/sec (1073741824 B / 00:25.112768)
>>>avg: 41.52
>>>
>>>Test2
>>>The I did the same on a connected DRBD device (protocol C), 
>>>also ext3: [Cyl: 2747 to 6151]
>>>
>>>39.05 MB/sec (1073741824 B / 00:26.226047)
>>>35.95 MB/sec (1073741824 B / 00:28.483070)
>>>36.48 MB/sec (1073741824 B / 00:28.068153)
>>>avg: 37.16
>>>
>>>At first I was satisfied with the outcome that DRBD [with protocol C]
>>>costs you about 10% of your throughput with sequential writes.
>>>
>>>Test3
>>>But the I did the same test with DRBD disconnect and got these
>>>      
>>>
>>numbers:
>>    
>>
>>>[Cyl: 2747 to 6151]
>>>39.63 MB/sec (1073741824 B / 00:25.840004)
>>>40.30 MB/sec (1073741824 B / 00:25.406312)
>>>39.82 MB/sec (1073741824 B / 00:25.713998)
>>>avg: 39.91
>>>
>>>I aked myself: Why is it 4% below the first test ?
>>>
>>>Assumption: Maybe because the mirrored partition is behind the
>>>           root partition, and harddisk are slower on the outer 
>>>           cylinders than on the inner cylinders.
>>>
>>>Test4:
>>>So I unloaded the DRBD module and mounted the backing storage devices
>>>on the mountpoints directly! [Cyl: 2747 to 6151]
>>>39.65 MB/sec (1073741824 B / 00:25.823633)
>>>38.54 MB/sec (1073741824 B / 00:26.570280)
>>>37.26 MB/sec (1073741824 B / 00:27.479914)
>>>avg: 38.48
>>>
>>>Test3 was 3.5% faster than Test4. This could be explained by the fact
>>>that DRBD sometimes triggers the immediate write of buffers to disk.
>>>
>>>The DRBD mirroring overhead, thus Test4 to Test2 is 3.4% which is
>>>      
>>>
>>smaller
>>    
>>
>>>then the performance differences within the disk device Test1 to
>>>      
>>>
>>Test4 
>>    
>>
>>>is 7.3%
>>>
>>>CPU Usage:
>>>I monitored CPU Usage on the secondary system using the "top"
>>>      
>>>
>>utitily
>>    
>>
>>>and the hightes value for the drbd_receiver was 7.7%.
>>>
>>>Resync performance:
>>>For the Customer I configured the syncer to run with 10MB/sec, this
>>>makes sure that the Customer's application will continue to run
>>>during a resync operation. For testing purpose I set the 
>>>resync rate to 40M and got a resync rate in the area of 33MByte/sec.
>>>
>>>Effect of JumboFrames / 9000 MTU
>>>I repeated Test2 with an MTU of 1500 Byte and got these numbers:
>>>36.27 MB/sec (1073741824 B / 00:28.234703)
>>>36.22 MB/sec (1073741824 B / 00:28.270442)
>>>36.41 MB/sec (1073741824 B / 00:28.121841)
>>>On the secondary system the CPU system time's highest point was 7%,
>>>and the spotted maximum on the drbd_receiver thread was 9.7%
>>>So it seems the the JumboFrames only ease the task of the secondary
>>>      
>>>
>>node,
>>    
>>
>>>but do not improve performance.
>>>
>>>Test Setup:
>>>Linux 2.6.9
>>>DRBD 0.7.5
>>>Writes were this command: ./dm -a 0x00 -o /mnt/ha0/1Gfile -b1M -m -p
>>>      
>>>
>>-y -s1G
>>    
>>
>>>dm is from drbd-0.7.5.tar.gz in the benchmark directory.
>>>
>>>Conclusion:
>>>===========
>>>The Performance inhomogeneity within a single disk drive can be
>>>bigger (measusred 7.3%) than the loss of performance caused by DRBD
>>>mirroring (measured 3.4%).
>>>
>>>This only holds true it the limiting factor is the performance of
>>>the disk. In other words: Your network link and your CPU needs to 
>>>be strong enough.
>>>
>>>-phil
>>> 
>>>
>>>      
>>>
>>_______________________________________________
>>drbd-user mailing list
>>drbd-user at lists.linbit.com
>>http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>>    
>>
>
>_______________________________________________
>drbd-user mailing list
>drbd-user at lists.linbit.com
>http://lists.linbit.com/mailman/listinfo/drbd-user
>
>  
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20041123/13392a85/attachment.htm>


More information about the drbd-user mailing list