Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi there, I find your comment about S-ATA RAID controllers highly interesting - I am struggling bigtime with getting anywhere near reasonable speeds using a NFS - DRDB - 3Ware RAID chain. What do you mean exactly with "slow", and would you be able to share any pointers on how to pinpoint why my server pair is so painfully slow? Best regards & TIA Jan Philipp Reisner wrote: >Hi, > >Again I had the chance to set up a DRBD Cluster for a LINBIT >Customer. It was the first time I a had one of these new SATA >devices really under my fingers [ withouh any of these enterprise >ready RAID controlers, which are in reality rather slow... ] > >The machines are some DELLs with a single P4 w 2.8 GHz and an >"Intel Corp. 6300ESB SATA Storage Controller" IDE Controller, >two "Intel Corp. 82547GI Gigabit Ethernet Controller" NICs >and 512 MB RAM, the disk calles itsef "ST380011AS" Seagate >Baracuda. > >At first the performance of the disk was miserable, in the >area of ~5 MB/sec, as it turned out the reason for this was >because we used the LINUX's common IDE (PATA). > >Then we tried the libata/ata_piix driver combination, and suddenly >we got a write performance in the area of 40MB/sec. > >BTW, with libata suddenly the disk appear as SCSI disk ! >[ -> changing all config files from "hdc" to "sda" ] > >Networksetup: >e1000 driver, the machines connected with a straight >cat5E cable, forced the cards into "speed 1000" with >ethtool, and set the MTU to 9000 aka Jumboframes. > >I am interested in raw data throughput, so I did sequential >writes on an ext3 Filesystem. > >Test1 >I wrote a 1GB File (with sync) to the root partition >[Cyl: 250 to 1466] 3 times: > >43.35 MB/sec (1073741824 B / 00:23.621594) >40.43 MB/sec (1073741824 B / 00:25.328009) >40.78 MB/sec (1073741824 B / 00:25.112768) >avg: 41.52 > >Test2 >The I did the same on a connected DRBD device (protocol C), >also ext3: [Cyl: 2747 to 6151] > >39.05 MB/sec (1073741824 B / 00:26.226047) >35.95 MB/sec (1073741824 B / 00:28.483070) >36.48 MB/sec (1073741824 B / 00:28.068153) >avg: 37.16 > >At first I was satisfied with the outcome that DRBD [with protocol C] >costs you about 10% of your throughput with sequential writes. > >Test3 >But the I did the same test with DRBD disconnect and got these numbers: >[Cyl: 2747 to 6151] >39.63 MB/sec (1073741824 B / 00:25.840004) >40.30 MB/sec (1073741824 B / 00:25.406312) >39.82 MB/sec (1073741824 B / 00:25.713998) >avg: 39.91 > >I aked myself: Why is it 4% below the first test ? > >Assumption: Maybe because the mirrored partition is behind the > root partition, and harddisk are slower on the outer > cylinders than on the inner cylinders. > >Test4: >So I unloaded the DRBD module and mounted the backing storage devices >on the mountpoints directly! [Cyl: 2747 to 6151] >39.65 MB/sec (1073741824 B / 00:25.823633) >38.54 MB/sec (1073741824 B / 00:26.570280) >37.26 MB/sec (1073741824 B / 00:27.479914) >avg: 38.48 > >Test3 was 3.5% faster than Test4. This could be explained by the fact >that DRBD sometimes triggers the immediate write of buffers to disk. > >The DRBD mirroring overhead, thus Test4 to Test2 is 3.4% which is smaller >then the performance differences within the disk device Test1 to Test4 >is 7.3% > >CPU Usage: > I monitored CPU Usage on the secondary system using the "top" utitily > and the hightes value for the drbd_receiver was 7.7%. > >Resync performance: > For the Customer I configured the syncer to run with 10MB/sec, this > makes sure that the Customer's application will continue to run > during a resync operation. For testing purpose I set the > resync rate to 40M and got a resync rate in the area of 33MByte/sec. > >Effect of JumboFrames / 9000 MTU > I repeated Test2 with an MTU of 1500 Byte and got these numbers: > 36.27 MB/sec (1073741824 B / 00:28.234703) > 36.22 MB/sec (1073741824 B / 00:28.270442) > 36.41 MB/sec (1073741824 B / 00:28.121841) > On the secondary system the CPU system time's highest point was 7%, > and the spotted maximum on the drbd_receiver thread was 9.7% > So it seems the the JumboFrames only ease the task of the secondary node, > but do not improve performance. > >Test Setup: > Linux 2.6.9 > DRBD 0.7.5 > Writes were this command: ./dm -a 0x00 -o /mnt/ha0/1Gfile -b1M -m -p -y -s1G > dm is from drbd-0.7.5.tar.gz in the benchmark directory. > >Conclusion: >=========== > The Performance inhomogeneity within a single disk drive can be > bigger (measusred 7.3%) than the loss of performance caused by DRBD > mirroring (measured 3.4%). > > This only holds true it the limiting factor is the performance of > the disk. In other words: Your network link and your CPU needs to > be strong enough. > >-phil > >