[DRBD-user] Slow Lustre performance over DRBD
somsak_sr at thaigrid.or.th
Wed Jan 28 18:21:12 CET 2009
More ifnromation, I just try iperf and the performance is quite low
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 2.01 GBytes 1.73 Gbits/sec
I use pre-compiled version of Iperf from rpmforge.
Oops!, did I mentioned that I'm using CentOS4 on x4500 machines, and
those MDSs machine are CentOS5? I have to use different OS because poor
driver support of X4500 on RHEL5.
About IPOIB connected mode, it seems that the infiniband is already
running in connected mode.
[root at storage-0-3 ~]# cat /sys/class/net/ib1/mode
I will try to use OFED1.4 instead, to see if it improve the performance.
Do I need to upgrade firmware to maximize the performance? Right now it is
[root at storage-0-3 ~]# cat /sys/class/infiniband/mthca0/fw_ver
Sorry if the question is becoming irrelevant to the list.
Somsak Sriprayoonsakul wrote:
> Thanks all for sharing.
> About the webminar, I have look through that but maybe I miss
> something. I have already adjust sndbuf, al-extend, no*flush,
> max-buffers, unplug-watermark. The only parameters that really improve
> performance are sndbuf-size and no*flush.
> I admit that I'm not Infiniband expert. But the performance of pure
> Lustre over IB is really amazing without tweaking anything, so I
> thought IB should doing ok already. I will double check that again.
> I use the infiniband tool came with RHEL4, but the kernel-ib package
> was taken from Lustre. rpm -qi kernel-ib returns
> Name : kernel-ib Relocations: (not relocatable)
> Version : 1.3 Vendor: OpenFabrics
> Release : 2.6.9_67.0.22.EL_lustre.1.6.6smp Build Date: Fri 12
> Sep 2008 06:08:31 AM ICT
> So I guess I'm using OFED 1.3.
> Currently my MTU is 65520, do I need to increase it?
> For the performance, I got
> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]
> 65536 5000 623.60 623.58
> from ib_write_bw, I think it used to reach 700 at my first test, might
> because few things running over it (drbd, lustre) while testing.
> Robert Dunkley wrote:
>> Hi Somsak,
>> I use some DRBD systems running over Infiniband/IPOIB, try setting a
>> larger MTU and switch IPOIB to connected mode. What sort of raw
>> performance results do you get over IPOIB? (My own 20Gb setup shows
>> only about 700-1000MByte/sec with IPerf). What version of OFED are
>> you running?
>> -----Original Message-----
>> From: drbd-user-bounces at lists.linbit.com
>> [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Florian Haas
>> Sent: 28 January 2009 08:27
>> To: drbd-user at lists.linbit.com
>> Subject: Re: [DRBD-user] Slow Lustre performance over DRBD
>> can you please take a look at our performance tuning webinar
>> run the micro-benchmarks described there, and share your results?
>> Also, it would be helpful if you could provide network throughput test
>> results for your IPoIB connection you are using for DRBD replication.
>> Why are you using protocol A? Can you afford to lose updates on node
>> On 01/28/2009 07:14 AM, Somsak Sriprayoonsakul wrote:
>>> Dear List,
>>> I am setting up a 4 nodes Lustre cluster. The Cluster consist of
>>> 1. 2 nodes with a shared external storage of about 800GB, this one
>>> use DRBD. These nodes served as Lustre MDS in active passive mode.
>>> 2. 2 Sun X4500 nodes, which contains 48 disks of 750GB. These nodes
>>> served as Lustre OSSs. 2 boot disks was combined using RAID1 for OS
>>> installation. For the rest of 46 disks, I divided it into 6 group of
>>> RAID10 which contains 8 hdds per group (one group use 2 vacant
>>> partitions available after making RAID1 of OS drive). All the RAID were
>>> configured using software raid (this thumper does not support hardware
>>> Note that, both X4500 nodes were using Infiniband SDR (10Gbps)
>>> connecting to each client. Each client has DDR (20Gbps) infiniband
>>> I conducted a test by creating Lustre over the that 6 RAID10 (3
>>> from a
>>> node, another 3 from another node) without DRBD, and run iozone in
>>> parallel mode (-t 8 -+m) of 8 clients over Infiniband yield the total
>>> performance of about 1.3 - 1.4GB/s. I monitored the raid device by
>>> "iostat -d 2 -k), each RAID10 could deliver about 200+MB/s per RAID.
>>> Then I switch over to DRBD, one device per RAID group mirroring to
>>> neighbor node. Each node served 3 primary DRBD exported as Lustre OSSs.
>>> Then I conduct the same test again, but this time the performance is
>>> down to only about 350MB/s maximum. I did iostat on each machine and
>>> each RAID only deliver about 50+MB/s. This was the maximum I got from
>>> tweaking many parameters in drbd.conf. Attached with this e-mail is the
>>> current configuration used in this set-up.
>>> I think I already tweak and adjust all parameters I could
>>> already. I
>>> am aware that the performance over DRBD will be slower, but I think it
>>> should be at least about 600-700MB/s. I also test switching over to
>>> Gigabit Ethernet for DRBD but the performance is much worse, as
>>> Could anyone suggest the performance tuning for my set-up?
>> drbd-user mailing list
>> drbd-user at lists.linbit.com
> drbd-user mailing list
> drbd-user at lists.linbit.com
More information about the drbd-user