[DRBD-user] Slow Lustre performance over DRBD

[Somsak Sriprayoonsakul]

Dear List,

   I am setting up a 4 nodes Lustre cluster. The Cluster consist of

1. 2 nodes with a shared external storage of about 800GB, this one did't 
use DRBD. These nodes served as Lustre MDS in active passive mode.

2. 2 Sun X4500 nodes, which contains 48 disks of 750GB. These nodes 
served as Lustre OSSs. 2 boot disks was combined using RAID1 for OS 
installation. For the rest of 46 disks, I divided it into 6 group of 
RAID10 which contains 8 hdds per group (one group use 2 vacant 
partitions available after making RAID1 of OS drive). All the RAID were 
configured using software raid (this thumper does not support hardware 
raid).

   Note that, both X4500 nodes were using Infiniband SDR (10Gbps) 
connecting to each client. Each client has DDR (20Gbps) infiniband 
installed.

   I conducted a test by creating Lustre over the that 6 RAID10 (3 from 
a node, another 3 from another node) without DRBD, and run iozone in 
parallel mode (-t 8 -+m) of 8 clients over Infiniband yield the total 
performance of about 1.3 - 1.4GB/s. I monitored the raid device by 
"iostat -d 2 -k), each RAID10 could deliver about 200+MB/s per RAID.

   Then I switch over to DRBD, one device per RAID group mirroring to 
neighbor node. Each node served 3 primary DRBD exported as Lustre OSSs. 
Then I conduct the same test again, but this time the performance is 
down to only about 350MB/s maximum. I did iostat on each machine and 
each RAID only deliver about 50+MB/s. This was the maximum I got from 
tweaking many parameters in drbd.conf. Attached with this e-mail is the 
current configuration used in this set-up.
     I think I already tweak and adjust all parameters I could already. 
I am aware that the performance over DRBD will be slower, but I think it 
should be at least about 600-700MB/s. I also test switching over to 
Gigabit Ethernet for DRBD but the performance is much worse, as expected.

   Could anyone suggest the performance tuning for my set-up?


Rgds,



-- 

----------------------------------------------------------------------------------- 

Somsak Sriprayoonsakul
INOX (Innovative Extremist)
Operation manager

somsak at inox.co.th
----------------------------------------------------------------------------------- 



------------------------------------------------------------------------

#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd82/drbd.conf
#
global { 
  usage-count no; 
}

common {
  protocol A;
  syncer {
    rate 450M;
    al-extents 3389;
  }
  startup {
    wfc-timeout 300;
    degr-wfc-timeout 300;
  }
  net {
    #max-buffers 8192;
    #max-epoch-size  8192;
    sndbuf-size 716800;
    after-sb-0pri discard-younger-primary;
    after-sb-1pri consensus;
  }
  disk {
    no-disk-flushes;
    no-md-flushes;
  }
}

resource ost0 {
  device    /dev/drbd0;
  disk      /dev/md2;
  on storage-0-2.local {
    address   192.168.0.1:7789;
    meta-disk internal;
  }
  on storage-0-3.local {
    address   192.168.0.2:7789;
    meta-disk internal;
  }
}

resource ost1 {
  device    /dev/drbd1;
  disk      /dev/md3;
  on storage-0-2.local {
    address   192.168.0.1:7790;
    meta-disk internal;
  }
  on storage-0-3.local {
    address   192.168.0.2:7790;
    meta-disk internal;
  }
}

resource ost2 {
  device    /dev/drbd2;
  disk      /dev/md4;
  on storage-0-2.local {
    address   192.168.0.1:7791;
    meta-disk internal;
  }
  on storage-0-3.local {
    address   192.168.0.2:7791;
    meta-disk internal;
  }
}

resource ost3 {
  device    /dev/drbd3;
  disk      /dev/md5;
  on storage-0-2.local {
    address   192.168.0.1:7792;
    meta-disk internal;
  }
  on storage-0-3.local {
    address   192.168.0.2:7792;
    meta-disk internal;
  }
}

resource ost4 {
  device    /dev/drbd4;
  disk      /dev/md6;
  on storage-0-2.local {
    address   192.168.0.1:7793;
    meta-disk internal;
  }
  on storage-0-3.local {
    address   192.168.0.2:7793;
    meta-disk internal;
  }
}

resource ost5 {
  device    /dev/drbd5;
  disk      /dev/md7;
  on storage-0-2.local {
    address   192.168.0.1:7794;
    meta-disk internal;
  }
  on storage-0-3.local {
    address   192.168.0.2:7794;
    meta-disk internal;
  }
}