[DRBD-user] Some performance hints/questions

Christian Balzer chibi at gol.com
Tue Sep 8 06:21:50 CEST 2009

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hello,

I'm just setting up a new DRBD cluster pair on this hardware (x2):
Dual Quad Opteron 2378, 16GB RAM, 2x Intel SSD (32GB) on NV SATA, 
8x ST31500341AS (1.5TB SATA) on onboard LSI1068E, 2x Dualport 82575EB
(Intel-GigE) cards.
The SSDs hold the OS and DRBD meta data (flexible disk, 1GB), the 8
Seagates are a MD RAID5. The 4 GigE ports are in bonding mode 0 and thus
have a bandwidth of 3.5Gb/s according to iperf.
The standalone MD RAID5 gives me 200MB/s writes and 620MB/s reads.

Using DRBD 8.3.2 on a standard 2.6.27.31 kernel.
The relevant entries from drbd.conf are:
---
common {
  syncer {
    rate 200M;
    al-extents 3833;
  }
  disk {
    no-disk-barrier;
    no-disk-flushes;
    no-md-flushes;
    use-bmbv;
  }
  net {
    sndbuf-size 0;
    rcvbuf-size 0;
    max-buffers     16384;
    unplug-watermark   16384;
    max-epoch-size  16384;
}
---

Note the 3833 for extents, the website/manpage say 3843 but DRBD insisted 
on 10 less. ^_-
The initial sync hovered around 58MB/s while I had rcvbuf-size unspecified 
in the config and thus at the default. I tried various variations of 
sndbuf-size including 0 (auto-tune), but they had at best minor effects. 

After changing both of them to 0, speeds went up 113MB/s (both sync and 
later filesystem tests). I guess this should be a major hint to people 
using modern kernels, the defaults and manual tuning made things actually
a lot worse.

I'm still missing about 80MB/s performance wise and wonder where they 
could be hiding. And yes, I did change the scheduler to deadline, though
I expect CFQ on this kernel to be doing well enough, too.
TCP_Cork on or off had no measurable effects either, btw.

Will experiment more, but after the surprise speed doubling by just
letting the kernel handle the net buffer section I'm wondering what else
is there to poke that could yield a similar improvement.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/



More information about the drbd-user mailing list