Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello,
I'm just setting up a new DRBD cluster pair on this hardware (x2):
Dual Quad Opteron 2378, 16GB RAM, 2x Intel SSD (32GB) on NV SATA,
8x ST31500341AS (1.5TB SATA) on onboard LSI1068E, 2x Dualport 82575EB
(Intel-GigE) cards.
The SSDs hold the OS and DRBD meta data (flexible disk, 1GB), the 8
Seagates are a MD RAID5. The 4 GigE ports are in bonding mode 0 and thus
have a bandwidth of 3.5Gb/s according to iperf.
The standalone MD RAID5 gives me 200MB/s writes and 620MB/s reads.
Using DRBD 8.3.2 on a standard 2.6.27.31 kernel.
The relevant entries from drbd.conf are:
---
common {
syncer {
rate 200M;
al-extents 3833;
}
disk {
no-disk-barrier;
no-disk-flushes;
no-md-flushes;
use-bmbv;
}
net {
sndbuf-size 0;
rcvbuf-size 0;
max-buffers 16384;
unplug-watermark 16384;
max-epoch-size 16384;
}
---
Note the 3833 for extents, the website/manpage say 3843 but DRBD insisted
on 10 less. ^_-
The initial sync hovered around 58MB/s while I had rcvbuf-size unspecified
in the config and thus at the default. I tried various variations of
sndbuf-size including 0 (auto-tune), but they had at best minor effects.
After changing both of them to 0, speeds went up 113MB/s (both sync and
later filesystem tests). I guess this should be a major hint to people
using modern kernels, the defaults and manual tuning made things actually
a lot worse.
I'm still missing about 80MB/s performance wise and wonder where they
could be hiding. And yes, I did change the scheduler to deadline, though
I expect CFQ on this kernel to be doing well enough, too.
TCP_Cork on or off had no measurable effects either, btw.
Will experiment more, but after the surprise speed doubling by just
letting the kernel handle the net buffer section I'm wondering what else
is there to poke that could yield a similar improvement.
Regards,
Christian
--
Christian Balzer Network/Systems Engineer
chibi at gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/