[DRBD-user] bonding more than two network cards still a bad idea?

J. Ryan Earl oss at jryanearl.us
Mon Oct 4 20:45:30 CEST 2010

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Mon, Oct 4, 2010 at 11:13 AM, Bart Coninckx <bart.coninckx at telenet.be>wrote:

> JR,
> thank you for this very elaborate and technically rich reply. I will
> certainly
> look into your suggestions about using Broadcom cards. I have one dual port
> Broadcom card in this server, but I was using one port combined with one
> port
> on an Intel e1000 dual port NIC in balanced-rr to provide for backup in the
> event a NIC goes down. Two port NICs usually share one chip for two ports,
> so
> in case of a problem with the chip, the complete DRBD would be out. Reality
> shows this might be a bad idea though: doing a bonnie++ test to the backend
> storage (RAID5 on 15K rpm disks) gives me a 255 MB/sec write performance,
> doing the same test on the DRBD device drops this to 77 MB/sec, even with
> the
> MTU set to 9000. It would be nice to get as close as possible to the
> theoretical maximum, so a lot needs to be done to get there.
> Step 1 would be changing everything to the broadcom NIC. Any other
> suggestions?

77MB/sec is low for a single GigE link if you backing store can do
250MB/sec.  I think you should test on your hardware with a single GigE--no
bonding--and work on getting close to the 110-120M/sec range before pursuing
bonding optimization.  Did you go through:
http://www.drbd.org/users-guide-emb/p-performance.html ?

I use the following network sysctl tuning:

# Tune TCP and network parameters
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
vm.min_free_kbytes = 65536
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 25000
net.ipv4.tcp_no_metrics_save = 1
sys.net.ipv4.route.flush = 1

This gives me up to 16MB TCP windows and considerable backlog to tolerate
latency with high-throughput.  It's tuned for 40gbit IPoIB, you could reduce
some of these numbers for slower connections...

Anyway, what NICs are you using?  Older interrupt-based NICs like the
e1000/e1000e (older Intel) and tg3 (older Broadcom) will not perform as well
as the newer RDMA-based hardware, but they should be well above the 77MB/sec
range.  Does your RAID controller have a power-backed write cache?  Have you
tried RAID10?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20101004/69c0ae80/attachment.htm>

More information about the drbd-user mailing list