<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#ffffff" text="#000000">
Hello DRBD users,<br>
<br>
I wanted to share my experience with DRBD being used over
(Infiniband) SDP, for I obtained results that may be interesting to
some of you.<br>
<br>
I first started very frustated by the fact that while SDP
(re-)synching performances were better than TCP's (~5%), general
write operations (e.g. bonnie++ or dd if=/dev/zero ...) were
horrible - ~50MB/s (SDP) vs. ~170MB/s (TCP) - the relevant DRBD
configuration being:<br>
<br>
net {<br>
max-buffers 8192;<br>
max-epoch-size 8192;<br>
sndbuf-size 1024k;<br>
}<br>
<br>
After much playing around, I found that two parameters affect write
performances drastically:<br>
<br>
1. DRBD's 'sdnbuf-size' parameter<br>
While 'sdnbuf-size' does not affect TCP performances when set above
a value of 1024k, it does so sensibly for SDP connections.<br>
The more you increase the 'sndbuf-size' value, the more SDP
performances increase: write (cf. dd if=/dev/zero ...) jumping from
~50MB/s (sndbuf-size=1024k) to ~120MB/s (sndbuf-size=10240k; the
maximum allowed).<br>
Fortunately, increasing this parameter does not affect small writes
performances (cf. bonnie++ -n ... ), neither when using TCP or SDP.<br>
<br>
2. SDP module's (ib_sdp) 'recv_poll' parameter<br>
The 'recv_poll' parameter (default: 700usec)
also affects SDP performances sensibly.<br>
I lowered it (step-by-step) down to 100usec and finally got ~230MB/s
raw write performances (cf. dd if=/dev/zero ...) and bonnie++ write
performances that are 10-15% above TCP performances.<br>
Fortunately again, lowering this parameter does not affect small
writes performances (cf. bonnie++ -n ... ) nor general TCP-via-SDP
performances (iperf being stable at 9.1GBit/s via SDP vs. 3.2GBit/s
via TCP).<br>
Also, CPU usage remains the same.<br>
Finally, nothing amiss shows up in the kernel log.<br>
<br>
In the end, I obtained performances that are consistent with what
the backing RAID array allows me to expect.<br>
<br>
I hope this may be of interest to some of you.<br>
If someone knowledgeable of DRBD and/or Infiniband modules
parameters were to read this post, I would happily have a feedback
about why those parameters affect DRBD (write) performances (but not
(re-)synch's).<br>
Eventually, maybe those results are only valid for the particular
hardware and/or software versions I have been playing with.<br>
<br>
Speaking of...<br>
<br>
Software version:<br>
- Debian Squeeze 6.0.2 64-bit (kernel 2.6.32)<br>
- Custom-built OFED 1.5.3.2 kernel modules (including patch related
to <a class="moz-txt-link-freetext" href="http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2240">http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2240</a>)<br>
- Custom-built DRBD 8.3.11 kernel module (along with re-packaged
DRBD 8.3.11 user-space utilities)<br>
<br>
Hardware:<br>
- Motherboard: Supermicro X7DWN+<br>
- CPU: Intel Xeon L5410 (2x) [HT disabled, VT enabled]<br>
- RAM: 32GB DDR2-667 (16x2GB)<br>
- NIC: Mellanox MT25204 [InfiniHost III Lx HCA] (ib_rdma_bw
~1.5GB/s; ib_rdma_lat ~2usec)<br>
- HD/RAID: ICP 5165BR with 16x1TB HDs (RAID-6 - LVM2 - DRBD)<br>
<br>
Cheers<br>
<br>
<div class="moz-signature">-- <br>
<title>Cédric Dufour @ Idiap Research Institute</title>
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<style type="text/css">
DIV.signature {FONT:normal 11px sans-serif;COLOR:#000000;}
DIV.signature P {MARGIN:5px 0px;}
DIV.signature A {TEXT-DECORATION:none;COLOR:#000066;}
DIV.signature A:hover {TEXT-DECORATION:underline;COLOR:#000066;}
DIV.signature .name {FONT:bold 13px sans-serif;COLOR:#000066;}
DIV.signature .citation {FONT:italic normal 10px sans-serif;COLOR:#000000;}
DIV.signature .details {FONT:normal 12px monospace;COLOR:#000000;}
DIV.signature .label {FONT:bold 12px monospace;COLOR:#660000;}
</style>
<div class="signature">
<p class="name">Cédric Dufour @ Idiap Research Institute<br>
<span class="citation">"No complaint is enough praise"</span></p>
</div>
</div>
</body>
</html>