<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<font size="-1">Hi,<br>
<br>
I never messed with this options and don't have that bad results.<br>
<br>
You have so low write performance withr 16 spindles. You should
try to change your controller, I have better write performance
with half of the spindles. Try areca controller, that's what we
use.<br>
<br>
Regarding to Infiniband, in the next cluster setup, I will mess up
with this options and share my results.<br>
<br>
Thanks,<br>
</font><br>
On 29-07-2011 12:50, Cédric Dufour - Idiap Research Institute wrote:
<blockquote cite="mid:4E329E76.3050003@idiap.ch" type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
Hello DRBD users,<br>
<br>
I wanted to share my experience with DRBD being used over
(Infiniband) SDP, for I obtained results that may be interesting
to some of you.<br>
<br>
I first started very frustated by the fact that while SDP
(re-)synching performances were better than TCP's (~5%), general
write operations (e.g. bonnie++ or dd if=/dev/zero ...) were
horrible - ~50MB/s (SDP) vs. ~170MB/s (TCP) - the relevant DRBD
configuration being:<br>
<br>
net {<br>
max-buffers 8192;<br>
max-epoch-size 8192;<br>
sndbuf-size 1024k;<br>
}<br>
<br>
After much playing around, I found that two parameters affect
write performances drastically:<br>
<br>
1. DRBD's 'sdnbuf-size' parameter<br>
While 'sdnbuf-size' does not affect TCP performances when set
above a value of 1024k, it does so sensibly for SDP connections.<br>
The more you increase the 'sndbuf-size' value, the more SDP
performances increase: write (cf. dd if=/dev/zero ...) jumping
from ~50MB/s (sndbuf-size=1024k) to ~120MB/s (sndbuf-size=10240k;
the maximum allowed).<br>
Fortunately, increasing this parameter does not affect small
writes performances (cf. bonnie++ -n ... ), neither when using TCP
or SDP.<br>
<br>
2. SDP module's (ib_sdp) 'recv_poll' parameter<br>
The 'recv_poll' parameter (default: 700usec) also affects SDP
performances sensibly.<br>
I lowered it (step-by-step) down to 100usec and finally got
~230MB/s raw write performances (cf. dd if=/dev/zero ...) and
bonnie++ write performances that are 10-15% above TCP
performances.<br>
Fortunately again, lowering this parameter does not affect small
writes performances (cf. bonnie++ -n ... ) nor general TCP-via-SDP
performances (iperf being stable at 9.1GBit/s via SDP vs.
3.2GBit/s via TCP).<br>
Also, CPU usage remains the same.<br>
Finally, nothing amiss shows up in the kernel log.<br>
<br>
In the end, I obtained performances that are consistent with what
the backing RAID array allows me to expect.<br>
<br>
I hope this may be of interest to some of you.<br>
If someone knowledgeable of DRBD and/or Infiniband modules
parameters were to read this post, I would happily have a feedback
about why those parameters affect DRBD (write) performances (but
not (re-)synch's).<br>
Eventually, maybe those results are only valid for the particular
hardware and/or software versions I have been playing with.<br>
<br>
Speaking of...<br>
<br>
Software version:<br>
- Debian Squeeze 6.0.2 64-bit (kernel 2.6.32)<br>
- Custom-built OFED 1.5.3.2 kernel modules (including patch
related to <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2240">http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2240</a>)<br>
- Custom-built DRBD 8.3.11 kernel module (along with re-packaged
DRBD 8.3.11 user-space utilities)<br>
<br>
Hardware:<br>
- Motherboard: Supermicro X7DWN+<br>
- CPU: Intel Xeon L5410 (2x) [HT disabled, VT enabled]<br>
- RAM: 32GB DDR2-667 (16x2GB)<br>
- NIC: Mellanox MT25204 [InfiniHost III Lx HCA] (ib_rdma_bw
~1.5GB/s; ib_rdma_lat ~2usec)<br>
- HD/RAID: ICP 5165BR with 16x1TB HDs (RAID-6 - LVM2 - DRBD)<br>
<br>
Cheers<br>
<br>
<div class="moz-signature">-- <br>
<title>Cédric Dufour @ Idiap Research Institute</title>
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<style type="text/css">
DIV.signature {FONT:normal 11px sans-serif;COLOR:#000000;}
DIV.signature P {MARGIN:5px 0px;}
DIV.signature A {TEXT-DECORATION:none;COLOR:#000066;}
DIV.signature A:hover {TEXT-DECORATION:underline;COLOR:#000066;}
DIV.signature .name {FONT:bold 13px sans-serif;COLOR:#000066;}
DIV.signature .citation {FONT:italic normal 10px sans-serif;COLOR:#000000;}
DIV.signature .details {FONT:normal 12px monospace;COLOR:#000000;}
DIV.signature .label {FONT:bold 12px monospace;COLOR:#660000;}
</style>
<div class="signature">
<p class="name">Cédric Dufour @ Idiap Research Institute<br>
<span class="citation">"No complaint is enough praise"</span></p>
</div>
</div>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
drbd-user mailing list
<a class="moz-txt-link-abbreviated" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a>
<a class="moz-txt-link-freetext" href="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Igor Neves <a class="moz-txt-link-rfc2396E" href="mailto:igor.neves@3gnt.net"><igor.neves@3gnt.net></a>
3GNTW - Tecnologias de Informação, Lda
SIP: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
MSN: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
JID: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
PSTN: 00351 252377120
</pre>
</body>
</html>