On Thu, Dec 18, 2008 at 1:50 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<br>
</div>try and you'll find out. if you allow more CPUs,<br>
try the drbd cpu-mask setting again.<br>
<br>
just for the record, iirc, I have benchmarked Connected DRBD streaming<br>
writes at ~600 MByte/sec in our lab with 10GbE as well as Dolphin<br>
Interconnect. so the drbd code should be able to handle this.<br>
</blockquote><div><br>Hmm. Just moving the interrupts didn't help, so I upgraded the kernel to the current on <a href="http://kernel.org">kernel.org</a> - prior to this I was using the stock RHEL 5.2 kernel. The interesting thing after the upgrade is that the interrupts were now spread out evenly between both cores, and amongst the tweaks that you've mentioned, sndbuf-size 0 now had a positive effect, improving the performance to ~380-390MB/s average on both IPoIB and 10Gbe. The odd thing is that I see the sync rate spike sometimes (the first number) upwards to 700MB/s, so it's fluctuating a lot for whatever reason, and only averaging the aforementioned 380-390.<br>
<br>What sort of kernel/cpu/memory configuration allowed you to get that 600MB/s number?<br><br>Thanks,<br><br>-Gennadiy<br></div></div>