Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, Dec 19, 2008 at 09:24:32AM -0500, Gennadiy Nerubayev wrote: > On Thu, Dec 18, 2008 at 1:50 PM, Lars Ellenberg <lars.ellenberg at linbit.com> > wrote: > > > try and you'll find out. if you allow more CPUs, > try the drbd cpu-mask setting again. > > just for the record, iirc, I have benchmarked Connected DRBD streaming > writes at ~600 MByte/sec in our lab with 10GbE as well as Dolphin > Interconnect. so the drbd code should be able to handle this. > > > Hmm. Just moving the interrupts didn't help, so I upgraded the kernel to the > current on kernel.org - prior to this I was using the stock RHEL 5.2 kernel. > The interesting thing after the upgrade is that the interrupts were now spread > out evenly between both cores, and amongst the tweaks that you've mentioned, > sndbuf-size 0 now had a positive effect, improving the performance to > ~380-390MB/s average on both IPoIB and 10Gbe. The odd thing is that I see the > sync rate spike sometimes (the first number) upwards to 700MB/s, so it's > fluctuating a lot for whatever reason, and only averaging the aforementioned > 380-390. > > What sort of kernel/cpu/memory configuration allowed you to get that 600MB/s > number? uh. oh. I have to admit that this was probably not really realistical. sort of only writing 500MB (odirect, "synchronously"), as that was what fit into the controller cache... and it finished subsecond. that's where the number comes from ;) don't have a real storage backend in the lab capable of sustained writes in that performance range. (yet.) but, I think nothing special, actually, it was jumbo frames, disabling flow control, and huge max-buffers and the like, that did the trick, mostly, as well as allowing more than one core (as one single cpu was maxed out sometimes). -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed