[DRBD-user] Speeding up sync rate on fast links and storage

Lars Ellenberg lars.ellenberg at linbit.com
Fri Dec 19 19:39:20 CET 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Fri, Dec 19, 2008 at 09:24:32AM -0500, Gennadiy Nerubayev wrote:
> On Thu, Dec 18, 2008 at 1:50 PM, Lars Ellenberg <lars.ellenberg at linbit.com>
> wrote:
> 
> 
>     try and you'll find out. if you allow more CPUs,
>     try the drbd cpu-mask setting again.
> 
>     just for the record, iirc, I have benchmarked Connected DRBD streaming
>     writes at ~600 MByte/sec in our lab with 10GbE as well as Dolphin
>     Interconnect.  so the drbd code should be able to handle this.
> 
> 
> Hmm. Just moving the interrupts didn't help, so I upgraded the kernel to the
> current on kernel.org - prior to this I was using the stock RHEL 5.2 kernel.
> The interesting thing after the upgrade is that the interrupts were now spread
> out evenly between both cores, and amongst the tweaks that you've mentioned,
> sndbuf-size 0 now had a positive effect, improving the performance to
> ~380-390MB/s average on both IPoIB and 10Gbe. The odd thing is that I see the
> sync rate spike sometimes (the first number) upwards to 700MB/s, so it's
> fluctuating a lot for whatever reason, and only averaging the aforementioned
> 380-390.
> 
> What sort of kernel/cpu/memory configuration allowed you to get that 600MB/s
> number?

uh. oh.
I have to admit that this was probably not really realistical.
sort of only writing 500MB (odirect, "synchronously"), as that
was what fit into the controller cache...
and it finished subsecond. that's where the number comes from
 ;)
don't have a real storage backend in the lab capable of sustained
writes in that performance range. (yet.)

but, I think nothing special, actually,
it was jumbo frames, disabling flow control, and huge max-buffers
and the like, that did the trick, mostly, as well as allowing more than
one core (as one single cpu was maxed out sometimes).

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed



More information about the drbd-user mailing list