[DRBD-user] drbd performance with GbE in connected mode

Ross S. W. Walker rwalker at medallion.com
Mon Jan 15 20:37:51 CET 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


> -----Original Message-----
> From: drbd-user-bounces at lists.linbit.com 
> [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Ralf Gross
> Sent: Monday, January 15, 2007 12:56 PM
> To: drbd-user at lists.linbit.com
> Subject: Re: [DRBD-user] drbd performance with GbE in connected mode
> 
> Ross S. W. Walker schrieb:
> > > Both Raid's are able to write with >120MB/s in disconnected mode.
> > > I'm just wondering because I have seen people on the list that get
> > > ~95MB/s with a similar setup. 
> > 
> > There's lies, damn lies, then benchmarks. Pump a big enough 
> block size
> > to the storage till you get the throughput you like. Some 
> people throw
> > 1MB block size and get 115MB/s throughput.
> > 
> > I always like to use the block sizes my application uses. 
> If it's ext2
> > then 4K is a good benchmark, but different apps use different block
> > sizes.
> > 
> > Always ask the block size and io pattern for the benchmarks 
> people throw
> > around.
> 
> It seems that 
> 
> a. changing the net parameters did help 
> 
>  sndbuf-size       240k;
>  max-buffers      20480;
>  max-epoch-size   16384;
>  unplug-watermark 20480;
> 
> b. changing the bonding mode of the interfaces from balance-rr to
>    balance-xor did help too.
> 
> I now get about 85MB/s. Maybe that's by accident, but I could watch
> the write performance go down when increasing the sndbuf-size.

Well I can see increasing sndbuf as increasing latency, so it makes
sense that decreasing it would also decrease latency a bit, why not try
128K and see where that puts you. If you have direct connections and a
fast network, there is no real need to have a large sndbuf. If I was
using Prot A and a slow network of T1s then I would use a very large
sndbuf.

Statistically speaking though when doing a benchmark over a short period
of time 82-83-85 MB/s are about the same. I find that a 15 minute run
will normally get rid of the 3-5 MB/s swings between runs and narrow it
down to 1-2 MB/s swings.

It looks like you are approaching the part of tuning were you are
receiving diminishing returns and will need to do more and more tuning
to squeeze less and less out, so I would say that 85 MB/s is what your
gonna see unless you can find a way to run drbd with multiple paths,
which I don't think it has the capability to do.

Well let me know if you can squeeze any more out of it. You might want
to see if there is any filesystem optimizations you can do now to get
some extra performance out of it.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.




More information about the drbd-user mailing list