Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Wed, Dec 19, 2007 at 09:57:22AM -0800, Art Age Software wrote:
> Lars,
>
> Thanks for looking at this...
>
> > you have a very interesting maximum and a huge deviation there.
>
> Does this reflect a problem with my TCP stack? What might be causing
> the huge max?
sorry, my crystall ball is in cleansing atm.
> > but, lets use the 0.180 ms average rtt of 4k packets.
> >
> > averages from the dd commands below are
> >
> > drbd disconnected: 0.310 ms per 4k request
> > drbd connected 1.170 ms per 4k request
> > non-drbd 0.300 ms per 4k request
> >
> > I've also already seen non-drbd be slower than
> > drbd-unconnected on the same hardware,
> > there are funny effects in play.
> > but they are close within 3%, this is expected.
>
> Hmmm, it looks to me like non-drbd is **faster** than
> drbd-disconnected from my numbers, which I would expect. Am I
> mis-reading?
what I meant is that they are close, and that I have seen on various
setup either the exact same values, or one or the other showing less
latency. yes, obviously in _your_ setup, non-drbd apears to have less
latency, as is naively expected anyways. but I wanted to point out that
there sometimes you also see the counter-intuitive thing, namely
disconnected drbd showing less latency than the underlying device by
itself. which is where said "funny effects" come into play.
> > however your drbd-connected seems bad.
> > from ping rtt and non-drbd numbers we'd expect that
> > latency of drbd connected should be ~ 0.480 ms.
> > your measurement indicates it is worse than this
> > expectation by a factor of 2.5.
>
> Yes, this is the crux of the problem I am experiencing - now confirmed
> with MySQL out of the equation.
>
> > in all setups I have tuned so far,
> > the actual (measured) latency of drbd,
> > and the rough estimate given by said ping and dd commands
> > are very close.
> >
> > so I suspect your secondaries ("node2") io subsystem is slower.
> > please verify.
>
> The 2 nodes are identical - right down to the io subsystem (identical
> RAID-10 hardware with battery-backed write-back cache enabled and
> identical model hard drives).
did you _measure_ that.
or do you just "know" that.
because we already had the case that identical hardware showed a factor
100 different latency values. specifically, we just recently had two
DELL MD3000, one of which was showing 0.7 ms, the other one had
70.x ms. and yes, the additional latency was attributed to drbd at
first, as well. only that on the same boxes, we had also other storage,
and there it behaved nicely. we have not been able to track it down to
the cause, but believe that resetting the storage made the effect
"go away".
so do measure.
> Should I disable SELinux?
that is an interessting question.
I have no idea in how far this could affect latency.
--
: Lars Ellenberg Tel +43-1-8178292-55 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Vivenotgasse 48, A-1120 Vienna/Europe http://www.linbit.com :