Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Lars,
Thanks for looking at this...
> you have a very interesting maximum and a huge deviation there.
Does this reflect a problem with my TCP stack? What might be causing
the huge max?
> but, lets use the 0.180 ms average rtt of 4k packets.
>
> averages from the dd commands below are
>
> drbd disconnected: 0.310 ms per 4k request
> drbd connected 1.170 ms per 4k request
> non-drbd 0.300 ms per 4k request
>
> I've also already seen non-drbd be slower than
> drbd-unconnected on the same hardware,
> there are funny effects in play.
> but they are close within 3%, this is expected.
Hmmm, it looks to me like non-drbd is **faster** than
drbd-disconnected from my numbers, which I would expect. Am I
mis-reading?
> however your drbd-connected seems bad.
> from ping rtt and non-drbd numbers we'd expect that
> latency of drbd connected should be ~ 0.480 ms.
> your measurement indicates it is worse than this
> expectation by a factor of 2.5.
Yes, this is the crux of the problem I am experiencing - now confirmed
with MySQL out of the equation.
> in all setups I have tuned so far,
> the actual (measured) latency of drbd,
> and the rough estimate given by said ping and dd commands
> are very close.
>
> so I suspect your secondaries ("node2") io subsystem is slower.
> please verify.
The 2 nodes are identical - right down to the io subsystem (identical
RAID-10 hardware with battery-backed write-back cache enabled and
identical model hard drives).
> other than that, pinning of drbd related threads to one CPU,
> preferably the same where you pinned the NIC driver irq to,
> could help to reduce latency.
I have not pinned NIC driver IRQs. (I don't know how.) I have pinned
the DRBD-related threads to a single CPU core and the test results
reflect that configuration.
I'm really at a loss here. Do you have any other suggestions for
getting to the bottom of this?
Should I disable irqbalance daemon? (I tried it and it seemed to make
no difference).
Should I disable SELinux?
Thanks,
Sam