Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Wed, Aug 04, 2010 at 10:34:45PM +0200, Sebastian Hetze wrote: > On Wed, Aug 04, 2010 at 08:00:01PM +0200, Lars Ellenberg wrote: > > > With protocol B this can lead to an situation where the secondary node > > > becomes completely unusable. It looks like the secondary sends all IO > > > requests to the LVM layer and LVM can not manage the queue after a > > > certain point. > > > > Too bad. > > So you say this is a problem of LVM and nothing DRBD can do about, > right? Not necessarily. But I won't spend much time on the mailing list to find a workaround in DRBD, if we already have several (disabling barriers, or according to you, using a different protocol). > > > I would expect DRBD to use the flush method on LVM containers as > > > default. At least if protocol B is used. > > > > With kernels >= 2.6.24, a "flush" is implemented as "empty barrier", > > so if there is no barrier support, there will be no flush support > > either (except for maybe very few special cases). > > Well, since flush works with LVM, this might be one of these cases. No. LVM supports barriers, thus it supports flushes as well (as long as the physical disks can do them, too). > > > In my test setup with 10 DRBD resources the logger loop takes arround > > > 50 seconds to finish on the primary. While the primary is working with > > > load below 1, the secondary load raises up to 10 and stays there for a > > > couple of minutes. With only 10 resources the secondary recovers after > > > a while. > > > If you try the same simple test with 30 or more DRBD resources the > > > secondary will get a load of 40 and wont recover, at least not within > > > an hour. > > > > ;-) > > > > If they are hurting you, disable barriers, then. > > We will do that. > > What I still dont understand is why only the secondary has this trouble. > I thought DRBD would use the same disk settings for the primary, that > would mean the primary DRBD would use barriers and the underlying LVM > would get into the same troublesome situation. But this is not the case. > What is the difference here? The drbd inserted barriers are there to guarantee that write ordering on the secondary will not violate any write-after-write dependencies that we have on the primary. On the primary, DRBD does only insert barriers on meta-data updates. User (i.e. filesystem) supplied barriers are not supported yet, anyways. If we analyse many write-after-write dependencies, we have to insert many barriers. (or flushes; or drains). Your example workload does many tiny writes and fsyncs, causing many such dependencies, inserting many such barriers. BTW, double check your io scheduler behaviour. It may well be that some "idle wait" of your io scheduler is hurting you more than the barriers themselves. Try deadline and noop. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed