[DRBD-user] Problems after upgrade 8.2.0 to 8.3.0

Matteo Tescione matteo at rmnet.it
Mon Feb 9 19:18:46 CET 2009

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

HI Lars and folks,

Il 09/02/09 13:42, "Lars Ellenberg" <[IND]> ha scritto:

> On Sat, Feb 07, 2009 at 04:55:10PM -0800, John Du wrote:
>> I upgraded the kernel to the latest Red Hat release: 2.6.18-128.el5 and
>> that did not fix the problem.  I installed 8.2.7 and it behaves the same
>> as 8.3.  The system becomes too slow to use.  I rolled back to 8.2.0.
>> 8.2.0 works in terms of writing data to the underlying storage but it is
>> not stable in replicating data to the secondary node.  Unloading the
>> DRBD module of 8.2.0 locks up the host.  8.2.7 and 8.3 do not log nay
>> errors when they are slow.  The DRBD worker thread stays in state D and
>> other processes are blocked.
>> Here is some observation:
>> When running 8.2.0, the command "iostat" does not show the DRBD device.
>> With 8.3.0 or 8.2.7, "iostat" shows the DRBD device as one of the IO
>> devices.  On all our other server where DRBD are used, iostat does not
>> display the DRBD devices.
>> In other words, when iostat shows a DRBD divce (/dev/drbd1 in my case)
>> as one of the IO devices, the host system becomes very slow.
> in January 2008, we started to use blkdev_issue_flush and BIO_RW_BARRIER
> to increase reliability when using volatile disk caches.
> these changes have been first released with 8.0.9 and 8.2.5.
> 8.0.9 (2008-02-04) and 8.0.10 (2008-02-07) have been bad releases,
> and have been superseeded within days by 8.0.11 (2008-02-12).
> 8.0.12 (2008-04-24) was the first release that could switch off this
> additional reliability feature again, as it turned out that the
> "better", more "advanced", IO subsystem you have, the worse the
> performance impact is when turning on barriers/flushes.
> the options to do so are "no-disk-flushes" (backing store),
> "no-md-flushes" (meta data store).
> in the 8.2 series, 8.2.6 (2008-05-30) is the first release
> to support these keywords to switch off these additional
> reliability features again.
> 8.2.7, and 8.3.0 again add some more advanced handling
> for additional performance while maintaining reliability
> when running on volatile caches.
> again, these features can be switched of.
> They should only be switched off if you run on a "safe" device
> (non-volatile, battery backed cache -- or no cache at all).
> the important additional switch now is "no-disk-barrier".
> see also "man drbd.conf"
> all of this has _nothing_ to do with whether "iostats" shows
> statistics for drbd or not.
It's a long time since i've been using the no-disk-barrier, no-disk-flushes
and no drain to optimize the performance of our datastores (3ware 9650 w/
BBU cache) but it has no apparent correlations btw the 2 mentioned drbd
version (8.2.7 vs 8.3.0)

We are using Centos5.2 x86_64 with a custom kernel and the read performance
lost (about 20%) is only seen running 8.3.0 (same config&setup)

If you don't believe me I can produce logs and iostats during some IO test.

In the btw, iostats accounting for drbd devices is still crazy, even
reverting back to 8.2.7

Any suggestions?
So long and thank for all the fish
#Matteo Tescione
#RMnet srl

More information about the drbd-user mailing list