[DRBD-user] Problems after upgrade 8.2.0 to 8.3.0
jjohndu at gmail.com
Mon Feb 9 19:35:44 CET 2009
Lars Ellenberg wrote:
> On Sat, Feb 07, 2009 at 04:55:10PM -0800, John Du wrote:
>> I upgraded the kernel to the latest Red Hat release: 2.6.18-128.el5 and
>> that did not fix the problem. I installed 8.2.7 and it behaves the same
>> as 8.3. The system becomes too slow to use. I rolled back to 8.2.0.
>> 8.2.0 works in terms of writing data to the underlying storage but it is
>> not stable in replicating data to the secondary node. Unloading the
>> DRBD module of 8.2.0 locks up the host. 8.2.7 and 8.3 do not log nay
>> errors when they are slow. The DRBD worker thread stays in state D and
>> other processes are blocked.
>> Here is some observation:
>> When running 8.2.0, the command "iostat" does not show the DRBD device.
>> With 8.3.0 or 8.2.7, "iostat" shows the DRBD device as one of the IO
>> devices. On all our other server where DRBD are used, iostat does not
>> display the DRBD devices.
>> In other words, when iostat shows a DRBD divce (/dev/drbd1 in my case)
>> as one of the IO devices, the host system becomes very slow.
> in January 2008, we started to use blkdev_issue_flush and BIO_RW_BARRIER
> to increase reliability when using volatile disk caches.
> these changes have been first released with 8.0.9 and 8.2.5.
> 8.0.9 (2008-02-04) and 8.0.10 (2008-02-07) have been bad releases,
> and have been superseeded within days by 8.0.11 (2008-02-12).
> 8.0.12 (2008-04-24) was the first release that could switch off this
> additional reliability feature again, as it turned out that the
> "better", more "advanced", IO subsystem you have, the worse the
> performance impact is when turning on barriers/flushes.
> the options to do so are "no-disk-flushes" (backing store),
> "no-md-flushes" (meta data store).
> in the 8.2 series, 8.2.6 (2008-05-30) is the first release
> to support these keywords to switch off these additional
> reliability features again.
> 8.2.7, and 8.3.0 again add some more advanced handling
> for additional performance while maintaining reliability
> when running on volatile caches.
> again, these features can be switched of.
> They should only be switched off if you run on a "safe" device
> (non-volatile, battery backed cache -- or no cache at all).
> the important additional switch now is "no-disk-barrier".
> see also "man drbd.conf"
> all of this has _nothing_ to do with whether "iostats" shows
> statistics for drbd or not.
Thank you Lars for your detailed explanation. I'll try the parameters
you mentioned above. As I mentioned in my original message the process
queue reaches 100 on this node when running 8.3 and it is under 1 with
I still do not understand why iostat only shows DRBD devices on this
particular node with 8.2.7 and 8.3.0 but not other nodes with the same
hardware, same Linux Kernel and same DRBD version.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the drbd-user