[Drbd-dev] DRBD + DM = EIO.
lars.ellenberg at linbit.com
Thu May 27 11:01:25 CEST 2010
On Wed, May 26, 2010 at 12:21:27PM -0400, Ben Timby wrote:
> I posted a couple times to drbd-user, but I think this list is
> actually the correct forum for what I am experiencing. I will
> reiterate all the information have at this time below.
> I have two matched machines. They have 15 SATA hard drives in a raid 5
> array. I am using LVM to split this array into two volumes. I am then
> using DRBD to replicate these two volumes. On top of DRBD, I have two
> more LVM volumes, on which I can create (replicated) snapshots. These
> volumes each contain a single file system which is ext4, the size is
> 10.84TB per volume.
> The OS is CentOS 5.4, I am running DRBD 8.3.7, I built an RPM using
> the instructions provided in the DRBD users guide. I am using the
> 2.6.18-164.15.1.el5 kernel on an x86_64 processor.
> I am intermittently receiving the following error in /var/log/messages:
> May 10 00:05:11 ragoon6 kernel: block drbd1: p read: error=-5
> I tracked this down to the function drbd_endio_pri, after this error
> occurs, DRBD goes into diskless mode, shovelling reads/writes to it's
> peer. Once in diskless mode, I no longer receive this error, but I
> can't run this way.
> I removed DRBD from the mix, thus I have RAID -> LVM -> LVM -> EXT4,
> and I get no EIO errors.
> I found that I can immediately trigger this error by starting a raid
> rebuild on the underlying array (while DRBD is in the stack). I do
> this by executing the weekly cron job that is part of the mdadm
> package on CentOS:
> # /etc/cron.weekly/99-raid-check
> I rebuilt the RPM and added a call to dump_stack() in the
> drbd_endio_pri function. Below is the stack trace.
> I just started walking the stack trace in my kernel sources to try to
> locate the issue. However, I am hopeful that a DRBD developer can help
> me to find the (I am assuming) bug in interaction between
> device-mapper and DRBD.
You most likely hit
What is missing from that commit message is that a "failed"
(intentionally not served) READA will be considered
a real local IO error and cause DRBD to detach.
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
More information about the drbd-dev