[DRBD-user] TRIM/discard leads to secondary becoming diskless

Lars Ellenberg lars.ellenberg at linbit.com
Mon Oct 5 16:49:11 CEST 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Sun, Oct 04, 2015 at 08:36:37PM +0200, Olav Morken wrote:
> Hi,
> I'm testing DRBD on a couple of virtual machines running Debian
> Jessie, and noticed that every time a discard request is sent to the
> DRBD-device, the secondary DRBD host becomes diskless:
> > [  121.779367] block drbd2: discard: error=-95 s=0s

Please also reproduce with the out of tree DRBD 8.4.

Also, what exactly is the IO stack below DRBD?
If the lower level device below the receiving side does not support
discards, DRBD is supposed to fall back to zero-out.

If the lower level device below the sending (Primary) side does not
support discards, DRBD supposedly does not even announce discard to
upper layers, and these requests should never reach us.

If the lower level devices pretend to support discards,
but then fail with EOPNOTSUP, on the Primary side, we can propagate this
error to upper layers without detaching, assuming the upper layer will
then "deal with it appropriately".

But on the Secondary side, that's not so straight forward, (the discard
may have already been "successfully" completed on the primary side).
We cannot have a "write" (equivalent) succeed on the Primary,
but fail on the Secondary, and not do anything about it.

Which is why the Secondary side currently detaches, if it submitted a
discard because the lower level device announced support, but then
does not deliver, but returns an error.

That's just the way it currently is.  Just ignoring the error
on the secondary side is not going to help much, either.
Certainly not optimal.  Various options exist to better deal with this.  

At some point we'll likely retry ourselves internally,
or maybe always use some zero-out-with-unmap wrapper.

Best for now:
Don't use devices that pretend to do discards, but then don't deliver.
Or don't use discards in the first place.

I'm not sure if in-tree behaves the same as out-of-tree.
Our last pull request has been ignored
(which would not have helped your older kernel anyways).
Also we managed to produce merge bugs in the past.
So maybe out-of-tree behaves better. Maybe not.  Please try.

> > [  121.779376] block drbd2: disk( UpToDate -> Failed )
> > [  121.779382] block drbd2: Local IO failed in drbd_endio_write_sec_final. Detaching...
> > [  121.779866] block drbd2: bitmap WRITE of 0 pages took 0 jiffies
> > [  121.779873] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
> > [  121.779877] block drbd2: disk( Failed -> Diskless )
> The backing device for the DRBD-devices is a thin-provisioned LVM2
> volume, created using:
> > lvcreate -L 2G -T vg/dvb-testvm-01
> lvdisplay for the lvm-volume is available here:
> https://gist.github.com/olavmrk/de339e697b0be1ad5d19#file-lvdisplay
> This is a stock Debian Jessie install, using kernel version
> 3.16.7-ckt11-1.
> My drbd disk configuration:
> https://gist.github.com/olavmrk/de339e697b0be1ad5d19#file-testvm-01-res
> In my case, the discard happened as I was creating a file system on
> the DRBD-device:
> > mkfs.ext4 /dev/drbd/by-res/testvm-01
> I was also able to reproduce by running blkdiscard:
> > blkdiscard /dev/drbd/by-res/testvm-01
> error -95 seems to be -EOPNOTSUPP, which makes sense, since I have not
> made any effort in trying to support TRIM through the virtual
> machine.
> I assume that I am hitting this code path:
> https://github.com/torvalds/linux/blob/v4.3-rc4/drivers/block/drbd/drbd_worker.c#L151
> The same error seems to be dealt with when it happens on the primary:
> https://github.com/torvalds/linux/blob/v4.3-rc4/drivers/block/drbd/drbd_worker.c#L246
> https://github.com/torvalds/linux/blob/v4.3-rc4/drivers/block/drbd/drbd_req.c#L651
> I tried searching for this problem but couldn't find any mentions of
> it. I can't be the only one running DRBD on volumes that do not
> support the TRIM/discard operation, so I am wondering if anybody has
> any ideas about what actually triggers this bug on my setup?
> Best regards,
> Olav Morken

: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
please don't Cc me, but send to list   --   I'm subscribed

More information about the drbd-user mailing list