[DRBD-user] requests get stuck in secondary

Lars Ellenberg lars.ellenberg at linbit.com
Thu Dec 10 14:08:35 CET 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Mon, Nov 23, 2015 at 04:55:53PM +0100, Matyas Koszik wrote:
> Hi,
> I've recently upgraded from 8.3 to 8.4 (together with the whole system,
> from debian wheezy to jessie), and since then I see some rather
> strange behaviors. I have 4 resources defined, and they all exhibit this
> issue. From time to time, I see this in the logs:
> 2015-11-23 15:27:55 warning kernel: [216645.905347] block drbd4: Remote failed to finish a request within ko-count * timeout
> This indicates that the secondary did not complete a write in 4 * 20
> seconds. To further debug the issue I started running diskping and iostat
> and cat /proc/drbd on both nodes. The primary acted normally (iostat
> showed 100% util on the drbd device, indicating it's waiting for the
> secondary, and the drbd stats showed a lot of 'pe' requests).
> Looking at the stats from the secondary revealed some anomalies: up until
> the point the request got stuck everything was normal, but after that, I
> could see a request that's been sent to the local device (lo:1), but
> wasn't accounted for as coming from anywhere else (ua:0). The system
> didn't get out of this state without the reestablishment of the session.
> iostat on the secondary also didn't reveal anything anomalous, there were
> no stuck requests according to that source.
> I'm inclined to think it's software-related, since before the upgrade this
> exact same setup worked flawlessly... How can I debug this further?
> Current versions:
> version: 8.4.3 (api:1/proto:86-101)

You may want to upgrade.
I seem to remember potential false hits for the "timeout" detection.

But maybe it's simply because 8.4 is better at detecting these timeouts,
where 8.3 would not even have noticed much longer timeouts sometimes.

: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
please don't Cc me, but send to list   --   I'm subscribed

More information about the drbd-user mailing list