<div dir="ltr">Thanks Lars for the detailed answer and explanation.... and thanks Trevor</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 15, 2015 at 4:34 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Oct 15, 2015 at 03:02:18PM +0300, Ammar Sheikh Saleh wrote:<br>
> > "JBD: Spotted dirty metadata buffer" turns up several bugzillas from<br>
> > 2010 and 2012, suggesting kernel upgrade would help.<br>
><br>
</span><span class="">> When ==> During normal operation<br>
><br>
> What's the IO stack below DRBD ==> 4 Disks in JBOD<br>
><br>
> Kernel upgrade is not possible now ... but at the same time its not too<br>
> old ( v.2.6.32-431.20.3)<br>
<br>
</span>What the message is complaining about is that JBD (the journaling<br>
component of ext*) detects a buffer that is "dirty" (under write-out),<br>
when it did not expect it to be.<br>
<br>
This all happens before any IO request related to that buffer even<br>
reaches the block layer or DRBD.<br>
<br>
Maybe you have some "live dump" going on or do some other "unexpected"<br>
(by JBD) things. Or your kernel simply still has some bug/race in the<br>
related IO paths.<br>
<br>
In any case, this has nothing to do with DRBD, as the error message<br>
complains about something happening in the buffer/page cache,<br>
which is *above* DRBD.<br>
<br>
So whatever happens there, would happen just the same, even without DRBD.<br>
(Though, of course, DRBD changes timings and memory pressure, so it may<br>
make it more or less likely to trigger this).<br>
<span class=""><br>
> > On Thu, Oct 15, 2015 at 12:32:11PM +0300, Ammar Sheikh Saleh wrote:<br>
> > > Hi all,<br>
> > ><br>
> > > I have one cluster with 2 nodes in active-active - master/slave setup<br>
> > ><br>
> > > I see the following error messages in the system logs:<br>
> ><br>
> > *when*.<br>
> ><br>
> > After a hard crash?<br>
> > After a simulated Crash?<br>
> > After a switchover?<br>
> > After a failover?<br>
> > During normal operation?<br>
> ><br>
> > What's the IO stack below DRBD?<br>
> ><br>
</span>> > > Jan 5 22:02:18 lws1h1 kernel: EXT4-fs error (device drbd1): ext4_mb_generate_buddy: EXT4-fs: group 3795: 32532 blocks in bitmap, 29780 in gd<br>
<div class="HOEnZb"><div class="h5">> > > Jan 5 22:02:19 lws1h1 kernel: JBD: Spotted dirty metadata buffer (dev = drbd1, blocknr = 0). There's a risk of filesystem corruption in case of system crash.<br>
> > ><br>
> > > What is the appropriate action to fix this issue<br>
> ><br>
> > Did you even google for the message?<br>
> > "JBD: Spotted dirty metadata buffer" turns up several bugzillas from<br>
> > 2010 and 2012, suggesting kernel upgrade would help.<br>
<br>
<br>
--<br>
: Lars Ellenberg<br>
: <a href="http://www.LINBIT.com" rel="noreferrer" target="_blank">http://www.LINBIT.com</a> | Your Way to High Availability<br>
: DRBD, Linux-HA and Pacemaker support and consulting<br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
__<br>
please don't Cc me, but send to list -- I'm subscribed<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</div></div></blockquote></div><br></div>