Thanks for the quick reply,<div><br></div><div>My test handler currently isn't doing anything interesting, I just had it echo 'hello world' to a file which is located on a different drive than the LVM volume. The echo seems to have completed successfully as the file is written.</div>
<div><br></div><div>The end goal for the handler is to simply insert a row into a remote DB, other than that the default behavior on io-error of detaching is exactly what I would like to have happen.
</div><div><br>I just tried filtering out drbd in lvm.conf and that doesn't seem to be the issue. After another try I did a quick ps auxf this showed up:</div><div><br></div><div><div>root 340 0.0 0.0 21392 1284 ? Ss 12:59 0:00 udevd --daemon</div>
<div>root 415 0.0 0.0 21384 896 ? S 12:59 0:00 \_ udevd --daemon</div><div>root 1775 0.0 0.0 8448 724 ? D 13:04 0:00 | \_ /sbin/blkid -o udev -p /dev/drbd575</div></div><div>
<br></div>
<div>So it seems like udev is initiating the blkid call, could it be doing this before drbd has finished executing the handler?</div><div><br></div><div>Thanks,</div><div><br>Chris</div><div><br></div><div><br></div><div>
<br><div><br><div class="gmail_quote">On Thu, Apr 12, 2012 at 8:36 AM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>On Thu, Apr 12, 2012 at 08:21:38AM -0400, Chris Dickson wrote:<br>
> Hello,<br>
><br>
> Someone please chime in if my method of simulating io-errors is too<br>
> complicated and there is an easier way.<br>
><br>
> I've been trying to simulate IO errors with drbd 8.4.1 by creating a device<br>
> mapper with dmsetup. I create the device mapper from a 1GB LVM volume that<br>
> was initialized with internal meta data and synced:<br>
><br>
> dmsetup create bad_disk << EOF<br>
> 0 1000 linear /dev/vg0/vol575 0<br>
> 1000 1 error<br>
> 1001 2096151 linear /dev/vg0/vol575 1001<br>
> EOF<br>
><br>
> I can now successfully start the drbd device backed by the bad_disk device<br>
> mapper and it shows Connected and UpToDate. When I change its role to<br>
> Primary, dmesg shows my IO error that I set at block 1000 and my custom<br>
> local-io-error script is called successfully. The drbd device is also set<br>
> to a disk state of Diskless.<br>
><br>
> It's at this moment that all other operations attempted on the device will<br>
> hang. Somewhere during or shortly after the io-error handler something ties<br>
> up the device and nothing I can do can free it... the first dmesg problem I<br>
> can see is this:<br>
><br>
> INFO: task blkid:1945 blocked for more than 120 seconds.<br>
><br>
> It might not be drbd, LVM is involved and also my manually created device<br>
> mapper on top of it. I wanted to throw this out there if anyone has tried<br>
> the same thing and encountered the error or if I'm doing something overtly<br>
> wrong.<br>
<br>
</div></div>What is your io error handler trying to do?<br>
<br>
It is run synchronously from a drbd kernel thread which<br>
also is (may be) necessary to process further IO requests on that drbd.<br>
<br>
If you trigger synchronous IO on that drbd from the handler,<br>
you deadlock on yourself.<br>
<br>
You may not even be aware of it: if you do any lvm commands,<br>
they will scan all devices (not filtered), and by doing so,<br>
may trigger IO there.<br>
<br>
If you do not expect to use DRBD as a PV,<br>
please reject drbd from your filter in lvm.conf.<br>
<br>
If that does not help already, and you try to do anything "interesting"<br>
from that handler, consider backgrounding it.<br>
<br>
Better yet, tell us what you actually want to achieve,<br>
any we may be able to suggest a solution.<br>
<span><font color="#888888"><br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</font></span></blockquote></div><br></div></div>