<div>Very detailed answer Lars, however we are missing the basic info from Peter. </div>
<div> </div>
<div>Peter, do you use Xen on top of DRBD (or some other virtualisation technology)? Do you use heartbeat to start/stop resources?</div>
<div>If you use your own HA RA you should not return control to heartbeat before VM is really shutdown. Even after that you usually will notice such log messages, and thus have to sleep for ~1 second.</div>
<div> </div>
<div>Regards.<br></div>
<div class="gmail_quote">2009/3/16 Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div class="im">On Sun, Mar 15, 2009 at 11:20:56PM +0100, Peter Funk wrote:<br>> Hello,<br>><br>> today I tried to find out, who is the "someone" in the following<br>> drbd syslog messages:<br>> .... State change failed: Device is held open by someone<br>
> ... drbd0: state = { cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate r--- }<br>> ... drbd0: wanted = { cs:Connected st:Secondary/Secondary ds:UpToDate/UpToDate r--- }<br>><br>> Unfortunately I had no success.<br>
><br>> Searching the archives I've noticed similar questions came up<br>> here on the list earlier:<br>> <a href="http://lists.linbit.com/pipermail/drbd-user/2008-November/010706.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2008-November/010706.html</a><br>
> <a href="http://lists.linbit.com/pipermail/drbd-user/2008-August/009954.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2008-August/009954.html</a><br>> <a href="http://lists.linbit.com/pipermail/drbd-user/2007-August/007338.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2007-August/007338.html</a><br>
> (I'm not using ocfs2: The answer Lars Ellenberg gave in the<br>> last citiation didn't apply to my situation here).<br>><br>> I've looked into source and found in drbd_main.c, that the message<br>
> above is given, if( ns.role == Secondary && mdev->open_cnt )<br>><br>> Can you imagine a race scenario, where the reference counter ``open_cnt``<br>> might be incremented or decremented wrong? Especially on a dual<br>
> quad core (menaing it has 8 active CPU cores) SMP machine?<br><br></div>No.<br>Well. Unless ...<br><br>You can check with a tight loop.<br>just after reboot, and no one has accessed the drbd so far,<br>and drbd is primary, do<br>
for i in `seq 8`; do<br> ( i=1000; while let --i; do : < /dev/drbd0 ; done ) &<br>done<br><br>if you did only do that, and you cannot make it secondary after that,<br>well, then it is probably time to use an atomic_t for open_cnt.<br>
<br>.release operations are serialized in some generic layer.<br>or at least, have been serialized, iirc -- though of course<br>I may be wrong, or the kernel has changed without me noticing.<br>
<div class="im"><br>> Any hints, how to find out?<br><br></div>if the filesystem is still mounted ...<br>oh well. unmount it. not lazy, but really.<br><br>of course you also tried lsof and fuser.<br><br>if that did not pick up anything,<br>
<br>If nfs was involved, try<br> killall -9 nfsd<br> killall -9 lockd<br> echo 0 > /proc/fs/nfsd/threads<br><br>if lvm/dmsetup/kpartx/multipath/udev is involved, try<br> dmsetup ls --tree -o inverted<br>
and check if there are dependencies from drbd.<br><br>if loop/cryptoloop/etc is involved, check if one of those<br>is still accessing them.<br><br>if some virtualization tecknique is in use, shut down/destroy all<br>
containers/VMs that may have been accessing that drbd during their<br>life time.<br><font color="#888888"><br>--<br>: Lars Ellenberg<br>: LINBIT | Your Way to High Availability<br>: DRBD/HA support and consulting <a href="http://www.linbit.com/" target="_blank">http://www.linbit.com</a><br>
<br>DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>__<br>please don't Cc me, but send to list -- I'm subscribed<br></font>
<div>
<div></div>
<div class="h5">_______________________________________________<br>drbd-user mailing list<br><a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br><a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</div></div></blockquote></div><br>