<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'><font size="3">Hi Felix,</font><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">I am using DRBD with pacemaker+heartbeat for a HA cluster. There are no mounted filesystems at this time. Below is a copy of the kernel log after I attempted to stop the drbd service:</div><div id=""><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177694.517283] block drbd0: Requested state change failed by peer : Refusing to be Primary while peer is not outdated (-7)</font></div><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177694.873466] block drbd0: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) disk( UpToDate -> Outdated ) pdsk( UpToDate -> DUnknown )</font></div><div id=""><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177694.873540] block drbd0: short read expecting header on sock: r=-512</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.029784] block drbd0: meta connection shut down by peer.</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.195502] block drbd0: asender terminated</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.195526] block drbd0: Terminating asender thread</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.207524] block drbd0: Connection closed</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.207594] block drbd0: conn( Disconnecting -> StandAlone )</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.209382] block drbd0: receiver terminated</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.209396] block drbd0: Terminating receiver thread</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.209668] block drbd0: disk( Outdated -> Diskless )</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.210268] block drbd0: drbd_bm_resize called with capacity == 0</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.211205] block drbd0: worker terminated</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:14 node1 kernel: [177695.211208] block drbd0: Terminating worker thread</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177696.418762] block drbd1: Requested state change failed by peer: Refusing to be Primary while peer is not outdated (-7)</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177696.722826] block drbd1: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) disk( UpToDate -> Outdated ) pdsk( UpToDate -> DUnknown )</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177696.722871] block drbd1: short read expecting header on sock: r=-512</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177696.880462] block drbd1: meta connection shut down by peer.</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037039] block drbd1: asender terminated</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037067] block drbd1: Terminating asender thread</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037568] block drbd1: Connection closed</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037616] block drbd1: conn( Disconnecting -> StandAlone )</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037714] block drbd1: receiver terminated</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037716] block drbd1: Terminating receiver thread</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.037990] block drbd1: disk( Outdated -> Diskless )</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.038598] block drbd1: drbd_bm_resize called with capacity == 0</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.040081] block drbd1: worker terminated</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:44:16 node1 kernel: [177697.040083] block drbd1: Terminating worker thread</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:40 node1 kernel: [177841.014578] INFO: task rmmod:21024 blocked for more than 120 seconds.</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:40 node1 kernel: [177841.206355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634666] rmmod D 00000000ffffffff 0 21024 20992 0x00000004</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634695] ffff88020c43bc78 0000000000000086 0000000000015e00 0000000000015e00</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634709] ffff880214c8dfd0 ffff88020c43bfd8 0000000000015e00 ffff880214c8dc00</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634713] 0000000000015e00 ffff88020c43bfd8 0000000000015e00 ffff880214c8dfd0</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634720] Call Trace:</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634777] [<ffffffff8155e67d>] schedule_timeout+0x22d/0x300</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634822] [<ffffffff8102e779>] ? native_smp_send_reschedule+0x49/0x60</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634832] [<ffffffff8104ce56>] ? resched_task+0x76/0x90</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634839] [<ffffffff8105dd2b>] ? try_to_wake_up+0x2fb/0x480</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634843] [<ffffffff8155d7f6>] wait_for_common+0xd6/0x180</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634847] [<ffffffff8105deb0>] ? default_wake_function+0x0/0x20</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634850] [<ffffffff8155d95d>] wait_for_completion+0x1d/0x20</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634859] [<ffffffff81081a35>] flush_cpu_workqueue+0x65/0xa0</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634862] [<ffffffff81081bb0>] ? wq_barrier_func+0x0/0x20</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634866] [<ffffffff81081d2c>] flush_workqueue+0x4c/0x80</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634878] [<ffffffff8109f520>] ? __try_stop_module+0x0/0x50</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634890] [<ffffffff810b7c04>] __stop_machine+0xf4/0x120</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634894] [<ffffffff8109f520>] ? __try_stop_module+0x0/0x50</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634897] [<ffffffff810b7e5e>] stop_machine+0x3e/0x60</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634900] [<ffffffff8109e834>] ? find_module+0x34/0x70</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634922] [<ffffffff8109fe4e>] sys_delete_module+0x17e/0x270</font></div><div id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><font face="'courier new', courier, monaco, monospace, sans-serif" size="1">Jan 26 15:46:41 node1 kernel: [177841.634936] [<ffffffff81013172>] system_call_fastpath+0x16/0x1b</font></div></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; " id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; " id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown">I am using the DRBD 8.3.7 backport from this repository:</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; " id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><a href="https://launchpad.net/~ubuntu-ha/+archive/ppa">https://launchpad.net/~ubuntu-ha/+archive/ppa</a></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; " id="aeaoofnhgocdbnbeljkmbjdmhbcokfdb-mousedown"><br></div><div id="" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Thanks,</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Andrew<br><br><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Felix Frank" <ff@mpexnet.de><br><b>To: </b>"Andrew Martin" <amartin@xes-inc.com><br><b>Cc: </b>drbd-user@lists.linbit.com<br><b>Sent: </b>Tuesday, January 10, 2012 2:11:13 AM<br><b>Subject: </b>Re: [DRBD-user] Removing DRBD Kernel Module Blocks<br><br>Hi,<br><br>On 01/09/2012 08:03 PM, Andrew Martin wrote:<br>> Shutting these VMs down gracefully takes over 30 minutes as the "rmmod<br>> drbd" command in /etc/rc0.d/K08drbd blocks. Moreover, it seems that<br>> access to any information related to the kernel modules blocks as well<br>> (e.g. "lsmod", accesses to /proc/modules, etc). Do you have any ideas on<br>> what is causing the removal of this module to block and how to resolve it?<br><br>what's using those DRBDs and is it cleanly shut down before the drbd<br>service is being stopped?<br><br>Are there filesystems in your block devices? Do they get unmounted in time?<br><br>Are there any incriminating kernel log entries after this slow shutdown?<br><br>Cheers,<br>Felix<br></div><br></div></div></body></html>