Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Not sure how the output is supposed to look like, but what I have now seems readable and provides quite some info. Does anyone can give me pointers on what to look for? Where can one determine what is the offending process? Cheers, BC On 29 Jul 2014, at 11:57, Richard Weinberger <richard at nod.at> wrote: > Am 29.07.2014 11:37, schrieb Bart Coninckx: >> That is what I actually did at first but I had no output getting to the redirected file, > > Depending on how broken your disto's netcat is, socat often does a better job. > i.e. socat udp-recv:PORT - > > Thanks, > //richard > >> BC >> >> >> >> On 29 Jul 2014, at 09:21, Richard Weinberger <richard.weinberger at gmail.com> wrote: >> >>> On Mon, Jul 28, 2014 at 7:11 PM, Bart Coninckx <bart.coninckx at telenet.be> wrote: >>>> >>>> On 18 Jul 2014, at 22:10, Bart Coninckx <bart.coninckx at telenet.be> wrote: >>>> >>>>> >>>>> On 18 Jul 2014, at 17:30, Lars Ellenberg <lars.ellenberg at linbit.com> wrote: >>>>> >>>>>> On Fri, Jul 18, 2014 at 03:03:50PM +0200, Bart Coninckx wrote: >>>>>>> >>>>>>> On 17 Jul 2014, at 17:07, Lars Ellenberg <lars.ellenberg at linbit.com> wrote: >>>>>>> >>>>>>>> On Thu, Jul 17, 2014 at 04:04:51PM +0200, Bart Coninckx wrote: >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> >>>>>>>>> I seem to experience random kernel panics on two CentOS 6.5 servers. >>>>>>>>> These do not seem to happen when DRBD is stopped on one node. Both >>>>>>>>> servers have Infiniband cards installed for the DRBD connections. I >>>>>>>>> upgraded the 8.4.4 installation to 8.4.5 to no avail. >>>>>>>>> Can someone provide me more hints on effectively troubleshoot this problem? >>>>>>>> >>>>>>>> hook up a serial console or "equivalent". >>>>>>>> >>>>>>>> Capture panic messages. >>>>>>>> Have them analysed. >>>>>> >>>>>>> Which parties are best approached for analysing? >>>>>> >>>>>> This list. >>>>>> Distribution support. >>>>>> LINBIT support. >>>>>> The maintainer (or corresponding list) of the driver that causes the panic. >>>>>> Your resident linux guru. >>>>>> >>>>>> Depends on the capture, I guess ;-) >>>>>> >>>>>> -- >>>>>> : Lars Ellenberg >>>>>> : LINBIT | Your Way to High Availability >>>>>> : DRBD/HA support and consulting http://www.linbit.com >>>>>> >>>>>> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. >>>>>> __ >>>>>> please don't Cc me, but send to list -- I'm subscribed >>>>>> _______________________________________________ >>>>>> drbd-user mailing list >>>>>> drbd-user at lists.linbit.com >>>>>> http://lists.linbit.com/mailman/listinfo/drbd-user >>>>> >>>>> I configured netconsole. I'm a happy panic hunter now ;-) >>>>> >>>>> BC >>>>> _______________________________________________ >>>>> drbd-user mailing list >>>>> drbd-user at lists.linbit.com >>>>> http://lists.linbit.com/mailman/listinfo/drbd-user >>>> >>>> >>>> OK, I trapped a kernel panic output. Am I correct in assuming that the problem is the problem is SCST related as this is mentioned quite early in the panic log? : >>> >>> Please never capture netconsole logs using syslog. The result is a broken mess. >>> Netcat woks better. :) >>> >>>> >>>> Jul 28 19:02:19 storage3 BUG: unable to handle kernel >>>> Jul 28 19:02:19 storage3 BUG: unable to handle kernel >>>> Jul 28 19:02:19 storage3 NULL pointer dereference >>>> Jul 28 19:02:19 storage3 NULL pointer dereference >>>> Jul 28 19:02:19 storage3 at 00000000000000c4 >>>> Jul 28 19:02:19 storage3 at 00000000000000c4 >>>> Jul 28 19:02:19 storage3 IP: >>>> Jul 28 19:02:19 storage3 IP: >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 PGD 0 >>>> Jul 28 19:02:19 storage3 PGD 0 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 Oops: 0002 [#1] >>>> Jul 28 19:02:19 storage3 Oops: 0002 [#1] >>>> Jul 28 19:02:19 storage3 SMP >>>> Jul 28 19:02:19 storage3 SMP >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 last sysfs file: /sys/devices/virtual/block/drbd11/range >>>> Jul 28 19:02:19 storage3 last sysfs file: /sys/devices/virtual/block/drbd11/range >>>> Jul 28 19:02:19 storage3 CPU 1 >>>> Jul 28 19:02:19 storage3 CPU 1 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 Modules linked in: >>>> Jul 28 19:02:19 storage3 Modules linked in: >>>> Jul 28 19:02:19 storage3 netconsole >>>> Jul 28 19:02:19 storage3 netconsole >>>> Jul 28 19:02:19 storage3 iscsi_scst >>>> Jul 28 19:02:19 storage3 iscsi_scst >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 scst_vdisk >>>> Jul 28 19:02:19 storage3 scst_vdisk >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 scst >>>> Jul 28 19:02:19 storage3 scst >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 crc32c_intel >>>> Jul 28 19:02:19 storage3 crc32c_intel >>>> Jul 28 19:02:19 storage3 drbd >>>> Jul 28 19:02:19 storage3 drbd >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 configfs >>>> Jul 28 19:02:19 storage3 configfs >>>> Jul 28 19:02:19 storage3 dm_mod >>>> Jul 28 19:02:19 storage3 dm_mod >>>> Jul 28 19:02:19 storage3 iw_nes >>>> Jul 28 19:02:19 storage3 iw_nes >>>> Jul 28 19:02:19 storage3 iw_cxgb3 >>>> Jul 28 19:02:19 storage3 iw_cxgb3 >>>> Jul 28 19:02:19 storage3 cxgb3 >>>> Jul 28 19:02:19 storage3 cxgb3 >>>> Jul 28 19:02:19 storage3 mdio >>>> Jul 28 19:02:19 storage3 mdio >>>> Jul 28 19:02:19 storage3 rdma_ucm >>>> Jul 28 19:02:19 storage3 rdma_ucm >>>> Jul 28 19:02:19 storage3 ib_ucm >>>> Jul 28 19:02:19 storage3 ib_ucm >>>> Jul 28 19:02:19 storage3 ib_uverbs >>>> >>>> Jul 28 19:02:19 storage3 ib_uverbs >>>> Jul 28 19:02:19 storage3 ib_srp >>>> Jul 28 19:02:19 storage3 ib_srp >>>> Jul 28 19:02:19 storage3 scsi_transport_srp >>>> Jul 28 19:02:19 storage3 scsi_transport_srp >>>> Jul 28 19:02:19 storage3 scsi_tgt >>>> Jul 28 19:02:19 storage3 scsi_tgt >>>> Jul 28 19:02:19 storage3 ib_qib >>>> Jul 28 19:02:19 storage3 ib_qib >>>> Jul 28 19:02:19 storage3 ib_iser >>>> Jul 28 19:02:19 storage3 ib_iser >>>> Jul 28 19:02:19 storage3 libiscsi >>>> Jul 28 19:02:19 storage3 libiscsi >>>> Jul 28 19:02:19 storage3 rdma_cm >>>> Jul 28 19:02:19 storage3 rdma_cm >>>> Jul 28 19:02:19 storage3 iw_cm >>>> Jul 28 19:02:19 storage3 iw_cm >>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi >>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi >>>> Jul 28 19:02:19 storage3 ib_ipath >>>> Jul 28 19:02:19 storage3 ib_ipath >>>> Jul 28 19:02:19 storage3 ib_addr >>>> Jul 28 19:02:19 storage3 ib_addr >>>> Jul 28 19:02:19 storage3 ib_ipoib >>>> Jul 28 19:02:19 storage3 ib_ipoib >>>> Jul 28 19:02:19 storage3 ib_cm >>>> Jul 28 19:02:19 storage3 ib_cm >>>> Jul 28 19:02:19 storage3 ipv6 >>>> Jul 28 19:02:19 storage3 ipv6 >>>> Jul 28 19:02:19 storage3 ib_umad >>>> Jul 28 19:02:19 storage3 ib_umad >>>> Jul 28 19:02:19 storage3 mlx4_ib >>>> Jul 28 19:02:19 storage3 mlx4_ib >>>> Jul 28 19:02:19 storage3 mlx4_en >>>> Jul 28 19:02:19 storage3 mlx4_en >>>> Jul 28 19:02:19 storage3 mlx4_core >>>> Jul 28 19:02:19 storage3 mlx4_core >>>> Jul 28 19:02:19 storage3 ib_sa >>>> Jul 28 19:02:19 storage3 ib_sa >>>> Jul 28 19:02:19 storage3 libcrc32c >>>> Jul 28 19:02:19 storage3 libcrc32c >>>> Jul 28 19:02:19 storage3 iTCO_wdt >>>> Jul 28 19:02:19 storage3 iTCO_wdt >>>> Jul 28 19:02:19 storage3 iTCO_vendor_support >>>> Jul 28 19:02:19 storage3 iTCO_vendor_support >>>> Jul 28 19:02:19 storage3 serio_raw >>>> Jul 28 19:02:19 storage3 serio_raw >>>> Jul 28 19:02:19 storage3 wmi >>>> Jul 28 19:02:19 storage3 wmi >>>> Jul 28 19:02:19 storage3 acpi_pad >>>> Jul 28 19:02:19 storage3 acpi_pad >>>> Jul 28 19:02:19 storage3 sg >>>> Jul 28 19:02:19 storage3 sg >>>> Jul 28 19:02:19 storage3 ib_mthca >>>> Jul 28 19:02:19 storage3 ib_mthca >>>> Jul 28 19:02:19 storage3 ib_mad >>>> Jul 28 19:02:19 storage3 ib_mad >>>> Jul 28 19:02:19 storage3 ib_core >>>> Jul 28 19:02:19 storage3 ib_core >>>> Jul 28 19:02:19 storage3 igb >>>> Jul 28 19:02:19 storage3 igb >>>> Jul 28 19:02:19 storage3 dca >>>> Jul 28 19:02:19 storage3 dca >>>> Jul 28 19:02:19 storage3 i2c_algo_bit >>>> Jul 28 19:02:19 storage3 i2c_algo_bit >>>> >>>> Jul 28 19:02:19 storage3 ptp >>>> Jul 28 19:02:19 storage3 ptp >>>> Jul 28 19:02:19 storage3 pps_core >>>> Jul 28 19:02:19 storage3 pps_core >>>> Jul 28 19:02:19 storage3 i2c_i801 >>>> Jul 28 19:02:19 storage3 i2c_i801 >>>> Jul 28 19:02:19 storage3 i2c_core >>>> Jul 28 19:02:19 storage3 i2c_core >>>> Jul 28 19:02:19 storage3 lpc_ich >>>> Jul 28 19:02:19 storage3 lpc_ich >>>> Jul 28 19:02:19 storage3 mfd_core >>>> Jul 28 19:02:19 storage3 mfd_core >>>> Jul 28 19:02:19 storage3 shpchp >>>> Jul 28 19:02:19 storage3 shpchp >>>> Jul 28 19:02:19 storage3 ext4 >>>> Jul 28 19:02:19 storage3 ext4 >>>> Jul 28 19:02:19 storage3 jbd2 >>>> Jul 28 19:02:19 storage3 jbd2 >>>> Jul 28 19:02:19 storage3 mbcache >>>> Jul 28 19:02:19 storage3 mbcache >>>> Jul 28 19:02:19 storage3 sd_mod >>>> Jul 28 19:02:19 storage3 sd_mod >>>> Jul 28 19:02:19 storage3 crc_t10dif >>>> Jul 28 19:02:19 storage3 crc_t10dif >>>> Jul 28 19:02:19 storage3 isci >>>> Jul 28 19:02:19 storage3 isci >>>> Jul 28 19:02:19 storage3 libsas >>>> Jul 28 19:02:19 storage3 libsas >>>> Jul 28 19:02:19 storage3 scsi_transport_sas >>>> Jul 28 19:02:19 storage3 scsi_transport_sas >>>> Jul 28 19:02:19 storage3 ahci >>>> Jul 28 19:02:19 storage3 ahci >>>> Jul 28 19:02:19 storage3 3w_sas >>>> Jul 28 19:02:19 storage3 3w_sas >>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole] >>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole] >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Not tainted 2.6.32-scst #1 >>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Not tainted 2.6.32-scst #1 >>>> Jul 28 19:02:19 storage3 Supermicro X9DRW >>>> Jul 28 19:02:19 storage3 Supermicro X9DRW >>>> Jul 28 19:02:19 storage3 /X9DRW >>>> Jul 28 19:02:19 storage3 /X9DRW >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 RIP: 0010:[<ffffffffa07d656d>] >>>> Jul 28 19:02:19 storage3 RIP: 0010:[<ffffffffa07d656d>] >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 RSP: 0018:ffff88045a8d7760 EFLAGS: 00010286 >>>> Jul 28 19:02:19 storage3 RSP: 0018:ffff88045a8d7760 EFLAGS: 00010286 >>>> Jul 28 19:02:19 storage3 RAX: 0000000000000030 RBX: 0000000000000000 RCX: ffff880460924800 >>>> Jul 28 19:02:19 storage3 RAX: 0000000000000030 RBX: 0000000000000000 RCX: ffff880460924800 >>>> Jul 28 19:02:19 storage3 RDX: ffff880460924940 RSI: 0000000000000000 RDI: 0000000000000000 >>>> Jul 28 19:02:19 storage3 RDX: ffff880460924940 RSI: 0000000000000000 RDI: 0000000000000000 >>>> Jul 28 19:02:19 storage3 RBP: ffff88045a8d7760 R08: 0000000000000000 R09: ffff88046bdc8020 >>>> Jul 28 19:02:19 storage3 RBP: ffff88045a8d7760 R08: 0000000000000000 R09: ffff88046bdc8020 >>>> Jul 28 19:02:19 storage3 R10: ffff880028403ec0 R11: 0000000000000000 R12: ffffea00118246c0 >>>> Jul 28 19:02:19 storage3 R10: ffff880028403ec0 R11: 0000000000000000 R12: ffffea00118246c0 >>>> Jul 28 19:02:19 storage3 R13: ffff8804765d55c0 R14: 000000000000000e R15: 000000000000000b >>>> Jul 28 19:02:19 storage3 R13: ffff8804765d55c0 R14: 000000000000000e R15: 000000000000000b >>>> Jul 28 19:02:19 storage3 FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000 >>>> >>>> Jul 28 19:02:19 storage3 FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000 >>>> Jul 28 19:02:19 storage3 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >>>> Jul 28 19:02:19 storage3 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4 CR3: 000000046216c000 CR4: 00000000001407e0 >>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4 CR3: 000000046216c000 CR4: 00000000001407e0 >>>> Jul 28 19:02:19 storage3 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>> Jul 28 19:02:19 storage3 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>> Jul 28 19:02:19 storage3 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>> Jul 28 19:02:19 storage3 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>> Jul 28 19:02:19 storage3 Process drbd_w_ws008 (pid: 6392, threadinfo ffff88045a8d6000, task ffff8804753ed500) >>>> Jul 28 19:02:19 storage3 Process drbd_w_ws008 (pid: 6392, threadinfo ffff88045a8d6000, task ffff8804753ed500) >>>> Jul 28 19:02:19 storage3 Stack: >>>> Jul 28 19:02:19 storage3 Stack: >>>> Jul 28 19:02:19 storage3 ffff88045a8d7790 >>>> Jul 28 19:02:19 storage3 ffff88045a8d7790 >>>> Jul 28 19:02:19 storage3 ffffffff8144ee30 >>>> Jul 28 19:02:19 storage3 ffffffff8144ee30 >>>> Jul 28 19:02:19 storage3 0000000000000000 >>>> Jul 28 19:02:19 storage3 0000000000000000 >>>> Jul 28 19:02:19 storage3 ffff8804765d55c0 >>>> Jul 28 19:02:19 storage3 ffff8804765d55c0 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>" >>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>" >>>> Jul 28 19:02:19 storage3 0000000000000002 >>>> Jul 28 19:02:19 storage3 0000000000000002 >>>> Jul 28 19:02:19 storage3 ffff88046bdc86e0 >>>> Jul 28 19:02:19 storage3 ffff88046bdc86e0 >>>> Jul 28 19:02:19 storage3 ffff88045a8d77b0 >>>> Jul 28 19:02:19 storage3 ffff88045a8d77b0 >>>> Jul 28 19:02:19 storage3 ffffffff8144f5ae >>>> Jul 28 19:02:19 storage3 ffffffff8144f5ae >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>" >>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>" >>>> Jul 28 19:02:19 storage3 ffff88045a8d77e0 >>>> Jul 28 19:02:19 storage3 ffff88045a8d77e0 >>>> Jul 28 19:02:19 storage3 ffff8804765d55c0 >>>> Jul 28 19:02:19 storage3 ffff8804765d55c0 >>>> Jul 28 19:02:19 storage3 ffff88045a8d77d0 >>>> Jul 28 19:02:19 storage3 ffff88045a8d77d0 >>>> Jul 28 19:02:19 storage3 ffffffff8144f66b >>>> Jul 28 19:02:19 storage3 ffffffff8144f66b >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 Call Trace: >>>> Jul 28 19:02:19 storage3 Call Trace: >>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] skb_release_data+0x90/0x130 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] skb_release_data+0x90/0x130 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] __kfree_skb+0x1e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] __kfree_skb+0x1e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] consume_skb+0x3b/0x80 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] consume_skb+0x3b/0x80 >>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] dev_kfree_skb_any+0x3d/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] dev_kfree_skb_any+0x3d/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> Jul 28 19:02:19 storage3 Code: >>>> Jul 28 19:02:19 storage3 Code: >>>> Jul 28 19:02:19 storage3 89 >>>> Jul 28 19:02:19 storage3 89 >>>> Jul 28 19:02:19 storage3 e5 >>>> Jul 28 19:02:19 storage3 e5 >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 1f >>>> >>>> Jul 28 19:02:19 storage3 1f >>>> Jul 28 19:02:19 storage3 44 >>>> Jul 28 19:02:19 storage3 44 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 e8 >>>> Jul 28 19:02:19 storage3 e8 >>>> Jul 28 19:02:19 storage3 52 >>>> Jul 28 19:02:19 storage3 52 >>>> Jul 28 19:02:19 storage3 e9 >>>> Jul 28 19:02:19 storage3 e9 >>>> Jul 28 19:02:19 storage3 ff >>>> Jul 28 19:02:19 storage3 ff >>>> Jul 28 19:02:19 storage3 ff >>>> Jul 28 19:02:19 storage3 ff >>>> Jul 28 19:02:19 storage3 31 >>>> Jul 28 19:02:19 storage3 31 >>>> Jul 28 19:02:19 storage3 c0 >>>> Jul 28 19:02:19 storage3 c0 >>>> Jul 28 19:02:19 storage3 c9 >>>> Jul 28 19:02:19 storage3 c9 >>>> Jul 28 19:02:19 storage3 c3 >>>> Jul 28 19:02:19 storage3 c3 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 66 >>>> Jul 28 19:02:19 storage3 2e >>>> Jul 28 19:02:19 storage3 2e >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 1f >>>> Jul 28 19:02:19 storage3 1f >>>> Jul 28 19:02:19 storage3 84 >>>> Jul 28 19:02:19 storage3 84 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 55 >>>> Jul 28 19:02:19 storage3 55 >>>> Jul 28 19:02:19 storage3 48 >>>> Jul 28 19:02:19 storage3 48 >>>> Jul 28 19:02:19 storage3 89 >>>> Jul 28 19:02:19 storage3 89 >>>> Jul 28 19:02:19 storage3 e5 >>>> Jul 28 19:02:19 storage3 e5 >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 1f >>>> Jul 28 19:02:19 storage3 1f >>>> >>>> Jul 28 19:02:19 storage3 44 >>>> Jul 28 19:02:19 storage3 44 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 48 >>>> Jul 28 19:02:19 storage3 48 >>>> Jul 28 19:02:19 storage3 8b >>>> Jul 28 19:02:19 storage3 8b >>>> Jul 28 19:02:19 storage3 7f >>>> Jul 28 19:02:19 storage3 7f >>>> Jul 28 19:02:19 storage3 38 >>>> Jul 28 19:02:19 storage3 38 >>>> Jul 28 19:02:19 storage3 unparseable log message: "<f0> " >>>> Jul 28 19:02:19 storage3 unparseable log message: "<f0> " >>>> Jul 28 19:02:19 storage3 ff >>>> Jul 28 19:02:19 storage3 ff >>>> Jul 28 19:02:19 storage3 8f >>>> Jul 28 19:02:19 storage3 8f >>>> Jul 28 19:02:19 storage3 c4 >>>> Jul 28 19:02:19 storage3 c4 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 0f >>>> Jul 28 19:02:19 storage3 94 >>>> Jul 28 19:02:19 storage3 94 >>>> Jul 28 19:02:19 storage3 c0 >>>> Jul 28 19:02:19 storage3 c0 >>>> Jul 28 19:02:19 storage3 84 >>>> Jul 28 19:02:19 storage3 84 >>>> Jul 28 19:02:19 storage3 c0 >>>> Jul 28 19:02:19 storage3 c0 >>>> Jul 28 19:02:19 storage3 74 >>>> Jul 28 19:02:19 storage3 74 >>>> Jul 28 19:02:19 storage3 63 >>>> Jul 28 19:02:19 storage3 63 >>>> Jul 28 19:02:19 storage3 8b >>>> Jul 28 19:02:19 storage3 8b >>>> Jul 28 19:02:19 storage3 8f >>>> Jul 28 19:02:19 storage3 8f >>>> Jul 28 19:02:19 storage3 10 >>>> Jul 28 19:02:19 storage3 10 >>>> Jul 28 19:02:19 storage3 01 >>>> Jul 28 19:02:19 storage3 01 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 00 >>>> Jul 28 19:02:19 storage3 85 >>>> Jul 28 19:02:19 storage3 85 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 RIP >>>> Jul 28 19:02:19 storage3 RIP >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 RSP <ffff88045a8d7760> >>>> Jul 28 19:02:19 storage3 RSP <ffff88045a8d7760> >>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4 >>>> >>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4 >>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dc ]--- >>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dc ]--- >>>> Jul 28 19:02:19 storage3 Kernel panic - not syncing: Fatal exception in interrupt >>>> Jul 28 19:02:19 storage3 Kernel panic - not syncing: Fatal exception in interrupt >>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G D --------------- 2.6.32-scst #1 >>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G D --------------- 2.6.32-scst #1 >>>> Jul 28 19:02:19 storage3 Call Trace: >>>> Jul 28 19:02:19 storage3 Call Trace: >>>> Jul 28 19:02:19 storage3 [<ffffffff81524d7d>] ? panic+0xa7/0x16f >>>> Jul 28 19:02:19 storage3 [<ffffffff81524d7d>] ? panic+0xa7/0x16f >>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180 >>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80 >>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0 >>>> >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> Jul 28 19:02:19 storage3 ------------[ cut here ]------------ >>>> Jul 28 19:02:19 storage3 ------------[ cut here ]------------ >>>> Jul 28 19:02:19 storage3 WARNING: at arch/x86/kernel/smp.c:118 native_smp_send_reschedule+0x5c/0x60() (Tainted: G D --------------- ) >>>> Jul 28 19:02:19 storage3 WARNING: at arch/x86/kernel/smp.c:118 native_smp_send_reschedule+0x5c/0x60() (Tainted: G D --------------- ) >>>> Jul 28 19:02:19 storage3 Hardware name: X9DRW >>>> Jul 28 19:02:19 storage3 Hardware name: X9DRW >>>> Jul 28 19:02:19 storage3 Modules linked in: >>>> Jul 28 19:02:19 storage3 Modules linked in: >>>> Jul 28 19:02:19 storage3 netconsole >>>> Jul 28 19:02:19 storage3 netconsole >>>> Jul 28 19:02:19 storage3 iscsi_scst >>>> Jul 28 19:02:19 storage3 iscsi_scst >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 scst_vdisk >>>> Jul 28 19:02:19 storage3 scst_vdisk >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 scst >>>> Jul 28 19:02:19 storage3 scst >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 crc32c_intel >>>> >>>> Jul 28 19:02:19 storage3 crc32c_intel >>>> Jul 28 19:02:19 storage3 drbd >>>> Jul 28 19:02:19 storage3 drbd >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 (U) >>>> Jul 28 19:02:19 storage3 configfs >>>> Jul 28 19:02:19 storage3 configfs >>>> Jul 28 19:02:19 storage3 dm_mod >>>> Jul 28 19:02:19 storage3 dm_mod >>>> Jul 28 19:02:19 storage3 iw_nes >>>> Jul 28 19:02:19 storage3 iw_nes >>>> Jul 28 19:02:19 storage3 iw_cxgb3 >>>> Jul 28 19:02:19 storage3 iw_cxgb3 >>>> Jul 28 19:02:19 storage3 cxgb3 >>>> Jul 28 19:02:19 storage3 cxgb3 >>>> Jul 28 19:02:19 storage3 mdio >>>> Jul 28 19:02:19 storage3 mdio >>>> Jul 28 19:02:19 storage3 rdma_ucm >>>> Jul 28 19:02:19 storage3 rdma_ucm >>>> Jul 28 19:02:19 storage3 ib_ucm >>>> Jul 28 19:02:19 storage3 ib_ucm >>>> Jul 28 19:02:19 storage3 ib_uverbs >>>> Jul 28 19:02:19 storage3 ib_uverbs >>>> Jul 28 19:02:19 storage3 ib_srp >>>> Jul 28 19:02:19 storage3 ib_srp >>>> Jul 28 19:02:19 storage3 scsi_transport_srp >>>> Jul 28 19:02:19 storage3 scsi_transport_srp >>>> Jul 28 19:02:19 storage3 scsi_tgt >>>> Jul 28 19:02:19 storage3 scsi_tgt >>>> Jul 28 19:02:19 storage3 ib_qib >>>> Jul 28 19:02:19 storage3 ib_qib >>>> Jul 28 19:02:19 storage3 ib_iser >>>> Jul 28 19:02:19 storage3 ib_iser >>>> Jul 28 19:02:19 storage3 libiscsi >>>> Jul 28 19:02:19 storage3 libiscsi >>>> Jul 28 19:02:19 storage3 rdma_cm >>>> Jul 28 19:02:19 storage3 rdma_cm >>>> Jul 28 19:02:19 storage3 iw_cm >>>> Jul 28 19:02:19 storage3 iw_cm >>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi >>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi >>>> Jul 28 19:02:19 storage3 ib_ipath >>>> Jul 28 19:02:19 storage3 ib_ipath >>>> Jul 28 19:02:19 storage3 ib_addr >>>> Jul 28 19:02:19 storage3 ib_addr >>>> Jul 28 19:02:19 storage3 ib_ipoib >>>> Jul 28 19:02:19 storage3 ib_ipoib >>>> Jul 28 19:02:19 storage3 ib_cm >>>> Jul 28 19:02:19 storage3 ib_cm >>>> Jul 28 19:02:19 storage3 ipv6 >>>> Jul 28 19:02:19 storage3 ipv6 >>>> Jul 28 19:02:19 storage3 ib_umad >>>> Jul 28 19:02:19 storage3 ib_umad >>>> Jul 28 19:02:19 storage3 mlx4_ib >>>> Jul 28 19:02:19 storage3 mlx4_ib >>>> Jul 28 19:02:19 storage3 mlx4_en >>>> Jul 28 19:02:19 storage3 mlx4_en >>>> Jul 28 19:02:19 storage3 mlx4_core >>>> Jul 28 19:02:19 storage3 mlx4_core >>>> Jul 28 19:02:19 storage3 ib_sa >>>> Jul 28 19:02:19 storage3 ib_sa >>>> Jul 28 19:02:19 storage3 libcrc32c >>>> Jul 28 19:02:19 storage3 libcrc32c >>>> Jul 28 19:02:19 storage3 iTCO_wdt >>>> Jul 28 19:02:19 storage3 iTCO_wdt >>>> >>>> Jul 28 19:02:19 storage3 iTCO_vendor_support >>>> Jul 28 19:02:19 storage3 iTCO_vendor_support >>>> Jul 28 19:02:19 storage3 serio_raw >>>> Jul 28 19:02:19 storage3 serio_raw >>>> Jul 28 19:02:19 storage3 wmi >>>> Jul 28 19:02:19 storage3 wmi >>>> Jul 28 19:02:19 storage3 acpi_pad >>>> Jul 28 19:02:19 storage3 acpi_pad >>>> Jul 28 19:02:19 storage3 sg >>>> Jul 28 19:02:19 storage3 sg >>>> Jul 28 19:02:19 storage3 ib_mthca >>>> Jul 28 19:02:19 storage3 ib_mthca >>>> Jul 28 19:02:19 storage3 ib_mad >>>> Jul 28 19:02:19 storage3 ib_mad >>>> Jul 28 19:02:19 storage3 ib_core >>>> Jul 28 19:02:19 storage3 ib_core >>>> Jul 28 19:02:19 storage3 igb >>>> Jul 28 19:02:19 storage3 igb >>>> Jul 28 19:02:19 storage3 dca >>>> Jul 28 19:02:19 storage3 dca >>>> Jul 28 19:02:19 storage3 i2c_algo_bit >>>> Jul 28 19:02:19 storage3 i2c_algo_bit >>>> Jul 28 19:02:19 storage3 ptp >>>> Jul 28 19:02:19 storage3 ptp >>>> Jul 28 19:02:19 storage3 pps_core >>>> Jul 28 19:02:19 storage3 pps_core >>>> Jul 28 19:02:19 storage3 i2c_i801 >>>> Jul 28 19:02:19 storage3 i2c_i801 >>>> Jul 28 19:02:19 storage3 i2c_core >>>> Jul 28 19:02:19 storage3 i2c_core >>>> Jul 28 19:02:19 storage3 lpc_ich >>>> Jul 28 19:02:19 storage3 lpc_ich >>>> Jul 28 19:02:19 storage3 mfd_core >>>> Jul 28 19:02:19 storage3 mfd_core >>>> Jul 28 19:02:19 storage3 shpchp >>>> Jul 28 19:02:19 storage3 shpchp >>>> Jul 28 19:02:19 storage3 ext4 >>>> Jul 28 19:02:19 storage3 ext4 >>>> Jul 28 19:02:19 storage3 jbd2 >>>> Jul 28 19:02:19 storage3 jbd2 >>>> Jul 28 19:02:19 storage3 mbcache >>>> Jul 28 19:02:19 storage3 mbcache >>>> Jul 28 19:02:19 storage3 sd_mod >>>> Jul 28 19:02:19 storage3 sd_mod >>>> Jul 28 19:02:19 storage3 crc_t10dif >>>> Jul 28 19:02:19 storage3 crc_t10dif >>>> Jul 28 19:02:19 storage3 isci >>>> Jul 28 19:02:19 storage3 isci >>>> Jul 28 19:02:19 storage3 libsas >>>> Jul 28 19:02:19 storage3 libsas >>>> Jul 28 19:02:19 storage3 scsi_transport_sas >>>> Jul 28 19:02:19 storage3 scsi_transport_sas >>>> Jul 28 19:02:19 storage3 ahci >>>> Jul 28 19:02:19 storage3 ahci >>>> Jul 28 19:02:19 storage3 3w_sas >>>> Jul 28 19:02:19 storage3 3w_sas >>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole] >>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole] >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 >>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G D --------------- 2.6.32-scst #1 >>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G D --------------- 2.6.32-scst #1 >>>> Jul 28 19:02:19 storage3 Call Trace: >>>> Jul 28 19:02:19 storage3 Call Trace: >>>> Jul 28 19:02:19 storage3 <IRQ> >>>> >>>> Jul 28 19:02:19 storage3 <IRQ> >>>> Jul 28 19:02:19 storage3 [<ffffffff81071a17>] ? warn_slowpath_common+0x87/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81071a17>] ? warn_slowpath_common+0x87/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81071a6a>] ? warn_slowpath_null+0x1a/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff81071a6a>] ? warn_slowpath_null+0x1a/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8103094c>] ? native_smp_send_reschedule+0x5c/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8103094c>] ? native_smp_send_reschedule+0x5c/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8105dd08>] ? scheduler_tick+0x208/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff8105dd08>] ? scheduler_tick+0x208/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff810ac6c0>] ? tick_sched_timer+0x0/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff810ac6c0>] ? tick_sched_timer+0x0/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81083d8e>] ? update_process_times+0x6e/0x90 >>>> Jul 28 19:02:19 storage3 [<ffffffff81083d8e>] ? update_process_times+0x6e/0x90 >>>> Jul 28 19:02:19 storage3 [<ffffffff810ac726>] ? tick_sched_timer+0x66/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff810ac726>] ? tick_sched_timer+0x66/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109f2ee>] ? __run_hrtimer+0x8e/0x1a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109f2ee>] ? __run_hrtimer+0x8e/0x1a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff810a66ef>] ? ktime_get_update_offsets+0x4f/0xd0 >>>> Jul 28 19:02:19 storage3 [<ffffffff810a66ef>] ? ktime_get_update_offsets+0x4f/0xd0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109f656>] ? hrtimer_interrupt+0xe6/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109f656>] ? hrtimer_interrupt+0xe6/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff81031cfd>] ? local_apic_timer_interrupt+0x3d/0x70 >>>> Jul 28 19:02:19 storage3 [<ffffffff81031cfd>] ? local_apic_timer_interrupt+0x3d/0x70 >>>> Jul 28 19:02:19 storage3 [<ffffffff81036f39>] ? native_apic_msr_eoi_write+0x19/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff81036f39>] ? native_apic_msr_eoi_write+0x19/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8152eb85>] ? smp_apic_timer_interrupt+0x45/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8152eb85>] ? smp_apic_timer_interrupt+0x45/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100bb53>] ? apic_timer_interrupt+0x13/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100bb53>] ? apic_timer_interrupt+0x13/0x20 >>>> Jul 28 19:02:19 storage3 <EOI> >>>> Jul 28 19:02:19 storage3 <EOI> >>>> Jul 28 19:02:19 storage3 [<ffffffff81524e22>] ? panic+0x14c/0x16f >>>> Jul 28 19:02:19 storage3 [<ffffffff81524e22>] ? panic+0x14c/0x16f >>>> Jul 28 19:02:19 storage3 [<ffffffff81524daf>] ? panic+0xd9/0x16f >>>> Jul 28 19:02:19 storage3 [<ffffffff81524daf>] ? panic+0xd9/0x16f >>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470 >>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180 >>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst] >>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80 >>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80 >>>> >>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> >>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib] >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40 >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd] >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 >>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dd ]--- >>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dd ]--- >>>> >>>> >>>> Thank you, >>>> >>>> BC >>>> >>>> _______________________________________________ >>>> drbd-user mailing list >>>> drbd-user at lists.linbit.com >>>> http://lists.linbit.com/mailman/listinfo/drbd-user >>> >>> >>> >>> -- >>> Thanks, >>> //richard >>