[DRBD-user] kernel panics when connected

Richard Weinberger richard at nod.at
Tue Jul 29 11:57:18 CEST 2014

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Am 29.07.2014 11:37, schrieb Bart Coninckx:
> That is what I actually did at first but I had no output getting to the redirected file,

Depending on how broken your disto's netcat is, socat often does a better job.
i.e. socat udp-recv:PORT -

Thanks,
//richard

> BC
> 
> 
> 
> On 29 Jul 2014, at 09:21, Richard Weinberger <richard.weinberger at gmail.com> wrote:
> 
>> On Mon, Jul 28, 2014 at 7:11 PM, Bart Coninckx <bart.coninckx at telenet.be> wrote:
>>>
>>> On 18 Jul 2014, at 22:10, Bart Coninckx <bart.coninckx at telenet.be> wrote:
>>>
>>>>
>>>> On 18 Jul 2014, at 17:30, Lars Ellenberg <lars.ellenberg at linbit.com> wrote:
>>>>
>>>>> On Fri, Jul 18, 2014 at 03:03:50PM +0200, Bart Coninckx wrote:
>>>>>>
>>>>>> On 17 Jul 2014, at 17:07, Lars Ellenberg <lars.ellenberg at linbit.com> wrote:
>>>>>>
>>>>>>> On Thu, Jul 17, 2014 at 04:04:51PM +0200, Bart Coninckx wrote:
>>>>>>>>  Hi all,
>>>>>>>>
>>>>>>>>
>>>>>>>> I seem to experience random kernel panics on two CentOS 6.5 servers.
>>>>>>>> These do not seem to happen when DRBD is stopped on one node. Both
>>>>>>>> servers have Infiniband cards installed for the DRBD connections. I
>>>>>>>> upgraded the 8.4.4 installation to 8.4.5 to no avail.
>>>>>>>> Can someone provide me more hints on effectively troubleshoot this problem?
>>>>>>>
>>>>>>> hook up a serial console or "equivalent".
>>>>>>>
>>>>>>> Capture panic messages.
>>>>>>> Have them analysed.
>>>>>
>>>>>> Which parties are best approached for analysing?
>>>>>
>>>>> This list.
>>>>> Distribution support.
>>>>> LINBIT support.
>>>>> The maintainer (or corresponding list) of the driver that causes the panic.
>>>>> Your resident linux guru.
>>>>>
>>>>> Depends on the capture, I guess ;-)
>>>>>
>>>>> --
>>>>> : Lars Ellenberg
>>>>> : LINBIT | Your Way to High Availability
>>>>> : DRBD/HA support and consulting http://www.linbit.com
>>>>>
>>>>> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
>>>>> __
>>>>> please don't Cc me, but send to list   --   I'm subscribed
>>>>> _______________________________________________
>>>>> drbd-user mailing list
>>>>> drbd-user at lists.linbit.com
>>>>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>>>
>>>> I configured netconsole. I'm a happy panic hunter now ;-)
>>>>
>>>> BC
>>>> _______________________________________________
>>>> drbd-user mailing list
>>>> drbd-user at lists.linbit.com
>>>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>>
>>>
>>> OK, I trapped a kernel panic output. Am I correct in assuming that the problem is the problem is SCST related as this is mentioned quite early in the panic log? :
>>
>> Please never capture netconsole logs using syslog. The result is a broken mess.
>> Netcat woks better. :)
>>
>>>
>>> Jul 28 19:02:19 storage3 BUG: unable to handle kernel
>>> Jul 28 19:02:19 storage3 BUG: unable to handle kernel
>>> Jul 28 19:02:19 storage3 NULL pointer dereference
>>> Jul 28 19:02:19 storage3 NULL pointer dereference
>>> Jul 28 19:02:19 storage3 at 00000000000000c4
>>> Jul 28 19:02:19 storage3 at 00000000000000c4
>>> Jul 28 19:02:19 storage3 IP:
>>> Jul 28 19:02:19 storage3 IP:
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 PGD 0
>>> Jul 28 19:02:19 storage3 PGD 0
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 Oops: 0002 [#1]
>>> Jul 28 19:02:19 storage3 Oops: 0002 [#1]
>>> Jul 28 19:02:19 storage3 SMP
>>> Jul 28 19:02:19 storage3 SMP
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 last sysfs file: /sys/devices/virtual/block/drbd11/range
>>> Jul 28 19:02:19 storage3 last sysfs file: /sys/devices/virtual/block/drbd11/range
>>> Jul 28 19:02:19 storage3 CPU 1
>>> Jul 28 19:02:19 storage3 CPU 1
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 Modules linked in:
>>> Jul 28 19:02:19 storage3 Modules linked in:
>>> Jul 28 19:02:19 storage3 netconsole
>>> Jul 28 19:02:19 storage3 netconsole
>>> Jul 28 19:02:19 storage3 iscsi_scst
>>> Jul 28 19:02:19 storage3 iscsi_scst
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 scst_vdisk
>>> Jul 28 19:02:19 storage3 scst_vdisk
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 scst
>>> Jul 28 19:02:19 storage3 scst
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 crc32c_intel
>>> Jul 28 19:02:19 storage3 crc32c_intel
>>> Jul 28 19:02:19 storage3 drbd
>>> Jul 28 19:02:19 storage3 drbd
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 configfs
>>> Jul 28 19:02:19 storage3 configfs
>>> Jul 28 19:02:19 storage3 dm_mod
>>> Jul 28 19:02:19 storage3 dm_mod
>>> Jul 28 19:02:19 storage3 iw_nes
>>> Jul 28 19:02:19 storage3 iw_nes
>>> Jul 28 19:02:19 storage3 iw_cxgb3
>>> Jul 28 19:02:19 storage3 iw_cxgb3
>>> Jul 28 19:02:19 storage3 cxgb3
>>> Jul 28 19:02:19 storage3 cxgb3
>>> Jul 28 19:02:19 storage3 mdio
>>> Jul 28 19:02:19 storage3 mdio
>>> Jul 28 19:02:19 storage3 rdma_ucm
>>> Jul 28 19:02:19 storage3 rdma_ucm
>>> Jul 28 19:02:19 storage3 ib_ucm
>>> Jul 28 19:02:19 storage3 ib_ucm
>>> Jul 28 19:02:19 storage3 ib_uverbs
>>>
>>> Jul 28 19:02:19 storage3 ib_uverbs
>>> Jul 28 19:02:19 storage3 ib_srp
>>> Jul 28 19:02:19 storage3 ib_srp
>>> Jul 28 19:02:19 storage3 scsi_transport_srp
>>> Jul 28 19:02:19 storage3 scsi_transport_srp
>>> Jul 28 19:02:19 storage3 scsi_tgt
>>> Jul 28 19:02:19 storage3 scsi_tgt
>>> Jul 28 19:02:19 storage3 ib_qib
>>> Jul 28 19:02:19 storage3 ib_qib
>>> Jul 28 19:02:19 storage3 ib_iser
>>> Jul 28 19:02:19 storage3 ib_iser
>>> Jul 28 19:02:19 storage3 libiscsi
>>> Jul 28 19:02:19 storage3 libiscsi
>>> Jul 28 19:02:19 storage3 rdma_cm
>>> Jul 28 19:02:19 storage3 rdma_cm
>>> Jul 28 19:02:19 storage3 iw_cm
>>> Jul 28 19:02:19 storage3 iw_cm
>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi
>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi
>>> Jul 28 19:02:19 storage3 ib_ipath
>>> Jul 28 19:02:19 storage3 ib_ipath
>>> Jul 28 19:02:19 storage3 ib_addr
>>> Jul 28 19:02:19 storage3 ib_addr
>>> Jul 28 19:02:19 storage3 ib_ipoib
>>> Jul 28 19:02:19 storage3 ib_ipoib
>>> Jul 28 19:02:19 storage3 ib_cm
>>> Jul 28 19:02:19 storage3 ib_cm
>>> Jul 28 19:02:19 storage3 ipv6
>>> Jul 28 19:02:19 storage3 ipv6
>>> Jul 28 19:02:19 storage3 ib_umad
>>> Jul 28 19:02:19 storage3 ib_umad
>>> Jul 28 19:02:19 storage3 mlx4_ib
>>> Jul 28 19:02:19 storage3 mlx4_ib
>>> Jul 28 19:02:19 storage3 mlx4_en
>>> Jul 28 19:02:19 storage3 mlx4_en
>>> Jul 28 19:02:19 storage3 mlx4_core
>>> Jul 28 19:02:19 storage3 mlx4_core
>>> Jul 28 19:02:19 storage3 ib_sa
>>> Jul 28 19:02:19 storage3 ib_sa
>>> Jul 28 19:02:19 storage3 libcrc32c
>>> Jul 28 19:02:19 storage3 libcrc32c
>>> Jul 28 19:02:19 storage3 iTCO_wdt
>>> Jul 28 19:02:19 storage3 iTCO_wdt
>>> Jul 28 19:02:19 storage3 iTCO_vendor_support
>>> Jul 28 19:02:19 storage3 iTCO_vendor_support
>>> Jul 28 19:02:19 storage3 serio_raw
>>> Jul 28 19:02:19 storage3 serio_raw
>>> Jul 28 19:02:19 storage3 wmi
>>> Jul 28 19:02:19 storage3 wmi
>>> Jul 28 19:02:19 storage3 acpi_pad
>>> Jul 28 19:02:19 storage3 acpi_pad
>>> Jul 28 19:02:19 storage3 sg
>>> Jul 28 19:02:19 storage3 sg
>>> Jul 28 19:02:19 storage3 ib_mthca
>>> Jul 28 19:02:19 storage3 ib_mthca
>>> Jul 28 19:02:19 storage3 ib_mad
>>> Jul 28 19:02:19 storage3 ib_mad
>>> Jul 28 19:02:19 storage3 ib_core
>>> Jul 28 19:02:19 storage3 ib_core
>>> Jul 28 19:02:19 storage3 igb
>>> Jul 28 19:02:19 storage3 igb
>>> Jul 28 19:02:19 storage3 dca
>>> Jul 28 19:02:19 storage3 dca
>>> Jul 28 19:02:19 storage3 i2c_algo_bit
>>> Jul 28 19:02:19 storage3 i2c_algo_bit
>>>
>>> Jul 28 19:02:19 storage3 ptp
>>> Jul 28 19:02:19 storage3 ptp
>>> Jul 28 19:02:19 storage3 pps_core
>>> Jul 28 19:02:19 storage3 pps_core
>>> Jul 28 19:02:19 storage3 i2c_i801
>>> Jul 28 19:02:19 storage3 i2c_i801
>>> Jul 28 19:02:19 storage3 i2c_core
>>> Jul 28 19:02:19 storage3 i2c_core
>>> Jul 28 19:02:19 storage3 lpc_ich
>>> Jul 28 19:02:19 storage3 lpc_ich
>>> Jul 28 19:02:19 storage3 mfd_core
>>> Jul 28 19:02:19 storage3 mfd_core
>>> Jul 28 19:02:19 storage3 shpchp
>>> Jul 28 19:02:19 storage3 shpchp
>>> Jul 28 19:02:19 storage3 ext4
>>> Jul 28 19:02:19 storage3 ext4
>>> Jul 28 19:02:19 storage3 jbd2
>>> Jul 28 19:02:19 storage3 jbd2
>>> Jul 28 19:02:19 storage3 mbcache
>>> Jul 28 19:02:19 storage3 mbcache
>>> Jul 28 19:02:19 storage3 sd_mod
>>> Jul 28 19:02:19 storage3 sd_mod
>>> Jul 28 19:02:19 storage3 crc_t10dif
>>> Jul 28 19:02:19 storage3 crc_t10dif
>>> Jul 28 19:02:19 storage3 isci
>>> Jul 28 19:02:19 storage3 isci
>>> Jul 28 19:02:19 storage3 libsas
>>> Jul 28 19:02:19 storage3 libsas
>>> Jul 28 19:02:19 storage3 scsi_transport_sas
>>> Jul 28 19:02:19 storage3 scsi_transport_sas
>>> Jul 28 19:02:19 storage3 ahci
>>> Jul 28 19:02:19 storage3 ahci
>>> Jul 28 19:02:19 storage3 3w_sas
>>> Jul 28 19:02:19 storage3 3w_sas
>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole]
>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole]
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Not tainted 2.6.32-scst #1
>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Not tainted 2.6.32-scst #1
>>> Jul 28 19:02:19 storage3 Supermicro X9DRW
>>> Jul 28 19:02:19 storage3 Supermicro X9DRW
>>> Jul 28 19:02:19 storage3 /X9DRW
>>> Jul 28 19:02:19 storage3 /X9DRW
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 RIP: 0010:[<ffffffffa07d656d>]
>>> Jul 28 19:02:19 storage3 RIP: 0010:[<ffffffffa07d656d>]
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 RSP: 0018:ffff88045a8d7760  EFLAGS: 00010286
>>> Jul 28 19:02:19 storage3 RSP: 0018:ffff88045a8d7760  EFLAGS: 00010286
>>> Jul 28 19:02:19 storage3 RAX: 0000000000000030 RBX: 0000000000000000 RCX: ffff880460924800
>>> Jul 28 19:02:19 storage3 RAX: 0000000000000030 RBX: 0000000000000000 RCX: ffff880460924800
>>> Jul 28 19:02:19 storage3 RDX: ffff880460924940 RSI: 0000000000000000 RDI: 0000000000000000
>>> Jul 28 19:02:19 storage3 RDX: ffff880460924940 RSI: 0000000000000000 RDI: 0000000000000000
>>> Jul 28 19:02:19 storage3 RBP: ffff88045a8d7760 R08: 0000000000000000 R09: ffff88046bdc8020
>>> Jul 28 19:02:19 storage3 RBP: ffff88045a8d7760 R08: 0000000000000000 R09: ffff88046bdc8020
>>> Jul 28 19:02:19 storage3 R10: ffff880028403ec0 R11: 0000000000000000 R12: ffffea00118246c0
>>> Jul 28 19:02:19 storage3 R10: ffff880028403ec0 R11: 0000000000000000 R12: ffffea00118246c0
>>> Jul 28 19:02:19 storage3 R13: ffff8804765d55c0 R14: 000000000000000e R15: 000000000000000b
>>> Jul 28 19:02:19 storage3 R13: ffff8804765d55c0 R14: 000000000000000e R15: 000000000000000b
>>> Jul 28 19:02:19 storage3 FS:  0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
>>>
>>> Jul 28 19:02:19 storage3 FS:  0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
>>> Jul 28 19:02:19 storage3 CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
>>> Jul 28 19:02:19 storage3 CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4 CR3: 000000046216c000 CR4: 00000000001407e0
>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4 CR3: 000000046216c000 CR4: 00000000001407e0
>>> Jul 28 19:02:19 storage3 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> Jul 28 19:02:19 storage3 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> Jul 28 19:02:19 storage3 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> Jul 28 19:02:19 storage3 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> Jul 28 19:02:19 storage3 Process drbd_w_ws008 (pid: 6392, threadinfo ffff88045a8d6000, task ffff8804753ed500)
>>> Jul 28 19:02:19 storage3 Process drbd_w_ws008 (pid: 6392, threadinfo ffff88045a8d6000, task ffff8804753ed500)
>>> Jul 28 19:02:19 storage3 Stack:
>>> Jul 28 19:02:19 storage3 Stack:
>>> Jul 28 19:02:19 storage3 ffff88045a8d7790
>>> Jul 28 19:02:19 storage3 ffff88045a8d7790
>>> Jul 28 19:02:19 storage3 ffffffff8144ee30
>>> Jul 28 19:02:19 storage3 ffffffff8144ee30
>>> Jul 28 19:02:19 storage3 0000000000000000
>>> Jul 28 19:02:19 storage3 0000000000000000
>>> Jul 28 19:02:19 storage3 ffff8804765d55c0
>>> Jul 28 19:02:19 storage3 ffff8804765d55c0
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>"
>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>"
>>> Jul 28 19:02:19 storage3 0000000000000002
>>> Jul 28 19:02:19 storage3 0000000000000002
>>> Jul 28 19:02:19 storage3 ffff88046bdc86e0
>>> Jul 28 19:02:19 storage3 ffff88046bdc86e0
>>> Jul 28 19:02:19 storage3 ffff88045a8d77b0
>>> Jul 28 19:02:19 storage3 ffff88045a8d77b0
>>> Jul 28 19:02:19 storage3 ffffffff8144f5ae
>>> Jul 28 19:02:19 storage3 ffffffff8144f5ae
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>"
>>> Jul 28 19:02:19 storage3 unparseable log message: "<d>"
>>> Jul 28 19:02:19 storage3 ffff88045a8d77e0
>>> Jul 28 19:02:19 storage3 ffff88045a8d77e0
>>> Jul 28 19:02:19 storage3 ffff8804765d55c0
>>> Jul 28 19:02:19 storage3 ffff8804765d55c0
>>> Jul 28 19:02:19 storage3 ffff88045a8d77d0
>>> Jul 28 19:02:19 storage3 ffff88045a8d77d0
>>> Jul 28 19:02:19 storage3 ffffffff8144f66b
>>> Jul 28 19:02:19 storage3 ffffffff8144f66b
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 Call Trace:
>>> Jul 28 19:02:19 storage3 Call Trace:
>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] skb_release_data+0x90/0x130
>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] skb_release_data+0x90/0x130
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] __kfree_skb+0x1e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] __kfree_skb+0x1e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] consume_skb+0x3b/0x80
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] consume_skb+0x3b/0x80
>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] dev_kfree_skb_any+0x3d/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] dev_kfree_skb_any+0x3d/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>>
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>> Jul 28 19:02:19 storage3 Code:
>>> Jul 28 19:02:19 storage3 Code:
>>> Jul 28 19:02:19 storage3 89
>>> Jul 28 19:02:19 storage3 89
>>> Jul 28 19:02:19 storage3 e5
>>> Jul 28 19:02:19 storage3 e5
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 1f
>>>
>>> Jul 28 19:02:19 storage3 1f
>>> Jul 28 19:02:19 storage3 44
>>> Jul 28 19:02:19 storage3 44
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 e8
>>> Jul 28 19:02:19 storage3 e8
>>> Jul 28 19:02:19 storage3 52
>>> Jul 28 19:02:19 storage3 52
>>> Jul 28 19:02:19 storage3 e9
>>> Jul 28 19:02:19 storage3 e9
>>> Jul 28 19:02:19 storage3 ff
>>> Jul 28 19:02:19 storage3 ff
>>> Jul 28 19:02:19 storage3 ff
>>> Jul 28 19:02:19 storage3 ff
>>> Jul 28 19:02:19 storage3 31
>>> Jul 28 19:02:19 storage3 31
>>> Jul 28 19:02:19 storage3 c0
>>> Jul 28 19:02:19 storage3 c0
>>> Jul 28 19:02:19 storage3 c9
>>> Jul 28 19:02:19 storage3 c9
>>> Jul 28 19:02:19 storage3 c3
>>> Jul 28 19:02:19 storage3 c3
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 66
>>> Jul 28 19:02:19 storage3 2e
>>> Jul 28 19:02:19 storage3 2e
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 1f
>>> Jul 28 19:02:19 storage3 1f
>>> Jul 28 19:02:19 storage3 84
>>> Jul 28 19:02:19 storage3 84
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 55
>>> Jul 28 19:02:19 storage3 55
>>> Jul 28 19:02:19 storage3 48
>>> Jul 28 19:02:19 storage3 48
>>> Jul 28 19:02:19 storage3 89
>>> Jul 28 19:02:19 storage3 89
>>> Jul 28 19:02:19 storage3 e5
>>> Jul 28 19:02:19 storage3 e5
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 1f
>>> Jul 28 19:02:19 storage3 1f
>>>
>>> Jul 28 19:02:19 storage3 44
>>> Jul 28 19:02:19 storage3 44
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 48
>>> Jul 28 19:02:19 storage3 48
>>> Jul 28 19:02:19 storage3 8b
>>> Jul 28 19:02:19 storage3 8b
>>> Jul 28 19:02:19 storage3 7f
>>> Jul 28 19:02:19 storage3 7f
>>> Jul 28 19:02:19 storage3 38
>>> Jul 28 19:02:19 storage3 38
>>> Jul 28 19:02:19 storage3 unparseable log message: "<f0> "
>>> Jul 28 19:02:19 storage3 unparseable log message: "<f0> "
>>> Jul 28 19:02:19 storage3 ff
>>> Jul 28 19:02:19 storage3 ff
>>> Jul 28 19:02:19 storage3 8f
>>> Jul 28 19:02:19 storage3 8f
>>> Jul 28 19:02:19 storage3 c4
>>> Jul 28 19:02:19 storage3 c4
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 0f
>>> Jul 28 19:02:19 storage3 94
>>> Jul 28 19:02:19 storage3 94
>>> Jul 28 19:02:19 storage3 c0
>>> Jul 28 19:02:19 storage3 c0
>>> Jul 28 19:02:19 storage3 84
>>> Jul 28 19:02:19 storage3 84
>>> Jul 28 19:02:19 storage3 c0
>>> Jul 28 19:02:19 storage3 c0
>>> Jul 28 19:02:19 storage3 74
>>> Jul 28 19:02:19 storage3 74
>>> Jul 28 19:02:19 storage3 63
>>> Jul 28 19:02:19 storage3 63
>>> Jul 28 19:02:19 storage3 8b
>>> Jul 28 19:02:19 storage3 8b
>>> Jul 28 19:02:19 storage3 8f
>>> Jul 28 19:02:19 storage3 8f
>>> Jul 28 19:02:19 storage3 10
>>> Jul 28 19:02:19 storage3 10
>>> Jul 28 19:02:19 storage3 01
>>> Jul 28 19:02:19 storage3 01
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 00
>>> Jul 28 19:02:19 storage3 85
>>> Jul 28 19:02:19 storage3 85
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 RIP
>>> Jul 28 19:02:19 storage3 RIP
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 RSP <ffff88045a8d7760>
>>> Jul 28 19:02:19 storage3 RSP <ffff88045a8d7760>
>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4
>>>
>>> Jul 28 19:02:19 storage3 CR2: 00000000000000c4
>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dc ]---
>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dc ]---
>>> Jul 28 19:02:19 storage3 Kernel panic - not syncing: Fatal exception in interrupt
>>> Jul 28 19:02:19 storage3 Kernel panic - not syncing: Fatal exception in interrupt
>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G      D    ---------------    2.6.32-scst #1
>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G      D    ---------------    2.6.32-scst #1
>>> Jul 28 19:02:19 storage3 Call Trace:
>>> Jul 28 19:02:19 storage3 Call Trace:
>>> Jul 28 19:02:19 storage3 [<ffffffff81524d7d>] ? panic+0xa7/0x16f
>>> Jul 28 19:02:19 storage3 [<ffffffff81524d7d>] ? panic+0xa7/0x16f
>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180
>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180
>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130
>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80
>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0
>>>
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>> Jul 28 19:02:19 storage3 ------------[ cut here ]------------
>>> Jul 28 19:02:19 storage3 ------------[ cut here ]------------
>>> Jul 28 19:02:19 storage3 WARNING: at arch/x86/kernel/smp.c:118 native_smp_send_reschedule+0x5c/0x60() (Tainted: G      D    ---------------   )
>>> Jul 28 19:02:19 storage3 WARNING: at arch/x86/kernel/smp.c:118 native_smp_send_reschedule+0x5c/0x60() (Tainted: G      D    ---------------   )
>>> Jul 28 19:02:19 storage3 Hardware name: X9DRW
>>> Jul 28 19:02:19 storage3 Hardware name: X9DRW
>>> Jul 28 19:02:19 storage3 Modules linked in:
>>> Jul 28 19:02:19 storage3 Modules linked in:
>>> Jul 28 19:02:19 storage3 netconsole
>>> Jul 28 19:02:19 storage3 netconsole
>>> Jul 28 19:02:19 storage3 iscsi_scst
>>> Jul 28 19:02:19 storage3 iscsi_scst
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 scst_vdisk
>>> Jul 28 19:02:19 storage3 scst_vdisk
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 scst
>>> Jul 28 19:02:19 storage3 scst
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 crc32c_intel
>>>
>>> Jul 28 19:02:19 storage3 crc32c_intel
>>> Jul 28 19:02:19 storage3 drbd
>>> Jul 28 19:02:19 storage3 drbd
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 (U)
>>> Jul 28 19:02:19 storage3 configfs
>>> Jul 28 19:02:19 storage3 configfs
>>> Jul 28 19:02:19 storage3 dm_mod
>>> Jul 28 19:02:19 storage3 dm_mod
>>> Jul 28 19:02:19 storage3 iw_nes
>>> Jul 28 19:02:19 storage3 iw_nes
>>> Jul 28 19:02:19 storage3 iw_cxgb3
>>> Jul 28 19:02:19 storage3 iw_cxgb3
>>> Jul 28 19:02:19 storage3 cxgb3
>>> Jul 28 19:02:19 storage3 cxgb3
>>> Jul 28 19:02:19 storage3 mdio
>>> Jul 28 19:02:19 storage3 mdio
>>> Jul 28 19:02:19 storage3 rdma_ucm
>>> Jul 28 19:02:19 storage3 rdma_ucm
>>> Jul 28 19:02:19 storage3 ib_ucm
>>> Jul 28 19:02:19 storage3 ib_ucm
>>> Jul 28 19:02:19 storage3 ib_uverbs
>>> Jul 28 19:02:19 storage3 ib_uverbs
>>> Jul 28 19:02:19 storage3 ib_srp
>>> Jul 28 19:02:19 storage3 ib_srp
>>> Jul 28 19:02:19 storage3 scsi_transport_srp
>>> Jul 28 19:02:19 storage3 scsi_transport_srp
>>> Jul 28 19:02:19 storage3 scsi_tgt
>>> Jul 28 19:02:19 storage3 scsi_tgt
>>> Jul 28 19:02:19 storage3 ib_qib
>>> Jul 28 19:02:19 storage3 ib_qib
>>> Jul 28 19:02:19 storage3 ib_iser
>>> Jul 28 19:02:19 storage3 ib_iser
>>> Jul 28 19:02:19 storage3 libiscsi
>>> Jul 28 19:02:19 storage3 libiscsi
>>> Jul 28 19:02:19 storage3 rdma_cm
>>> Jul 28 19:02:19 storage3 rdma_cm
>>> Jul 28 19:02:19 storage3 iw_cm
>>> Jul 28 19:02:19 storage3 iw_cm
>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi
>>> Jul 28 19:02:19 storage3 scsi_transport_iscsi
>>> Jul 28 19:02:19 storage3 ib_ipath
>>> Jul 28 19:02:19 storage3 ib_ipath
>>> Jul 28 19:02:19 storage3 ib_addr
>>> Jul 28 19:02:19 storage3 ib_addr
>>> Jul 28 19:02:19 storage3 ib_ipoib
>>> Jul 28 19:02:19 storage3 ib_ipoib
>>> Jul 28 19:02:19 storage3 ib_cm
>>> Jul 28 19:02:19 storage3 ib_cm
>>> Jul 28 19:02:19 storage3 ipv6
>>> Jul 28 19:02:19 storage3 ipv6
>>> Jul 28 19:02:19 storage3 ib_umad
>>> Jul 28 19:02:19 storage3 ib_umad
>>> Jul 28 19:02:19 storage3 mlx4_ib
>>> Jul 28 19:02:19 storage3 mlx4_ib
>>> Jul 28 19:02:19 storage3 mlx4_en
>>> Jul 28 19:02:19 storage3 mlx4_en
>>> Jul 28 19:02:19 storage3 mlx4_core
>>> Jul 28 19:02:19 storage3 mlx4_core
>>> Jul 28 19:02:19 storage3 ib_sa
>>> Jul 28 19:02:19 storage3 ib_sa
>>> Jul 28 19:02:19 storage3 libcrc32c
>>> Jul 28 19:02:19 storage3 libcrc32c
>>> Jul 28 19:02:19 storage3 iTCO_wdt
>>> Jul 28 19:02:19 storage3 iTCO_wdt
>>>
>>> Jul 28 19:02:19 storage3 iTCO_vendor_support
>>> Jul 28 19:02:19 storage3 iTCO_vendor_support
>>> Jul 28 19:02:19 storage3 serio_raw
>>> Jul 28 19:02:19 storage3 serio_raw
>>> Jul 28 19:02:19 storage3 wmi
>>> Jul 28 19:02:19 storage3 wmi
>>> Jul 28 19:02:19 storage3 acpi_pad
>>> Jul 28 19:02:19 storage3 acpi_pad
>>> Jul 28 19:02:19 storage3 sg
>>> Jul 28 19:02:19 storage3 sg
>>> Jul 28 19:02:19 storage3 ib_mthca
>>> Jul 28 19:02:19 storage3 ib_mthca
>>> Jul 28 19:02:19 storage3 ib_mad
>>> Jul 28 19:02:19 storage3 ib_mad
>>> Jul 28 19:02:19 storage3 ib_core
>>> Jul 28 19:02:19 storage3 ib_core
>>> Jul 28 19:02:19 storage3 igb
>>> Jul 28 19:02:19 storage3 igb
>>> Jul 28 19:02:19 storage3 dca
>>> Jul 28 19:02:19 storage3 dca
>>> Jul 28 19:02:19 storage3 i2c_algo_bit
>>> Jul 28 19:02:19 storage3 i2c_algo_bit
>>> Jul 28 19:02:19 storage3 ptp
>>> Jul 28 19:02:19 storage3 ptp
>>> Jul 28 19:02:19 storage3 pps_core
>>> Jul 28 19:02:19 storage3 pps_core
>>> Jul 28 19:02:19 storage3 i2c_i801
>>> Jul 28 19:02:19 storage3 i2c_i801
>>> Jul 28 19:02:19 storage3 i2c_core
>>> Jul 28 19:02:19 storage3 i2c_core
>>> Jul 28 19:02:19 storage3 lpc_ich
>>> Jul 28 19:02:19 storage3 lpc_ich
>>> Jul 28 19:02:19 storage3 mfd_core
>>> Jul 28 19:02:19 storage3 mfd_core
>>> Jul 28 19:02:19 storage3 shpchp
>>> Jul 28 19:02:19 storage3 shpchp
>>> Jul 28 19:02:19 storage3 ext4
>>> Jul 28 19:02:19 storage3 ext4
>>> Jul 28 19:02:19 storage3 jbd2
>>> Jul 28 19:02:19 storage3 jbd2
>>> Jul 28 19:02:19 storage3 mbcache
>>> Jul 28 19:02:19 storage3 mbcache
>>> Jul 28 19:02:19 storage3 sd_mod
>>> Jul 28 19:02:19 storage3 sd_mod
>>> Jul 28 19:02:19 storage3 crc_t10dif
>>> Jul 28 19:02:19 storage3 crc_t10dif
>>> Jul 28 19:02:19 storage3 isci
>>> Jul 28 19:02:19 storage3 isci
>>> Jul 28 19:02:19 storage3 libsas
>>> Jul 28 19:02:19 storage3 libsas
>>> Jul 28 19:02:19 storage3 scsi_transport_sas
>>> Jul 28 19:02:19 storage3 scsi_transport_sas
>>> Jul 28 19:02:19 storage3 ahci
>>> Jul 28 19:02:19 storage3 ahci
>>> Jul 28 19:02:19 storage3 3w_sas
>>> Jul 28 19:02:19 storage3 3w_sas
>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole]
>>> Jul 28 19:02:19 storage3 [last unloaded: netconsole]
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3
>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G      D    ---------------    2.6.32-scst #1
>>> Jul 28 19:02:19 storage3 Pid: 6392, comm: drbd_w_ws008 Tainted: G      D    ---------------    2.6.32-scst #1
>>> Jul 28 19:02:19 storage3 Call Trace:
>>> Jul 28 19:02:19 storage3 Call Trace:
>>> Jul 28 19:02:19 storage3 <IRQ>
>>>
>>> Jul 28 19:02:19 storage3 <IRQ>
>>> Jul 28 19:02:19 storage3 [<ffffffff81071a17>] ? warn_slowpath_common+0x87/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81071a17>] ? warn_slowpath_common+0x87/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81071a6a>] ? warn_slowpath_null+0x1a/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff81071a6a>] ? warn_slowpath_null+0x1a/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8103094c>] ? native_smp_send_reschedule+0x5c/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8103094c>] ? native_smp_send_reschedule+0x5c/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8105dd08>] ? scheduler_tick+0x208/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff8105dd08>] ? scheduler_tick+0x208/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff810ac6c0>] ? tick_sched_timer+0x0/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff810ac6c0>] ? tick_sched_timer+0x0/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81083d8e>] ? update_process_times+0x6e/0x90
>>> Jul 28 19:02:19 storage3 [<ffffffff81083d8e>] ? update_process_times+0x6e/0x90
>>> Jul 28 19:02:19 storage3 [<ffffffff810ac726>] ? tick_sched_timer+0x66/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff810ac726>] ? tick_sched_timer+0x66/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109f2ee>] ? __run_hrtimer+0x8e/0x1a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109f2ee>] ? __run_hrtimer+0x8e/0x1a0
>>> Jul 28 19:02:19 storage3 [<ffffffff810a66ef>] ? ktime_get_update_offsets+0x4f/0xd0
>>> Jul 28 19:02:19 storage3 [<ffffffff810a66ef>] ? ktime_get_update_offsets+0x4f/0xd0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109f656>] ? hrtimer_interrupt+0xe6/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff8109f656>] ? hrtimer_interrupt+0xe6/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff81031cfd>] ? local_apic_timer_interrupt+0x3d/0x70
>>> Jul 28 19:02:19 storage3 [<ffffffff81031cfd>] ? local_apic_timer_interrupt+0x3d/0x70
>>> Jul 28 19:02:19 storage3 [<ffffffff81036f39>] ? native_apic_msr_eoi_write+0x19/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff81036f39>] ? native_apic_msr_eoi_write+0x19/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8152eb85>] ? smp_apic_timer_interrupt+0x45/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8152eb85>] ? smp_apic_timer_interrupt+0x45/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8100bb53>] ? apic_timer_interrupt+0x13/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100bb53>] ? apic_timer_interrupt+0x13/0x20
>>> Jul 28 19:02:19 storage3 <EOI>
>>> Jul 28 19:02:19 storage3 <EOI>
>>> Jul 28 19:02:19 storage3 [<ffffffff81524e22>] ? panic+0x14c/0x16f
>>> Jul 28 19:02:19 storage3 [<ffffffff81524e22>] ? panic+0x14c/0x16f
>>> Jul 28 19:02:19 storage3 [<ffffffff81524daf>] ? panic+0xd9/0x16f
>>> Jul 28 19:02:19 storage3 [<ffffffff81524daf>] ? panic+0xd9/0x16f
>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81529052>] ? oops_end+0xf2/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff81049edb>] ? no_context+0xfb/0x260
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a165>] ? __bad_area_nosemaphore+0x125/0x1e0
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e3da>] ? ipoib_send+0x36a/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a233>] ? bad_area_nosemaphore+0x13/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470
>>> Jul 28 19:02:19 storage3 [<ffffffff8104a96f>] ? __do_page_fault+0x30f/0x470
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8116eb1d>] ? __kmalloc_node+0x4d/0x60
>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180
>>> Jul 28 19:02:19 storage3 [<ffffffff8144d35a>] ? __alloc_skb+0x7a/0x180
>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8152af3e>] ? do_page_fault+0x3e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81528325>] ? page_fault+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffffa07d656d>] ? iscsi_put_page_callback+0xd/0x90 [iscsi_scst]
>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130
>>> Jul 28 19:02:19 storage3 [<ffffffff8144ee30>] ? skb_release_data+0x90/0x130
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f5ae>] ? __kfree_skb+0x1e/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80
>>> Jul 28 19:02:19 storage3 [<ffffffff8144f66b>] ? consume_skb+0x3b/0x80
>>>
>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff81459a7d>] ? dev_kfree_skb_any+0x3d/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>>
>>> Jul 28 19:02:19 storage3 [<ffffffffa033de58>] ? poll_tx+0xc8/0x190 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033e4f5>] ? ipoib_send+0x485/0x7a0 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffffa033bfd3>] ? ipoib_start_xmit+0x2b3/0x600 [ib_ipoib]
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e382>] ? dev_queue_xmit+0x122/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff8145df84>] ? dev_hard_start_xmit+0x224/0x480
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff814ac187>] ? tcp_init_tso_segs+0x37/0x50
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8147996a>] ? sch_direct_xmit+0x15a/0x1c0
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff8145e488>] ? dev_queue_xmit+0x228/0x320
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81463c1d>] ? neigh_connected_output+0xbd/0x100
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81497e87>] ? ip_finish_output+0x237/0x310
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81498018>] ? ip_output+0xb8/0xc0
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81497315>] ? ip_local_out+0x25/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff814977f0>] ? ip_queue_xmit+0x190/0x420
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814acace>] ? tcp_transmit_skb+0x40e/0x7b0
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814af030>] ? tcp_write_xmit+0x230/0xa90
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff814afbb0>] ? __tcp_push_pending_frames+0x30/0xe0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149ee33>] ? tcp_push_pending_frames+0x33/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f685>] ? do_tcp_setsockopt+0x425/0x4a0
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff8149f75a>] ? tcp_setsockopt+0x2a/0x30
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff81448214>] ? sock_common_setsockopt+0x14/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c8fab>] ? drbd_setsockopt.clone.1+0x4b/0x60 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05c90c3>] ? wait_for_work+0x103/0x2b0 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffff8109abc0>] ? autoremove_wake_function+0x0/0x40
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05ce1c6>] ? drbd_worker+0x176/0x370 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6cd>] ? drbd_thread_setup+0x2d/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffffa05eb6a0>] ? drbd_thread_setup+0x0/0x110 [drbd]
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a826>] ? kthread+0x96/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8109a790>] ? kthread+0x0/0xa0
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>> Jul 28 19:02:19 storage3 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dd ]---
>>> Jul 28 19:02:19 storage3 ---[ end trace c3efbd66c8d497dd ]---
>>>
>>>
>>> Thank you,
>>>
>>> BC
>>>
>>> _______________________________________________
>>> drbd-user mailing list
>>> drbd-user at lists.linbit.com
>>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>>
>>
>> -- 
>> Thanks,
>> //richard
>

More information about the drbd-user mailing list