Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Tue, Oct 13, 2009 at 11:04:43AM +0200, Massimo CtRiX Cetra wrote:
>
> Hello all,
>
> i just had an oops that you may find attached and which is very similar
> to this that i already posted to LKML
> (
> this is the reference:
> http://marc.info/?l=linux-kernel&m=125501488220358&w=4
> ).
>
> Kernel is 2.6.31.2, drbd is 8.3.4 which mirrors a partition with ocfs2.
>
> The oops in the LKML mail were triggered by php, this one seems to be
> triggered by DRBD.
> There is also a kernel bug entry here:
> http://bugzilla.kernel.org/show_bug.cgi?id=14378
>
> Now, before bugging again LKML people, i'm asking if this may be in some
> ways related to DRBD.
> (my guess is that it's a kernel bug).
>
> Network card is OK, as well as the switch.
>
> Max
> [60257.728500] ------------[ cut here ]------------
> [60257.728500] WARNING: at net/core/skbuff.c:398 skb_release_head_state+0x64/0xc8()
This is _NOT_ and oops.
it is a _WARNING_.
different thing.
specifically, it is this warning:
static void skb_release_head_state(struct sk_buff *skb)
{
skb_dst_drop(skb);
#ifdef CONFIG_XFRM
secpath_put(skb->sp);
#endif
if (skb->destructor) {
WARN_ON(in_irq()); <=====
skb->destructor(skb);
}
so apparently virtio_net does something
that skb_release_head_state does not expect,
namely destruct from irq context.
No, I don't think DRBD has anything in particular to do with these
warnings, we do not touch skbs directly.
If it comes down to pointing fingers,
from the information at hand, it looks like an issue with virtio_net.
possibly it is even because you try to use netconsole and virtio_net
at the same time, and they do interfere?
using netconsole to debug network issues may not be the smartest move.
don't do that, then ;)
> [60257.728500] Hardware name:
> [60257.728500] Modules linked in: ocfs2 jbd2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue crc32c netconsole configfs drbd cn loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw virtio_net pcspkr psmouse virtio_balloon parport_pc parport button processor i2c_piix4 i2c_core evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod ide_cd_mod cdrom virtio_blk ata_generic ata_piix libata scsi_mod virtio_pci virtio_ring virtio piix ide_pci_generic ide_core floppy thermal fan thermal_sys
> [60257.728500] Pid: 1814, comm: drbd1_receiver Tainted: G W 2.6.31.2-vserver-navynet #1
> [60257.728500] Call Trace:
> [60257.728500] <IRQ> [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
> [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
> [60257.728500] [<ffffffff81049ae1>] ? warn_slowpath_common+0x77/0xa3
> [60257.728500] [<ffffffff81253cd5>] ? skb_release_head_state+0x64/0xc8
> [60257.728500] [<ffffffff81253a1a>] ? __kfree_skb+0x9/0x7d
> [60257.728500] [<ffffffffa01e2139>] ? free_old_xmit_skbs+0x51/0x6e [virtio_net]
> [60257.728500] [<ffffffffa01e2c85>] ? start_xmit+0x26/0xf2 [virtio_net]
> [60257.728500] [<ffffffff8126934f>] ? netpoll_send_skb+0xd2/0x205
> [60257.728500] [<ffffffffa02a2216>] ? write_msg+0x90/0xeb [netconsole]
> [60257.728500] [<ffffffff81049f06>] ? __call_console_drivers+0x5e/0x6f
> [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
> [60257.728500] [<ffffffff8104a082>] ? release_console_sem+0x115/0x1ba
> [60257.728500] [<ffffffff8104a632>] ? vprintk+0x2f2/0x34b
> [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
> [60257.728500] [<ffffffff81308309>] ? printk+0x4e/0x5d
> [60257.728500] [<ffffffff8102b49d>] ? kvm_clock_read+0x4d/0x52
> [60257.728500] [<ffffffff81070b62>] ? getnstimeofday+0x55/0xaf
> [60257.728500] [<ffffffff81062683>] ? ktime_get_ts+0x21/0x49
> [60257.728500] [<ffffffff810626b7>] ? ktime_get+0xc/0x41
> [60257.728500] [<ffffffff81062788>] ? hrtimer_interrupt+0x9c/0x146
> [60257.728500] [<ffffffff81024a4b>] ? smp_apic_timer_interrupt+0x80/0x93
> [60257.728500] [<ffffffff81011663>] ? apic_timer_interrupt+0x13/0x20
> [60257.728500] <EOI> [<ffffffff81157811>] ? cap_socket_recvmsg+0x0/0x3
> [60257.728500] [<ffffffff8130a9eb>] ? _spin_unlock_irq+0xd/0x31
> [60257.728500] [<ffffffff8103e6de>] ? finish_task_switch+0x5b/0xec
> [60257.728500] [<ffffffff81308f73>] ? thread_return+0x47/0xd5
> [60257.728500] [<ffffffff81293cc8>] ? tcp_current_mss+0x3f/0x5a
> [60257.728500] [<ffffffff81309213>] ? schedule_timeout+0x21/0x197
> [60257.728500] [<ffffffff8104f482>] ? local_bh_disable+0xe/0x10
> [60257.728500] [<ffffffff8130a7bf>] ? _spin_lock_bh+0x13/0x29
> [60257.728500] [<ffffffff8124f6ba>] ? release_sock+0x19/0xc3
> [60257.728500] [<ffffffff8124fd9b>] ? sk_wait_data+0x85/0xca
> [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
> [60257.728500] [<ffffffff812892ab>] ? tcp_prequeue_process+0x9d/0xac
> [60257.728500] [<ffffffff8128a4ae>] ? tcp_recvmsg+0x421/0xabd
> [60257.728500] [<ffffffff8118d7cb>] ? cfq_add_rq_rb+0xd2/0xe5
> [60257.728500] [<ffffffff8124eb4a>] ? sock_common_recvmsg+0x30/0x45
> [60257.728500] [<ffffffff8124cde9>] ? sock_recvmsg+0xf7/0x18d
> [60257.728500] [<ffffffff8105f762>] ? autoremove_wake_function+0x0/0x2e
> [60257.728500] [<ffffffff8130a825>] ? _spin_lock_irqsave+0x25/0x41
> [60257.728500] [<ffffffffa0271e4d>] ? _drbd_send_cmd+0x11d/0x1eb [drbd]
> [60257.728500] [<ffffffff811851cf>] ? blk_rq_map_sg+0x12d/0x276
> [60257.728500] [<ffffffffa025c7d8>] ? drbd_recv+0x74/0x147 [drbd]
> [60257.728500] [<ffffffffa025c001>] ? drbd_may_finish_epoch+0x2c4/0x46e [drbd]
> [60257.728500] [<ffffffffa025ee5d>] ? drbd_recv_header+0x18/0xc8 [drbd]
> [60257.728500] [<ffffffffa025ef39>] ? drbdd+0x2c/0x1d6 [drbd]
> [60257.728500] [<ffffffffa026212b>] ? drbdd_init+0xf2/0x14a [drbd]
> [60257.728500] [<ffffffffa0273275>] ? drbd_thread_setup+0x16f/0x231 [drbd]
> [60257.728500] [<ffffffff81011b9a>] ? child_rip+0xa/0x20
> [60257.728500] [<ffffffff811b1e73>] ? vgacon_cursor+0x0/0x1a4
> [60257.728500] [<ffffffffa0273106>] ? drbd_thread_setup+0x0/0x231 [drbd]
> [60257.728500] [<ffffffff81011b90>] ? child_rip+0x0/0x20
> [60257.728500] ---[ end trace 694817acca794f2c ]---
120 kB of virtually identical stuff deleted.
btw, gzip does a good job in deflating repetitive logs ;)
--
: Lars Ellenberg
: LINBIT HA-Solutions GmbH
: DRBD®/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed