Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello, After upgrading a pair of Xen/DRBD machines from Debian lenny to squeeze they started rebooting spontaneously once or twice a day. One machine reboots first and the other one follows some 10 seconds later. Kernel panic traces causing the reboot are attached. It looks like the skb_copy_bits problem again but the solution from etch/lenny (ethtool -K xenbr0 sg off) doesn't work any more: http://www.gossamer-threads.com/lists/drbd/users/16962 Or could this be some new problem altogether? Kernel and DRBD versions involved: /proc/version Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-31) (ben at decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Tue Mar 8 00:01:30 UTC 2011 /proc/drbd version: 8.3.7 (api:88/proto:86-91) srcversion: EE47D8BF18AC166BE219757 dpkg -l ii drbd8-utils 2:8.3.7-2.1 RAID 1 over tcp/ip for Linux utilities drbdadm -V DRBDADM_BUILDTAG=GIT-hash:\ ea9e28dbff98e331a62bcbcc63a6135808fe2917\ build\ by\ root at NB-XP13822\,\ 2010-07-15\ 12:07:52 DRBDADM_API_VERSION=88 DRBD_KERNEL_VERSION_CODE=0x080307 DRBDADM_VERSION_CODE=0x080307 DRBDADM_VERSION=8.3.7 -- Valentin -------------- next part -------------- [68024.237607] BUG: unable to handle kernel paging request at ffff88001db01028 [68024.240422] IP: [<ffffffff81256620>] skb_copy_bits+0x114/0x1e0 [68024.240422] PGD 1002067 PUD 1006067 PMD 1cf067 PTE 10000004887065 [68024.240422] Oops: 0000 [#1] SMP [68024.240422] last sysfs file: /sys/hypervisor/version/minor [68024.240422] CPU 0 [68024.240422] Modules linked in: xt_physdev hmac sha1_generic ipmi_devintf drbd lru_cache cn xen_evtchn xenfs ip6t_REJECT ip6t_LOG ipt_REJECT ipt_LOG xt_tcpudp xt_multiport nf_conntrack_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge bonding 8021q garp stp loop radeon ttm snd_pcm snd_timer drm_kms_helper psmouse snd drm soundcore i2c_algo_bit snd_page_alloc ipmi_si i2c_core hpilo i5000_edac ipmi_msghandler pcspkr evdev serio_raw edac_core hpwdt rng_core i5k_amb shpchp pci_hotplug container processor acpi_processor button ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod usbhid hid sg sr_mod cdrom ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore nls_base bnx2 cciss scsi_mod thermal thermal_sys [last unloaded: scsi_wait_scan] [68024.240422] Pid: 2495, comm: drbd6_receiver Not tainted 2.6.32-5-xen-amd64 #1 ProLiant DL360 G5 [68024.240422] RIP: e030:[<ffffffff81256620>] [<ffffffff81256620>] skb_copy_bits+0x114/0x1e0 [68024.240422] RSP: e02b:ffff88001f76b680 EFLAGS: 00010206 [68024.240422] RAX: ffff88001db01000 RBX: ffff880012707030 RCX: 00000000000005a8 [68024.240422] RDX: 0000000000000042 RSI: ffff88001db01028 RDI: ffff880012707140 [68024.240422] RBP: 00000000000005a8 R08: 0000000000000020 R09: 0000000000000014 [68024.240422] R10: dead000000200200 R11: dead000000100100 R12: 0000000000000042 [68024.240422] R13: 00000000000005a8 R14: 0000000000000042 R15: ffff8800027c9ce8 [68024.240422] FS: 00007fa47966b700(0000) GS:ffff880003322000(0000) knlGS:0000000000000000 [68024.240422] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [68024.240422] CR2: ffff88001db01028 CR3: 000000001f7a2000 CR4: 0000000000002660 [68024.240422] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [68024.240422] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [68024.240422] Process drbd6_receiver (pid: 2495, threadinfo ffff88001f76a000, task ffff880001c20e20) [68024.240422] Stack: [68024.240422] ffff880012707000 ffff88001f76a000 ffff880012707140 000005ea8127ce7e [68024.240422] <0> 0000000000000000 ffff88001e88410c ffff88001e884120 ffff8800027c9ce8 [68024.240422] <0> ffff8800027c9ce8 0000000000000000 00000000000005a8 ffffffff8125824a [68024.240422] Call Trace: [68024.240422] [<ffffffff8125824a>] ? __pskb_pull_tail+0x81/0x28f [68024.240422] [<ffffffff81287dc6>] ? ip_finish_output+0x0/0x64 [68024.240422] [<ffffffff812601a9>] ? dev_queue_xmit+0xae/0x38d [68024.240422] [<ffffffff812879c3>] ? ip_queue_xmit+0x311/0x386 [68024.240422] [<ffffffff8128f2e5>] ? sk_stream_alloc_skb+0x2f/0xd5 [68024.240422] [<ffffffff810e86da>] ? __kmalloc_node_track_caller+0xfa/0x11b [68024.240422] [<ffffffff8128f2e5>] ? sk_stream_alloc_skb+0x2f/0xd5 [68024.240422] [<ffffffff81298373>] ? tcp_transmit_skb+0x648/0x687 [68024.240422] [<ffffffff81299c55>] ? tcp_retransmit_skb+0x4a1/0x5aa [68024.240422] [<ffffffff81299ef6>] ? tcp_xmit_retransmit_queue+0x198/0x223 [68024.240422] [<ffffffff81295ac3>] ? tcp_ack+0x1744/0x1952 [68024.240422] [<ffffffff8129a3e7>] ? tcp_write_xmit+0x466/0x96c [68024.240422] [<ffffffff81296221>] ? tcp_validate_incoming+0x1ba/0x2be [68024.240422] [<ffffffff8129690e>] ? tcp_rcv_established+0x5e9/0x6d9 [68024.240422] [<ffffffff8129d9bf>] ? tcp_v4_do_rcv+0x1bb/0x376 [68024.240422] [<ffffffff8130d199>] ? _spin_lock_bh+0x9/0x25 [68024.240422] [<ffffffff8128d76e>] ? tcp_prequeue_process+0x73/0x89 [68024.240422] [<ffffffff8128e9be>] ? tcp_recvmsg+0x53e/0xa9e [68024.240422] [<ffffffff81252b6a>] ? sock_common_recvmsg+0x30/0x45 [68024.240422] [<ffffffff812510b9>] ? sock_recvmsg+0xa6/0xbe [68024.240422] [<ffffffff81065eb2>] ? autoremove_wake_function+0x0/0x2e [68024.240422] [<ffffffff811803c5>] ? generic_unplug_device+0x32/0x34 [68024.240422] [<ffffffffa00f62c4>] ? dm_table_unplug_all+0x4b/0xb4 [dm_mod] [68024.240422] [<ffffffffa034196f>] ? drbd_recv+0x74/0x147 [drbd] [68024.240422] [<ffffffffa00f4ad0>] ? dm_unplug_all+0x3b/0x4c [dm_mod] [68024.240422] [<ffffffffa0341a5a>] ? drbd_recv_header+0x18/0xc8 [drbd] [68024.240422] [<ffffffffa0348f61>] ? drbdd_init+0x111/0x292 [drbd] [68024.240422] [<ffffffffa0355701>] ? drbd_thread_setup+0x2b/0xf8 [drbd] [68024.240422] [<ffffffffa03556d6>] ? drbd_thread_setup+0x0/0xf8 [drbd] [68024.240422] [<ffffffff81065be5>] ? kthread+0x79/0x81 [68024.240422] [<ffffffff81012baa>] ? child_rip+0xa/0x20 [68024.240422] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b [68024.240422] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 [68024.240422] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 [68024.240422] Code: ff 41 8b 97 cc 00 00 00 49 03 9f d0 00 00 00 49 63 f4 49 63 cd 48 8b 7c 24 10 8b 54 13 10 48 01 d6 49 63 d6 48 29 d6 48 8d 34 30 <f3> a4 48 8b 44 24 08 ff 48 1c 44 29 ed 0f 84 a1 00 00 00 45 01 [68024.240422] RIP [<ffffffff81256620>] skb_copy_bits+0x114/0x1e0 [68024.240422] RSP <ffff88001f76b680> [68024.240422] CR2: ffff88001db01028 [68024.240422] ---[ end trace 4d2cdb50ee807d85 ]--- [68024.240422] Kernel panic - not syncing: Fatal exception in interrupt [68024.240422] Pid: 2495, comm: drbd6_receiver Tainted: G D 2.6.32-5-xen-amd64 #1 [68024.240422] Call Trace: [68024.240422] [<ffffffff8130b298>] ? panic+0x86/0x143 [68024.240422] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [68024.240422] [<ffffffff811bad25>] ? vgacon_cursor+0x0/0x140 [68024.240422] [<ffffffff8100e63d>] ? xen_force_evtchn_callback+0x9/0xa [68024.240422] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [68024.240422] [<ffffffff811bad25>] ? vgacon_cursor+0x0/0x140 [68024.240422] [<ffffffff8130d13a>] ? _spin_unlock_irqrestore+0xd/0xe [68024.240422] [<ffffffff8130dfd5>] ? oops_end+0xa7/0xb4 [68024.240422] [<ffffffff810333a4>] ? no_context+0x1e9/0x1f8 [68024.240422] [<ffffffff81033559>] ? __bad_area_nosemaphore+0x1a6/0x1ca [68024.240422] [<ffffffffa02e5abc>] ? ipt_do_table+0x5ee/0x61e [ip_tables] [68024.240422] [<ffffffff8100c369>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [68024.240422] [<ffffffff81032cc7>] ? spurious_fault+0x12c/0x1c4 [68024.240422] [<ffffffff8130f39f>] ? do_page_fault+0x69/0x2fc [68024.240422] [<ffffffff8130d4b5>] ? page_fault+0x25/0x30 [68024.240422] [<ffffffff81256620>] ? skb_copy_bits+0x114/0x1e0 [68024.240422] [<ffffffff8125824a>] ? __pskb_pull_tail+0x81/0x28f [68024.240422] [<ffffffff81287dc6>] ? ip_finish_output+0x0/0x64 [68024.240422] [<ffffffff812601a9>] ? dev_queue_xmit+0xae/0x38d [68024.240422] [<ffffffff812879c3>] ? ip_queue_xmit+0x311/0x386 [68024.240422] [<ffffffff8128f2e5>] ? sk_stream_alloc_skb+0x2f/0xd5 [68024.240422] [<ffffffff810e86da>] ? __kmalloc_node_track_caller+0xfa/0x11b [68024.240422] [<ffffffff8128f2e5>] ? sk_stream_alloc_skb+0x2f/0xd5 [68024.240422] [<ffffffff81298373>] ? tcp_transmit_skb+0x648/0x687 [68024.240422] [<ffffffff81299c55>] ? tcp_retransmit_skb+0x4a1/0x5aa [68024.240422] [<ffffffff81299ef6>] ? tcp_xmit_retransmit_queue+0x198/0x223 [68024.240422] [<ffffffff81295ac3>] ? tcp_ack+0x1744/0x1952 [68024.240422] [<ffffffff8129a3e7>] ? tcp_write_xmit+0x466/0x96c [68024.240422] [<ffffffff81296221>] ? tcp_validate_incoming+0x1ba/0x2be [68024.240422] [<ffffffff8129690e>] ? tcp_rcv_established+0x5e9/0x6d9 [68024.240422] [<ffffffff8129d9bf>] ? tcp_v4_do_rcv+0x1bb/0x376 [68024.240422] [<ffffffff8130d199>] ? _spin_lock_bh+0x9/0x25 [68024.240422] [<ffffffff8128d76e>] ? tcp_prequeue_process+0x73/0x89 [68024.240422] [<ffffffff8128e9be>] ? tcp_recvmsg+0x53e/0xa9e [68024.240422] [<ffffffff81252b6a>] ? sock_common_recvmsg+0x30/0x45 [68024.240422] [<ffffffff812510b9>] ? sock_recvmsg+0xa6/0xbe [68024.240422] [<ffffffff81065eb2>] ? autoremove_wake_function+0x0/0x2e [68024.240422] [<ffffffff811803c5>] ? generic_unplug_device+0x32/0x34 [68024.240422] [<ffffffffa00f62c4>] ? dm_table_unplug_all+0x4b/0xb4 [dm_mod] [68024.240422] [<ffffffffa034196f>] ? drbd_recv+0x74/0x147 [drbd] [68024.240422] [<ffffffffa00f4ad0>] ? dm_unplug_all+0x3b/0x4c [dm_mod] [68024.240422] [<ffffffffa0341a5a>] ? drbd_recv_header+0x18/0xc8 [drbd] [68024.240422] [<ffffffffa0348f61>] ? drbdd_init+0x111/0x292 [drbd] [68024.240422] [<ffffffffa0355701>] ? drbd_thread_setup+0x2b/0xf8 [drbd] [68024.240422] [<ffffffffa03556d6>] ? drbd_thread_setup+0x0/0xf8 [drbd] [68024.240422] [<ffffffff81065be5>] ? kthread+0x79/0x81 [68024.240422] [<ffffffff81012baa>] ? child_rip+0xa/0x20 [68024.240422] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b [68024.240422] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 [68024.240422] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 -------------- next part -------------- [68004.752025] block drbd3: PingAck did not arrive in time. [68004.754879] block drbd3: short read expecting header on sock: r=-512 [68009.460027] block drbd1: PingAck did not arrive in time. [68009.462843] block drbd1: short read expecting header on sock: r=-512 [68010.536026] block drbd0: PingAck did not arrive in time. [68010.538870] block drbd0: short read expecting header on sock: r=-512 [68010.896026] block drbd4: PingAck did not arrive in time. [68010.898842] block drbd4: short read expecting header on sock: r=-512 [68011.956026] block drbd5: PingAck did not arrive in time. [68011.958890] block drbd5: short read expecting header on sock: r=-512 [68011.976027] block drbd6: PingAck did not arrive in time. [68011.979813] block drbd6: error receiving Data, l: 4120! [68016.648063] BUG: unable to handle kernel paging request at ffff88001d9cb000 [68016.651738] IP: [<ffffffff81256620>] skb_copy_bits+0x114/0x1e0 [68016.652009] PGD 1002067 PUD 1006067 PMD 1ce067 PTE 0 [68016.652009] Oops: 0000 [#1] SMP [68016.652009] last sysfs file: /sys/hypervisor/version/minor [68016.652009] CPU 1 [68016.652009] Modules linked in: xt_physdev hmac sha1_generic ipmi_devintf drbd lru_cache cn xen_evtchn xenfs ip6t_REJECT ip6t_LOG ipt_REJECT ipt_LOG xt_tcpudp xt_multiport nf_conntrack_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_comment ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge bonding 8021q garp stp loop radeon ttm snd_pcm drm_kms_helper snd_timer drm i5000_edac snd soundcore i2c_algo_bit hpwdt edac_core psmouse snd_page_alloc hpilo serio_raw ipmi_si i2c_core evdev pcspkr ipmi_msghandler i5k_amb rng_core shpchp processor pci_hotplug container acpi_processor button ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod usbhid hid sg sr_mod cdrom ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore nls_base bnx2 cciss scsi_mod thermal thermal_sys [last unloaded: scsi_wait_scan] [68016.652009] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 #1 ProLiant DL360 G5 [68016.652009] RIP: e030:[<ffffffff81256620>] [<ffffffff81256620>] skb_copy_bits+0x114/0x1e0 [68016.652009] RSP: e02b:ffff880003343ba0 EFLAGS: 00010246 [68016.652009] RAX: ffff88001d9cb000 RBX: ffff88001f3f1040 RCX: 0000000000000588 [68016.652009] RDX: 0000000000000062 RSI: ffff88001d9cb000 RDI: ffff88001f3f1160 [68016.652009] RBP: 0000000000000588 R08: 0000000000000020 R09: ffffffff814eeb50 [68016.652009] R10: 000000000000000e R11: ffffffff812600fb R12: 0000000000000062 [68016.652009] R13: 0000000000000588 R14: 0000000000000062 R15: ffff88001507aae8 [68016.652009] FS: 00007f81c9af3740(0000) GS:ffff880003340000(0000) knlGS:0000000000000000 [68016.652009] CS: e033 DS: 002b ES: 002b CR0: 000000008005003b [68016.652009] CR2: ffff88001d9cb000 CR3: 000000001e15a000 CR4: 0000000000002660 [68016.652009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [68016.652009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [68016.652009] Process swapper (pid: 0, threadinfo ffff88001fd4e000, task ffff88001fd40e20) [68016.652009] Stack: [68016.652009] ffff88001f3f1000 ffff88001fd4e000 ffff88001f3f1160 000005ea8127ce7e [68016.652009] <0> 0000000000000001 ffff88001ed3950c ffff88001ed39520 ffff88001507aae8 [68016.652009] <0> ffff88001507aae8 0000000000000000 00000000000005a8 ffffffff8125824a [68016.652009] Call Trace: [68016.652009] <IRQ> [68016.652009] [<ffffffff8125824a>] ? __pskb_pull_tail+0x81/0x28f [68016.652009] [<ffffffff81287dc6>] ? ip_finish_output+0x0/0x64 [68016.652009] [<ffffffff812601a9>] ? dev_queue_xmit+0xae/0x38d [68016.652009] [<ffffffff812879c3>] ? ip_queue_xmit+0x311/0x386 [68016.652009] [<ffffffff8102ddcc>] ? pvclock_clocksource_read+0x3a/0x8b [68016.652009] [<ffffffff812a9e2d>] ? inet_sk_rebuild_header+0x30/0x358 [68016.652009] [<ffffffff81066fd5>] ? run_posix_cpu_timers+0x25/0x6ea [68016.652009] [<ffffffff8106a1ff>] ? sched_clock_local+0x13/0x74 [68016.652009] [<ffffffff81298373>] ? tcp_transmit_skb+0x648/0x687 [68016.652009] [<ffffffff812975c1>] ? tcp_established_options+0x2d/0xa9 [68016.652009] [<ffffffff81299c55>] ? tcp_retransmit_skb+0x4a1/0x5aa [68016.652009] [<ffffffff8129b685>] ? tcp_write_timer+0x0/0x197 [68016.652009] [<ffffffff8129b18b>] ? tcp_retransmit_timer+0x46f/0x56b [68016.652009] [<ffffffff8129b71b>] ? tcp_write_timer+0x96/0x197 [68016.652009] [<ffffffff8105b5ff>] ? run_timer_softirq+0x1c9/0x268 [68016.652009] [<ffffffff81054cab>] ? __do_softirq+0xdd/0x1a6 [68016.652009] [<ffffffff81012cac>] ? call_softirq+0x1c/0x30 [68016.652009] [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c [68016.652009] [<ffffffff81054b1b>] ? irq_exit+0x36/0x76 [68016.652009] [<ffffffff811f2269>] ? xen_evtchn_do_upcall+0x33/0x42 [68016.652009] [<ffffffff81012cfe>] ? xen_do_hypervisor_callback+0x1e/0x30 [68016.652009] <EOI> [68016.652009] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001 [68016.652009] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001 [68016.652009] [<ffffffff8100ec1a>] ? xen_vcpuop_set_next_event+0x0/0x60 [68016.652009] [<ffffffff8100e6bb>] ? xen_safe_halt+0xc/0x15 [68016.652009] [<ffffffff8100bfc7>] ? xen_idle+0x37/0x40 [68016.652009] [<ffffffff81010eb1>] ? cpu_idle+0xa2/0xda [68016.652009] [<ffffffff8100ec99>] ? xen_irq_enable_direct_end+0x0/0x7 [68016.652009] [<ffffffff813019e6>] ? cpu_bringup+0x6d/0x72 [68016.652009] Code: ff 41 8b 97 cc 00 00 00 49 03 9f d0 00 00 00 49 63 f4 49 63 cd 48 8b 7c 24 10 8b 54 13 10 48 01 d6 49 63 d6 48 29 d6 48 8d 34 30 <f3> a4 48 8b 44 24 08 ff 48 1c 44 29 ed 0f 84 a1 00 00 00 45 01 [68016.652009] RIP [<ffffffff81256620>] skb_copy_bits+0x114/0x1e0 [68016.652009] RSP <ffff880003343ba0> [68016.652009] CR2: ffff88001d9cb000 [68016.652009] ---[ end trace e2e5c512f09892f2 ]--- [68016.652009] Kernel panic - not syncing: Fatal exception in interrupt [68016.652009] Pid: 0, comm: swapper Tainted: G D 2.6.32-5-xen-amd64 #1 [68016.652009] Call Trace: [68016.652009] <IRQ> [<ffffffff8130b298>] ? panic+0x86/0x143 [68016.652009] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [68016.652009] [<ffffffff811bad25>] ? vgacon_cursor+0x0/0x140 [68016.652009] [<ffffffff8100e63d>] ? xen_force_evtchn_callback+0x9/0xa [68016.652009] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [68016.652009] [<ffffffff811bad25>] ? vgacon_cursor+0x0/0x140 [68016.652009] [<ffffffff8130d13a>] ? _spin_unlock_irqrestore+0xd/0xe [68016.652009] [<ffffffff8130dfd5>] ? oops_end+0xa7/0xb4 [68016.652009] [<ffffffff810333a4>] ? no_context+0x1e9/0x1f8 [68016.652009] [<ffffffff81033559>] ? __bad_area_nosemaphore+0x1a6/0x1ca [68016.652009] [<ffffffffa02ebabc>] ? ipt_do_table+0x5ee/0x61e [ip_tables] [68016.652009] [<ffffffff8100c369>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [68016.652009] [<ffffffff812600fb>] ? dev_queue_xmit+0x0/0x38d [68016.652009] [<ffffffff81032cc7>] ? spurious_fault+0x12c/0x1c4 [68016.652009] [<ffffffff8130f39f>] ? do_page_fault+0x69/0x2fc [68016.652009] [<ffffffff8130d4b5>] ? page_fault+0x25/0x30 [68016.652009] [<ffffffff812600fb>] ? dev_queue_xmit+0x0/0x38d [68016.652009] [<ffffffff81256620>] ? skb_copy_bits+0x114/0x1e0 [68016.652009] [<ffffffff8125824a>] ? __pskb_pull_tail+0x81/0x28f [68016.652009] [<ffffffff81287dc6>] ? ip_finish_output+0x0/0x64 [68016.652009] [<ffffffff812601a9>] ? dev_queue_xmit+0xae/0x38d [68016.652009] [<ffffffff812879c3>] ? ip_queue_xmit+0x311/0x386 [68016.652009] [<ffffffff8102ddcc>] ? pvclock_clocksource_read+0x3a/0x8b [68016.652009] [<ffffffff812a9e2d>] ? inet_sk_rebuild_header+0x30/0x358 [68016.652009] [<ffffffff81066fd5>] ? run_posix_cpu_timers+0x25/0x6ea [68016.652009] [<ffffffff8106a1ff>] ? sched_clock_local+0x13/0x74 [68016.652009] [<ffffffff81298373>] ? tcp_transmit_skb+0x648/0x687 [68016.652009] [<ffffffff812975c1>] ? tcp_established_options+0x2d/0xa9 [68016.652009] [<ffffffff81299c55>] ? tcp_retransmit_skb+0x4a1/0x5aa [68016.652009] [<ffffffff8129b685>] ? tcp_write_timer+0x0/0x197 [68016.652009] [<ffffffff8129b18b>] ? tcp_retransmit_timer+0x46f/0x56b [68016.652009] [<ffffffff8129b71b>] ? tcp_write_timer+0x96/0x197 [68016.652009] [<ffffffff8105b5ff>] ? run_timer_softirq+0x1c9/0x268 [68016.652009] [<ffffffff81054cab>] ? __do_softirq+0xdd/0x1a6 [68016.652009] [<ffffffff81012cac>] ? call_softirq+0x1c/0x30 [68016.652009] [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c [68016.652009] [<ffffffff81054b1b>] ? irq_exit+0x36/0x76 [68016.652009] [<ffffffff811f2269>] ? xen_evtchn_do_upcall+0x33/0x42 [68016.652009] [<ffffffff81012cfe>] ? xen_do_hypervisor_callback+0x1e/0x30 [68016.652009] <EOI> [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001 [68016.652009] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001 [68016.652009] [<ffffffff8100ec1a>] ? xen_vcpuop_set_next_event+0x0/0x60 [68016.652009] [<ffffffff8100e6bb>] ? xen_safe_halt+0xc/0x15 [68016.652009] [<ffffffff8100bfc7>] ? xen_idle+0x37/0x40 [68016.652009] [<ffffffff81010eb1>] ? cpu_idle+0xa2/0xda [68017.030437] [<ffffffff8100ec99>] ? xen_irq_enable_direct_end+0x0/0x7 [68017.030437] [<ffffffff813019e6>] ? cpu_bringup+0x6d/0x72