Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello, Just recently saw this message in my syslog, using drbd 8.4.1. As you can also see I'm runing as Xen dom0. The crash seems to be in the ip_route_output_slow call... is this a kernel bug or a drbd problem? I also attached the error in a text file for convenience. I noticed that Lars patched drbd 8.3 with a comment regarding a null pointer dereference, could this be that issue? I really would love to use 8.3, but I'm reliant on the drbd volume minor numbers going above 255. Would anyone possibly know a way to tweak 8.3 to allow larger minor numbers? Thanks, Chris The error: Jun 26 10:17:45 vicks0 kernel: [239010.496650] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c0 Jun 26 10:17:45 vicks0 kernel: [239010.496814] IP: [<ffffffff815707d5>] ip_route_output_slow+0x345/0x540 Jun 26 10:17:45 vicks0 kernel: [239010.496908] PGD 1ecb894067 PUD 1ecaf06067 PMD 0 Jun 26 10:17:45 vicks0 kernel: [239010.496996] Oops: 0000 [#1] SMP Jun 26 10:17:45 vicks0 kernel: [239010.497069] CPU 1 Jun 26 10:17:45 vicks0 kernel: [239010.497080] Modules linked in: iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ebt_arp ebt_ip xt_physdev iptable_filter ip_tables drbd(O) libcrc32c bonding ebtable_broute bridge stp ebtable_filter ebtables x_tables psmouse amd64_edac_mod serio_raw edac_core fam15h_power joydev sp5100_tco edac_mce_amd k10temp i2c_piix4 xen_wdt mac_hid xen_pciback xen_netback xen_gntdev xen_gntalloc xenfs xen_evtchn xen_blkback lp parport dm_snapshot usbhid hid mpt2sas(O) pata_atiixp scsi_transport_sas raid_class ixgbe(O) dca e1000e Jun 26 10:17:45 vicks0 kernel: [239010.497877] Jun 26 10:17:45 vicks0 kernel: [239010.497931] Pid: 27041, comm: drbd_r_r4537 Tainted: G O 3.2.0-23-generic #36-Ubuntu ASUSTeK Computer INC. KGP(M)E-D16/KGP(M)E-D16 Jun 26 10:17:45 vicks0 kernel: [239010.498084] RIP: e030:[<ffffffff815707d5>] [<ffffffff815707d5>] ip_route_output_slow+0x345/0x540 Jun 26 10:17:45 vicks0 kernel: [239010.498210] RSP: e02b:ffff8809ef679820 EFLAGS: 00010246 Jun 26 10:17:45 vicks0 kernel: [239010.498281] RAX: 0000000000000000 RBX: ffff8809f0d14658 RCX: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498395] RDX: ffff880c7801f440 RSI: 0000000000000000 RDI: ffff880c7801f440 Jun 26 10:17:45 vicks0 kernel: [239010.498509] RBP: ffff8809ef6798b0 R08: 00000000ac1111b9 R09: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498624] R10: ffff8809ef679858 R11: 00000000ac1111b9 R12: 00000000b91110ac Jun 26 10:17:45 vicks0 kernel: [239010.498738] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498857] FS: 00007f0fa16c1700(0000) GS:ffff881f9e0aa000(0000) knlGS:0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498976] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 26 10:17:45 vicks0 kernel: [239010.499048] CR2: 00000000000000c0 CR3: 0000001ec6b02000 CR4: 0000000000000660 Jun 26 10:17:45 vicks0 kernel: [239010.499167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.499282] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 26 10:17:45 vicks0 kernel: [239010.499397] Process drbd_r_r4537 (pid: 27041, threadinfo ffff8809ef678000, task ffff881ecd352de0) Jun 26 10:17:45 vicks0 kernel: [239010.499519] Stack: Jun 26 10:17:45 vicks0 kernel: [239010.499573] ffff8809ef679840 ffffffff81578909 ffff8809f25d9f00 ffff8809ef679858 Jun 26 10:17:45 vicks0 kernel: [239010.499708] ffff880900000000 b91111ac81578a6c ffff8809ef679860 ffff8809fd010010 Jun 26 10:17:45 vicks0 kernel: [239010.499842] ffff880c7801f440 ffff881ec7d8a440 ffff8809eb336598 ffff881ec9e15140 Jun 26 10:17:45 vicks0 kernel: [239010.499977] Call Trace: Jun 26 10:17:45 vicks0 kernel: [239010.500040] [<ffffffff81578909>] ? ip_local_out+0x29/0x30 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815718a0>] __ip_route_output_key+0xb0/0x1b0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815719c7>] ip_route_output_flow+0x27/0x70 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81578cbe>] ip_queue_xmit+0x3ae/0x410 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81590d99>] tcp_transmit_skb+0x359/0x580 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815926ac>] tcp_send_ack+0x8c/0xe0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81583680>] tcp_cleanup_rbuf+0x70/0x120 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8158457b>] tcp_recvmsg+0x63b/0xba0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815a853b>] inet_recvmsg+0x6b/0x80 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8165c579>] ? _raw_spin_unlock_bh+0x19/0x20 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8152780d>] sock_recvmsg+0x11d/0x140 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8100a29d>] ? xen_force_evtchn_callback+0xd/0x10 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8100aa82>] ? check_events+0x12/0x20 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8100aa6f>] ? xen_restore_fl_direct_reloc+0x4/0x4 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81004c62>] ? xen_mc_flush+0xb2/0x1c0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81003ec3>] ? xen_write_msr_safe+0xa3/0xc0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81012728>] ? __switch_to+0x138/0x360 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217712>] drbd_recv+0x82/0x140 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa02177e6>] drbd_recv_all+0x16/0x40 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217834>] drbd_recv_all_warn+0x24/0x70 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021a93f>] read_in_block+0x16f/0x350 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217712>] ? drbd_recv+0x82/0x140 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021b979>] recv_resync_read+0x29/0x230 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021bc1c>] receive_RSDataReply+0x9c/0x250 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217834>] ? drbd_recv_all_warn+0x24/0x70 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021f8d6>] drbdd+0xb6/0x170 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0220e30>] drbdd_init+0x40/0xb0 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa02294c9>] drbd_thread_setup+0xa9/0x160 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81056183>] ? finish_task_switch+0x53/0xf0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81666bf4>] kernel_thread_helper+0x4/0x10 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81664ca3>] ? int_ret_from_sys_call+0x7/0x1b Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8165c97c>] ? retint_restore_args+0x5/0x6 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81666bf0>] ? gs_change+0x13/0x13 Jun 26 10:17:45 vicks0 kernel: [239010.500069] Code: 83 79 18 01 7e 08 3c 01 0f 84 da 01 00 00 44 8b 6b 18 45 85 ed 0f 84 db 00 00 00 0f b6 4d a9 48 8d 04 c9 45 31 ff 4c 8b 6c c2 60 <41> 8b 85 c0 00 00 00 89 03 e9 cf fd ff ff 0f 1f 44 00 00 31 d2 Jun 26 10:17:45 vicks0 kernel: [239010.500069] RIP [<ffffffff815707d5>] ip_route_output_slow+0x345/0x540 Jun 26 10:17:45 vicks0 kernel: [239010.500069] RSP <ffff8809ef679820> Jun 26 10:17:45 vicks0 kernel: [239010.500069] CR2: 00000000000000c0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] ---[ end trace 903d87b699d75f9f ]--- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20120626/6bc812d8/attachment.htm> -------------- next part -------------- Jun 26 10:17:45 vicks0 kernel: [239010.496650] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c0 Jun 26 10:17:45 vicks0 kernel: [239010.496814] IP: [<ffffffff815707d5>] ip_route_output_slow+0x345/0x540 Jun 26 10:17:45 vicks0 kernel: [239010.496908] PGD 1ecb894067 PUD 1ecaf06067 PMD 0 Jun 26 10:17:45 vicks0 kernel: [239010.496996] Oops: 0000 [#1] SMP Jun 26 10:17:45 vicks0 kernel: [239010.497069] CPU 1 Jun 26 10:17:45 vicks0 kernel: [239010.497080] Modules linked in: iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ebt_arp ebt_ip xt_physdev iptable_filter ip_tables drbd(O) libcrc32c bonding ebtable_broute bridge stp ebtable_filter ebtables x_tables psmouse amd64_edac_mod serio_raw edac_core fam15h_power joydev sp5100_tco edac_mce_amd k10temp i2c_piix4 xen_wdt mac_hid xen_pciback xen_netback xen_gntdev xen_gntalloc xenfs xen_evtchn xen_blkback lp parport dm_snapshot usbhid hid mpt2sas(O) pata_atiixp scsi_transport_sas raid_class ixgbe(O) dca e1000e Jun 26 10:17:45 vicks0 kernel: [239010.497877] Jun 26 10:17:45 vicks0 kernel: [239010.497931] Pid: 27041, comm: drbd_r_r4537 Tainted: G O 3.2.0-23-generic #36-Ubuntu ASUSTeK Computer INC. KGP(M)E-D16/KGP(M)E-D16 Jun 26 10:17:45 vicks0 kernel: [239010.498084] RIP: e030:[<ffffffff815707d5>] [<ffffffff815707d5>] ip_route_output_slow+0x345/0x540 Jun 26 10:17:45 vicks0 kernel: [239010.498210] RSP: e02b:ffff8809ef679820 EFLAGS: 00010246 Jun 26 10:17:45 vicks0 kernel: [239010.498281] RAX: 0000000000000000 RBX: ffff8809f0d14658 RCX: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498395] RDX: ffff880c7801f440 RSI: 0000000000000000 RDI: ffff880c7801f440 Jun 26 10:17:45 vicks0 kernel: [239010.498509] RBP: ffff8809ef6798b0 R08: 00000000ac1111b9 R09: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498624] R10: ffff8809ef679858 R11: 00000000ac1111b9 R12: 00000000b91110ac Jun 26 10:17:45 vicks0 kernel: [239010.498738] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498857] FS: 00007f0fa16c1700(0000) GS:ffff881f9e0aa000(0000) knlGS:0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.498976] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 26 10:17:45 vicks0 kernel: [239010.499048] CR2: 00000000000000c0 CR3: 0000001ec6b02000 CR4: 0000000000000660 Jun 26 10:17:45 vicks0 kernel: [239010.499167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 26 10:17:45 vicks0 kernel: [239010.499282] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 26 10:17:45 vicks0 kernel: [239010.499397] Process drbd_r_r4537 (pid: 27041, threadinfo ffff8809ef678000, task ffff881ecd352de0) Jun 26 10:17:45 vicks0 kernel: [239010.499519] Stack: Jun 26 10:17:45 vicks0 kernel: [239010.499573] ffff8809ef679840 ffffffff81578909 ffff8809f25d9f00 ffff8809ef679858 Jun 26 10:17:45 vicks0 kernel: [239010.499708] ffff880900000000 b91111ac81578a6c ffff8809ef679860 ffff8809fd010010 Jun 26 10:17:45 vicks0 kernel: [239010.499842] ffff880c7801f440 ffff881ec7d8a440 ffff8809eb336598 ffff881ec9e15140 Jun 26 10:17:45 vicks0 kernel: [239010.499977] Call Trace: Jun 26 10:17:45 vicks0 kernel: [239010.500040] [<ffffffff81578909>] ? ip_local_out+0x29/0x30 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815718a0>] __ip_route_output_key+0xb0/0x1b0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815719c7>] ip_route_output_flow+0x27/0x70 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81578cbe>] ip_queue_xmit+0x3ae/0x410 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81590d99>] tcp_transmit_skb+0x359/0x580 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815926ac>] tcp_send_ack+0x8c/0xe0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81583680>] tcp_cleanup_rbuf+0x70/0x120 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8158457b>] tcp_recvmsg+0x63b/0xba0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff815a853b>] inet_recvmsg+0x6b/0x80 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8165c579>] ? _raw_spin_unlock_bh+0x19/0x20 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8152780d>] sock_recvmsg+0x11d/0x140 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8100a29d>] ? xen_force_evtchn_callback+0xd/0x10 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8100aa82>] ? check_events+0x12/0x20 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8100aa6f>] ? xen_restore_fl_direct_reloc+0x4/0x4 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81004c62>] ? xen_mc_flush+0xb2/0x1c0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81003ec3>] ? xen_write_msr_safe+0xa3/0xc0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81012728>] ? __switch_to+0x138/0x360 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217712>] drbd_recv+0x82/0x140 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa02177e6>] drbd_recv_all+0x16/0x40 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217834>] drbd_recv_all_warn+0x24/0x70 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021a93f>] read_in_block+0x16f/0x350 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217712>] ? drbd_recv+0x82/0x140 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021b979>] recv_resync_read+0x29/0x230 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021bc1c>] receive_RSDataReply+0x9c/0x250 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0217834>] ? drbd_recv_all_warn+0x24/0x70 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa021f8d6>] drbdd+0xb6/0x170 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa0220e30>] drbdd_init+0x40/0xb0 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffffa02294c9>] drbd_thread_setup+0xa9/0x160 [drbd] Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81056183>] ? finish_task_switch+0x53/0xf0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81666bf4>] kernel_thread_helper+0x4/0x10 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81664ca3>] ? int_ret_from_sys_call+0x7/0x1b Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff8165c97c>] ? retint_restore_args+0x5/0x6 Jun 26 10:17:45 vicks0 kernel: [239010.500069] [<ffffffff81666bf0>] ? gs_change+0x13/0x13 Jun 26 10:17:45 vicks0 kernel: [239010.500069] Code: 83 79 18 01 7e 08 3c 01 0f 84 da 01 00 00 44 8b 6b 18 45 85 ed 0f 84 db 00 00 00 0f b6 4d a9 48 8d 04 c9 45 31 ff 4c 8b 6c c2 60 <41> 8b 85 c0 00 00 00 89 03 e9 cf fd ff ff 0f 1f 44 00 00 31 d2 Jun 26 10:17:45 vicks0 kernel: [239010.500069] RIP [<ffffffff815707d5>] ip_route_output_slow+0x345/0x540 Jun 26 10:17:45 vicks0 kernel: [239010.500069] RSP <ffff8809ef679820> Jun 26 10:17:45 vicks0 kernel: [239010.500069] CR2: 00000000000000c0 Jun 26 10:17:45 vicks0 kernel: [239010.500069] ---[ end trace 903d87b699d75f9f ]---