Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I would guess that you have TSO and scatter/gather enabled in your Dom0 -- this crash is an unfortunate confluence of a number of behaviours that is only exposed when you are using Xen _and_ have scatter/gather enabled _and_ lose the link to the peer. The problem is that when you have scatter/gather enabled, DRBD uses the zero-copy interface to the network stack -- the network stack has the annoying tendency to keep a reference to the pages of requests even after the TCP connection they were sent on is terminated _but_ DRBD makes the assumption that once the TCP connection to the peer is terminated, the pages of all pending requests are no longer referenced by the network stack and it can safely complete any pending I/O requests upwards. Normally this is fine too, _but_ when you are running Xen and the request comes from blkback you will have a problem because when the I/O is completed to blkback, it unmaps the grant reference it has on the guests pages -- this effectively removes the page from Dom0 -- later on, if the network stack decides to use the cached page it has and you blow up with this oops. So.. there are several problem areas here that conspire: 1. DRBD is violating the block I/O contract -- when it completes an I/O, NOONE underneath it should have a reference to any of the pages in the request - this is only a problem when zero-copy network send is used. 2. blkback is freeing (unmapping) pages when they are still referenced by lower layers -- really it should not do this until all the underlying references are gone (but it's assuming the lower layer followed the block I/O contract) I think you have to disable scatter/gather support in the network layer until a proper fix can be determined... Simon > -----Original Message----- > From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user- > bounces at lists.linbit.com] On Behalf Of Valentin Vidic > Sent: Tuesday, March 10, 2009 8:56 AM > To: drbd-user at lists.linbit.com > Subject: [DRBD-user] Kernel panic in skb_copy_bits > > Hi, > > Recently I've started getting the following kernel panic: > > [ 366.834266] drbd2: peer( Secondary -> Primary ) > [ 383.920322] drbd1: PingAck did not arrive in time. > [ 383.925828] drbd1: peer( Secondary -> Unknown ) conn( Connected -> > NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 ) > [ 383.941857] drbd1: asender terminated > [ 383.943390] drbd1: short read expecting header on sock: r=-512 > [ 383.958256] drbd1: Terminating asender thread > [ 383.967285] drbd1: Creating new current UUID > [ 383.975075] drbd1: Connection closed > [ 383.979220] drbd1: helper command: /sbin/drbdadm outdate-peer minor- > 1 > [ 383.993212] drbd1: helper command: /sbin/drbdadm outdate-peer minor- > 1 exit code 5 (0x500) > [ 384.003508] drbd1: outdate-peer helper returned 5 (peer is > unreachable, assumed to be dead) > [ 384.013728] drbd1: pdsk( DUnknown -> Outdated ) > [ 384.020778] drbd1: susp( 1 -> 0 ) > [ 384.031938] drbd1: conn( NetworkFailure -> Unconnected ) > [ 384.038132] drbd1: receiver terminated > [ 384.042484] drbd1: Restarting receiver thread > [ 384.047507] drbd1: receiver (re)started > [ 384.051945] drbd1: conn( Unconnected -> WFConnection ) > [ 388.193364] drbd2: PingAck did not arrive in time. > [ 388.198874] drbd2: peer( Primary -> Unknown ) conn( Connected -> > NetworkFailure ) pdsk( UpToDate -> DUnknown ) > [ 388.218319] drbd2: asender terminated > [ 388.218336] drbd2: short read expecting header on sock: r=-512 > [ 388.229284] drbd2: Terminating asender thread > [ 388.237285] drbd2: Connection closed > [ 388.241294] drbd2: conn( NetworkFailure -> Unconnected ) > [ 388.246663] drbd2: receiver terminated > [ 388.250993] drbd2: Restarting receiver thread > [ 388.256014] drbd2: receiver (re)started > [ 388.260454] drbd2: conn( Unconnected -> WFConnection ) > [ 390.258485] drbd0: PingAck did not arrive in time. > [ 390.264035] drbd0: peer( Secondary -> Unknown ) conn( Connected -> > NetworkFailure ) pdsk( UpToDate -> DUnknown ) susp( 0 -> 1 ) > [ 390.279992] drbd0: asender terminated > [ 390.279999] drbd0: short read expecting header on sock: r=-512 > [ 390.287996] drbd0: Creating new current UUID > [ 390.298491] drbd0: Terminating asender thread > [ 390.303527] drbd0: Connection closed > [ 390.307532] drbd0: helper command: /sbin/drbdadm outdate-peer minor- > 0 > [ 390.322060] drbd0: helper command: /sbin/drbdadm outdate-peer minor- > 0 exit code 5 (0x500) > [ 390.332083] drbd0: outdate-peer helper returned 5 (peer is > unreachable, assumed to be dead) > [ 390.347642] drbd0: pdsk( DUnknown -> Outdated ) > [ 390.352958] drbd0: susp( 1 -> 0 ) > [ 390.357022] drbd0: conn( NetworkFailure -> Unconnected ) > [ 390.363215] drbd0: receiver terminated > [ 390.370564] drbd0: Restarting receiver thread > [ 390.375561] drbd0: receiver (re)started > [ 390.379983] drbd0: conn( Unconnected -> WFConnection ) > [ 393.705742] device vif4.0 entered promiscuous mode > [ 393.719435] xenbr0: port 2(vif4.0) entering learning state > [ 393.725380] drbd2: susp( 0 -> 1 ) > [ 393.725411] drbd2: helper command: /sbin/drbdadm outdate-peer minor- > 2 > [ 393.732256] drbd2: helper command: /sbin/drbdadm outdate-peer minor- > 2 exit code 5 (0x500) > [ 393.732266] drbd2: outdate-peer helper returned 5 (peer is > unreachable, assumed to be dead) > [ 393.732280] drbd2: role( Secondary -> Primary ) pdsk( DUnknown -> > Outdated ) > [ 393.732304] drbd2: susp( 1 -> 0 ) > [ 393.732497] drbd2: Creating new current UUID > [ 393.803216] xenbr0: topology change detected, propagating > [ 393.810787] xenbr0: port 2(vif4.0) entering forwarding state > [ 394.209915] blkback: ring-ref 8, event-channel 20, protocol 1 > (x86_64-abi) > [ 395.180474] BUG: unable to handle kernel paging request at > ffff88001ddbf000 > [ 395.188634] IP: [<ffffffff803c0011>] skb_copy_bits+0x139/0x216 > [ 395.188634] PGD 1a6a067 PUD 1a6b067 PMD 1b5a067 PTE 0 > [ 395.200918] Oops: 0000 [1] SMP > [ 395.200918] CPU 0 > [ 395.200918] Modules linked in: xt_physdev sha1_generic drbd cn > ipt_REJECT xt_tcpudp xt_multiport nf_conntrack_ipv4 xt_state > nf_conntrack iptable_filter ip_tables x_tables ipv6 bridge bonding > dm_mod ipmi_si ipmi_devintf ipmi_msghandler 8021q loop parport_pc > psmouse usbhid rng_core i2c_i801 hid parport container i2c_core > serio_raw ff_memless button pcspkr iTCO_wdt i5000_edac edac_core shpchp > pci_hotplug evdev ext3 jbd mbcache sg sr_mod cdrom ide_pci_generic > ide_core ata_piix sd_mod ses enclosure floppy ata_generic libata bnx2 > dock ehci_hcd firmware_class uhci_hcd megaraid_sas scsi_mod thermal > processor fan thermal_sys > [ 395.280972] Pid: 0, comm: swapper Not tainted 2.6.26-1-xen-amd64 #1 > [ 395.280972] RIP: e030:[<ffffffff803c0011>] [<ffffffff803c0011>] > skb_copy_bits+0x139/0x216 > [ 395.280972] RSP: e02b:ffffffff80595c40 EFLAGS: 00010286 > [ 395.280972] RAX: 0000000000000062 RBX: ffff88001e4623a8 RCX: > 0000000000000588 > [ 395.280972] RDX: ffff880001554800 RSI: ffff88001ddbf000 RDI: > ffff8800015541a0 > [ 395.280972] RBP: 0000000000000588 R08: 0000000000000588 R09: > 00000000000005ea > [ 395.280972] R10: ffff880080000000 R11: ffff88001e4623a8 R12: > 0000000000000062 > [ 395.280972] R13: 0000000000000001 R14: 0000000000000062 R15: > ffff8800015541a0 > [ 395.280972] FS: 00007f23afba4730(0000) GS:ffffffff80539000(0000) > knlGS:0000000000000000 > [ 395.280972] CS: e033 DS: 0000 ES: 0000 > [ 395.280972] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 395.280972] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > [ 395.280972] Process swapper (pid: 0, threadinfo ffffffff80552000, > task ffffffff804fe460) > [ 395.280972] Stack: ffff880001554000 ffffffff803c0c2c > 0000000000000010 ffff88001e136000 > [ 395.280972] ffff88001e4623a8 ffff88001e4623a8 0000000000000000 > 0000000000000000 > [ 395.280972] 00000000000005a8 ffffffff803c0d17 0000000200000000 > ffff88001e136000 > [ 395.280972] Call Trace: > [ 395.280972] <IRQ> [<ffffffff803c0c2c>] ? > pskb_expand_head+0xde/0x143 > [ 395.280972] [<ffffffff803c0d17>] ? __pskb_pull_tail+0x86/0x290 > [ 395.280972] [<ffffffff803c8a60>] ? dev_queue_xmit+0x153/0x3eb > [ 395.280972] [<ffffffff803e9dbd>] ? ip_queue_xmit+0x29a/0x2ed > [ 395.280972] [<ffffffff8040a69b>] ? > inet_sk_rebuild_header+0xf2/0x32f > [ 395.280972] [<ffffffff8020e7b4>] ? get_nsec_offset+0x9/0x2c > [ 395.280972] [<ffffffff8020e7b4>] ? get_nsec_offset+0x9/0x2c > [ 395.280972] [<ffffffff8020e810>] ? local_clock+0x39/0x83 > [ 395.280972] [<ffffffff803f9d18>] ? tcp_transmit_skb+0x731/0x76e > [ 395.280972] [<ffffffff8020e8e3>] ? sched_clock+0x15/0x36 > [ 395.280972] [<ffffffff803fa9f7>] ? tcp_retransmit_skb+0x4a7/0x5b1 > [ 395.280972] [<ffffffff803fd039>] ? tcp_write_timer+0x557/0x77e > [ 395.280972] [<ffffffff8020ee50>] ? timer_interrupt+0x401/0x415 > [ 395.280972] [<ffffffff80235dd6>] ? __mod_timer+0xd4/0xe3 > [ 395.280972] [<ffffffff803fcae2>] ? tcp_write_timer+0x0/0x77e > [ 395.280972] [<ffffffff8023569f>] ? run_timer_softirq+0x190/0x237 > [ 395.280972] [<ffffffff80231c94>] ? __do_softirq+0x77/0x103 > [ 395.280972] [<ffffffff8020c13c>] ? call_softirq+0x1c/0x28 > [ 395.280972] [<ffffffff8020e08a>] ? do_softirq+0x55/0xbb > [ 395.280972] [<ffffffff8020e16d>] ? do_IRQ+0x7d/0x9a > [ 395.280972] [<ffffffff8037d6c4>] ? evtchn_do_upcall+0x13c/0x1fc > [ 395.280972] [<ffffffff8020bbde>] ? do_hypervisor_callback+0x1e/0x30 > [ 395.280972] <EOI> [<ffffffff8020e795>] ? xen_safe_halt+0x90/0xa6 > [ 395.280972] [<ffffffff8020a0c8>] ? xen_idle+0x2e/0x66 > [ 395.280972] [<ffffffff80209cd6>] ? cpu_idle+0x97/0xb9 > [ 395.280972] > [ 395.280972] > [ 395.280972] Code: 00 00 00 00 00 88 ff ff 48 c1 e6 0c 48 01 c6 8b 83 > c8 00 00 00 8b 44 07 20 4c 89 ff 48 01 c6 49 63 c4 48 01 c6 49 63 c6 48 > 29 c6 <f3> a4 65 48 8b 04 25 10 00 00 00 ff 88 44 e0 ff ff 44 29 c5 0f > [ 395.280972] RIP [<ffffffff803c0011>] skb_copy_bits+0x139/0x216 > [ 395.280972] RSP <ffffffff80595c40> > [ 395.280972] CR2: ffff88001ddbf000 > [ 395.280972] ---[ end trace 3b4a27cfc0a95b4b ]--- > [ 395.280972] Kernel panic - not syncing: Aiee, killing interrupt > handler! > > It happens on two nodes running Xen on DRBD managed by Heartbeat: > * node1 with drbd0 and drbd1 as Primary > * node2 with drbd2 as Primary > > The above output happens if I kill node2. Device drbd2 is then > activated on node1 and domU is started on it but right after that > kernel panics and node1 also reboots. The problem is reproducible and > the kernel panic output is always the same. > > I get this with different versions of Debian kernels (2.6.18, 2.6.26), > Xen (3.0.3, 3.2) and DRBD (8.0.13, 8.0.14). > > Any ideas what could be causing this? Could it be that node1 is > trying to retransmit a DRBD/TCP packet on a broken connection to > node2? > > -- > Valentin > _______________________________________________ > drbd-user mailing list > drbd-user at lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user