Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi, I'm using DRBD 8.4.8-1 with linux-2.6.32 kernel, on 2 nodes. # cat /proc/drbd version: 8.4.8-1 (api:1/proto:86-101) GIT-hash: 22b4c802192646e433d3f7399d578ec7fecc6272 When running a kind of system test (detach/attach loop in high system load), NULL pointer deref occurs on a node at drbd_submit_peer_request. One thing I notice is the following 2 assertion failure about the same drbd device (drbd6) is seen many times before NULL deref occurs. Jan 18 12:19:29 HOSTA_101 kernel: : [161157.608191] block drbd6: ASSERT( i >= 0 ) in drivers/block/drbd/drbd_int.h:2270 Jan 18 12:19:29 HOSTA_101 kernel: : [161158.332840] block drbd6: ASSERT( atomic_read(&device->local_cnt) ) in drivers/block/drbd/drbd_actlog.c:691 These makes me the suspicion that device->local_cnt is invalid count. (the former assertion failure occurs in put_ldev() in drbd code I'm using). Here's syslog snippet of BUG. Jan 18 12:20:38 HOSTA_101 kernel: : [161228.600846] BUG: unable to handle kernel NULL pointer dereference at (null) Jan 18 12:20:38 HOSTA_101 kernel: : [161228.601807] IP: [<ffffffffa010f07d>] drbd_submit_peer_request+0x8d/0x4c0 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.610078] PGD 765c6067 PUD 7645d067 PMD 0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.610078] Oops: 0000 [#1] PREEMPT SMP Jan 18 12:20:38 HOSTA_101 kernel: : [161228.610078] last sysfs file: /sys/devices/virtual/block/drbd7/removable Jan 18 12:20:38 HOSTA_101 kernel: : [161228.644817] CPU 4 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.660713] Modules linked in: e1000 igb mlx4_core mlx4_en virtio_net virtio_balloon ipmi_msghandler ipmi_watchdog kplugdr libcrc32c crc32c drbd scsi_transport_iscsi libiscsi libiscsi_tcp iscsi_tcp [last unloaded: ipmi_msghandler] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.692781] Pid: 25503, comm: drbd_r_r4 Tainted: G W 2.6.32.59.cge 60-A64-N1.07 KVM Jan 18 12:20:38 HOSTA_101 kernel: : [161228.710234] RIP: 0010:[<ffffffffa010f07d>] [<ffffffffa010f07d>] drbd_submit_peer_request+0x8d/0x4c0 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.714138] RSP: 0018:ffff88004fc09d90 EFLAGS: 00010286 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.725343] RAX: 0000000000000000 RBX: ffff88004fc09e60 RCX: ffff8800376bf8c0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.725343] RDX: ffff88007b863800 RSI: 0000000000011200 RDI: 0000000000000246 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.738464] RBP: ffff88004fc09df0 R08: 0000000000000000 R09: 0000000000000000 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.738464] R10: 0000000000000000 R11: 0000000000000000 R12: ffffea0001a1b550 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.757103] R13: 0000000000064000 R14: 0000000000043400 R15: 0000000000000064 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.777606] FS: 0000000000000000(0000) GS:ffff880001d00000(0000) knlGS:0000000000000000 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.783633] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Jan 18 12:20:38 HOSTA_101 kernel: : [161228.796394] CR2: 0000000000000000 CR3: 000000004f345000 CR4: 00000000000406e0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.804852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.809335] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.809335] Process drbd_r_r4 (pid: 25503, threadinfo ffff88004fc08000, task ffff88007b6bb2a0) Jan 18 12:20:38 HOSTA_101 kernel: : [161228.838090] Stack: Jan 18 12:20:38 HOSTA_101 kernel: : [161228.838090] 0000000000000000 0000000000000001 0000000000000002 ffff88007b863800 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.838090] <0> ffff88004ed31810 ffff8800376bf8c0 ffff88004fc09df0 ffff88004fc09e60 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.838090] <0> ffff88007b863800 ffff88004ed31810 ffff88007b863800 0000000000043400 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.852971] Call Trace: Jan 18 12:20:38 HOSTA_101 kernel: : [161228.856040] [<ffffffffa01126ab>] receive_RSDataReply+0x13b/0x490 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.856040] [<ffffffffa0110880>] drbd_receiver+0x100/0x2e0 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.856040] [<ffffffffa0124740>] ? drbd_thread_setup+0x0/0x110 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.856040] [<ffffffffa012476d>] drbd_thread_setup+0x2d/0x110 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.856040] [<ffffffffa0124740>] ? drbd_thread_setup+0x0/0x110 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.871833] [<ffffffff811251b6>] kthread+0x96/0xa0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.871833] [<ffffffff81125120>] ? kthread+0x0/0xa0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.871833] [<ffffffff810b445a>] child_rip+0xa/0x20 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.880486] [<ffffffff81125120>] ? kthread+0x0/0xa0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.880486] [<ffffffff81125120>] ? kthread+0x0/0xa0 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.885895] [<ffffffff810b4450>] ? child_rip+0x0/0x20 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.887926] Code: 00 00 00 e8 76 45 0f e1 48 85 c0 48 89 45 c8 4c 8b 4d a0 0f 84 f6 03 00 00 48 8b 4d c8 4c 89 31 48 8b 55 b8 48 8b 82 80 00 00 00 <48> 8b 00 48 89 41 10 48 8b 45 a8 48 89 41 20 48 8b 55 c0 48 c7 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.894609] RIP [<ffffffffa010f07d>] drbd_submit_peer_request+0x8d/0x4c0 [drbd] Jan 18 12:20:38 HOSTA_101 kernel: : [161228.894609] RSP <ffff88004fc09d90> Jan 18 12:20:38 HOSTA_101 kernel: : [161228.894609] CR2: 0000000000000000 Jan 18 12:20:38 HOSTA_101 kernel: : [161228.924389] ---[ end trace d6d8759a31519f4d ]--- Any help really appreciated. Thanks a lot. Regards, Tadashi