<div dir="ltr"><div><div><div><div>Hello,<br><br>I get the following kernel panic (and automatic reboot) while the node is becoming primary. The info below appears on the console then the node reboots instantly (I have captured it using a serial port console). <br>
</div>Here is the full output from tail -f /var/log/messages :<br><br>Jun 17 15:36:38 NODE-CONTB kernel: drbd: initialized. Version: 8.4.4 (api:1/proto:86-101)<br>Jun 17 15:36:38 NODE-CONTB kernel: drbd: GIT-hash: 599f286440bd633d15d5ff985204aff4bccffadd build by phil@Build64R6, 2013-10-14 15:33:06<br>
Jun 17 15:36:38 NODE-CONTB kernel: drbd: registered as block device major 147<br>Jun 17 15:36:38 NODE-CONTB multipathd: drbd0: add path (uevent)<br>Jun 17 15:36:38 NODE-CONTB multipathd: drbd0: failed to get path uid<br>Jun 17 15:36:38 NODE-CONTB multipathd: uevent trigger error<br>
Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Starting worker thread (from drbdsetup [7425])<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: disk( Diskless -> Attaching ) <br>Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Method to ensure write ordering: drain<br>
Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: max BIO size = 1048576<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: drbd_bm_resize called with capacity == 209708728<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: resync bitmap: bits=26213591 words=409588 pages=800<br>
Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: size = 100 GB (104854364 KB)<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: bitmap READ of 800 pages took 30 jiffies<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: recounting of set bits took additional 4 jiffies<br>
Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: 104 MB (26624 bits) marked out-of-sync by on disk bit-map.<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: disk( Attaching -> UpToDate ) <br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: attached to UUIDs 825D19B481BE18C5:0000000000000000:85A1FF27F6205E06:7A8DAA94E8F6092A<br>
Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: conn( StandAlone -> Unconnected ) <br>Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Starting receiver thread (from drbd_w_repdata [7427])<br>Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: receiver (re)started<br>
Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: conn( Unconnected -> WFConnection ) <br>WARN: stdin/stdout is not a TTY; using /dev/consoleJun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Handshake successful: Agreed network protocol version 101<br>
Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Agreed to support TRIM on protocol level<br>Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Peer authenticated using 20 bytes HMAC<br>Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: conn( WFConnection -> WFReportParams ) <br>
Jun 17 15:36:38 NODE-CONTB kernel: drbd repdata: Starting asender thread (from drbd_r_repdata [7442])<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: drbd_sync_handshake:<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: self 825D19B481BE18C4:0000000000000000:85A1FF27F6205E06:7A8DAA94E8F6092A bits:26624 flags:0<br>
Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: peer 825D19B481BE18C4:0000000000000000:85A1FF27F6205E06:7A8DAA94E8F6092A bits:0 flags:0<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: uuid_compare()=1 by rule 40<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent ) <br>
Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 133(1), total 133; compression: 100.0%<br>Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 133(1), total 133; compression: 100.0%<br>
Jun 17 15:36:38 NODE-CONTB kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0<br>BUG: unable to handle kernel NULL pointer dereference at 0000000000000038<br>IP: [<ffffffff81446dd0>] sock_ioctl+0x30/0x280<br>
PGD 900ece067 PUD 900eda067 PMD 0 <br>Oops: 0000 [#1] SMP <br>last sysfs file: /sys/devices/virtual/block/drbd0/range<br>CPU 25 <br>Modules linked in: drbd(U) autofs4 sctp libcrc32c dlm configfs dm_round_robin cpufreq_ondemand freq_table pcc_cpufreq bonding 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_multipath microcode power_meter iTCO_wdt iTCO_vendor_support hpilo hpwdt osst st ch tg3 ptp pps_core sg serio_raw lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom mptsas mptscsih mptbase scsi_transport_sas hpsa pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]<br>
<br>Pid: 7451, comm: drbdadm Not tainted 2.6.32-431.5.1.el6.x86_64 #1 HP ProLiant DL360p Gen8<br>RIP: 0010:[<ffffffff81446dd0>] [<ffffffff81446dd0>] sock_ioctl+0x30/0x280<br>RSP: 0018:ffff880900f7de38 EFLAGS: 00010282<br>
RAX: 0000000000000000 RBX: 0000000000005401 RCX: 00007fff07020d00<br>RDX: 00007fff07020d00 RSI: 0000000000005401 RDI: ffff88092cbbe980<br>RBP: ffff880900f7de58 R08: ffffffff8166f200 R09: 0000000000000000<br>R10: 0000000000000001 R11: 0000000000000001 R12: 00007fff07020d00<br>
R13: 00007fff07020d00 R14: ffff8809295c8780 R15: 0000000000000000<br>FS: 00007fb831ebf700(0000) GS:ffff880049320000(0000) knlGS:0000000000000000<br>CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br>CR2: 0000000000000038 CR3: 000000092acb2000 CR4: 00000000000407e0<br>
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br>Process drbdadm (pid: 7451, threadinfo ffff880900f7c000, task ffff880914940080)<br>
Stack:<br> ffff88092cbbe980 ffff8809295c87c8 00007fff07020d00 0000000000000000<br><d> ffff880900f7de98 ffffffff8119db32 ffff8809304c4880 ffff88092d02c780<br><d> 0000000000000000 0000000000000000 0000000000000000 ffff88092cbbe980<br>
Call Trace:<br> [<ffffffff8119db32>] vfs_ioctl+0x22/0xa0<br> [<ffffffff8119dcd4>] do_vfs_ioctl+0x84/0x580<br> [<ffffffff8119e251>] sys_ioctl+0x81/0xa0<br> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br>
Code: 83 ec 20 48 89 1c 24 4c 89 64 24 08 4c 89 6c 24 10 4c 89 74 24 18 0f 1f 44 00 00 4c 8b b7 a0 00 00 00 89 f3 49 89 d4 49 8b 46 38 <4c> 8b 68 38 8d 83 10 76 ff ff 83 f8 0f 76 51 8d 83 00 75 ff ff <br>RIP [<ffffffff81446dd0>] sock_ioctl+0x30/0x280<br>
RSP <ffff880900f7de38><br>CR2: 0000000000000038<br><br><br></div>The system is running RHEL 6.5 with DRBD 8.4 from EPEL. In this setup I have DRBD replicating between two SAN-backed RedHat clusters (similar with the setup from <a href="http://www.drbd.org/users-guide/s-pacemaker-floating-peers.html">http://www.drbd.org/users-guide/s-pacemaker-floating-peers.html</a> but with RH cluster instead of Pacemaker). The kernel panic occurs on the second node while trying to start the drbd service after I forcefully reboot the first cluster node that was running DRBD. I can failover gracefully the DRBD service between the two nodes but rebooting one node will cause a kernel panic on the other node while starting DRBD. <br>
</div>I am not even sure that the problem is in DRBD or in the kernel but the process seems to be drbdadm. <br></div>Have you seen something similar?<br><br>Best regards,<br><br>Radu<br><div><br><div><br><div><div><div><div>
<br></div></div></div></div></div></div></div>