<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Greetings,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My team is seeing kernel panics when using DRBD inside an RHCS cluster. These appear quite similar, perhaps identical, to the panics reported by others on this list. We would greatly appreciate any assistance with them.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Environment:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">CentOS 6.0<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Kernel 2.6.32-71.29.1.el6.x86_64<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">RHCS 3.0.12<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">DRBD 8.4.0 (built from sources, on the target system)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Steps to reproduce:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Configure and start a two-node RHCS cluster with DRBD as a cluster resource.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Shut down the node on which DRBD is primary (“node 1”). (Backup node “2” takes over.)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">Start node 1 back up.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in">As soon as node 1 reaches the DRBD init script, node 2 panics. 100% reproducible.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Panic details:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">BUG: unable to handle kernel NULL pointer dereference at 0000000000000038<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">IP: [<ffffffff813fda60>] sock_ioctl+0x30/0x280<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">PGD 211f3b067 PUD 1c97fd067 PMD 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Oops: 0000 [#1] SMP
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">last sysfs file: /sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/local_cpus<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">CPU 0
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Modules linked in: dlm configfs sctp drbd(U) libcrc32c sunrpc ipv6 vsock(U) ext3 jbd xfs exportfs dm_mirror dm_region_hash dm_log ppdev
parport_pc parport e1000 vmware_balloon vmci(U) i2c_piix4 i2c_core sg shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi ata_generic pata_acpi ata_piix dm_mod [last unloaded: configfs]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Pid: 8640, comm: drbdadm Not tainted 2.6.32-71.29.1.el6.x86_64 #1 VMware Virtual Platform<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RIP: 0010:[<ffffffff813fda60>] [<ffffffff813fda60>] sock_ioctl+0x30/0x280<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RSP: 0018:ffff88016a889e38 EFLAGS: 00010282<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RAX: 0000000000000000 RBX: 0000000000005401 RCX: 00007fffbd9d44d0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RDX: 00007fffbd9d44d0 RSI: 0000000000005401 RDI: ffff88021c751c00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RBP: ffff88016a889e58 R08: ffffffff81536380 R09: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">R10: 0000000000000001 R11: 0000000000000001 R12: 00007fffbd9d44d0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">R13: 00007fffbd9d44d0 R14: ffff880221da8a00 R15: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">FS: 00007fe9a7eff700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">CR2: 0000000000000038 CR3: 000000016a8f5000 CR4: 00000000000006f0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Process drbdadm (pid: 8640, threadinfo ffff88016a888000, task ffff88016f5d8100)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Stack:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">ffff88021c751c00 ffff880221da8a48 00007fffbd9d44d0 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New""><0> ffff88016a889e98 ffffffff8117fdf2 ffff880236c4d880 ffff880221dc3740<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New""><0> 0000000000000000 0000000000000000 0000000000000000 ffff88021c751c00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Call Trace:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">[<ffffffff8117fdf2>] vfs_ioctl+0x22/0xa0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">[<ffffffff8117ff94>] do_vfs_ioctl+0x84/0x580<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">[<ffffffff81180511>] sys_ioctl+0x81/0xa0<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">[<ffffffff81013172>] system_call_fastpath+0x16/0x1b<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">Code: 83 ec 20 48 89 1c 24 4c 89 64 24 08 4c 89 6c 24 10 4c 89 74 24 18 0f 1f 44 00 00 4c 8b b7 a0 00 00 00 89 f3 49 89 d4 49 8b 46 38
<4c> 8b 68 38 8d 83 10 76 ff ff 83 f8 0f 76 51 8d 83 00 75 ff ff <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RIP [<ffffffff813fda60>] sock_ioctl+0x30/0x280<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">RSP <ffff88016a889e38><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New"">CR2: 0000000000000038<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">All help is greatly appreciated....<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Steven Roth<o:p></o:p></p>
<p class="MsoNormal">Hewlett-Packard Company<o:p></o:p></p>
<p class="MsoNormal"><a href="mailto:steve.roth@hp.com">steve.roth@hp.com</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>