<div><div>Hi folks,</div><div><br></div><div>We are running a file-server cluster using centos6, drbd, corosync, and pacemaker. Each time drbd is started, a kernel oops is triggered (see below). This doesn't seem to cause the system any problems as it completes the sync normally and the system is then stable & functioning. I'm thinking that this is a known issue ( <a href="http://lists.linbit.com/pipermail/drbd-user/2011-August/016628.html">http://lists.linbit.com/pipermail/drbd-user/2011-August/016628.html</a> & <a href="http://elrepo.org/bugs/view.php?id=210">http://elrepo.org/bugs/view.php?id=210</a>) but I'm nervous that this is something more serious. My question is, is this something that can be safely ignored or if not, how to approach fixing it?</div>
<div><br></div><div>Thanks!</div><div>-Ryan</div><div><br></div><div><div>I have the 'no-disk-barrier' option enabled in /etc/drbd.conf.</div><div><div><br></div><div>[admin@idaho ~]$ rpm -qa |grep drbd</div><div>
<div><div>drbd83-utils-8.3.12-1.el6.elrepo.i686</div><div>kmod-drbd83-8.3.12-1.el6.elrepo.i686</div></div><div><br></div><div><div>[admin@idaho ~]$ more /etc/redhat-release </div><div>CentOS release 6.2 (Final)</div></div>
</div><div><br></div><div><div>[admin@idaho ~]$ uname -r</div><div>2.6.32-220.7.1.el6.i686</div></div></div></div><div><br></div><div><br></div><div>/var/log/messages output:</div><div>Apr 4 10:53:42 idaho kernel: drbd: initialized. Version: 8.3.12 (api:88/proto:86-96)</div>
</div><div>Apr 4 10:53:42 idaho kernel: drbd: GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by dag@Build32R6, 2011-11-20 10:55:07</div><div>Apr 4 10:53:42 idaho kernel: drbd: registered as block device major 147</div>
<div>Apr 4 10:53:42 idaho kernel: drbd: minor_table @ 0xf3beea00</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: Starting worker thread (from cqueue [2042])</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: disk( Diskless -> Attaching )</div>
<div>Apr 4 10:53:42 idaho kernel: block drbd0: Found 6 transactions (324 active extents) in activity log.</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: Method to ensure write ordering: flush</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: max BIO size = 65536</div>
<div>Apr 4 10:53:42 idaho kernel: block drbd0: drbd_bm_resize called with capacity == 1757746384</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: resync bitmap: bits=219718298 words=6866198 pages=6706</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: size = 838 GB (878873192 KB)</div>
<div>Apr 4 10:53:42 idaho attrd: [1796]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)</div><div>Apr 4 10:53:42 idaho lrmd: [1795]: info: RA output: (drbd0:0:start:stdout) #012#012</div>
<div>Apr 4 10:53:42 idaho attrd: [1796]: notice: attrd_perform_update: Sent update 4: probe_complete=true</div><div>Apr 4 10:53:42 idaho crmd: [1798]: info: process_lrm_event: LRM operation kansas_stonith_start_0 (call=13, rc=0, cib-update=17, confirmed=true) ok</div>
<div>Apr 4 10:53:42 idaho kernel: block drbd0: bitmap READ of 6706 pages took 302 jiffies</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: recounting of set bits took additional 57 jiffies</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div>
<div>Apr 4 10:53:42 idaho kernel: block drbd0: disk( Attaching -> Outdated )</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: attached to UUIDs 8CBCEE5F323F2202:0000000000000000:37E7C7784E3A2062:37E6C7784E3A2063</div>
<div>Apr 4 10:53:42 idaho kernel: ------------[ cut here ]------------</div><div>Apr 4 10:53:42 idaho kernel: WARNING: at block/blk-core.c:1296 __make_request+0x4fa/0x570() (Not tainted)</div><div>Apr 4 10:53:42 idaho kernel: Hardware name: PowerEdge 1850</div>
<div>Apr 4 10:53:42 idaho kernel: block: BARRIER is deprecated, use FLUSH/FUA instead</div><div>Apr 4 10:53:42 idaho kernel: Modules linked in: drbd(U) autofs4 bonding ipv6 ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_multiport iptable_filter ip_tables nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs e1000 microcode dcdbas sg serio_raw iTCO_wdt iTCO_vendor_support e752x_edac edac_core ext4 mbcache jbd2 sd_mod crc_t10dif megaraid_mbox megaraid_mm sr_mod cdrom pata_sil680 video output pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]</div>
<div>Apr 4 10:53:42 idaho kernel: Pid: 2042, comm: cqueue Not tainted 2.6.32-220.7.1.el6.i686 #1</div><div>Apr 4 10:53:42 idaho kernel: Call Trace:</div><div><div>Apr 4 10:53:42 idaho kernel: [<c0454d71>] ? warn_slowpath_common+0x81/0xc0</div>
<div>Apr 4 10:53:42 idaho kernel: [<c05dfe6a>] ? __make_request+0x4fa/0x570</div><div>Apr 4 10:53:42 idaho kernel: [<c05dfe6a>] ? __make_request+0x4fa/0x570</div><div>Apr 4 10:53:42 idaho kernel: [<c0454e43>] ? warn_slowpath_fmt+0x33/0x40</div>
<div>Apr 4 10:53:42 idaho kernel: [<c05dfe6a>] ? __make_request+0x4fa/0x570</div><div>Apr 4 10:53:42 idaho kernel: [<c047ba2e>] ? up+0x2e/0x40</div><div>Apr 4 10:53:42 idaho kernel: [<f7ff7276>] ? drbd_queue_work+0x36/0x60 [drbd]</div>
<div>Apr 4 10:53:42 idaho kernel: [<f7ffd32d>] ? __drbd_set_state+0x80d/0x1010 [drbd]</div><div>Apr 4 10:53:42 idaho kernel: [<c05de835>] ? generic_make_request+0x395/0x600</div><div>Apr 4 10:53:42 idaho kernel: [<c0427ea3>] ? lapic_next_event+0x13/0x20</div>
<div>Apr 4 10:53:42 idaho kernel: [<c048401c>] ? clockevents_program_event+0x8c/0x120</div><div>Apr 4 10:53:42 idaho kernel: [<c05deb19>] ? submit_bio+0x79/0x120</div><div>Apr 4 10:53:42 idaho kernel: [<c0558da8>] ? bio_add_page+0x38/0x50</div>
<div>Apr 4 10:53:42 idaho kernel: [<f7ff4995>] ? _drbd_md_sync_page_io+0x105/0x310 [drbd]</div><div>Apr 4 10:53:42 idaho kernel: [<f7ff4ce4>] ? drbd_md_sync_page_io+0x144/0x610 [drbd]</div><div>Apr 4 10:53:42 idaho kernel: [<f7ff2603>] ? drbd_md_get_buffer+0x33/0x100 [drbd]</div>
<div>Apr 4 10:53:42 idaho kernel: [<c0465065>] ? del_timer+0x65/0xb0</div><div>Apr 4 10:53:42 idaho kernel: [<f7ffdd29>] ? drbd_md_sync+0x1f9/0x640 [drbd]</div><div>Apr 4 10:53:42 idaho kernel: [<f800789e>] ? _drbd_set_state.clone.0+0x4e/0x60 [drbd]</div>
<div>Apr 4 10:53:42 idaho kernel: [<f800d9a0>] ? drbd_nl_disk_conf+0x1040/0x11d0 [drbd]</div><div>Apr 4 10:53:42 idaho kernel: [<c04083a7>] ? __switch_to+0xd7/0x1a0</div><div>Apr 4 10:53:42 idaho kernel: [<c082e960>] ? schedule+0x3c0/0xae0</div>
<div>Apr 4 10:53:42 idaho kernel: [<f8008ddf>] ? drbd_connector_callback+0x10f/0x260 [drbd]</div><div>Apr 4 10:53:42 idaho kernel: [<c06ad7b0>] ? cn_queue_wrapper+0x0/0x30</div><div>Apr 4 10:53:42 idaho kernel: [<c06ad7c6>] ? cn_queue_wrapper+0x16/0x30</div>
<div>Apr 4 10:53:42 idaho kernel: [<c0471a7b>] ? worker_thread+0x11b/0x230</div><div>Apr 4 10:53:42 idaho kernel: [<c04761d0>] ? autoremove_wake_function+0x0/0x40</div><div>Apr 4 10:53:42 idaho kernel: [<c0471960>] ? worker_thread+0x0/0x230</div>
<div>Apr 4 10:53:42 idaho kernel: [<c0475f94>] ? kthread+0x74/0x80</div><div>Apr 4 10:53:42 idaho kernel: [<c0475f20>] ? kthread+0x0/0x80</div><div>Apr 4 10:53:42 idaho kernel: [<c0409fff>] ? kernel_thread_helper+0x7/0x10</div>
<div>Apr 4 10:53:42 idaho kernel: ---[ end trace 362fbb8d1eec5850 ]---</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: Barriers not supported on meta data device - disabling</div><div>Apr 4 10:53:42 idaho lrmd: [1795]: info: RA output: (drbd0:0:start:stdout)</div>
<div>Apr 4 10:53:42 idaho kernel: block drbd0: conn( StandAlone -> Unconnected )</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: Starting receiver thread (from drbd0_worker [2064])</div><div>Apr 4 10:53:42 idaho kernel: block drbd0: receiver (re)started</div>
<div>Apr 4 10:53:42 idaho kernel: block drbd0: conn( Unconnected -> WFConnection )</div></div><div><div>Apr 4 10:53:43 idaho kernel: block drbd0: Starting asender thread (from drbd0_receiver [2121])</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: data-integrity-alg: <not-used></div>
<div>Apr 4 10:53:43 idaho kernel: block drbd0: drbd_sync_handshake:</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: self 8CBCEE5F323F2202:0000000000000000:37E7C7784E3A2062:37E6C7784E3A2063 bits:0 flags:0</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: peer F5F5A2EF289C4C79:8CBCEE5F323F2203:37E7C7784E3A2063:37E6C7784E3A2063 bits:10988 flags:0</div>
<div>Apr 4 10:53:43 idaho kernel: block drbd0: uuid_compare()=-1 by rule 50</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )</div>
<div>Apr 4 10:53:43 idaho kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID )</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: updated sync uuid 8CBDEE5F323F2202:0000000000000000:37E7C7784E3A2062:37E6C7784E3A2063</div>
<div>Apr 4 10:53:43 idaho kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)</div>
<div>Apr 4 10:53:43 idaho kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )</div><div>Apr 4 10:53:43 idaho kernel: block drbd0: Began resync as SyncTarget (will sync 43964 KB [10991 bits set]).</div>
<div>Apr 4 10:53:45 idaho kernel: block drbd0: Resync done (total 1 sec; paused 0 sec; 43964 K/sec)</div><div>Apr 4 10:53:45 idaho kernel: block drbd0: updated UUIDs F5F5A2EF289C4C78:0000000000000000:8CBDEE5F323F2202:8CBCEE5F323F2203</div>
<div>Apr 4 10:53:45 idaho kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )</div><div>Apr 4 10:53:45 idaho kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0</div>
<div>Apr 4 10:53:45 idaho crm-unfence-peer.sh[2179]: invoked for drbd0</div><div>Apr 4 10:53:45 idaho kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)</div><div>Apr 4 10:53:46 idaho kernel: block drbd0: bitmap WRITE of 6666 pages took 742 jiffies</div>
<div>Apr 4 10:53:46 idaho kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>Apr 4 10:53:46 idaho kernel: Bridge firewalling registered</div><div>Apr 4 10:53:57 idaho abrtd: Sending an email...</div>
<div>Apr 4 10:53:57 idaho abrtd: Email was sent to: root@localhost</div><div>Apr 4 10:53:57 idaho abrtd: Can't open file '/var/spool/abrt/oops-2012-02-28-09:12:18-1892-1/uid': No such file or directory</div>
<div>Apr 4 10:53:57 idaho abrtd: DUP_OF_DIR: /var/spool/abrt/oops-2012-02-28-09:12:18-1892-1</div><div>Apr 4 10:53:57 idaho abrtd: Dump directory is a duplicate of /var/spool/abrt/oops-2012-02-28-09:12:18-1892-1</div><div>
Apr 4 10:53:57 idaho abrtd: Deleting dump directory oops-2012-04-04-10:53:43-1755-0 (dup of oops-2012-02-28-09:12:18-1892-1), sending dbus signal</div><div>Apr 4 10:56:52 idaho ntpd[1639]: synchronized to 208.139.196.164, stratum 2</div>
<div>Apr 4 10:56:52 idaho ntpd[1639]: time reset +0.306834 s</div><div>Apr 4 10:56:52 idaho ntpd[1639]: kernel time sync status change 2001</div></div><div><br></div>-- <br><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(48,70,157);font-family:Verdana;font-size:9px">
<span style="letter-spacing:0px"><b>Ryan C Shannon | Senior System Administrator |<span> </span></b><a href="http://www.visionlink.org/" style="color:rgb(42,93,176)" target="_blank"><span style="text-decoration:underline"><b>VisionLink, Inc.</b></span></a><b> </b></span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(128,128,128);font-family:Verdana;font-size:9px"><span style="letter-spacing:0px">First National Center | 3101 Iris Avenue, Suite 240 | Boulder CO, 80301 | </span>Office 303.402.0170 x180</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(128,128,128);min-height:11px;font-family:Verdana;font-size:9px"><span style="letter-spacing:0px"></span></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;min-height:11px;color:rgb(128,128,128);font-family:Verdana;font-size:9px">
<span style="border-collapse:collapse;color:rgb(187,3,3);font-family:verdana;font-size:x-small"><b><span style="color:rgb(128,128,128);font-family:Verdana;font-size:9px;border-collapse:separate;font-weight:normal"><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(255,73,58);display:inline!important">
<span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold">Other ways to stay in touch -</span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"><a href="http://www.visionlinkblog.org/" style="color:rgb(42,93,176)" target="_blank"><span style="color:rgb(47,70,157)">Blog</span></a></span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold">|</span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"><a href="http://twitter.com/VisionLink" style="color:rgb(42,93,176)" target="_blank"><span style="color:rgb(47,70,157)">Twitter</span></a></span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold">|</span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"><a href="http://www.facebook.com/VisionLink.CommunityOS" style="color:rgb(42,93,176)" target="_blank"><span style="color:rgb(47,70,157)">Facebook</span></a></span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold">|</span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"><a href="http://www.linkedin.com/company/visionlink" style="color:rgb(42,93,176)" target="_blank"><span style="color:rgb(47,70,157)">LinkedIn</span></a></span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold">|</span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"> </span><span style="color:rgb(187,3,3);font-family:verdana;font-size:10px;font-weight:bold"><a href="http://www.visionlink.org/" style="color:rgb(42,93,176)" target="_blank"><span style="color:rgb(47,70,157)">Web</span></a></span></span></div>
</span></b></span><span style="letter-spacing:0px"></span></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font color="#BB0303" face="verdana"><span style="border-collapse:collapse"><b><span style="border-collapse:separate"><div style="color:rgb(80,0,80);font-family:Verdana;font-size:9px;font-weight:normal">
</div></span></b></span></font></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(255,73,58);font-family:Verdana;font-size:9px"><span style="letter-spacing:0px"><br></span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(179,179,179);font-family:Verdana;font-size:9px"><span style="letter-spacing:0px">This message and any attachments may contain information that is privileged, </span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(179,179,179);font-family:Verdana;font-size:9px"><span style="letter-spacing:0px">confidential or exempt from disclosure under applicable law or agreement. If you </span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(179,179,179);font-family:Verdana;font-size:9px"><span style="letter-spacing:0px">have received this message in error, please reply and delete the message and </span></div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;color:rgb(179,179,179);font-family:Verdana;font-size:9px"><span style="letter-spacing:0px">any attachments without opening the attachment. Thank you.</span></div>
<br>