Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi folks, We are running a file-server cluster using centos6, drbd, corosync, and pacemaker. Each time drbd is started, a kernel oops is triggered (see below). This doesn't seem to cause the system any problems as it completes the sync normally and the system is then stable & functioning. I'm thinking that this is a known issue ( http://lists.linbit.com/pipermail/drbd-user/2011-August/016628.html & http://elrepo.org/bugs/view.php?id=210) but I'm nervous that this is something more serious. My question is, is this something that can be safely ignored or if not, how to approach fixing it? Thanks! -Ryan I have the 'no-disk-barrier' option enabled in /etc/drbd.conf. [admin at idaho ~]$ rpm -qa |grep drbd drbd83-utils-8.3.12-1.el6.elrepo.i686 kmod-drbd83-8.3.12-1.el6.elrepo.i686 [admin at idaho ~]$ more /etc/redhat-release CentOS release 6.2 (Final) [admin at idaho ~]$ uname -r 2.6.32-220.7.1.el6.i686 /var/log/messages output: Apr 4 10:53:42 idaho kernel: drbd: initialized. Version: 8.3.12 (api:88/proto:86-96) Apr 4 10:53:42 idaho kernel: drbd: GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by dag at Build32R6, 2011-11-20 10:55:07 Apr 4 10:53:42 idaho kernel: drbd: registered as block device major 147 Apr 4 10:53:42 idaho kernel: drbd: minor_table @ 0xf3beea00 Apr 4 10:53:42 idaho kernel: block drbd0: Starting worker thread (from cqueue [2042]) Apr 4 10:53:42 idaho kernel: block drbd0: disk( Diskless -> Attaching ) Apr 4 10:53:42 idaho kernel: block drbd0: Found 6 transactions (324 active extents) in activity log. Apr 4 10:53:42 idaho kernel: block drbd0: Method to ensure write ordering: flush Apr 4 10:53:42 idaho kernel: block drbd0: max BIO size = 65536 Apr 4 10:53:42 idaho kernel: block drbd0: drbd_bm_resize called with capacity == 1757746384 Apr 4 10:53:42 idaho kernel: block drbd0: resync bitmap: bits=219718298 words=6866198 pages=6706 Apr 4 10:53:42 idaho kernel: block drbd0: size = 838 GB (878873192 KB) Apr 4 10:53:42 idaho attrd: [1796]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Apr 4 10:53:42 idaho lrmd: [1795]: info: RA output: (drbd0:0:start:stdout) #012#012 Apr 4 10:53:42 idaho attrd: [1796]: notice: attrd_perform_update: Sent update 4: probe_complete=true Apr 4 10:53:42 idaho crmd: [1798]: info: process_lrm_event: LRM operation kansas_stonith_start_0 (call=13, rc=0, cib-update=17, confirmed=true) ok Apr 4 10:53:42 idaho kernel: block drbd0: bitmap READ of 6706 pages took 302 jiffies Apr 4 10:53:42 idaho kernel: block drbd0: recounting of set bits took additional 57 jiffies Apr 4 10:53:42 idaho kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map. Apr 4 10:53:42 idaho kernel: block drbd0: disk( Attaching -> Outdated ) Apr 4 10:53:42 idaho kernel: block drbd0: attached to UUIDs 8CBCEE5F323F2202:0000000000000000:37E7C7784E3A2062:37E6C7784E3A2063 Apr 4 10:53:42 idaho kernel: ------------[ cut here ]------------ Apr 4 10:53:42 idaho kernel: WARNING: at block/blk-core.c:1296 __make_request+0x4fa/0x570() (Not tainted) Apr 4 10:53:42 idaho kernel: Hardware name: PowerEdge 1850 Apr 4 10:53:42 idaho kernel: block: BARRIER is deprecated, use FLUSH/FUA instead Apr 4 10:53:42 idaho kernel: Modules linked in: drbd(U) autofs4 bonding ipv6 ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_multiport iptable_filter ip_tables nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs e1000 microcode dcdbas sg serio_raw iTCO_wdt iTCO_vendor_support e752x_edac edac_core ext4 mbcache jbd2 sd_mod crc_t10dif megaraid_mbox megaraid_mm sr_mod cdrom pata_sil680 video output pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Apr 4 10:53:42 idaho kernel: Pid: 2042, comm: cqueue Not tainted 2.6.32-220.7.1.el6.i686 #1 Apr 4 10:53:42 idaho kernel: Call Trace: Apr 4 10:53:42 idaho kernel: [<c0454d71>] ? warn_slowpath_common+0x81/0xc0 Apr 4 10:53:42 idaho kernel: [<c05dfe6a>] ? __make_request+0x4fa/0x570 Apr 4 10:53:42 idaho kernel: [<c05dfe6a>] ? __make_request+0x4fa/0x570 Apr 4 10:53:42 idaho kernel: [<c0454e43>] ? warn_slowpath_fmt+0x33/0x40 Apr 4 10:53:42 idaho kernel: [<c05dfe6a>] ? __make_request+0x4fa/0x570 Apr 4 10:53:42 idaho kernel: [<c047ba2e>] ? up+0x2e/0x40 Apr 4 10:53:42 idaho kernel: [<f7ff7276>] ? drbd_queue_work+0x36/0x60 [drbd] Apr 4 10:53:42 idaho kernel: [<f7ffd32d>] ? __drbd_set_state+0x80d/0x1010 [drbd] Apr 4 10:53:42 idaho kernel: [<c05de835>] ? generic_make_request+0x395/0x600 Apr 4 10:53:42 idaho kernel: [<c0427ea3>] ? lapic_next_event+0x13/0x20 Apr 4 10:53:42 idaho kernel: [<c048401c>] ? clockevents_program_event+0x8c/0x120 Apr 4 10:53:42 idaho kernel: [<c05deb19>] ? submit_bio+0x79/0x120 Apr 4 10:53:42 idaho kernel: [<c0558da8>] ? bio_add_page+0x38/0x50 Apr 4 10:53:42 idaho kernel: [<f7ff4995>] ? _drbd_md_sync_page_io+0x105/0x310 [drbd] Apr 4 10:53:42 idaho kernel: [<f7ff4ce4>] ? drbd_md_sync_page_io+0x144/0x610 [drbd] Apr 4 10:53:42 idaho kernel: [<f7ff2603>] ? drbd_md_get_buffer+0x33/0x100 [drbd] Apr 4 10:53:42 idaho kernel: [<c0465065>] ? del_timer+0x65/0xb0 Apr 4 10:53:42 idaho kernel: [<f7ffdd29>] ? drbd_md_sync+0x1f9/0x640 [drbd] Apr 4 10:53:42 idaho kernel: [<f800789e>] ? _drbd_set_state.clone.0+0x4e/0x60 [drbd] Apr 4 10:53:42 idaho kernel: [<f800d9a0>] ? drbd_nl_disk_conf+0x1040/0x11d0 [drbd] Apr 4 10:53:42 idaho kernel: [<c04083a7>] ? __switch_to+0xd7/0x1a0 Apr 4 10:53:42 idaho kernel: [<c082e960>] ? schedule+0x3c0/0xae0 Apr 4 10:53:42 idaho kernel: [<f8008ddf>] ? drbd_connector_callback+0x10f/0x260 [drbd] Apr 4 10:53:42 idaho kernel: [<c06ad7b0>] ? cn_queue_wrapper+0x0/0x30 Apr 4 10:53:42 idaho kernel: [<c06ad7c6>] ? cn_queue_wrapper+0x16/0x30 Apr 4 10:53:42 idaho kernel: [<c0471a7b>] ? worker_thread+0x11b/0x230 Apr 4 10:53:42 idaho kernel: [<c04761d0>] ? autoremove_wake_function+0x0/0x40 Apr 4 10:53:42 idaho kernel: [<c0471960>] ? worker_thread+0x0/0x230 Apr 4 10:53:42 idaho kernel: [<c0475f94>] ? kthread+0x74/0x80 Apr 4 10:53:42 idaho kernel: [<c0475f20>] ? kthread+0x0/0x80 Apr 4 10:53:42 idaho kernel: [<c0409fff>] ? kernel_thread_helper+0x7/0x10 Apr 4 10:53:42 idaho kernel: ---[ end trace 362fbb8d1eec5850 ]--- Apr 4 10:53:42 idaho kernel: block drbd0: Barriers not supported on meta data device - disabling Apr 4 10:53:42 idaho lrmd: [1795]: info: RA output: (drbd0:0:start:stdout) Apr 4 10:53:42 idaho kernel: block drbd0: conn( StandAlone -> Unconnected ) Apr 4 10:53:42 idaho kernel: block drbd0: Starting receiver thread (from drbd0_worker [2064]) Apr 4 10:53:42 idaho kernel: block drbd0: receiver (re)started Apr 4 10:53:42 idaho kernel: block drbd0: conn( Unconnected -> WFConnection ) Apr 4 10:53:43 idaho kernel: block drbd0: Starting asender thread (from drbd0_receiver [2121]) Apr 4 10:53:43 idaho kernel: block drbd0: data-integrity-alg: <not-used> Apr 4 10:53:43 idaho kernel: block drbd0: drbd_sync_handshake: Apr 4 10:53:43 idaho kernel: block drbd0: self 8CBCEE5F323F2202:0000000000000000:37E7C7784E3A2062:37E6C7784E3A2063 bits:0 flags:0 Apr 4 10:53:43 idaho kernel: block drbd0: peer F5F5A2EF289C4C79:8CBCEE5F323F2203:37E7C7784E3A2063:37E6C7784E3A2063 bits:10988 flags:0 Apr 4 10:53:43 idaho kernel: block drbd0: uuid_compare()=-1 by rule 50 Apr 4 10:53:43 idaho kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) Apr 4 10:53:43 idaho kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID ) Apr 4 10:53:43 idaho kernel: block drbd0: updated sync uuid 8CBDEE5F323F2202:0000000000000000:37E7C7784E3A2062:37E6C7784E3A2063 Apr 4 10:53:43 idaho kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 Apr 4 10:53:43 idaho kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0) Apr 4 10:53:43 idaho kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) Apr 4 10:53:43 idaho kernel: block drbd0: Began resync as SyncTarget (will sync 43964 KB [10991 bits set]). Apr 4 10:53:45 idaho kernel: block drbd0: Resync done (total 1 sec; paused 0 sec; 43964 K/sec) Apr 4 10:53:45 idaho kernel: block drbd0: updated UUIDs F5F5A2EF289C4C78:0000000000000000:8CBDEE5F323F2202:8CBCEE5F323F2203 Apr 4 10:53:45 idaho kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) Apr 4 10:53:45 idaho kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 Apr 4 10:53:45 idaho crm-unfence-peer.sh[2179]: invoked for drbd0 Apr 4 10:53:45 idaho kernel: block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0) Apr 4 10:53:46 idaho kernel: block drbd0: bitmap WRITE of 6666 pages took 742 jiffies Apr 4 10:53:46 idaho kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map. Apr 4 10:53:46 idaho kernel: Bridge firewalling registered Apr 4 10:53:57 idaho abrtd: Sending an email... Apr 4 10:53:57 idaho abrtd: Email was sent to: root at localhost Apr 4 10:53:57 idaho abrtd: Can't open file '/var/spool/abrt/oops-2012-02-28-09:12:18-1892-1/uid': No such file or directory Apr 4 10:53:57 idaho abrtd: DUP_OF_DIR: /var/spool/abrt/oops-2012-02-28-09:12:18-1892-1 Apr 4 10:53:57 idaho abrtd: Dump directory is a duplicate of /var/spool/abrt/oops-2012-02-28-09:12:18-1892-1 Apr 4 10:53:57 idaho abrtd: Deleting dump directory oops-2012-04-04-10:53:43-1755-0 (dup of oops-2012-02-28-09:12:18-1892-1), sending dbus signal Apr 4 10:56:52 idaho ntpd[1639]: synchronized to 208.139.196.164, stratum 2 Apr 4 10:56:52 idaho ntpd[1639]: time reset +0.306834 s Apr 4 10:56:52 idaho ntpd[1639]: kernel time sync status change 2001 -- *Ryan C Shannon | Senior System Administrator | **VisionLink, Inc.*<http://www.visionlink.org/> * * First National Center | 3101 Iris Avenue, Suite 240 | Boulder CO, 80301 | Office 303.402.0170 x180 * Other ways to stay in touch - Blog <http://www.visionlinkblog.org/> | Twitter <http://twitter.com/VisionLink> | Facebook<http://www.facebook.com/VisionLink.CommunityOS> | LinkedIn <http://www.linkedin.com/company/visionlink> | Web<http://www.visionlink.org/> * * * This message and any attachments may contain information that is privileged, confidential or exempt from disclosure under applicable law or agreement. If you have received this message in error, please reply and delete the message and any attachments without opening the attachment. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20120404/0a8dba5d/attachment.htm>