Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Wed, Feb 29, 2012 at 01:33:56PM +0100, Dirk wrote: > Hi folks, > > I have setup a 2 node cluster on CentOS 6 using DRBD (dual primary) > and RHCS with CLVM and GFS2. So far, RHCS is just set up with node > definition to make GFS2 usable. > I have compiled DRBD 8.4.1 myself against the Xen 4.1.2 kernel from > Steve Haigh I am using. Make sure your DRBD module compiled *WITHOUT* warnings. If you see anything similar to drbd_main.c: In function ‘conn_new_minor’: drbd_main.c:2799: warning: passing argument 2 of ‘blk_queue_make_request’ from incompatible pointer type include/linux/blkdev.h:928: note: expected ‘int (*)(struct request_queue *, struct bio *)’ but argument is of type ‘void (*)(struct request_queue *, struct bio *)’ in there, then you are using a binary incompatilbe DRBD module, and likely need this patch to fix the build: http://git.drbd.org/gitweb.cgi?p=drbd-8.4.git;a=commit;h=55a81dcc13bf199cfa1bb6695267deec5adc92ca What happens is, rest of kernel expects an int return value, this drbd module was compiled for void. The return value becomes some "random garbage" in some register. Which may happen to be the expected value (0), or not (!= 0). In which case the retry loop in generic_make_request() will just call it again, with the same parameters, which may reference structures that just now have been freed ... *BOOM* ... You may of course also go back to 8.3.12. > I have followed the current DRBD documentation concerning usage in a > Redhat Cluster and CLVM/GFS2 to the bit, but every time I start the > Cluster and mount the DRBD based GFS2 volume, after short time > (approx. 1 minute) the kernel of one node oopses like that: > > >Message from syslogd at pclus3cent6-01 at Feb 29 12:39:35 ... > > kernel:Oops: 0000 [#1] SMP > > > >Message from syslogd at pclus3cent6-01 at Feb 29 12:39:35 ... > > kernel:last sysfs file: /sys/kernel/dlm/VirtSpace01/control > > > >Message from syslogd at pclus3cent6-01 at Feb 29 12:39:35 ... > > kernel:Stack: > > > >Message from syslogd at pclus3cent6-01 at Feb 29 12:39:35 ... > > kernel:Call Trace: > > > >Message from syslogd at pclus3cent6-01 at Feb 29 12:39:35 ... > > kernel:Code: 89 5d d8 4c 89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 > >7d f8 0f 1f 44 00 00 4c 8b 6f 58 49 89 fc 85 f6 89 f3 49 8b 7d 08 > >4d 8b 7d 00 <48> 8b 47 08 4d 8b 37 48 8b 40 48 75 0e 41 f6 44 24 > >18 01 ba fb > > > >Message from syslogd at pclus3cent6-01 at Feb 29 12:39:35 ... > > kernel:CR2: 0000000000000008 > > > > and the message log contains > > >Feb 29 12:39:35 pclus3cent6-01 kernel: Modules linked in: gfs2 dlm > >configfs drbd ebtable_nat ebtables tun libcrc32c sunrpc bridge stp > >llc bonding ipt_REJECT ipt_LOG xt_recent ip6t_REJECT > >nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables > >ipv6 xen_gntdev xen_evtchn xenfs bnx2 ics932s401 ibmaem ibmpex > >ipmi_msghandler serio_raw pcspkr i2c_i801 iTCO_wdt > >iTCO_vendor_support i5k_amb hwmon i5000_edac edac_core ioatdma dca > >e1000e ses enclosure sg shpchp ext4 mbcache jbd2 sr_mod cdrom > >sd_mod crc_t10dif ata_generic pata_acpi ata_piix aacraid radeon > >ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror > >dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4] > > > >Feb 29 12:39:35 pclus3cent6-01 kernel: [<ffffffffa04de865>] > >drbd_make_request+0x325/0x330 [drbd] > > I am sure I have seen a thread concerning this googling through mail > archives, but I do not find it any more, so please bear with me if > the problem has been already solved once. > > Is this a known issue with DRBD 8.4.1 on CentOS 6? > What can I do to troubleshoot this? I do not have any idea on where > to start yet. > > Any hint or help is appreciated. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com