Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Forwarding this from linux-ha list: -------- Original Message -------- Subject: Re: [Linux-HA] heartbeat 2.0.8: lockups Date: Mon, 19 Feb 2007 09:57:16 -0500 From: Gerry Reno <greno at verizon.net> Reply-To: General Linux-HA mailing list <linux-ha at lists.linux-ha.org> To: General Linux-HA mailing list <linux-ha at lists.linux-ha.org> References: <12392854.6367231171759462886.JavaMail.root at vms074.mailsrvcs.net> <26ef5e70702190352p4d6d24cajb31b28edbe0d1885 at mail.gmail.com> <45D9B8CD.70907 at verizon.net> Gerry Reno wrote: > Andrew Beekhof wrote: >> so what are we looking at here? what time did the lockup occur? >> >> On 2/18/07, greno at verizon.net <greno at verizon.net> wrote: >>> I've been running heartbeat on my two nodes for almost two weeks and >>> everything is functioning as it is supposed to with the exception >>> that I am getting frequent lockups on the primary server. It >>> doesn't matter which server that I make the primary it will >>> eventually be locked up. The lockups are very hard. There is no >>> response of any kind out of the locked up machine. Sometimes the >>> drive light will be on and sometimes not. The lockups are occurring >>> at times of disk access such as during backups or right after I ftp >>> a file or tar file over to another machine from the drbd array. >>> There is very little in the logs. It just shows a big gap and then >>> a syslog restart for when I cold booted the server to bring it back >>> up. I'm going to attach dmesg output and /var/log/messages output >>> for both servers. What should I do to track down the source of this >>> problem? >>> >>> heartbeat-2.0.8-1.fc6 >>> drbd-0.7.23-15.fc6.at >>> >>> Other info: >>> drbd is running over logical volume which is over a RAID-1 md array >>> on each server. >>> >>> Both servers were rock stable prior to installing HA. >>> >>> > Andrew, > The lockup occurred about 18:07. If you search for 'restart' in the > log you should find it. It was about a 7 minute gap. This one > occurred when I ftp'd a file. > > Gerry Andrew, I just went and checked on the primary and this was showing in a terminal window: Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: Oops: 0000 [#1] Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: SMP Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: CPU: 0 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: EIP: 0061:[<c042e17d>] Tainted: GF VLI Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: EFLAGS: 00010202 (2.6.19-1.2895.fc6xen #1) Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: EIP is at put_pid+0x6/0x20 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: eax: 000000ab ebx: 00000008 ecx: c1b84140 edx: 000000ab Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: esi: c1b840c0 edi: c1b840c0 ebp: e3957be0 esp: e9ea0f0c Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: ds: 007b es: 007b ss: 0069 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: Process IPaddr (pid: 29397, ti=e9ea0000 task=e70464d0 task.ti=e9ea0000) Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: Stack: c046c707 00000000 00000000 d0e76988 ed7e8520 e3957be0 d76b4b40 00000000 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: e1f53d20 c046a03a c0467578 d76b4b40 000001ff 00000004 c041fdbd 00000000 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: 00000000 e7046978 d76b4b40 e70464d0 00000001 c0420fa0 00000000 c04442e3 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: Call Trace: Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: [<c046c707>] __fput+0x12f/0x190 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: [<c046a03a>] filp_close+0x52/0x59 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: [<c041fdbd>] put_files_struct+0x65/0xa7 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: [<c0420fa0>] do_exit+0x246/0x787 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: [<c042156e>] sys_exit_group+0x0/0xd Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:30 2007 ... grp-01-30-02 kernel: [<c0404efb>] syscall_call+0x7/0xb Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:31 2007 ... grp-01-30-02 kernel: [<00c5d402>] 0xc5d402 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:31 2007 ... grp-01-30-02 kernel: ======================= Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:31 2007 ... grp-01-30-02 kernel: Code: 00 77 09 8d 1c 06 85 db 7e 15 eb a8 83 c3 08 81 c6 00 80 00 00 31 c9 81 fb 74 53 68 c0 72 cc 89 f8 5b 5e 5f c3 85 c0 89 c2 74 19 <8b> 00 48 74 0a 90 ff 0a 0f 94 c0 84 c0 74 0a a1 28 2d 83 c0 e9 Message from syslogd at grp-01-30-02 at Sun Feb 18 23:09:31 2007 ... grp-01-30-02 kernel: EIP: [<c042e17d>] put_pid+0x6/0x20 SS:ESP 0069:e9ea0f0c _______________________________________________ Linux-HA mailing list Linux-HA at lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems