<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.12.0">
</HEAD>
<BODY>
I'm really getting desparate on this, as we are currently not in a high availability state with our server, so I thought I'd include some more info. Attached is my drbd.conf. Also, I am running RHEL5 2.6.18-8.1.14.el5 on both systems. Below is a capture from my system messages log from the original failure:<BR>
<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234300760, limit=234300736<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234300800, limit=234300736<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234300864, limit=234300736<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234300928, limit=234300736<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234300992, limit=234300736<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234301016, limit=234300736<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:50 arc-swilliamslx kernel: drbd0: rw=0, want=234300744, limit=234300736<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: attempt to access beyond end of device<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: drbd0: rw=0, want=234303728, limit=234300736<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0): ext3_free_branches: Read failure, inode=14209948, block=<BR>
29287965<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: Aborting journal on device drbd0.<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0) in ext3_reserve_inode_write: Journal has aborted<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0) in ext3_truncate: Journal has aborted<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0) in ext3_reserve_inode_write: Journal has aborted<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0) in ext3_orphan_del: Journal has aborted<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0) in ext3_reserve_inode_write: Journal has aborted<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0) in ext3_delete_inode: Journal has aborted<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: __journal_remove_journal_head: freeing b_committed_data<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: ext3_abort called.<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: EXT3-fs error (device drbd0): ext3_journal_start_sb: Detected aborted journal<BR>
Feb 7 05:41:57 arc-swilliamslx kernel: Remounting filesystem read-only<BR>
(I believe this is where heartbeat stepped in and failed over to the other server, postgresql was down due to the read-only mount)<BR>
Feb 7 05:42:32 arc-swilliamslx kernel: __journal_remove_journal_head: freeing b_committed_data<BR>
Feb 7 05:42:32 arc-swilliamslx kernel: drbd0: role( Primary -> Secondary ) <BR>
Feb 7 05:42:32 arc-swilliamslx kernel: drbd0: Writing meta data super block now.<BR>
Feb 7 05:42:34 arc-swilliamslx kernel: drbd0: peer( Secondary -> Primary ) <BR>
<BR>
Any help will be greatly appreciated,<BR>
Doug<BR>
<BR>
On Thu, 2008-02-07 at 14:04 -0500, Doug Knight wrote:<BR>
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">Hi list,</FONT><BR>
<FONT COLOR="#000000">I had one of my HA systems, running drbd 8.0.1, issue an error on its drbd0 device (see title). We recently resized the underlying partition using gparted to include the partition immediately following it (verified that the new, larger partitions were identical, and ran the command to fix the meta-data, suggested when drbd was restarted). We did this on both systems, and everything seemed OK for a few days. This morning we got the error, heartbeat detected it, and migrated resources to the other system, no problem. I took drbd down on both systems, mounted and set primary drbd0 on the system with the issue, and did an fsck -fvn /dev/drbd0 on it (unmounted). I get the following:</FONT><BR>
<BR>
<FONT COLOR="#000000">The filesystem size (according to the superblock) is 29288495 blocks</FONT><BR>
<FONT COLOR="#000000">The physical size of the device is 29287592 blocks</FONT><BR>
<FONT COLOR="#000000">Either the superblock or the partition table is likely to be corrupt!</FONT><BR>
<BR>
<FONT COLOR="#000000">So, I then ran fsck without the -n to correct. Now, drbd seems to be completely hosed up. If I do a ./drbd start, the system locks up. If I do the drbdadm adjust pgsql, it locks up the system too. I went as far as to shutdown drbd, remove the kernel module, delete the sda5 partition and recreate it, starting over, and it still locks up the system when I try to bring up drbd. What I'd like to do is fix the issue on this system, and let it get back in sync with the other system. So 1) How do I get drbd back and functioning on the system where the issue occurred?, and 2) Do I need to do anything to the system that is currently running OK (due to the partition resize, etc)?</FONT><BR>
<BR>
<FONT COLOR="#000000">Thanks,</FONT><BR>
<FONT COLOR="#000000">Doug Knight</FONT><BR>
<FONT COLOR="#000000">WSI Corp</FONT><BR>
<FONT COLOR="#000000">Andover, MA 01945 </FONT>
<PRE>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">drbd-user mailing list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A></FONT>
<FONT COLOR="#000000"><A HREF="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>