[DRBD-user] DRBD attempts to access beyond end of device

Nate Seif seifn at rstechnicalservices.com
Thu Feb 28 17:46:11 CET 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.



On Thu, 28 Feb 2008, Nate Seif wrote:

>
>
> On Wed, 27 Feb 2008, Lars Ellenberg wrote:
>
>>  On Wed, Feb 27, 2008 at 01:07:04PM -0500, Nate Seif wrote:
>> > 
>> > 
>> >  On Wed, 27 Feb 2008, Lars Ellenberg wrote:
>> > 
>> > >  On Tue, Feb 26, 2008 at 04:14:36PM -0500, Nate Seif wrote:
>> > > >  Hello all:
>> > > >  I intermittently experience the errors below while running DRBD and 
>> > > >  would
>> > > >  like to correct whatever condition is causing DRBD to randomly lose 
>> > > >  pages.
>> > > >  My hard disks and partitions are identical and have never given me
>> > > >  problems previously. I don't see any other disk I/O errors in my 
>> > > >  logs. And
>> > > >  it appears that occassionally (not always) these errors are preceded 
>> > > >  by a
>> > > >  resync of the two disks.
>> > > > 
>> > > >  Why would DRBD "attempt to access beyond end of device"?
>> > > > 
>> > > >  I am running DRBD 8.06 on Gentoo Linux as I could not get my latest
>> > > >  Gentoo kernel to load the DRBD module where version > 8.06. Metadata 
>> > > >  is
>> > > >  "internal" and I'm running Protocol C. I'd be happy to post my 
>> > > >  drbd.conf
>> > > >  page if necessary.
>> > > > 
>> > > > 
>> > > >  Feb 26 08:21:46 <hostname> attempt to access beyond end of device
>> > > >  Feb 26 08:21:46 <hostname> drbd0: rw=1, want=211992584, 
>> > > >  limit=211986944
>> > > >  Feb 26 08:21:46 <hostname> Buffer I/O error on device drbd0, logical 
>> > > >  block
>> > > >  26499072
>> > > >  Feb 26 08:21:46 <hostname> lost page write due to I/O error on drbd0
>> > > >  Feb 26 08:21:46 <hostname> attempt to access beyond end of device
>> > > >  Feb 26 08:21:46 <hostname> drbd0: rw=1, want=211992592, 
>> > > >  limit=211986944
>> > > >  Feb 26 08:21:46 <hostname> Buffer I/O error on device drbd0, logical 
>> > > >  block
>> > > >  26499073
>> > > >  Feb 26 08:21:46 <hostname> lost page write due to I/O error on drbd0
>> > > > 
>> > > > 
>> > > >  Any ideas, tips, help, etc. is much appreciated. Thank you -
>> > > 
>> > >  let me guess:
>> > >  you did mkfs /dev/sda1, not mkfs /dev/drbd0?
>> > >  well, you screwed up.
>> > 
>> >  I did NOT mkfs on /dev/hda4. (I have DRBD running on a pair of IDE/PATA
>> >  disks and no SATA drives in either system.)
>> > 
>> >  I partitioned my disks with fdisk. I have identical drives with
>> >  identically sized partitions. I compiled the DRBD module, started DRBD,
>> >  mounted /dev/drbd0 (not /dev/hda4), and formatted drbd0 with an ext3
>> >  file system on the primary only after I got DRBD up and running months
>> >  ago.
>>
>>  please do
>>
>>       tune2fs -l /dev/mapper/vg00--bk1-root |
>>   grep -e ^Block.count: -e ^Block.size:
>
> I do not have RAID on either system and /dev/mapper does not exist on either 
> machine. I have a single, identical hard drive in each system where /dev/hda4 
> is the partition DRBD uses. Can I change the tune2fs command you suggested 
> above to get the bytes my ext3 FS thinks it's occupying?
>
>
>>
>>  you get two numbers.
>>  multiply those, you get the size (in bytes)
>>  your ext3 thinks it is occupying.
>>  which is the size of the partition you run the mkfs on, at the time of
>>  the mkfs run, unless you used special options.
>>
>>  now, do
>>   grep -e hda4 -e drbd0 /proc/partitions
>
>
> # grep -e hda4 -e drbd0 /proc/partitions
>    3     4  105996744 hda4
> 147     0  105993472 drbd0
> #
>
>
>
>>
>>  you again get two numbers, this time unit is kilo byte.
>>  that is the size of the partitions as the kernel sees them now.
>>  according to the logs above (the limit= is unit sectors),
>>  drbd0 will be 105993472 kB.
>>  I dare say hda4 will be somewhat larger, my best guess, given the
>>  information I have, is that hda4 will be 105996740 kB.
>>  and that this also matches what the tune2fs reports.
>
>
> I imagine ext3 and the kernel need to have or use the same number for file 
> system size on /dev/drbd0. If these numbers differ, then I get the errors I 
> reported, correct? How (or is it possible?) to know whether the size as ext3 
> sees it is correct or the kernel size is correct? Could a corrupted inode be 
> responsible for this problem? How do I avoid this problem in the future? Can 
> I run e2fsck on /dev/drbd0 to fix such a problem?
>
>
> ASIDE: I backed up my data from the primary side. Primary and secondary 
> machines went to "Primary/Unknown" and "Unknown/Secondary" after copying a 
> little less than 10 GB of data and drbd reports "NetworkFailure." All NICs 
> involved seem to be working fine - I can ping and copy files to/from both 
> computers but DRBD is disconnected.

I'm not sure if this is germaine to the current discussion, but I spotted 
the following error(s) in my logs on the primary computer with /dev/drbd0 
mounted for r/w access at about the same time DRBD went to a 
"NetworkFailure" status:

Feb 27 17:04:11 <hostname> drbd0: PingAck did not arrive in time.
Feb 27 17:04:11 <hostname> drbd0: peer( Secondary -> Unknown ) conn( 
Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Feb 27 17:04:11 <hostname> drbd0: Creating new current UUID
Feb 27 17:04:11 <hostname> drbd0: asender terminated
Feb 27 17:04:11 <hostname> drbd0: short read expecting header on sock: 
r=-512
Feb 27 17:04:13 <hostname> drbd0: tl_clear()
Feb 27 17:04:13 <hostname> drbd0: Connection closed
Feb 27 17:04:13 <hostname> drbd0: Writing meta data super block now.
Feb 27 17:04:13 <hostname> drbd0: conn( NetworkFailure -> Unconnected )
Feb 27 17:04:13 <hostname> drbd0: receiver terminated
Feb 27 17:04:13 <hostname> drbd0: receiver (re)started
Feb 27 17:04:13 <hostname> drbd0: conn( Unconnected -> WFConnection )
.
.
.
Feb 27 17:07:56 <hostname> drbd0: conn( WFConnection -> WFReportParams )
Feb 27 17:07:56 <hostname> drbd0: Handshake successful: DRBD Network 
Protocol version 86
Feb 27 17:07:56 <hostname> drbd0: peer( Unknown -> Secondary ) conn( 
WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
Feb 27 17:08:06 <hostname> drbd0: PingAck did not arrive in time.
Feb 27 17:08:06 <hostname> drbd0: peer( Secondary -> Unknown ) conn( 
WFBitMapS -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Feb 27 17:08:06 <hostname> drbd0: asender terminated
Feb 27 17:08:18 <hostname> drbd0: short sent ReportBitMap size=4096 
sent=3800
Feb 27 17:08:18 <hostname> drbd0: Writing meta data super block now.
Feb 27 17:08:18 <hostname> drbd0: BUG! md_sync_timer expired! Worker calls 
drbd_md_sync().
Feb 27 17:08:18 <hostname> drbd0: BUG! md_sync_timer expired! Worker calls 
drbd_md_sync().
Feb 27 17:08:18 <hostname> drbd0: tl_clear()
Feb 27 17:08:18 <hostname> drbd0: Connection closed
Feb 27 17:08:18 <hostname> drbd0: conn( NetworkFailure -> Unconnected )
Feb 27 17:08:18 <hostname> drbd0: receiver terminated
Feb 27 17:08:18 <hostname> drbd0: receiver (re)started
Feb 27 17:08:18 <hostname> drbd0: conn( Unconnected -> WFConnection )
.
.
.
Feb 27 17:09:09 <hostname> drbd0: conn( WFConnection -> WFReportParams )
Feb 27 17:09:09 <hostname> drbd0: Handshake successful: DRBD Network 
Protocol version 86
Feb 27 17:09:09 <hostname> drbd0: peer( Unknown -> Secondary ) conn( 
WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )
Feb 27 17:09:19 <hostname> drbd0: meta connection shut down by peer.
Feb 27 17:09:19 <hostname> drbd0: peer( Secondary -> Unknown ) conn( 
WFBitMapS -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Feb 27 17:09:19 <hostname> drbd0: asender terminated
Feb 27 17:09:25 <hostname> drbd0: short sent ReportBitMap size=4096 
sent=1152
Feb 27 17:09:25 <hostname> drbd0: Writing meta data super block now.
Feb 27 17:09:25 <hostname> drbd0: BUG! md_sync_timer expired! Worker calls 
drbd_md_sync().
Feb 27 17:09:25 <hostname> drbd0: BUG! md_sync_timer expired! Worker calls 
drbd_md_sync().
Feb 27 17:09:25 <hostname> drbd0: BUG! md_sync_timer expired! Worker calls 
drbd_md_sync().



>
>
> Nate
>
>
>> 
>> >  My logs did not start recording these errors until several weeks ago.
>>
>>  which probably only means that the file system slowly filled up,
>>  and now starts actually _using_ those areas which are no longer there,
>>  because they are now occupied by the drbd meta data.
>>
>>  -- 
>> :  Lars Ellenberg                           http://www.linbit.com :
>> :  DRBD/HA support and consulting             sales at linbit.com :
>> :  LINBIT Information Technologies GmbH      Tel +43-1-8178292-0  :
>> :  Vivenotgasse 48, A-1120 Vienna/Europe     Fax +43-1-8178292-82 :
>>  __
>>  please use the "List-Reply" function of your email client.
>>  _______________________________________________
>>  drbd-user mailing list
>>  drbd-user at lists.linbit.com
>>  http://lists.linbit.com/mailman/listinfo/drbd-user
>> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>



More information about the drbd-user mailing list