[DRBD-user] DRBD Errors

Mark Olliver mpo at thermeoneurope.com
Thu Sep 28 12:26:30 CEST 2006

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi All,

Did anyone have any other ideas?


Just for Clarity bellow is a copy of my setup.

These two servers are connected via ipsec vpn one based in a London colo the
other in a Dublin colo. 

Current Primary Server
[root at uk-openvz2 ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 35669.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 293.3 GB, 293391564800 bytes
255 heads, 63 sectors/track, 35669 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          25      200781   83  Linux
/dev/sda2              26        1045     8193150   82  Linux swap
/dev/sda3            1046        1810     6144862+  83  Linux
/dev/sda4            1811       35669   271972417+   5  Extended
/dev/sda5            1811        2447     5116671   83  Linux
/dev/sda6            2448       35669   266855683+  83  Linux

[root at uk-openvz2 ~]# mount
/dev/sda3 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda5 on /var type ext3 (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/drbd0 on /mnt/drbd0 type ext3 (rw)


#
# drbd.conf
#
#
global {
        minor-count 5;
}
resource vz1 {
  protocol A;
  #incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt
-f";
  startup {
    wfc-timeout  60;
    degr-wfc-timeout 60;    # 2 minutes.
   }
  disk {
    on-io-error   detach;
  }
  net {
    sndbuf-size 512k;
    timeout     60;
    connect-int 10;
    ping-int    10;
    max-buffers 2048;
    on-disconnect reconnect;
  }
  syncer {
    rate 4M;
    group 1;
    al-extents 257;
  }

  on uk-openvz2.thermeon.eu {
    device     /dev/drbd0;
    disk       /dev/sda6;
    address    192.168.240.4:7789;
    meta-disk  internal;
  }

  on ie-openvz1.thermeon.eu {
    device    /dev/drbd0;
    disk      /dev/sda6;
    address   192.168.242.1:7789;
    meta-disk internal;
  }
}

[root at uk-openvz2 ~]# cat /proc/drbd
version: 0.7.19 (api:78/proto:74)
SVN Revision: 2212 build by root at cp1.thermeoneurope.com, 2006-05-30 12:57:03
 0: cs:WFConnection st:Primary/Unknown ld:Consistent
    ns:0 nr:0 dw:2234604 dr:3164897 al:1751 bm:1751 lo:0 pe:0 ua:0 ap:0
 1: cs:Unconfigured
 2: cs:Unconfigured
 3: cs:Unconfigured
 4: cs:Unconfigured

Ext3 format done on /mnt/drbd0


Secondary Setup
[root at ie-openvz1 ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 71338.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 586.7 GB, 586783129600 bytes
255 heads, 63 sectors/track, 71338 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          25      200781   83  Linux
/dev/sda2              26        1045     8193150   82  Linux swap
/dev/sda3            1046        1810     6144862+  83  Linux
/dev/sda4            1811       71338   558483660    5  Extended
/dev/sda5            1811        2447     5116671   83  Linux
/dev/sda6            2448       35669   266855683+  83  Linux
/dev/sda7           35670       38102    19543041   83  Linux

Command (m for help):

[root at ie-openvz1 ~]# mount
/dev/sda3 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/sda5 on /var type ext3 (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/drbd1 on /mnt/drbd1 type ext3 (rw)

[root at ie-openvz1 ~]# cat /proc/drbd
version: 0.7.19 (api:78/proto:74)
SVN Revision: 2212 build by root at cp-bk.thermeoneurope.com, 2006-05-30
15:22:21
 0: cs:Unconfigured
 1: cs:WFConnection st:Primary/Unknown ld:Consistent
    ns:0 nr:0 dw:1852292 dr:782157 al:224 bm:481 lo:0 pe:0 ua:0 ap:0
 2: cs:Unconfigured
 3: cs:Unconfigured
 4: cs:Unconfigured

Drbd1 is on a different partition and would talk to another server, drbd0 is
the one that had problems

#
# drbd.conf
#
#
global {
        minor-count 5;
}
resource vz1 {
  protocol A;
  #incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt
-f";
  startup {
    wfc-timeout  60;
    degr-wfc-timeout 60;    # 2 minutes.
  }
  disk {
    on-io-error   detach;
  }
  net {
    sndbuf-size 512k;
    timeout     60;
    connect-int 10;
   ping-int    10;
    max-buffers 2048;
    on-disconnect reconnect;
  }
  syncer {
    rate 4M;
    group 1;
    al-extents 257;
  }
  on uk-openvz2.thermeon.eu {
    device     /dev/drbd0;
    disk       /dev/sda6;
    address    192.168.240.4:7789;
    meta-disk  internal;
  }
  on ie-openvz1.thermeon.eu {
    device    /dev/drbd0;
    disk      /dev/sda6;
    address   192.168.242.1:7789;
    meta-disk internal;
  }
}
resource disk2 {
  protocol A;
  #incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt
-f";
  startup {
    wfc-timeout  60;
    degr-wfc-timeout 60;    # 2 minutes.
  }
  disk {
    on-io-error   detach;
  }
  net {
    sndbuf-size 512k;
    timeout     60;
    connect-int 10;
    ping-int    10;
    max-buffers 2048;
    on-disconnect reconnect;
  }
  syncer {
    rate 4M;
    group 2;
    al-extents 257;
}
  on uk-openvz3.thermeon.eu {
    device     /dev/drbd1;
    disk       /dev/sda6;
    address    192.168.240.5:7789;
    meta-disk  internal;
  }
  on ie-openvz1.thermeon.eu {
    device    /dev/drbd1;
    disk      /dev/sda7;
    address   192.168.242.1:7790;
    meta-disk internal;
  }
}

No format was done on this device other than for real partitions.



*******************************************************************
Mark Olliver BSc (Hons) 
Thermeon Europe Ltd
 
V-Card
http://www.thermeoneurope.com/e-Card/card.html?uid=mpo

-----Original Message-----
From: drbd-user-bounces at linbit.com [mailto:drbd-user-bounces at linbit.com] On
Behalf Of Lars Ellenberg
Sent: 27 September 2006 16:26
To: drbd-user at linbit.com
Subject: Re: [DRBD-user] DRBD Errors

/ 2006-09-27 14:18:52 +0100
\ Mark Olliver:
> Hi All,
> 
>  
> 
> I stopped my primary device last night and turned it off, then made the
> secondary device primary. After which I mounted as normal and started
using
> it.
> 
> This morning I noticed that something went wrong over night and I had lots
> of file corruptions. Upon checking the logs I found the message bellow:
> 
>  
> 
> Sep 27 01:04:48 ie-openvz1 kernel: attempt to access beyond end of device
> 
> Sep 27 01:04:48 ie-openvz1 kernel: drbd0: rw=0, want=2148540848,
> limit=533449216

it is just an educated guess, but:
you created your file system on the lower level device.
you put drbd on top of that, using "meta-disk internal".
you did not shrink your file system before that.
oops.

recover:
  secondary# heartbeat stop
  primary# heartbeat stop
  
  verify drbd is Secondary/Secondary
  
  both# drbdadm down all
  
  verify drbd is Unconfigured
  
  good-data# fsck /dev/lower-level
  
  shrink the file system.
  good-date# "whatever-resize" /dev/lower-level minus-130MB
  
  both# start drbd.
  
  other-node# drbdadm invalidate all
  
  wait for full sync
  
  both# start heartbeat.

or, since you are in openvz, maybe you just want to redeploy?

-- 
: Lars Ellenberg                                  Tel +43-1-8178292-0  :
: LINBIT Information Technologies GmbH            Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe   http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user




More information about the drbd-user mailing list