[DRBD-user] Critical Fail :: bio would need to, but cannot, be split :: No Xen

Mike Peachey mike.peachey at jennic.com
Mon May 17 13:45:56 CEST 2010

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


May 17 08:00:02 carl kernel: block drbd0: bio would need to, but cannot,
be split: (vcnt=2,idx=0,size=8192,sector=1810514356)
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455629
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455630
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455631
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455632
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11


And on and on... I/O errors at NFS client.

Two servers.
Each has 3ware 9650SE-12ML Hardware RAID6 (256K stripes).
RAID6 presented to OS as /dev/sdb (3TB)
GPT 3TB single primary partition.
/dev/sdb on each machine is a member of /dev/drbd0 (r0)
drbd0 is an LVM2 Physical Volume and the only volume in the vg.
There are 13 Logical Volumes.
Each LV is XFS.
Each is mounted and exported via NFSv3 and Samba.

/etc/drbd.conf:
############################################################################
global {
    usage-count yes;
}
common {
}
resource r0 {
  protocol C;
  syncer {
    rate 110M;
    verify-alg "md5";
  }
  handlers {
    pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
    pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
    local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger
; halt -f";
    split-brain "/usr/lib/drbd/notify-split-brain.sh root";
    out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
  }
  startup {
    degr-wfc-timeout 120;    # 2 minutes.
    outdated-wfc-timeout 2;  # 2 seconds.
  }
  disk {
    on-io-error   detach;
    max-bio-bvecs 1;
  }
  net {
    cram-hmac-alg "sha1";
    shared-secret "redacted";
    after-sb-0pri disconnect;
    after-sb-1pri disconnect;
    after-sb-2pri disconnect;
    rr-conflict disconnect;
    data-integrity-alg "md5";
  }
  on carl {
    address    192.168.0.1:7788;
    device     /dev/drbd0;
    disk       /dev/sdb;
    meta-disk  internal;
  }
  on lenny {
    address    192.168.0.2:7788;
    device     /dev/drbd0;
    disk       /dev/sdb;
    meta-disk  internal;
  }
}
############################################################################

12:39:38 carl:~$ pvdisplay
  --- Physical volume ---
  PV Name               /dev/drbd0
  VG Name               vg0
  PV Size               2.73 TiB / not usable 104.65 MiB
  Allocatable           yes
  PE Size               256.00 MiB
  Total PE              11175
  Free PE               4198
  Allocated PE          6977
  PV UUID               9DYozY-RwHA-cFJn-I25c-MWEx-6uoK-CsiZs5

12:40:20 carl:~$ vgdisplay
  --- Volume group ---
  VG Name               vg0
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  19
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                13
  Open LV               12
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               2.73 TiB
  PE Size               256.00 MiB
  Total PE              11175
  Alloc PE / Size       6977 / 1.70 TiB
  Free  PE / Size       4198 / 1.02 TiB
  VG UUID               LPJ1T6-l17x-71rV-rrOR-u4qC-ah9g-eqT6Wd

12:40:23 carl:~$ cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757

12:40:41 carl:~$ drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ ea9e28dbff98e331a62bcbcc63a6135808fe2917\
build\ by\ root at carl.jennic.com\,\ 2010-03-30\ 14:45:34
DRBDADM_API_VERSION=88
DRBD_KERNEL_VERSION_CODE=0x080307
DRBDADM_VERSION_CODE=0x080307
DRBDADM_VERSION=8.3.7

12:39:26 lenny:~$ cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
root at lenny.jennic.com, 2010-03-30 14:46:04

12:41:00 lenny:~$ drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ ea9e28dbff98e331a62bcbcc63a6135808fe2917\
build\ by\ root at lenny.jennic.com\,\ 2010-03-30\ 14:46:25
DRBDADM_API_VERSION=88
DRBD_KERNEL_VERSION_CODE=0x080307
DRBDADM_VERSION_CODE=0x080307
DRBDADM_VERSION=8.3.7

12:41:26 carl:~$ uname -a
Linux carl 2.6.33.4-files #1 SMP Sun May 16 18:09:31 BST 2010 x86_64
Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz GenuineIntel GNU/Linux

12:42:12 lenny:~$ uname -a
Linux lenny 2.6.29.6-ioacct #1 SMP Tue Mar 30 14:23:04 BST 2010 x86_64
Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz GenuineIntel GNU/Linux


Lenny: lvm2-2.02.40
Carl: lvm2-2.02.64

Previous to the weekend Carl was identical to Lenny and there was no
sign of this problem suggesting it could be related to the lvm update,
but I don't know.

Am working on trying to get the partition table, but it's giving me some
trouble. Will provide ASAP.


-- 
Kind Regards,

__________________________________________________

Mike Peachey, IT Systems Administrator
Tel: +44 114 281 2655
Fax: +44 114 281 2951
Jennic Ltd, Furnival Street, Sheffield, S1 4QT, UK
Comp Reg No: 3191371 - Registered In England
http://www.jennic.com
__________________________________________________



More information about the drbd-user mailing list