Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
May 17 08:00:02 carl kernel: block drbd0: bio would need to, but cannot,
be split: (vcnt=2,idx=0,size=8192,sector=1810514356)
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455629
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455630
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455631
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
May 17 08:00:02 carl kernel: Buffer I/O error on device dm-11, logical
block 41455632
May 17 08:00:02 carl kernel: lost page write due to I/O error on dm-11
And on and on... I/O errors at NFS client.
Two servers.
Each has 3ware 9650SE-12ML Hardware RAID6 (256K stripes).
RAID6 presented to OS as /dev/sdb (3TB)
GPT 3TB single primary partition.
/dev/sdb on each machine is a member of /dev/drbd0 (r0)
drbd0 is an LVM2 Physical Volume and the only volume in the vg.
There are 13 Logical Volumes.
Each LV is XFS.
Each is mounted and exported via NFSv3 and Samba.
/etc/drbd.conf:
############################################################################
global {
usage-count yes;
}
common {
}
resource r0 {
protocol C;
syncer {
rate 110M;
verify-alg "md5";
}
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger
; halt -f";
split-brain "/usr/lib/drbd/notify-split-brain.sh root";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
}
startup {
degr-wfc-timeout 120; # 2 minutes.
outdated-wfc-timeout 2; # 2 seconds.
}
disk {
on-io-error detach;
max-bio-bvecs 1;
}
net {
cram-hmac-alg "sha1";
shared-secret "redacted";
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
data-integrity-alg "md5";
}
on carl {
address 192.168.0.1:7788;
device /dev/drbd0;
disk /dev/sdb;
meta-disk internal;
}
on lenny {
address 192.168.0.2:7788;
device /dev/drbd0;
disk /dev/sdb;
meta-disk internal;
}
}
############################################################################
12:39:38 carl:~$ pvdisplay
--- Physical volume ---
PV Name /dev/drbd0
VG Name vg0
PV Size 2.73 TiB / not usable 104.65 MiB
Allocatable yes
PE Size 256.00 MiB
Total PE 11175
Free PE 4198
Allocated PE 6977
PV UUID 9DYozY-RwHA-cFJn-I25c-MWEx-6uoK-CsiZs5
12:40:20 carl:~$ vgdisplay
--- Volume group ---
VG Name vg0
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 19
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 13
Open LV 12
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 256.00 MiB
Total PE 11175
Alloc PE / Size 6977 / 1.70 TiB
Free PE / Size 4198 / 1.02 TiB
VG UUID LPJ1T6-l17x-71rV-rrOR-u4qC-ah9g-eqT6Wd
12:40:23 carl:~$ cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757
12:40:41 carl:~$ drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ ea9e28dbff98e331a62bcbcc63a6135808fe2917\
build\ by\ root at carl.jennic.com\,\ 2010-03-30\ 14:45:34
DRBDADM_API_VERSION=88
DRBD_KERNEL_VERSION_CODE=0x080307
DRBDADM_VERSION_CODE=0x080307
DRBDADM_VERSION=8.3.7
12:39:26 lenny:~$ cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
root at lenny.jennic.com, 2010-03-30 14:46:04
12:41:00 lenny:~$ drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ ea9e28dbff98e331a62bcbcc63a6135808fe2917\
build\ by\ root at lenny.jennic.com\,\ 2010-03-30\ 14:46:25
DRBDADM_API_VERSION=88
DRBD_KERNEL_VERSION_CODE=0x080307
DRBDADM_VERSION_CODE=0x080307
DRBDADM_VERSION=8.3.7
12:41:26 carl:~$ uname -a
Linux carl 2.6.33.4-files #1 SMP Sun May 16 18:09:31 BST 2010 x86_64
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz GenuineIntel GNU/Linux
12:42:12 lenny:~$ uname -a
Linux lenny 2.6.29.6-ioacct #1 SMP Tue Mar 30 14:23:04 BST 2010 x86_64
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz GenuineIntel GNU/Linux
Lenny: lvm2-2.02.40
Carl: lvm2-2.02.64
Previous to the weekend Carl was identical to Lenny and there was no
sign of this problem suggesting it could be related to the lvm update,
but I don't know.
Am working on trying to get the partition table, but it's giving me some
trouble. Will provide ASAP.
--
Kind Regards,
__________________________________________________
Mike Peachey, IT Systems Administrator
Tel: +44 114 281 2655
Fax: +44 114 281 2951
Jennic Ltd, Furnival Street, Sheffield, S1 4QT, UK
Comp Reg No: 3191371 - Registered In England
http://www.jennic.com
__________________________________________________