Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Excuse the top post:
You seem to think that the mount of /dev/drbd0 on /home on alpha has
something to do with the fdisk output for /dev/sda6. It doesn't. You
probably have done some of the necessary steps out of order.
Yes, drbd appears OK, but the size of /dev/drbd0 on both servers will be
identical. You don't display its size on either server.
After drbd is running and the resources up to date (they are), then you make
the fs on the drbd device and mount it. Since you think the size of /home is
547286323, maybe your system does too (IE, /home was mounted before you made
the file system on /dev/drbd0).
What is in your drbd conf?
Is the drbd resource on /dev/sda or another disk?
The very first error is write beyond the end of the device, so something was
done out of order.
Dan in Atlanta
-----Original Message-----
From: drbd-user-bounces at lists.linbit.com
[mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Keith Christian
Sent: Thursday, August 05, 2010 12:30 PM
To: drbd-user at lists.linbit.com
Subject: [DRBD-user] Buffer I/O and lost page write errors
On an otherwise apparently healthy DRBD server pair, (named
alpha_server and beta_server in the captures below) I see the
following errors in /var/log/messages when data is rsync'ed from a
different machine to this drbd pair.
The first indication of a problem is that the rsync process on the
machine sending to this drbd pair reports a Read-only file system. I
don't know if "read only" is a misnomer from rsync's perspective (the
mount command shows /home mounted as rw (Line 68 below.)
rsync: failed to set times on "/home/user/userdata1000":
Read-only file system (30)
The machine sending the data has around 200gb in use in /home and the
targets have more than twice that in space available, so it it's not a
space problem.
The DRBD partitions are of the same size but the /home partition is
slightly different in size on alpha and beta, alpha's is 547286323
blocks (Line 53) and beta's is 32 blocks larger, 547286355 blocks
(Line 82.)
As I understand it, this shouldn't be a problem.
Any suggestions for further steps to diagnose/fix appreciated.
1 Aug 4 00:07:22 alpha_server kernel: drbd0: rw=1, want=1094572648,
limit=1094572647
2 Aug 4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821580
3 Aug 4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
4 Aug 4 00:07:22 alpha_server kernel: drbd0: rw=1, want=1094572656,
limit=1094572647
5 Aug 4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821581
6 Aug 4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
7 Aug 4 00:07:22 alpha_server kernel: drbd0: rw=1, want=1094572664,
limit=1094572647
8 Aug 4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821582
9 Aug 4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
10 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572672,
limit=1094572647
11 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821583
12 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
13 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572680,
limit=1094572647
14 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821584
15 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
16 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572688,
limit=1094572647
17 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821585
18 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
19 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572696,
limit=1094572647
20 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821586
21 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
22 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572704,
limit=1094572647
23 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821587
24 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
25 Aug 4 00:07:23 alpha_server kernel: Aborting journal on device
drbd0.
26 Aug 4 00:07:23 alpha_server kernel: EXT3-fs error (device drbd0):
ext3_journal_start_sb: Detected aborted journal
27
28
29
30
31 cat /proc/drbd (alpha_server)
32 version: 8.3.2 (api:88/proto:86-90)
33 GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
34 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
35 ns:613059302 nr:8307628 dw:78246580 dr:630584548 al:64211
bm:46024 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
36
37
38
39
40
41 fdisk -l (alpha_server)
42
43 Disk /dev/sda: 584.6 GB, 584652423168 bytes
44 255 heads, 63 sectors/track, 71079 cylinders
45 Units = cylinders of 16065 * 512 = 8225280 bytes
46
47 Device Boot Start End Blocks Id System
48 /dev/sda1 * 1 13 104391 83 Linux
49 /dev/sda2 14 2563 20482875 83 Linux
50 /dev/sda3 2564 2818 2048287+ 82 Linux
swap / Solaris
51 /dev/sda4 2819 71079 548306482+ 5 Extended
52 /dev/sda5 2819 2945 1020096 83 Linux
53 /dev/sda6 2946 71079 547286323+ 83 Linux
54
55
56
57
58
59 mount (alpha_server)
60 /dev/sda2 on / type ext3 (rw)
61 proc on /proc type proc (rw)
62 sysfs on /sys type sysfs (rw)
63 devpts on /dev/pts type devpts (rw,gid=5,mode=620)
64 /dev/sda1 on /boot type ext3 (rw)
65 tmpfs on /dev/shm type tmpfs (rw)
66 none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
67 sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
68 /dev/drbd0 on /home type ext3 (rw)
69
70
71
72
73
74 fdisk -l (beta_server)
75
76 Disk /dev/sda: 1348.4 GB, 1348485513216 bytes
77 255 heads, 63 sectors/track, 163944 cylinders
78 Units = cylinders of 16065 * 512 = 8225280 bytes
79
80 Device Boot Start End Blocks Id System
81 /dev/sda1 * 1 13 104391 83 Linux
82 /dev/sda2 14 68147 547286355 83 Linux
83 /dev/sda3 68148 70697 20482875 83 Linux
84 /dev/sda4 70698 163944 749006527+ 5 Extended
85 /dev/sda5 70698 70952 2048256 82 Linux
swap / Solaris
86 /dev/sda6 70953 71079 1020096 83 Linux
87 /dev/sda7 71080 163944 745938081 83 Linux
88
89
90
91
92
93 mount (beta_server)
94 /dev/sda3 on / type ext3 (rw)
95 proc on /proc type proc (rw)
96 sysfs on /sys type sysfs (rw)
97 devpts on /dev/pts type devpts (rw,gid=5,mode=620)
98 /dev/sda7 on /spare type ext3 (rw)
99 /dev/sda1 on /boot type ext3 (rw)
100 tmpfs on /dev/shm type tmpfs (rw)
101 none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
102
103
104
105
106
107 cat /proc/drbd (beta_server)
108 version: 8.3.2 (api:88/proto:86-90)
109 GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
110 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
111 ns:0 nr:547286323 dw:547286323 dr:0 al:0 bm:33404 lo:0 pe:0
ua:0 ap:0 ep:1 wo:b oos:0
112
=====END OF LISTING=====
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user