Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On an otherwise apparently healthy DRBD server pair, (named
alpha_server and beta_server in the captures below) I see the
following errors in /var/log/messages when data is rsync'ed from a
different machine to this drbd pair.
The first indication of a problem is that the rsync process on the
machine sending to this drbd pair reports a Read-only file system. I
don't know if "read only" is a misnomer from rsync's perspective (the
mount command shows /home mounted as rw (Line 68 below.)
rsync: failed to set times on "/home/user/userdata1000":
Read-only file system (30)
The machine sending the data has around 200gb in use in /home and the
targets have more than twice that in space available, so it it's not a
space problem.
The DRBD partitions are of the same size but the /home partition is
slightly different in size on alpha and beta, alpha's is 547286323
blocks (Line 53) and beta's is 32 blocks larger, 547286355 blocks
(Line 82.)
As I understand it, this shouldn't be a problem.
Any suggestions for further steps to diagnose/fix appreciated.
1 Aug 4 00:07:22 alpha_server kernel: drbd0: rw=1,
want=1094572648, limit=1094572647
2 Aug 4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821580
3 Aug 4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
4 Aug 4 00:07:22 alpha_server kernel: drbd0: rw=1,
want=1094572656, limit=1094572647
5 Aug 4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821581
6 Aug 4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
7 Aug 4 00:07:22 alpha_server kernel: drbd0: rw=1,
want=1094572664, limit=1094572647
8 Aug 4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821582
9 Aug 4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
10 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572672, limit=1094572647
11 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821583
12 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
13 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572680, limit=1094572647
14 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821584
15 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
16 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572688, limit=1094572647
17 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821585
18 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
19 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572696, limit=1094572647
20 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821586
21 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
22 Aug 4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572704, limit=1094572647
23 Aug 4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821587
24 Aug 4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
25 Aug 4 00:07:23 alpha_server kernel: Aborting journal on device drbd0.
26 Aug 4 00:07:23 alpha_server kernel: EXT3-fs error (device
drbd0): ext3_journal_start_sb: Detected aborted journal
27
28
29
30
31 cat /proc/drbd (alpha_server)
32 version: 8.3.2 (api:88/proto:86-90)
33 GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
34 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
35 ns:613059302 nr:8307628 dw:78246580 dr:630584548 al:64211
bm:46024 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
36
37
38
39
40
41 fdisk -l (alpha_server)
42
43 Disk /dev/sda: 584.6 GB, 584652423168 bytes
44 255 heads, 63 sectors/track, 71079 cylinders
45 Units = cylinders of 16065 * 512 = 8225280 bytes
46
47 Device Boot Start End Blocks Id System
48 /dev/sda1 * 1 13 104391 83 Linux
49 /dev/sda2 14 2563 20482875 83 Linux
50 /dev/sda3 2564 2818 2048287+ 82
Linux swap / Solaris
51 /dev/sda4 2819 71079 548306482+ 5 Extended
52 /dev/sda5 2819 2945 1020096 83 Linux
53 /dev/sda6 2946 71079 547286323+ 83 Linux
54
55
56
57
58
59 mount (alpha_server)
60 /dev/sda2 on / type ext3 (rw)
61 proc on /proc type proc (rw)
62 sysfs on /sys type sysfs (rw)
63 devpts on /dev/pts type devpts (rw,gid=5,mode=620)
64 /dev/sda1 on /boot type ext3 (rw)
65 tmpfs on /dev/shm type tmpfs (rw)
66 none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
67 sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
68 /dev/drbd0 on /home type ext3 (rw)
69
70
71
72
73
74 fdisk -l (beta_server)
75
76 Disk /dev/sda: 1348.4 GB, 1348485513216 bytes
77 255 heads, 63 sectors/track, 163944 cylinders
78 Units = cylinders of 16065 * 512 = 8225280 bytes
79
80 Device Boot Start End Blocks Id System
81 /dev/sda1 * 1 13 104391 83 Linux
82 /dev/sda2 14 68147 547286355 83 Linux
83 /dev/sda3 68148 70697 20482875 83 Linux
84 /dev/sda4 70698 163944 749006527+ 5 Extended
85 /dev/sda5 70698 70952 2048256 82
Linux swap / Solaris
86 /dev/sda6 70953 71079 1020096 83 Linux
87 /dev/sda7 71080 163944 745938081 83 Linux
88
89
90
91
92
93 mount (beta_server)
94 /dev/sda3 on / type ext3 (rw)
95 proc on /proc type proc (rw)
96 sysfs on /sys type sysfs (rw)
97 devpts on /dev/pts type devpts (rw,gid=5,mode=620)
98 /dev/sda7 on /spare type ext3 (rw)
99 /dev/sda1 on /boot type ext3 (rw)
100 tmpfs on /dev/shm type tmpfs (rw)
101 none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
102
103
104
105
106
107 cat /proc/drbd (beta_server)
108 version: 8.3.2 (api:88/proto:86-90)
109 GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
110 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
111 ns:0 nr:547286323 dw:547286323 dr:0 al:0 bm:33404 lo:0
pe:0 ua:0 ap:0 ep:1 wo:b oos:0
112
=====END OF LISTING=====