[DRBD-user] Buffer I/O and lost page write errors

Keith Christian keith1christian at gmail.com
Thu Aug 5 18:30:08 CEST 2010


On an otherwise apparently healthy DRBD server pair, (named
alpha_server and beta_server in the captures below) I see the
following errors in /var/log/messages when data is rsync'ed from a
different machine to this drbd pair.

The first indication of a problem is that the rsync process on the
machine sending to this drbd pair reports a Read-only file system.  I
don't know if "read only" is a misnomer from rsync's perspective (the
mount command shows /home mounted as rw (Line 68 below.)

        rsync: failed to set times on "/home/user/userdata1000":
Read-only file system (30)

The machine sending the data has around 200gb in use in /home and the
targets have more than twice that in space available, so it it's not a
space problem.

The DRBD partitions are of the same size but the /home partition is
slightly different in size on alpha and beta, alpha's is 547286323
blocks (Line 53) and beta's is 32 blocks larger, 547286355 blocks
(Line 82.)

As I understand it, this shouldn't be a problem.

Any suggestions for further steps to diagnose/fix appreciated.

     1	Aug  4 00:07:22 alpha_server kernel: drbd0: rw=1,
want=1094572648, limit=1094572647
     2	Aug  4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821580
     3	Aug  4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
     4	Aug  4 00:07:22 alpha_server kernel: drbd0: rw=1,
want=1094572656, limit=1094572647
     5	Aug  4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821581
     6	Aug  4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
     7	Aug  4 00:07:22 alpha_server kernel: drbd0: rw=1,
want=1094572664, limit=1094572647
     8	Aug  4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821582
     9	Aug  4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
    10	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572672, limit=1094572647
    11	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821583
    12	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    13	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572680, limit=1094572647
    14	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821584
    15	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    16	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572688, limit=1094572647
    17	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821585
    18	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    19	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572696, limit=1094572647
    20	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821586
    21	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    22	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1,
want=1094572704, limit=1094572647
    23	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821587
    24	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    25	Aug  4 00:07:23 alpha_server kernel: Aborting journal on device drbd0.
    26	Aug  4 00:07:23 alpha_server kernel: EXT3-fs error (device
drbd0): ext3_journal_start_sb: Detected aborted journal
    27	
    28	
    29	
    30	
    31	cat /proc/drbd (alpha_server)
    32	version: 8.3.2 (api:88/proto:86-90)
    33	GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
    34	 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
    35	    ns:613059302 nr:8307628 dw:78246580 dr:630584548 al:64211
bm:46024 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
    36	
    37	
    38	
    39	
    40	
    41	fdisk -l (alpha_server)
    42	
    43	Disk /dev/sda: 584.6 GB, 584652423168 bytes
    44	255 heads, 63 sectors/track, 71079 cylinders
    45	Units = cylinders of 16065 * 512 = 8225280 bytes
    46	
    47	   Device Boot      Start         End      Blocks   Id  System
    48	   /dev/sda1   *           1          13      104391   83  Linux
    49	   /dev/sda2              14        2563    20482875   83  Linux
    50	   /dev/sda3            2564        2818     2048287+  82
Linux swap / Solaris
    51	   /dev/sda4            2819       71079   548306482+   5  Extended
    52	   /dev/sda5            2819        2945     1020096   83  Linux
    53	   /dev/sda6            2946       71079   547286323+  83  Linux
    54	
    55	
    56	
    57	
    58	
    59	mount (alpha_server)
    60	/dev/sda2 on / type ext3 (rw)
    61	proc on /proc type proc (rw)
    62	sysfs on /sys type sysfs (rw)
    63	devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    64	/dev/sda1 on /boot type ext3 (rw)
    65	tmpfs on /dev/shm type tmpfs (rw)
    66	none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    67	sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    68	/dev/drbd0 on /home type ext3 (rw)
    69	
    70	
    71	
    72	
    73	
    74	fdisk -l (beta_server)
    75	
    76	Disk /dev/sda: 1348.4 GB, 1348485513216 bytes
    77	255 heads, 63 sectors/track, 163944 cylinders
    78	Units = cylinders of 16065 * 512 = 8225280 bytes
    79	
    80	   Device Boot      Start         End      Blocks   Id  System
    81	   /dev/sda1   *           1          13      104391   83  Linux
    82	   /dev/sda2              14       68147   547286355   83  Linux
    83	   /dev/sda3           68148       70697    20482875   83  Linux
    84	   /dev/sda4           70698      163944   749006527+   5  Extended
    85	   /dev/sda5           70698       70952     2048256   82
Linux swap / Solaris
    86	   /dev/sda6           70953       71079     1020096   83  Linux
    87	   /dev/sda7           71080      163944   745938081   83  Linux
    88	
    89	
    90	
    91	
    92	
    93	mount (beta_server)
    94	/dev/sda3 on / type ext3 (rw)
    95	proc on /proc type proc (rw)
    96	sysfs on /sys type sysfs (rw)
    97	devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    98	/dev/sda7 on /spare type ext3 (rw)
    99	/dev/sda1 on /boot type ext3 (rw)
   100	tmpfs on /dev/shm type tmpfs (rw)
   101	none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
   102	
   103	
   104	
   105	
   106	
   107	cat /proc/drbd (beta_server)
   108	version: 8.3.2 (api:88/proto:86-90)
   109	GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
   110	 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
   111	     ns:0 nr:547286323 dw:547286323 dr:0 al:0 bm:33404 lo:0
pe:0 ua:0 ap:0 ep:1 wo:b oos:0
   112	

=====END OF LISTING=====


More information about the drbd-user mailing list