[DRBD-user] Buffer I/O and lost page write errors

Dan Barker dbarker at visioncomm.net
Thu Aug 5 19:20:57 CEST 2010


Excuse the top post:

You seem to think that the mount of /dev/drbd0 on /home on alpha has
something to do with the fdisk output for /dev/sda6. It doesn't. You
probably have done some of the necessary steps out of order.

Yes, drbd appears OK, but the size of /dev/drbd0 on both servers will be
identical. You don't display its size on either server.

After drbd is running and the resources up to date (they are), then you make
the fs on the drbd device and mount it. Since you think the size of /home is
547286323, maybe your system does too (IE, /home was mounted before you made
the file system on /dev/drbd0).

What is in your drbd conf? 
Is the drbd resource on /dev/sda or another disk? 
The very first error is write beyond the end of the device, so something was
done out of order.

Dan in Atlanta

-----Original Message-----
From: drbd-user-bounces at lists.linbit.com
[mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Keith Christian
Sent: Thursday, August 05, 2010 12:30 PM
To: drbd-user at lists.linbit.com
Subject: [DRBD-user] Buffer I/O and lost page write errors

On an otherwise apparently healthy DRBD server pair, (named
alpha_server and beta_server in the captures below) I see the
following errors in /var/log/messages when data is rsync'ed from a
different machine to this drbd pair.

The first indication of a problem is that the rsync process on the
machine sending to this drbd pair reports a Read-only file system.  I
don't know if "read only" is a misnomer from rsync's perspective (the
mount command shows /home mounted as rw (Line 68 below.)

        rsync: failed to set times on "/home/user/userdata1000":
Read-only file system (30)

The machine sending the data has around 200gb in use in /home and the
targets have more than twice that in space available, so it it's not a
space problem.

The DRBD partitions are of the same size but the /home partition is
slightly different in size on alpha and beta, alpha's is 547286323
blocks (Line 53) and beta's is 32 blocks larger, 547286355 blocks
(Line 82.)

As I understand it, this shouldn't be a problem.

Any suggestions for further steps to diagnose/fix appreciated.

     1	Aug  4 00:07:22 alpha_server kernel: drbd0: rw=1, want=1094572648,
limit=1094572647
     2	Aug  4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821580
     3	Aug  4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
     4	Aug  4 00:07:22 alpha_server kernel: drbd0: rw=1, want=1094572656,
limit=1094572647
     5	Aug  4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821581
     6	Aug  4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
     7	Aug  4 00:07:22 alpha_server kernel: drbd0: rw=1, want=1094572664,
limit=1094572647
     8	Aug  4 00:07:22 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821582
     9	Aug  4 00:07:22 alpha_server kernel: lost page write due to I/O
error on drbd0
    10	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572672,
limit=1094572647
    11	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821583
    12	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    13	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572680,
limit=1094572647
    14	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821584
    15	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    16	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572688,
limit=1094572647
    17	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821585
    18	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    19	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572696,
limit=1094572647
    20	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821586
    21	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    22	Aug  4 00:07:23 alpha_server kernel: drbd0: rw=1, want=1094572704,
limit=1094572647
    23	Aug  4 00:07:23 alpha_server kernel: Buffer I/O error on device
drbd0, logical block 136821587
    24	Aug  4 00:07:23 alpha_server kernel: lost page write due to I/O
error on drbd0
    25	Aug  4 00:07:23 alpha_server kernel: Aborting journal on device
drbd0.
    26	Aug  4 00:07:23 alpha_server kernel: EXT3-fs error (device drbd0):
ext3_journal_start_sb: Detected aborted journal
    27	
    28	
    29	
    30	
    31	cat /proc/drbd (alpha_server)
    32	version: 8.3.2 (api:88/proto:86-90)
    33	GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
    34	 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
    35	    ns:613059302 nr:8307628 dw:78246580 dr:630584548 al:64211
bm:46024 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
    36	
    37	
    38	
    39	
    40	
    41	fdisk -l (alpha_server)
    42	
    43	Disk /dev/sda: 584.6 GB, 584652423168 bytes
    44	255 heads, 63 sectors/track, 71079 cylinders
    45	Units = cylinders of 16065 * 512 = 8225280 bytes
    46	
    47	   Device Boot      Start         End      Blocks   Id  System
    48	   /dev/sda1   *           1          13      104391   83  Linux
    49	   /dev/sda2              14        2563    20482875   83  Linux
    50	   /dev/sda3            2564        2818     2048287+  82  Linux
swap / Solaris
    51	   /dev/sda4            2819       71079   548306482+   5  Extended
    52	   /dev/sda5            2819        2945     1020096   83  Linux
    53	   /dev/sda6            2946       71079   547286323+  83  Linux
    54	
    55	
    56	
    57	
    58	
    59	mount (alpha_server)
    60	/dev/sda2 on / type ext3 (rw)
    61	proc on /proc type proc (rw)
    62	sysfs on /sys type sysfs (rw)
    63	devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    64	/dev/sda1 on /boot type ext3 (rw)
    65	tmpfs on /dev/shm type tmpfs (rw)
    66	none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    67	sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    68	/dev/drbd0 on /home type ext3 (rw)
    69	
    70	
    71	
    72	
    73	
    74	fdisk -l (beta_server)
    75	
    76	Disk /dev/sda: 1348.4 GB, 1348485513216 bytes
    77	255 heads, 63 sectors/track, 163944 cylinders
    78	Units = cylinders of 16065 * 512 = 8225280 bytes
    79	
    80	   Device Boot      Start         End      Blocks   Id  System
    81	   /dev/sda1   *           1          13      104391   83  Linux
    82	   /dev/sda2              14       68147   547286355   83  Linux
    83	   /dev/sda3           68148       70697    20482875   83  Linux
    84	   /dev/sda4           70698      163944   749006527+   5  Extended
    85	   /dev/sda5           70698       70952     2048256   82  Linux
swap / Solaris
    86	   /dev/sda6           70953       71079     1020096   83  Linux
    87	   /dev/sda7           71080      163944   745938081   83  Linux
    88	
    89	
    90	
    91	
    92	
    93	mount (beta_server)
    94	/dev/sda3 on / type ext3 (rw)
    95	proc on /proc type proc (rw)
    96	sysfs on /sys type sysfs (rw)
    97	devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    98	/dev/sda7 on /spare type ext3 (rw)
    99	/dev/sda1 on /boot type ext3 (rw)
   100	tmpfs on /dev/shm type tmpfs (rw)
   101	none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
   102	
   103	
   104	
   105	
   106	
   107	cat /proc/drbd (beta_server)
   108	version: 8.3.2 (api:88/proto:86-90)
   109	GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild at v20z-x86-64.home.local, 2009-08-29 14:02:24
   110	 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
   111	     ns:0 nr:547286323 dw:547286323 dr:0 al:0 bm:33404 lo:0 pe:0
ua:0 ap:0 ep:1 wo:b oos:0
   112	

=====END OF LISTING=====
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user



More information about the drbd-user mailing list