<div dir="ltr"><div>Hello,<br><br>I encounter IO errors on my new DRBD installation.<br></div><div><br>Software : Debian 7, DRBD 8.4.4, kernel 3.10.23.<br>Hardware : LSI MegaRAID SAS 9271-8i.<br>Layout : DRBD on top of RAID arrays, LVM on top of DRBD.<br>
<br>DRBD main configuration items :<br>disk {<br> disk-barrier no;<br> disk-flushes no;<br> md-flushes no;<br>}<br>net {<br> protocol C;<br> max-buffers 131072;<br>}<br>resource r0 {<br> volume 0 {<br> device /dev/drbd0;<br>
disk /dev/c0v0;<br> meta-disk internal;<br> }<br> volume 1 {<br> device /dev/drbd1;<br> disk /dev/c0v1;<br> meta-disk internal;<br> }<br> on server1 {<br> address <a href="http://192.168.0.1:7788">192.168.0.1:7788</a>;<br>
}<br> on <a href="http://server2.urbackups.com">server2.urbackups.com</a> {<br> address <a href="http://192.168.0.2:7788">192.168.0.2:7788</a>;<br> }<br>}<br><br>The issue is reproductible.<br>2 times on the primary, when I created / mounted /played with an ext4 FS (over a LV), I got the following error :<br>
<br># Apr 12 12:30:56 server1 kernel: EXT4-fs (dm-1): barriers disabled<br># Apr 12 12:30:56 server1 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: nobarrier,user_xattr<br># Apr 12 12:30:57 server1 kernel: block drbd1: local WRITE IO error sector 10512+4088 on sdb<br>
# Apr 12 12:30:57 server1 kernel: block drbd1: disk( UpToDate -> Failed ) <br># Apr 12 12:30:57 server1 kernel: block drbd1: Local IO failed in __req_mod. Detaching...<br># Apr 12 12:30:57 server1 kernel: block drbd1: bitmap WRITE of 0 pages took 0 jiffies<br>
# Apr 12 12:30:57 server1 kernel: block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br># Apr 12 12:30:57 server1 kernel: block drbd1: disk( Failed -> Diskless ) <br># Apr 12 12:30:57 server1 kernel: drbd r0: sock was shut down by peer<br>
<br># Apr 12 12:51:16 server1 kernel: EXT4-fs (dm-1): barriers disabled<br># Apr 12 12:51:16 server1 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: nobarrier,user_xattr<br># Apr 12 12:52:02 server1 kernel: block drbd0: Remote failed to finish a request within ko-count * timeout<br>
# Apr 12 12:52:02 server1 kernel: block drbd0: peer( Secondary -> Unknown ) conn( Connected -> Timeout ) pdsk( UpToDate -> DUnknown ) <br># Apr 12 12:52:02 server1 kernel: block drbd1: Remote failed to finish a request within ko-count * timeout<br>
# Apr 12 12:52:02 server1 kernel: block drbd1: peer( Secondary -> Unknown ) conn( Connected -> Timeout ) pdsk( UpToDate -> DUnknown ) <br># Apr 12 12:52:02 server1 kernel: block drbd0: new current UUID 8EDD6F6B8037052F:B4686C2EAED0C5BD:0583520A709450C9:0582520A709450C9<br>
# Apr 12 12:52:02 server1 kernel: drbd r0: asender terminated<br># Apr 12 12:52:02 server1 kernel: drbd r0: Terminating drbd_a_r0<br># Apr 12 12:52:02 server1 kernel: block drbd1: helper command: /sbin/drbdadm pri-on-incon-degr minor-1<br>
# Apr 12 12:52:02 server1 kernel: block drbd1: helper command: /sbin/drbdadm pri-on-incon-degr minor-1 exit code 0 (0x0)<br># Apr 12 12:52:02 server1 kernel: drbd r0: Connection closed<br># Apr 12 12:52:02 server1 kernel: dm-1: WRITE SAME failed. Manually zeroing.<br>
# Apr 12 12:52:02 server1 kernel: block drbd1: 3 messages suppressed in /usr/src/drbd-8.4.4/drbd/drbd_req.c:1198.<br># Apr 12 12:52:02 server1 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 10256+8<br>
# Apr 12 12:52:02 server1 kernel: EXT4-fs error (device dm-1): ext4_wait_block_bitmap:466: comm kworker/u82:2: Cannot read block bitmap - block_group = 1, block_bitmap = 1026<br># Apr 12 12:52:02 server1 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 10512+560<br>
# Apr 12 12:52:02 server1 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 11072+560<br><br>I then had some difficulties to reproduct the issue.<br>So I decided to create another new FS, and to promote secondary to primary.<br>
When I mounted the FS on the new primary, I got :<br><br># Apr 12 22:45:39 server2 kernel: block drbd0: role( Secondary -> Primary ) <br># Apr 12 22:45:39 server2 kernel: block drbd0: new current UUID EF985892FD50BA49:48CD646E1D04AC5C:919314E021E03BF4:919214E021E03BF5<br>
# Apr 12 22:45:39 server2 kernel: block drbd1: role( Secondary -> Primary ) <br># Apr 12 22:45:39 server2 kernel: block drbd1: new current UUID 782430D91ABEAF91:480750117C05BF9C:41A69EE3B4A47B3A:41A59EE3B4A47B3B<br># Apr 12 22:45:48 server2 kernel: bio: create slab <bio-2> at 2<br>
# Apr 12 22:46:02 server2 kernel: EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)<br># Apr 12 22:46:02 server2 kernel: block drbd1: local WRITE IO error sector 16808192+4096 on sdb<br># Apr 12 22:46:02 server2 kernel: block drbd1: disk( UpToDate -> Failed ) <br>
# Apr 12 22:46:02 server2 kernel: block drbd1: Local IO failed in __req_mod. Detaching...<br># Apr 12 22:46:02 server2 kernel: block drbd1: helper command: /sbin/drbdadm pri-on-incon-degr minor-1<br># Apr 12 22:46:02 server2 kernel: dm-1: WRITE SAME failed. Manually zeroing.<br>
# Apr 12 22:46:02 server2 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 16808192+560<br># Apr 12 22:46:02 server2 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 16808752+560<br>
# Apr 12 22:46:02 server2 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 16809312+560<br># Apr 12 22:46:02 server2 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 16809872+560<br>
# Apr 12 22:46:02 server2 kernel: block drbd1: IO ERROR: neither local nor remote data, sector 16810432+560<br># Apr 12 22:46:02 server2 kernel: block drbd1: helper command: /sbin/drbdadm pri-on-incon-degr minor-1 exit code 0 (0x0)<br>
# Apr 12 22:46:02 server2 kernel: block drbd1: bitmap WRITE of 1 pages took 0 jiffies<br># Apr 12 22:46:02 server2 kernel: block drbd1: 4 KB (1 bits) marked out-of-sync by on disk bit-map.<br># Apr 12 22:46:02 server2 kernel: block drbd1: disk( Failed -> Diskless ) <br>
<br>No issue at all on RAID controllers' side, their logs are clean.<br><br>Sounds like my issue is the same as this one :<br><a href="http://lists.linbit.com/pipermail/drbd-user/2013-November/020419.html">http://lists.linbit.com/pipermail/drbd-user/2013-November/020419.html</a><br>
<br>Is it a known issue ?<br>With DRBD 8.4.4 ?<br>With kernel 3.10.x ?<br>With both ?<br><br>Could you help me please ?<br><br>Thank you very much,<br><br>Best regards,<br><br>Ben<br><br></div></div>