<div class="gmail_quote">On Thu, Sep 10, 2009 at 6:20 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
[snip]<br>
<br>
I've seen similar symptoms before, and it could be worked around by<br>
disabling offloading settings on the NICs used for the replication ;)<br>
I know, that interaction sounds a bit far-fetched, but those are the<br>
facts.<br>
<br>
# to view offload settings<br>
ethtool -k eth7<br>
# to switch them all off:<br>
ethtool -K eth7 rx off tx off sg off tso off<br>
<br>
<br>
</blockquote></div><br>[root@virtfedbis ~]# ethtool -k eth3<br>Offload parameters for eth3:<br>Cannot get device flags: Operation not supported<br>rx-checksumming: on<br>tx-checksumming: on<br>scatter-gather: on<br>tcp-segmentation-offload: on<br>
udp-fragmentation-offload: off<br>generic-segmentation-offload: on<br>generic-receive-offload: off<br>large-receive-offload: off<br><br>[root@virtfedbis ~]# ethtool -K eth3 rx off tx off sg off tso off<br><br>[root@virtfedbis ~]# ethtool -k eth3<br>
Offload parameters for eth3:<br>Cannot get device flags: Operation not supported<br>rx-checksumming: off<br>tx-checksumming: off<br>scatter-gather: off<br>tcp-segmentation-offload: off<br>udp-fragmentation-offload: off<br>
generic-segmentation-offload: on<br>generic-receive-offload: off<br>large-receive-offload: off<br><br>If I try the attach without doing the same settings on other peer eth3 I get:<br><br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: disk( Diskless -> Attaching ) <br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: Found 6 transactions (244 active extents) in activity log.<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: Method to ensure write ordering: barrier<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: max_segment_size ( = BIO size ) = 32768<br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: recounting of set bits took additional 1 jiffies<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: 920 MB (235520 bits) marked out-of-sync by on disk bit-map.<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: Marked additional 0 KB as out-of-sync based on AL.<br>
Sep 10 18:24:34 virtfedbis kernel: end_request: I/O error, dev cciss/c0d0, sector 0<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: meta data flush failed with status -95, disabling md-flushes<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: disk( Attaching -> Negotiating ) <br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: drbd_sync_handshake:<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: self D5C42445B9F5C227:0000000000000000:0DB564243F5AA9A3:377245292BBD1112 bits:235520 flags:0<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: peer A0332E51B243BEE1:D5C42445B9F5C227:FFEDAA5E725D8157:13925DF660B57F5D bits:309189 flags:0<br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: uuid_compare()=-1 by rule 50<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: Becoming sync target due to disk states.<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: conn( Connected -> WFBitMapT ) disk( Negotiating -> Outdated ) <br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID ) <br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)<br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) <br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: Began resync as SyncTarget (will sync 1236756 KB [309189 bits set]).<br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: Resync aborted.<br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> Failed ) <br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: Local IO failed. Detaching...<br>
Sep 10 18:24:34 virtfedbis kernel: block drbd0: disk( Failed -> Diskless ) <br>Sep 10 18:24:34 virtfedbis kernel: block drbd0: Notified peer that my disk is broken.<br><br>Even after setting same on other peer I get:<br>
<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: disk( Diskless -> Attaching ) <br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: Found 6 transactions (244 active extents) in activity log.<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: Method to ensure write ordering: barrier<br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: max_segment_size ( = BIO size ) = 32768<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: recounting of set bits took additional 1 jiffies<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: 920 MB (235520 bits) marked out-of-sync by on disk bit-map.<br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: Marked additional 0 KB as out-of-sync based on AL.<br>Sep 10 18:26:06 virtfedbis kernel: end_request: I/O error, dev cciss/c0d0, sector 0<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: meta data flush failed with status -95, disabling md-flushes<br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: disk( Attaching -> Negotiating ) <br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: drbd_sync_handshake:<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: self FAFACA8496A4ED9D:0000000000000000:0DB564243F5AA9A3:377245292BBD1112 bits:235520 flags:0<br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: peer A0332E51B243BEE1:FAFACA8496A4ED9D:D5C42445B9F5C227:FFEDAA5E725D8157 bits:310129 flags:0<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: uuid_compare()=-1 by rule 50<br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: Becoming sync target due to disk states.<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: conn( Connected -> WFBitMapT ) disk( Negotiating -> Outdated ) <br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID ) <br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0<br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)<br>
Sep 10 18:26:06 virtfedbis kernel: block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) <br>Sep 10 18:26:06 virtfedbis kernel: block drbd0: Began resync as SyncTarget (will sync 1240516 KB [310129 bits set]).<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: Resync aborted.<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> Failed ) <br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: Local IO failed. Detaching...<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: 1121 messages suppressed in /root/drbd-8.3.3rc1/dist/BUILD/drbd-8.3.3rc1/drbd/drbd_receiver.c:1573.<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: Can not write resync data to local disk.<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: Can not write resync data to local disk.<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: Can not write resync data to local disk.<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: Can not write resync data to local disk.<br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: drbd_rs_complete_io() called, but extent not found<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: disk( Failed -> Diskless ) <br>
Sep 10 18:26:07 virtfedbis kernel: block drbd0: Notified peer that my disk is broken.<br>Sep 10 18:26:07 virtfedbis kernel: block drbd0: Can not write resync data to local disk.<br><br><br>