[DRBD-user] PROBLEM:"Digest mismatch, buffer modified by upper layers during write" happend again and again

活在梦中 lixunix at foxmail.com
Fri Mar 28 09:37:53 CET 2014

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


hI:   i am from china,i have a  some serious problem with my drbd,we did a ha for our mysql database with drbd ,everything is ok when we did not use NIC bond,but when we bond the nic with mode 1(active-backup),the link between the drbd perfom strange,just disconect-unknown-connect,following is the log:
------------------------------


Mar 27 13:57:58 mysql03 kernel: block drbd0: updated sync UUID B0A60E84594BD05F:9C83E27AB8FC9A75:9C82E27AB8FC9A75:9C81E27AB8FC9A75
Mar 27 13:57:58 mysql03 kernel: block drbd0: Digest mismatch, buffer modified by upper layers during write: 999570272s +4096
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: sock was shut down by peer
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: peer( Secondary -> Unknown ) conn( SyncSource -> BrokenPipe )
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: short read (expected size 16)
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: asender terminated
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: Terminating drbd_a_clusterd
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: Connection closed
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: conn( BrokenPipe -> Unconnected )
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: receiver terminated
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: Restarting receiver thread
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: receiver (re)started
Mar 27 13:57:58 mysql03 kernel: drbd clusterdb_res: conn( Unconnected -> WFConnection )
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Handshake successful: Agreed network protocol version 101
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Agreed to support TRIM on protocol level
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Peer authenticated using 20 bytes HMAC
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: conn( WFConnection -> WFReportParams )
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Starting asender thread (from drbd_r_clusterd [60474])
Mar 27 13:57:59 mysql03 kernel: block drbd0: drbd_sync_handshake:
Mar 27 13:57:59 mysql03 kernel: block drbd0: self B0A60E84594BD05F:9C83E27AB8FC9A75:9C82E27AB8FC9A75:9C81E27AB8FC9A75 bits:155141533 flags:0
Mar 27 13:57:59 mysql03 kernel: block drbd0: peer 9C83E27AB8FC9A74:0000000000000000:86B283665E3AEF84:86B183665E3AEF85 bits:155141256 flags:0
Mar 27 13:57:59 mysql03 kernel: block drbd0: uuid_compare()=1 by rule 70
Mar 27 13:57:59 mysql03 kernel: block drbd0: Becoming sync source due to disk states.
Mar 27 13:57:59 mysql03 kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS )
Mar 27 13:57:59 mysql03 kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 952(1), total 952; compression: 100.0%
Mar 27 13:57:59 mysql03 kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 952(1), total 952; compression: 100.0%
Mar 27 13:57:59 mysql03 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0
Mar 27 13:57:59 mysql03 kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
Mar 27 13:57:59 mysql03 kernel: block drbd0: conn( WFBitMapS -> SyncSource )
Mar 27 13:57:59 mysql03 kernel: block drbd0: Began resync as SyncSource (will sync 620566148 KB [155141537 bits set]).
Mar 27 13:57:59 mysql03 kernel: block drbd0: updated sync UUID B0A60E84594BD05F:9C84E27AB8FC9A75:9C83E27AB8FC9A75:9C82E27AB8FC9A75
Mar 27 13:57:59 mysql03 kernel: block drbd0: Digest mismatch, buffer modified by upper layers during write: 999570424s +4096
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: sock was shut down by peer
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: peer( Secondary -> Unknown ) conn( SyncSource -> BrokenPipe )
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: short read (expected size 16)
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: asender terminated
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Terminating drbd_a_clusterd
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Connection closed
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: conn( BrokenPipe -> Unconnected )
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: receiver terminated
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Restarting receiver thread
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: receiver (re)started
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: conn( Unconnected -> WFConnection )
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Handshake successful: Agreed network protocol version 101
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Agreed to support TRIM on protocol level
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Peer authenticated using 20 bytes HMAC
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: conn( WFConnection -> WFReportParams )
Mar 27 13:57:59 mysql03 kernel: drbd clusterdb_res: Starting asender thread (from drbd_r_clusterd [60474])
Mar 27 13:57:59 mysql03 kernel: block drbd0: drbd_sync_handshake:





---------------------------------------------------------






and this hanppend again and again
we can not find where the problem is ,i have read the DRBD'S user guide,it recommand to use bond with mode active-backup,i want to know is it the problem for our situation?


the following  is our config:


resource  clusterdb_res {
  device        /dev/drbd0;
  disk          /dev/sda3;
  meta-disk     internal;
  on mysql03 {
    address     10.10.10.70:7791;
  }
  on mysql01 {
    address     10.10.10.71:7791;
  }
}


[root at mysql03 drbd.d]# cat global_common.conf
global {
        usage-count yes;
}


common  {
        protocol C;


        handlers {
                pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                #split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                #out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
                 }


        startup {
        }


        disk {
                on-io-error detach;
        }


        net {
                cram-hmac-alg sha1;
                shared-secret "UcfGroup";
                data-integrity-alg sha1;
                after-sb-0pri disconnect;
                after-sb-1pri disconnect;
                after-sb-2pri disconnect;
       }
       syncer {
                #on-no-data-accessible io-error;
                #verify-alg sha1;
                rate 1000M;
       }
}






-----------------------next is the  bond information-----


[root at mysql03 bonding]# cat bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0


Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:16:2d:88:28:08
Slave queue ID: 0


Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: f4:ce:46:a5:81:94
Slave queue ID: 0



------------------------------------------




10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether ac:16:2d:88:28:08 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.70/24 brd 10.10.10.255 scope global bond0
    inet 10.10.10.100/24 brd 10.10.10.255 scope global secondary bond0
    inet6 fe80::ae16:2dff:fe88:2808/64 scope link
       valid_lft forever preferred_lft forever





--------------------------------------release info--------------------------------------------------


[root at mysql03 bonding]# uname -a
Linux mysql03 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root at mysql03 bonding]# drbdadm -V
DRBDADM_BUILDTAG=GIT-hash:\ 599f286440bd633d15d5ff985204aff4bccffadd\ build\ by\ mockbuild at Build64R6\,\ 2013-10-25\ 09:52:00
DRBDADM_API_VERSION=1
DRBD_KERNEL_VERSION_CODE=0x080404
DRBDADM_VERSION_CODE=0x080404
DRBDADM_VERSION=8.4.4



-------------------------------------------------------------
so ,i am so appreciate  if you can give me so clue to solve the problem,if you want more information about our config ,just let me know, 3q


waiting online ~~~
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20140328/050e1c97/attachment.htm>


More information about the drbd-user mailing list