<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16735" name=GENERATOR></HEAD>
<BODY><!--ppd1000010-->
<DIV><FONT face=Verdana>
<DIV><FONT size=2><SPAN class=678255316-21102008>Can someone help please me
</SPAN>diagnose <SPAN class=678255316-21102008>a </SPAN>DRBD slow<SPAN
class=678255316-21102008> sync problem? </SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT><FONT
size=2><SPAN class=678255316-21102008>The servers have been syncing for 3-4 days
and are now only 12.5% sync'd. </SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>I'm getting conflicting results
from benchmark tests.</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>Summary:</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>netperf is
fast</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>iperf is
fast</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>bonnie++ is
fast</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>
<DIV><FONT size=2><SPAN class=678255316-21102008>dd|nc is extremely
slow</SPAN></FONT></DIV>drbd sync is extremely slow</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008>Details:</SPAN></FONT></DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2><SPAN class=678255316-21102008></SPAN></FONT> </DIV>
<DIV><FONT size=2>Here's iperf...<SPAN class=678255316-21102008> very
fast...</SPAN></FONT></DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT face="courier, monaco, monospace, sans-serif" size=2>[root@ha03
download]# iperf -t TCP_STREAM -c
2.1.1.2<BR>------------------------------------------------------------<BR>Client
connecting to 2.1.1.2, TCP port 5001<BR>TCP window size: 16.0 KByte
(default)<BR>------------------------------------------------------------<BR>[
3] local 2.1.1.1 port 59472 connected with 2.1.1.2 port 5001</FONT></DIV>
<DIV><FONT face="courier, monaco, monospace, sans-serif" size=2>[ ID]
Interval Transfer
Bandwidth<BR>[ 3] 0.0-127.4 sec 13.9 GBytes
939 Mbits/sec</FONT></DIV>
<DIV><FONT face="courier, monaco, monospace, sans-serif"><BR><FONT
size=2> </FONT></DIV></FONT>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2>Now netperf...<SPAN class=678255316-21102008> very
fast...</SPAN></FONT></DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT face="courier, monaco, monospace, sans-serif" size=2>[root@ha03
download]# netperf -H 2.1.1.2<BR>TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0
AF_INET to 2.1.1.2 (2.1.1.2) port 0 AF_INET<BR>Recv
Send Send<BR>Socket Socket Message
Elapsed<BR>Size Size Size
Time Throughput<BR>bytes bytes
bytes secs. 10^6bits/sec</FONT></DIV>
<DIV><FONT face="verdana, helvetica, sans-serif"><FONT size=2><FONT
face="courier, monaco, monospace, sans-serif"> 87380 16384
16384 10.03
938.80</FONT><BR></FONT></FONT></DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>bonnie++ on one server...
fast...</FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>Version
1.03 ------Sequential Output------
--Sequential Input-
--Random-<BR>
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--<BR>Machine Size K/sec %CP
K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP<BR>ha03.mycharts.md 6G
68212 90 187783 31 91553 17 73379 90 271470
28 1099
1<BR>
------Sequential Create------ --------Random
Create--------<BR>
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--<BR>
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec
%CP<BR>
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++<BR>ha03.mycharts.md,6G,68212,90,187783,31,91553,17,73379,90,271470,28,1098.6,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++
++,+++<BR></FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>bonnie++ on the other server...
fast...</FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>Version
1.03 ------Sequential Output------
--Sequential Input-
--Random-<BR>
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--<BR>Machine Size K/sec %CP
K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP<BR>ha04.mycharts.md 6G
69619 91 191459 31 83074 16 72740 89 265057 27
964.9
1<BR>
------Sequential Create------ --------Random
Create--------<BR>
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--<BR>
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec
%CP<BR>
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++<BR>ha04.mycharts.md,6G,69619,91,191459,31,83074,16,72740,89,265057,27,964.9,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,++++</FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>
<DIV><FONT size=+0><FONT size=2></FONT></FONT> </DIV>
<DIV><FONT size=+0><FONT size=2></FONT></FONT> </DIV>
<DIV><FONT size=+0><FONT size=2>Now the dd command...<SPAN
class=678255316-21102008> very slow...</SPAN></FONT></FONT></DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><FONT face="verdana, helvetica, sans-serif"><FONT size=2><FONT
face="courier, monaco, monospace, sans-serif">[root@ha03 download]# dd
if=/dev/zero bs=1M count=100 | nc 2.1.1.2 10000<BR>2+0 records in<BR>1+0 records
out<BR>1048576 bytes (1.0 MB) copied, 12.7668 seconds, 82.1
kB/s</FONT><BR></FONT></FONT></DIV>
<DIV><FONT size=2></FONT> </DIV>
<DIV><SPAN class=678255316-21102008><FONT
size=2></FONT></SPAN> </DIV></FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>drbd stats... very
slow...</FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>[root@ha04 download]# cat
/proc/drbd<BR>version: 8.0.13 (api:86/proto:86)<BR>GIT-hash:
ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by <A
href="mailto:buildsvn@c5-i386-build">buildsvn@c5-i386-build</A>, 2008-08-07
13:42:56<BR> 0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C
r---<BR> ns:0 nr:196 dw:196 dr:0 al:0 bm:1 lo:0 pe:0 ua:0
ap:0<BR> resync: used:0/61 hits:9
misses:1 starving:0 dirty:0
changed:1<BR> act_log: used:0/257
hits:0 misses:0 starving:0 dirty:0 changed:0<BR> 1: cs:SyncTarget
st:Secondary/Primary ds:Inconsistent/UpToDate C r---<BR> ns:0
nr:89232960 dw:89232960 dr:0 al:0 bm:5446 lo:0 pe:0 ua:0
ap:0<BR> [=>..................]
sync'ed: 13.8% (547719/634860)M<BR>
finish: 389:29:20 speed: 320 (316)
K/sec<BR> resync: used:0/61
hits:5571613 misses:5447 starving:0 dirty:0
changed:5447<BR> act_log: used:0/257
hits:0 misses:0 starving:0 dirty:0 changed:0<BR></FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT
size=2>drbd.conf...</FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2><BR>[root@ha04 download]# cat
/etc/drbd.conf<BR># drbd.conf</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2>global {<BR>
usage-count no;<BR>}</FONT></SPAN></DIV>
<DIV> </DIV><SPAN class=678255316-21102008><FONT size=2>
<DIV><BR>common {<BR> syncer {<BR> rate
33M;<BR> al-extents 257;<BR> }<BR>}</DIV>
<DIV> </DIV>
<DIV>resource mysql_ha {<BR> protocol C;<BR> handlers
{<BR> pri-on-incon-degr "echo o > /proc/sysrq-trigger ;
halt -f";<BR> pri-lost-after-sb "echo o >
/proc/sysrq-trigger ; halt -f";<BR> local-io-error "echo o
> /proc/sysrq-trigger ; halt -f";<BR> outdate-peer
"/usr/lib/heartbeat/drbd-peer-outdater -t 5";<BR> #pri-lost
"echo pri-lost. Have a look at the log files. | mail -s 'DRBD Alert'
root";<BR> # split-brain "echo split-brain. drbdadm --
--discard-my-data connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' <A
href="mailto:admin@pmcipa.com">admin@pmcipa.com</A>";<BR>
#out-of-sync "echo out-of-sync. drbdadm down $DRBD_RESOURCE. drbdadm ::::0
set-gi $DRBD_RESOURCE. drbdadm up $DRBD_RESOURCE. | mail -s 'DRBD Alert'
root";<BR> }<BR> startup {<BR> wfc-timeout
0; #
infinite<BR> degr-wfc-timeout 120; # 2
minutes.<BR> }<BR> disk {<BR>
on-io-error detach;<BR> }<BR> net
{<BR> cram-hmac-alg "sha1";<BR>
shared-secret "NerfBurger23!";<BR> after-sb-0pri
disconnect;<BR> after-sb-1pri
disconnect;<BR> after-sb-2pri
disconnect;<BR> rr-conflict disconnect;<BR>
# data-integrity-alg "md5";<BR> }<BR> on ha03.mycharts.md
{<BR> device
/dev/drbd0;<BR> disk
/dev/vg01/lv01;<BR> address
2.1.1.1:7788;<BR> meta-disk internal;<BR>
}<BR> on ha04.mycharts.md {<BR>
device /dev/drbd0;<BR>
disk /dev/vg01/lv01;<BR>
address 2.1.1.2:7788;<BR> meta-disk
internal;<BR> }<BR>}</DIV>
<DIV> </DIV>
<DIV>resource ftp_ha {<BR> protocol C;<BR> handlers
{<BR> pri-on-incon-degr "echo o > /proc/sysrq-trigger ;
halt -f";<BR> pri-lost-after-sb "echo o >
/proc/sysrq-trigger ; halt -f";<BR> local-io-error "echo o
> /proc/sysrq-trigger ; halt -f";<BR> outdate-peer
"/usr/lib/heartbeat/drbd-peer-outdater -t 5";<BR> #pri-lost
"echo pri-lost. Have a look at the log files. | mail -s 'DRBD Alert'
root";<BR> # split-brain "echo split-brain. drbdadm --
--discard-my-data connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' <A
href="mailto:admin@pmcipa.com">admin@pmcipa.com</A>";<BR>
#out-of-sync "echo out-of-sync. drbdadm down $DRBD_RESOURCE. drbdadm ::::0
set-gi $DRBD_RESOURCE. drbdadm up $DRBD_RESOURCE. | mail -s 'DRBD Alert'
root";<BR> }<BR> startup {<BR> wfc-timeout
0; #
infinite<BR> degr-wfc-timeout 120; # 2
minutes.<BR> }<BR> disk {<BR>
on-io-error detach;<BR> }<BR> net
{<BR> cram-hmac-alg "sha1";<BR>
shared-secret "NerfBurger23!";<BR> after-sb-0pri
disconnect;<BR> after-sb-1pri
disconnect;<BR> after-sb-2pri
disconnect;<BR> rr-conflict disconnect;<BR>
# data-integrity-alg "md5";<BR> }<BR> on ha03.mycharts.md
{<BR> device
/dev/drbd1;<BR> disk
/dev/vg01/lv02;<BR> address
2.1.1.1:7789;<BR> meta-disk internal;<BR>
}<BR> on ha04.mycharts.md {<BR>
device /dev/drbd1;<BR>
disk /dev/vg01/lv02;<BR>
address 2.1.1.2:7789;<BR> meta-disk
internal;<BR> }<BR>}<BR></FONT></SPAN></DIV>
<DIV><SPAN class=678255316-21102008><FONT size=2> </DIV></FONT></SPAN>
<DIV><SPAN class=678255316-21102008><FONT size=2> </DIV></FONT></SPAN>
<DIV><SPAN class=678255316-21102008><FONT
size=2></FONT></SPAN></FONT> </DIV></DIV>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt" align=left><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">--<BR></SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">Eric
Robinson<BR></SPAN></P><BR><BR><FONT color=silver><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana"><FONT size=1>Disclaimer - October 21, 2008</FONT></FONT></SPAN></FONT> <P></P><FONT size=1><FONT face=Verdana color=gray><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana">This email and any files <FONT size=1>transmitted</FONT> with it are confidential and intended solely for <STRONG>drbd-user@lists.linbit.com</STRONG></SPAN></FONT><FONT face=Verdana color=gray><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana">. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. <P>This disclaimer was added by Policy Patrol: <A href="http://www.policypatrol.com/">http://www.policypatrol.com</P></A></SPAN></FONT></FONT></BODY></HTML>