<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1713965336;
mso-list-type:hybrid;
mso-list-template-ids:-395272144 -242698638 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>Hey guys,<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I’m running into some performance issues with DRBD
8.3.2-6 installed on Centos 5.4 running kernel 2.6.18-164.11.1. I’m
trying to sync a disk across a datacenter with an average latency of 28.8 ms and
a maximum throughput of 135Mb/second (Transferred a 2GB file using SFTP). <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Right now I average 2Mbytes/s <o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal>version: 8.3.2 (api:88/proto:86-90)<o:p></o:p></p>
<p class=MsoNormal>GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by
mockbuild@v20z-x86-64.home.local, 2009-08-29 14:07:55<o:p></o:p></p>
<p class=MsoNormal> 0: cs:SyncSource ro:Primary/Secondary
ds:UpToDate/Inconsistent C r----<o:p></o:p></p>
<p class=MsoNormal> ns:11050876 nr:0 dw:0 dr:11059127 al:0
bm:674 lo:1 pe:4 ua:253 ap:0 ep:1 wo:b oos:2388363296<o:p></o:p></p>
<p class=MsoNormal>
[>....................] sync'ed: 0.5% (2332384/2343176)M<o:p></o:p></p>
<p class=MsoNormal> finish: 362:48:56
speed: 1,824 (1,908) K/sec<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I didn’t build the system but I’m contemplating
rebuilding the machine so I can redo the raid configuration to a RAID 10, but
right now it’s a six disk RAID 5. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>The RAID card information form ‘lspci’<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal>03:00.0 RAID bus controller: LSI Logic / Symbios Logic
MegaRAID SAS 1078 (rev 04)<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I’ve read several threads and tried numerous things
but I haven’t had any success or any change in speed or performance, it
seems like I’m pinned at 2MB/s <o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>sync protocol A, B and C <o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>added deadline /sys/block/sdb/queue/scheduler<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>tried an external meta-disk on a ram disk<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>syncer rate doesn’t seem to change the speed at
all<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>tried no-disk-flushes and no-md-flushes <o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>tried several net settings, changing buffer size<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>One interesting thing I did notice, I was initially
configuring this to backup each other’s drive, e.g. host1:/dev/sdb1 ->
host2:/dev/sdb2 and host2:/dev/sdb1 -> host1:/dev/sdb2, when I had two DRBD
syncs running, both were transferring at 2MB/s so a total 4MB/s, when I
disabled one of the synchronizations, the other transfer still ran at 2MB/s. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Here is my configuration.<o:p></o:p></p>
<p class=MsoNormal>drbd.conf<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal>global { usage-count no; }<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>resource r0 {<o:p></o:p></p>
<p class=MsoNormal> protocol C;<o:p></o:p></p>
<p class=MsoNormal> startup {<o:p></o:p></p>
<p class=MsoNormal>
wfc-timeout 0;<o:p></o:p></p>
<p class=MsoNormal>
degr-wfc-timeout 120;<o:p></o:p></p>
<p class=MsoNormal> }<o:p></o:p></p>
<p class=MsoNormal> disk {<o:p></o:p></p>
<p class=MsoNormal>
no-disk-flushes;<o:p></o:p></p>
<p class=MsoNormal>
no-md-flushes;<o:p></o:p></p>
<p class=MsoNormal> }<o:p></o:p></p>
<p class=MsoNormal> net {<o:p></o:p></p>
<p class=MsoNormal>
cram-hmac-alg "sha1";<o:p></o:p></p>
<p class=MsoNormal>
shared-secret "XXXXXX";<o:p></o:p></p>
<p class=MsoNormal> }<o:p></o:p></p>
<p class=MsoNormal> syncer { rate
25M; }<o:p></o:p></p>
<p class=MsoNormal> on backup01.domain.com
{<o:p></o:p></p>
<p class=MsoNormal>
device /dev/drbd0;<o:p></o:p></p>
<p class=MsoNormal>
disk /dev/sdb1;<o:p></o:p></p>
<p class=MsoNormal>
address 192.168.70.21:7788;<o:p></o:p></p>
<p class=MsoNormal>
meta-disk /dev/ram0[0];<o:p></o:p></p>
<p class=MsoNormal> }<o:p></o:p></p>
<p class=MsoNormal> on backup02.domain.com
{<o:p></o:p></p>
<p class=MsoNormal>
device /dev/drbd0;<o:p></o:p></p>
<p class=MsoNormal>
disk /dev/sdb2;<o:p></o:p></p>
<p class=MsoNormal>
address 192.168.70.22:7788;<o:p></o:p></p>
<p class=MsoNormal>
meta-disk /dev/ram[0];<o:p></o:p></p>
<p class=MsoNormal> }<o:p></o:p></p>
<p class=MsoNormal>}<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I think it’s IO related, but if you guys and gals can
give me any input that would be great, I noticed that it’s bottlenecking
at IO, I’m really thinking about rebuilding these systems with a RAID10,
do you think this will solve my problems? Or is there something else wrong?
Here is the iostat output from both machines. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Backup01 iostat -m<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal>avg-cpu: %user %nice %system %iowait
%steal %idle<o:p></o:p></p>
<p class=MsoNormal>
0.04 0.00 0.12
0.05 0.00 99.78<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Device:
tps MB_read/s MB_wrtn/s
MB_read MB_wrtn<o:p></o:p></p>
<p class=MsoNormal>sda
5.31
0.20
0.03 1425
182<o:p></o:p></p>
<p class=MsoNormal>sda1
0.70
0.05
0.00 375
0<o:p></o:p></p>
<p class=MsoNormal>sda2
0.93
0.07
0.00 496
4<o:p></o:p></p>
<p class=MsoNormal>sda3
1.29
0.05 0.00
365 25<o:p></o:p></p>
<p class=MsoNormal>sda4
0.00
0.00
0.00 0
0<o:p></o:p></p>
<p class=MsoNormal>sda5
0.26
0.00
0.01 1
59<o:p></o:p></p>
<p class=MsoNormal>sda6
1.07
0.01
0.01 90
41<o:p></o:p></p>
<p class=MsoNormal>sda7
0.75
0.01
0.01
92 51<o:p></o:p></p>
<p class=MsoNormal>sda8
0.01
0.00
0.00 0
0<o:p></o:p></p>
<p class=MsoNormal>sdb
46.61
1.84
0.00 12972
0<o:p></o:p></p>
<p class=MsoNormal>sdb1
45.23
1.83
0.00 12901
0<o:p></o:p></p>
<p class=MsoNormal>sdb2
1.29
0.01
0.00 70
0<o:p></o:p></p>
<p class=MsoNormal>drbd0
0.05
0.00
0.00 0
0<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Backup02<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal>avg-cpu: %user %nice %system %iowait
%steal %idle<o:p></o:p></p>
<p class=MsoNormal>
0.04 0.00 0.18
0.13 0.00 99.65<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Device:
tps MB_read/s MB_wrtn/s
MB_read MB_wrtn<o:p></o:p></p>
<p class=MsoNormal>sda
6.09
0.05
0.78 321
5530<o:p></o:p></p>
<p class=MsoNormal>sda1
0.02
0.00
0.00 0
0<o:p></o:p></p>
<p class=MsoNormal>sda2
0.31 0.01
0.00
41 3<o:p></o:p></p>
<p class=MsoNormal>sda3
0.79
0.02
0.00 119
32<o:p></o:p></p>
<p class=MsoNormal>sda4
0.00
0.00
0.00 0
0<o:p></o:p></p>
<p class=MsoNormal>sda5
0.35
0.00
0.01 1
62<o:p></o:p></p>
<p class=MsoNormal>sda6
1.01
0.02
0.00 110
30<o:p></o:p></p>
<p class=MsoNormal>sda7
3.60
0.01
0.76 46
5401<o:p></o:p></p>
<p class=MsoNormal>sda8
0.01
0.00
0.00 0
0<o:p></o:p></p>
<p class=MsoNormal>sdb
44.03
0.00 1.80
1 12794<o:p></o:p></p>
<p class=MsoNormal>sdb1
0.01
0.00
0.00 1
0<o:p></o:p></p>
<p class=MsoNormal>sdb2
44.02
0.00
1.80 0
12794<o:p></o:p></p>
<p class=MsoNormal>###############################################<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks in advance for your help. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Best Regards,<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Dan Lavu <o:p></o:p></p>
</div>
</body>
</html>