<HTML dir=ltr><HEAD><TITLE>Re: [DRBD-user] Weird DRBD performance problem</TITLE>
<META content="text/html; charset=unicode" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 9.00.8112.16440"></HEAD>
<BODY>
<DIV dir=ltr id=idOWAReplyText22534>
<DIV dir=ltr><FONT color=#000000 size=2 face="Courier New">Hi Lars and Everyone else,</FONT></DIV>
<DIV dir=ltr><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT size=2 face="Courier New">I put the no-disk-drain in the hope that it might help me find the problem. I leave it out on production.</FONT></DIV>
<DIV dir=ltr><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT size=2 face="Courier New">The run with the various dd's with different commands was interesting:</FONT></DIV>
<DIV dir=ltr><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">san1:/dev # dd if=/dev/zero of=/dev/sdb2 bs=1M count=32768 oflag=direct<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 38.9945 s, 881 MB/s<BR>san1:/dev # dd if=/dev/zero of=/dev/sdb2 bs=1M count=32768 oflag=dsync<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 88.8506 s, 387 MB/s<BR>san1:/dev # dd if=/dev/zero of=/dev/sdb2 bs=1M count=32768 conv=fsync<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 72.9384 s, 471 MB/s</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">Not sure why direct would be so much faster than the others. Something to look into. Anyone have some thoughts on that one?</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">Since I'm trying to figure out what is going on I decided to put the DRBD metadata on a RAM disk and see what happens and did two sets of runs in the same format as the above.</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">Here are the runs with the metadata on the regular RAID system:</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=direct<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 53.1025 s, 647 MB/s<BR>san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=dsync<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 147.199 s, 233 MB/s<BR>san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 conv=fsync<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 156.27 s, 220 MB/s</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">Here are the runs with the metadata on the RAM disk:</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=direct<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 53.6152 s, 641 MB/s<BR>san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 oflag=dsync<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 129.425 s, 265 MB/s<BR>san1:/dev/drbd/by-res # dd if=/dev/zero of=r0 bs=1M count=32768 conv=fsync<BR>32768+0 records in<BR>32768+0 records out<BR>34359738368 bytes (34 GB) copied, 148.076 s, 232 MB/s</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">I did run the tests about 10 times each and they came out to about the same numbers (+/- 15MB/s). Basically, the difference between the metadata being on the disk and something extremely fast doesn't make a difference in this setup. The odd thing is the DRBD overhead is just really high on this system for some reason and I can't seem to find the answer.</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">The firmware between the raid controllers isn't the same. The big difference is that these servers are using the dell H700 card and the other server are using the LSI Megaraid. Both cards are really made by LSI but Dell has its own firmware but they all use the same driver.</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">I also don't understand why the transactions per second as seem by iostat is so high on the drbd device but when it gets to the backing store the transactions drop. Also, when I just run the dd command on the backing store the transactions per second are not high and look just like they do in the drbd test run for the backing device. For some reason the drbd device shows a huge number of transactions.</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">Anymore have some more thoughts about where the overhead hit might be coming from?</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New">thanks</FONT></DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV>
<DIV dir=ltr><FONT face="Courier New"></FONT> </DIV></DIV>
<DIV dir=ltr><BR><FONT face="Courier New">
<HR tabIndex=-1>
</FONT><FONT face="Courier New"><FONT size=2><B>From:</B> drbd-user-bounces@lists.linbit.com on behalf of Lars Ellenberg<BR><B>Sent:</B> Thu 2/2/2012 7:11 AM<BR><B>To:</B> drbd-user@lists.linbit.com<BR><B>Subject:</B> Re: [DRBD-user] Weird DRBD performance problem<BR></FONT><BR></FONT></DIV>
<DIV>
<P><FONT size=2><FONT face="Courier New">On Wed, Feb 01, 2012 at 06:04:18PM -0700, Roof, Morey R. wrote:<BR>> Hi Everyone,<BR>> <BR>> I have a DRBD performance problem that has got me completely confused.<BR>> I hoping that someone can help with this one as my other servers that<BR>> use the same type of RAID cards and DRBD don't have this problem.<BR>> <BR>> For the hardware, I have two Dell R515 servers with the H700 card,<BR>> basically an LSI Megaraid based card, and running SLES 11 SP1. This<BR>> problem shows up on drbd 8.3.11, 8.3.12, and 8.4.1 but I haven't<BR>> tested other versions.<BR>> <BR>> here is the simple config I made based on the servers that don't have<BR>> any issues:<BR>> <BR>> global {<BR>> # We don't want to be bother by the usage count numbers<BR>> usage-count no;<BR>> }<BR>> common {<BR>> protocol C;<BR>> net {<BR>> cram-hmac-alg md5;<BR>> shared-secret "P4ss";<BR>> }<BR>> }<BR>> resource r0 {<BR>> on san1 {<BR>> device /dev/drbd0;<BR>> disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2;<BR>> address 10.60.60.1:63000;<BR>> flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part1;<BR>> }<BR>> on san2 {<BR>> device /dev/drbd0;<BR>> disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part2;<BR>> address 10.60.60.2:63000;<BR>> flexible-meta-disk /dev/disk/by-id/scsi-36782bcb0698b6e00167bb1d107a77a47-part1;<BR>> }<BR>> startup {<BR>> wfc-timeout 5;<BR>> }<BR>> syncer {<BR>> rate 50M;<BR>> cpu-mask 4;<BR>> }<BR>> disk {<BR>> on-io-error detach;<BR>> no-disk-barrier;<BR>> no-disk-flushes;<BR>> no-disk-drain;<BR><BR><BR>Will people please STOP using no-disk-drain. On most hardware, it does<BR>not provide measurable performance gain, but may risk data integrity<BR>because of potential violation of write-after-write dependencies!<BR><BR>> no-md-flushes;<BR>> }<BR>> }<BR>> <BR>> version: 8.3.11 (api:88/proto:86-96)<BR>> GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by phil@fat-tyre <</FONT><A href="mailto:phil@fat-tyre"><FONT face="Courier New">mailto:phil@fat-tyre</FONT></A><FONT face="Courier New">> , 2011-06-29 11:37:11<BR>> 0: cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown C r----s<BR>> ns:0 nr:0 dw:8501248 dr:551 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:n oos:3397375600<BR>> <BR>> So, when I'm running just with one server and no replication the performance hit with DRBD is huge. The backing device shows a throughput of:<BR>> ----<BR>> san1:~ # dd if=/dev/zero of=/dev/disk/by-id/scsi-36782bcb0698b6300167badae13f2884d-part2 bs=1M count=16384<BR><BR>Hope you are not writing to the page cache only?<BR>add oflag=direct, or oflag=dsync, or conv=fsync combinations thereof.<BR><BR>> san1:~ # dd if=/dev/zero of=/dev/drbd/by-res/r0 bs=1M count=16384<BR>> 16384+0 records in<BR>> 16384+0 records out<BR>> 17179869184 bytes (17 GB) copied, 93.457 s, 184 MB/s<BR><BR>See if moving the drbd meta data to raid 1 helps.<BR><BR>> -------<BR>> <BR>> using iostat I see part of the problem:<BR>> <BR>> avg-cpu: %user %nice %system %iowait %steal %idle<BR>> 0.08 0.00 16.76 0.00 0.00 83.17<BR>> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn<BR>> sda 0.00 0.00 0.00 0 0<BR>> sdb 20565.00 0.00 360.00 0 719<BR>> drbd0 737449.50 0.00 360.08 0 720<BR>> avg-cpu: %user %nice %system %iowait %steal %idle<BR>> 0.07 0.00 28.87 1.37 0.00 69.69<BR>> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn<BR>> sda 1.50 0.00 0.01 0 0<BR>> sdb 57859.50 0.00 177.22 0 354<BR>> drbd0 362787.00 0.00 177.14 0 354<BR>> <BR>> the drbd device is showing a TPS about 10x - 20x of the backing store.<BR>> When I do this on my other servers I don't see anything like it. The<BR>> working servers are also running the same kernel and drbd versions.<BR><BR>The rest of the IO stack is the same as well, including driver,<BR>firmware, settings, health of controller cache battery?<BR>Not implying anything, that's just something to check...<BR><BR>--<BR>: Lars Ellenberg<BR>: LINBIT | Your Way to High Availability<BR>: DRBD/HA support and consulting </FONT><A href="http://www.linbit.com/"><FONT face="Courier New">http://www.linbit.com</FONT></A><BR><FONT face="Courier New">_______________________________________________<BR>drbd-user mailing list<BR>drbd-user@lists.linbit.com<BR></FONT><A href="http://lists.linbit.com/mailman/listinfo/drbd-user"><FONT face="Courier New">http://lists.linbit.com/mailman/listinfo/drbd-user</FONT></A><BR></FONT></P></DIV></BODY></HTML>