<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-15">
</head>
<body bgcolor="#ffffff" text="#000000">
<div class="moz-text-plain" wrap="true"
style="font-family: -moz-fixed; font-size: 17px;" lang="x-western"><tt></tt><tt><span
class="moz-txt-citetags"></span></tt><span class="moz-txt-citetags"></span><span
class="moz-txt-citetags"></span><span class="moz-txt-citetags"></span>Hello
DRBD users,<br>
<br>
<br>
<tt>to measure the gain of an external metadisk</tt><tt><span
class="moz-txt-citetags"> </span>I have done some performance tests
with</tt><span class="moz-txt-citetags"> </span>two different setups.<span
class="moz-txt-citetags"> </span>Setup A having internal metadata an
a RAID 10 (SAS) and Setup B with<span class="moz-txt-citetags"> </span>additional
RAID 1 of SSDs used as metadata disk:<br>
<br>
Setup A:<span class="moz-txt-citetags"><tt><br>
</tt></span>Data : SAS RAID 10<span class="moz-txt-citetags"><tt><br>
</tt></span>Metadata : Internal SAS RAID 10<span
class="moz-txt-citetags"><tt><br>
</tt></span><tt><br>
</tt>Setup B:<span class="moz-txt-citetags"><tt><br>
</tt></span>Data : SAS RAID 10<span class="moz-txt-citetags"><tt><br>
</tt></span>Metadata : External SSD RAID 1<span
class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><br>
</span><br>
In both setups I used the same LSI controller with 512 MB RAM.<span
class="moz-txt-citetags"></span><tt><span class="moz-txt-citetags"><br>
<br>
</span>For the tests I restricted the available </tt><tt>kernel</tt><span
class="moz-txt-citetags"> </span>memory to 384 MB via mem boot<tt><span
class="moz-txt-citetags"> </span>param to </tt><tt>avoid</tt><tt><span
class="moz-txt-citetags"></span> measuring the </tt><tt>linux</tt><span
class="moz-txt-citetags"> </span> cache performance. <span
class="moz-smiley-s1" title=":)"><span>:)</span></span><span
class="moz-txt-citetags"><tt><br>
<br>
</tt></span>I did not tweak the standard drbd.conf that much, except
that I set a<span class="moz-txt-citetags"> </span>higher syncer rate.<tt><span
class="moz-txt-citetags"></span></tt><span class="moz-txt-citetags"><br>
<br>
</span>I can see an obvious difference if I run the test on the<span
class="moz-txt-citetags"> </span>standalone devices (SAS RAID 10, SSD
RAID 1), e.g.<span class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><tt><br>
</tt></span>SAS RAID 10 without DRBD (rate and latency):<span
class="moz-txt-citetags"><tt><br>
</tt></span>root@helios:/mnt/sas# tiotest -t 8 -f 2000 -r 20000<span
class="moz-txt-citetags"><tt><br>
</tt></span>[...]<tt><span class="moz-txt-citetags"></span></tt><span
class="moz-txt-citetags"></span><br>
| Random Write 625 MBs | 115.6 s | 5.404 MB/s | 0.7 % | 3.7 % |<span
class="moz-txt-citetags"><tt><br>
</tt></span>[...]<tt></tt><span class="moz-txt-citetags"><tt><br>
</tt></span>| Random Write | 5.335 ms | 4713.499 ms | 0.06813 |
0.00000 |<span class="moz-txt-citetags"><tt><br>
</tt></span>[...]<span class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><br>
</span>vs.<span class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><tt><br>
</tt></span>SSD RAID 1 without DRBD (rate and latency):<span
class="moz-txt-citetags"><tt><br>
</tt></span>root@helios:/mnt/ssd# tiotest -t 8 -f 2000 -r 20000<span
class="moz-txt-citetags"><br>
</span>[...]<tt><span class="moz-txt-citetags"></span><br>
</tt>| Random Write 625 MBs | 12.6 s | 49.643 MB/s | 0.8 % | 18.0 % |<span
class="moz-txt-citetags"><tt><br>
</tt></span>[...]<tt><span class="moz-txt-citetags"></span></tt><tt></tt><span
class="moz-txt-citetags"><tt><br>
</tt></span>| Random Write | 0.591 ms | 4022.635 ms | 0.00500 |
0.00000 |<span class="moz-txt-citetags"><tt><br>
</tt></span>[...]<br>
<tt><br>
-> The random write rate is ten times higher as well as the
latency is ten times lower for the SSDs.</tt><span
class="moz-txt-citetags"><tt><br>
<br>
</tt></span>there is nearly no difference between the internal/external
metadata<span class="moz-txt-citetags"> </span>setups. Using the same
tests as above I only get a performance<span class="moz-txt-citetags"> </span>improvement
of 10% (concerning random write):<br>
<span class="moz-txt-citetags"></span><span class="moz-txt-citetags"></span><br>
Setup A<tt><span class="moz-txt-citetags"></span></tt><tt> (metadisk
internal, rate and latency)</tt><span class="moz-txt-citetags"><tt>:<br>
</tt></span>root@helios:/mnt/drbd# tiotest -t 8 -f 2000 -r 20000<span
class="moz-txt-citetags"><tt><br>
</tt></span>[...]<tt><span class="moz-txt-citetags"></span></tt><span
class="moz-txt-citetags"><tt><br>
</tt></span>| Random Write 625 MBs | 112.8 s | 5.543 MB/s | 8.7 % |
193.6 % |<span class="moz-txt-citetags"><tt><br>
</tt></span>[...<tt>]<br>
<span class="moz-txt-citetags"></span></tt><tt></tt><span
class="moz-txt-citetags"><tt> </tt> </span>| Random Write | 5.063 ms
| 5566.188 ms | 0.02562 | <span class="moz-txt-citetags"></span>0.00000
|<span class="moz-txt-citetags"><tt><br>
</tt></span>[...]<span class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><br>
</span>vs.<span class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><tt><br>
</tt></span>Setup B (metadisk external, rate and latency):<span
class="moz-txt-citetags"><br>
</span>root@helios:/mnt/drbd# tiotest -t 8 -f 2000 -r 20000<span
class="moz-txt-citetags"><tt><br>
</tt></span>[...]<span class="moz-txt-citetags"><tt><br>
</tt></span>| Random Write 625 MBs | 102.3 s | 6.110 MB/s | 2.3 % |
109.5 % |<span class="moz-txt-citetags"><tt><br>
</tt></span>[...]<tt><span class="moz-txt-citetags"></span></tt><span
class="moz-txt-citetags"><br>
</span>| Random Write | 4.558 ms | 4089.111 ms | 0.04813 | <span
class="moz-txt-citetags"></span>0.00000 |<span class="moz-txt-citetags"><tt><br>
</tt></span>[...]<span class="moz-txt-citetags"><tt><br>
</tt></span><span class="moz-txt-citetags"><tt><br>
</tt></span>I already read Florian Haas' blog post<span
class="moz-txt-citetags"> </span>(<a class="moz-txt-link-freetext"
href="http://fghaas.wordpress.com/2009/08/20/internal-metadata-and-why-we-recommend-it/">http://fghaas.wordpress.com/2009/08/20/internal-metadata-and-why-we-recommend-it/</a>)<tt><span
class="moz-txt-citetags"> </span>and also </tt><tt><span
class="moz-txt-citetags"></span>got </tt><span
class="moz-txt-citetags"></span>the points made in february on this
list, but I'm wondering<span class="moz-txt-citetags"> </span>if the
effect of (not) using external metadata is really that small or<span
class="moz-txt-citetags"> </span>if I made some serious mistakes as
far as my test setup is concerned.<br>
<br>
<br>
Tanks in advance<br>
<br>
Patrick<br>
<tt></tt></div>
</body>
</html>