You're probably FSB bound. Without DRBD you're probably doing direct DMA transfers that don't affect your FSB. You need to split the memory and I/O traffic so they're not on the same bus. You'd do better with a CPU that has an integrated memory controller and separate I/O channels. The performance you're seeing isn't bad for your hardware.<div>
<br></div><div>-JR<br><br><div class="gmail_quote">On Mon, Jan 31, 2011 at 6:11 PM, Mark Petersen <span dir="ltr"><<a href="mailto:mpetersen@peak6.com">mpetersen@peak6.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal">Hi,</p><p class="MsoNormal"> </p><p class="MsoNormal">I’m trying to use DRBD with the backend storage on a FusionIO device (NAND Flash without disk controller overhead.) I’ve found that I’m taking a huge performance hit via testing with fio.</p>
<p class="MsoNormal"> </p><p class="MsoNormal">IOPS</p><p class="MsoNormal">`fio --directory=/fio/test --direct=1 --rw=randread --bs=4k --size=5G --numjobs=64 --runtime=10 --group_reporting --name=file1`</p><p class="MsoNormal">
raw device: 104k</p><p class="MsoNormal">drbd device: 75k</p><p class="MsoNormal"> </p><p class="MsoNormal">Bandwidth</p><p class="MsoNormal">`fio --directory=/fio/test --direct=1 --rw=randread --bs=1m --size=5G --numjobs=4 --runtime=100 --group_reporting --name=file1`</p>
<p class="MsoNormal">raw device: 750 MB/s</p><p class="MsoNormal">drbd device: 550-600 MB/s</p><p class="MsoNormal"> </p><p class="MsoNormal">I’ve tried drbd 8.3.9 and 8.3.10 and while 8.3.10 might be slightly better, it’s mostly insignificant (1-2%.)</p>
<p class="MsoNormal"> </p><p class="MsoNormal">The backend for writing is 2 x 1GB bonded round robin dedicated interfaces, but since I’m seeing a read hit I’m not concerned with this potential write bottle neck at the moment unless it’s somehow related, I plan to use 10 gig cards in the near future if this works out.</p>
<p class="MsoNormal"> </p><p class="MsoNormal">? The DRBD process seems be CPU bound, but performance is worse if I change cpu affinity to be more than 1 core. This is on a E5430 @ 2.66GHz. I imagine a faster CPU could improve performance, but I’m looking for other suggestions of course.</p>
<p class="MsoNormal"> </p><p class="MsoNormal">Other Relevant Details:</p><p class="MsoNormal">CentOS 5.5</p><p class="MsoNormal">Kernel 2.6.18-194</p><p class="MsoNormal">Filesystem XFS</p><p class="MsoNormal">drbd 8.3.9 and 8.3.10 compiled into rpm’s from source (any compile time tweaks I could use, such as CPU type?)</p>
<p class="MsoNormal"> </p><p class="MsoNormal">Thanks,</p><p class="MsoNormal">Mark</p></div><hr>
See <a href="http://www.peak6.com/email_disclaimer.php" target="_blank">http://www.peak6.com/email_disclaimer.php</a> for terms and conditions related to this email
</div><br>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br></div>