On Fri, Dec 19, 2008 at 1:39 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div></div><div class="Wj3C7c">On Fri, Dec 19, 2008 at 09:24:32AM -0500, Gennadiy Nerubayev wrote:<br>
> On Thu, Dec 18, 2008 at 1:50 PM, Lars Ellenberg <<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>><br>
> wrote:<br>
<br>
</div></div>uh. oh.<br>
I have to admit that this was probably not really realistical.<br>
sort of only writing 500MB (odirect, "synchronously"), as that<br>
was what fit into the controller cache...<br>
and it finished subsecond. that's where the number comes from<br>
;)<br>
don't have a real storage backend in the lab capable of sustained<br>
writes in that performance range. (yet.)<br>
<br>
but, I think nothing special, actually,<br>
it was jumbo frames, disabling flow control, and huge max-buffers<br>
and the like, that did the trick, mostly, as well as allowing more than<br>
one core (as one single cpu was maxed out sometimes).</blockquote><div><br>Small update:<br><br>500MB/s makes sense if it's a single burst. What I'm finding is that during a long sync, the speed fluctuates wildly, even though neither the network link nor the storage exhibit such fluctuations on their own. I made a graph showing this effect during a sync lasting ~40 minutes. A script ran cat /proc/drbd ran every second, taking the first speed value. The average after the first minute or two stabilized at ~385MB/s:<br>
<br><a href="http://www.panix.com/~jasonm/drbdspeed-big.jpg">http://www.panix.com/~jasonm/drbdspeed-big.jpg</a> (note: large image)<br></div></div><br>There's a definite pattern, so the question is why is the sync speed (mis)behaving like that..<br>
<br>Thanks,<br><br>-Gennadiy<br>