On Sat, May 2, 2009 at 2:06 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
nice. you are going into page cache here, mostly<br>
(how much RAM did you say you have?)<br>
</blockquote><div><br>
4GB on one node, 8GB on the other. I verified that benchmark numbers are the same on both nodes.<br>
<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 oflag=direct<br>
> 4194304000 bytes (4.2 GB) copied, 17.6633 seconds, 237 MB/s<br>
<br>
do variations.<br>
use 512k, 1M, 2M, 100M.<br>
</blockquote><div><br>
After rerunning these various sizes, there is no appreciable
performance difference from bs=4M, in either connected or disconnected
mode, across all flags. I didn't do connected benchmarks on a single core, as the resync performance dropped by 100MB/s with only one core..<br>
<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
repeat measurements.<br>
see how reproducable they are.</blockquote><div><br>
Very consistent, with a margin of error of a few megabytes one way or the other; no significant fluctuations. <br>
<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
also read up about the "al-extents" drbd.conf parameter, "max-buffers",<br>
and the rest of the usual tuning stuff. the activity log updates will<br>
slow you down on streaming writes.</blockquote><div><br>
Neither has any noticeable effect (buffered and fsync may actually be a
little slower with larger max-buffers), but I am running with the
metadata on a ramdisk..<br>
<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
what is your rtt, small packets, large packets?<br>
ping -w 5 -f -s 64 other-node<br>
ping -w 5 -f -s 5000 other-node</blockquote><div><br>
PING 10.0.0.3 (10.0.0.3) 64(92) bytes of data.<br>
.<br>
--- 10.0.0.3 ping statistics ---<br>
147010 packets transmitted, 147010 received, 0% packet loss, time 5001ms<br>
rtt min/avg/max/mdev = 0.020/0.023/0.153/0.003 ms, ipg/ewma 0.034/0.023 ms<br>
<br>
PING 10.0.0.3 (10.0.0.3) 5000(5028) bytes of data.<br>
<br>
--- 10.0.0.3 ping statistics ---<br>
70976 packets transmitted, 70976 received, 0% packet loss, time 5001ms<br>
rtt min/avg/max/mdev = 0.043/0.047/0.164/0.009 ms, ipg/ewma 0.070/0.049 ms<br>
<br>
Hope that helps,<br>
<br>
-Gennadiy<br>
</div>