<div dir="ltr">Hello,<br><br>I think I need your help :)<br><br>Servers :<br>2 identical servers, Xeon E5-2670 v2 2.5 GHz (40 cores), 128 GB of RAM, RAID LSI 9271-8i.<br><br>Storage :<br>RAID 10 storage, throughput with dd and bonnie++ gives 680 MB/s read and write (expected value).<br>
<br>Network :<br>Replication link is a a 10 Ge link, throughput with iperf (6 threads or more) gives 7.7 Gb/s (918 MB/s) bidirectional.<br>Could be tuned, but well, enough to sustain my RAID array at 680 MB/s.<br>Latency of 10.8 ms between the 2 servers.<br>
MTU 9012.<br><br>Metadata to suit RAID array layout :<br>drbdmeta 1 v08 /dev/c0v1 internal create-md --al-stripes 4 --al-stripe-size-kB 32<br>al-extents 6433<br><br>Software :<br>Debian stable / 7 Wheezy<br>Kernel 3.10.23<br>
DRBD 8.4.4<br><br>Problem :<br>I can't manage to reach 680 MB/s on initial replication.<br><br>Initial configuration :<br>protocol C<br>disk-barrier no<br>disk-flushes no<br>resync-rate 680M<br>c-plan-ahead 0 (to force max throughput at resync-rate value, for test purpose)<br>
# finish: 103:20:15 speed: 41,976 (41,976) want: 696,320 K/sec<br><br>Guides studied :<br><a href="http://www.drbd.org/users-guide/s-throughput-tuning.html">http://www.drbd.org/users-guide/s-throughput-tuning.html</a><br>
<a href="http://www.drbd.org/users-guide/s-latency-tuning.html">http://www.drbd.org/users-guide/s-latency-tuning.html</a><br><br>Tests done :<br>Setting max-buffers to its max value 131072 gives the best improvment :<br># finish: 16:25:16 speed: 264,036 (257,292) want: 696,320 K/sec<br>
Tuning of other parameters give let's say nothing at all (max-epoch-size, unplug-watermark, sndbuf-size, scheduler)...<br><br>Questions :<br>How to be able to sustain synchronisation at max RAID array throughput ?<br>
Perhaps first, how to find the bottleneck ?<br><br>Thank you very much for your support,<br><br>Best regards,<br><br>Ben<br><br></div>