<div dir="ltr">Hi, <div><br></div><div>Thank you for your help.</div><div><br></div><div>Here are the results of <span style="font-size:13px;font-family:arial,sans-serif">ls -1 /sys/block/*/queue/max_hw_</span><u style="font-size:13px;font-family:arial,sans-serif"></u><span style="font-size:13px;font-family:arial,sans-serif">sect</span><span style="font-size:13px;font-family:arial,sans-serif">ors_kb | xargs -i bash -c 'echo -n "{} " ; cat {}'</span></div><div><br></div><div>Server 1</div><div><div>/sys/block/dm-0/queue/max_hw_sectors_kb 32767</div><div>/sys/block/dm-1/queue/max_hw_sectors_kb 32767</div><div>/sys/block/dm-2/queue/max_hw_sectors_kb 32767</div><div><b>/sys/block/drbd0/queue/max_hw_sectors_kb 1024</b></div><div>/sys/block/loop0/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop1/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop2/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop3/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop4/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop5/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop6/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop7/queue/max_hw_sectors_kb 127</div><div>/sys/block/md0/queue/max_hw_sectors_kb 32767</div><div>/sys/block/md1/queue/max_hw_sectors_kb 32767</div><div>/sys/block/ram0/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram10/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram11/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram12/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram13/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram14/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram15/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram1/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram2/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram3/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram4/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram5/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram6/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram7/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram8/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram9/queue/max_hw_sectors_kb 512</div><div>/sys/block/sda/queue/max_hw_sectors_kb 32767</div><div>/sys/block/sdb/queue/max_hw_sectors_kb 32767</div><div>/sys/block/sr0/queue/max_hw_sectors_kb 128</div></div><div><br></div><div>Server 2 </div><div><div>/sys/block/dm-0/queue/max_hw_sectors_kb 32767</div><div>/sys/block/dm-1/queue/max_hw_sectors_kb 32767</div><div>/sys/block/dm-2/queue/max_hw_sectors_kb 32767</div><div><b>/sys/block/drbd0/queue/max_hw_sectors_kb 1024</b></div><div>/sys/block/loop0/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop1/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop2/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop3/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop4/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop5/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop6/queue/max_hw_sectors_kb 127</div><div>/sys/block/loop7/queue/max_hw_sectors_kb 127</div><div>/sys/block/md0/queue/max_hw_sectors_kb 32767</div><div>/sys/block/md1/queue/max_hw_sectors_kb 32767</div><div>/sys/block/ram0/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram10/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram11/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram12/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram13/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram14/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram15/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram1/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram2/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram3/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram4/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram5/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram6/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram7/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram8/queue/max_hw_sectors_kb 512</div><div>/sys/block/ram9/queue/max_hw_sectors_kb 512</div><div>/sys/block/sda/queue/max_hw_sectors_kb 32767</div><div>/sys/block/sdb/queue/max_hw_sectors_kb 32767</div><div>/sys/block/sr0/queue/max_hw_sectors_kb 128</div></div><div><br></div><div>This is the same value as my other DRBD cluster. Is this an expected value ?</div><div><br></div><div><br></div><div>I tried to run again the disk performance using oflag=direct (ran multiple times)</div><div><br></div><div><div>dd if=/dev/zero of=/home/oracle/output oflag=direct bs=384k count=1k; rm -f</div><div>1024+0 records in</div><div>1024+0 records out</div><div>402653184 bytes (403 MB) copied, 2.26735 s, 178 MB/s</div></div><div><br></div><div>Any other thoughts about my really slow sync ?</div><div><br></div><div><br></div><div>Regards</div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-10-29 21:33 GMT+11:00 Lutz Vieweg <span dir="ltr"><<a href="mailto:lvml@5t9.de" target="_blank">lvml@5t9.de</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 10/29/2014 05:34 AM, aurelien panizza wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I've got two servers connected via a switch with a dedicated 1Gb NIC. We have a SSD raid 1 on both<br>
servers (software RAID on Redhat 6.5).<br>
</blockquote>
<br>
</span><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Full sync between the primary and the secondary is from 3MB/s to 12MB/s<br>
</blockquote>
<br>
</span><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
iostat report 100%util on /dev/drbd0 on the secondary.<br>
</blockquote>
<br></span>
Check whether there's a block device with a low max_hw_sectors_kb value<br>
in your device stack:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
ls -1 /sys/block/*/queue/max_hw_<u></u>sectors_kb | xargs -i bash -c 'echo -n "{} " ; cat {}'<br>
</blockquote>
<br>
If you see small values for max_hw_sectors_kb in one of these, chances<br>
are that just inefficient smalls blocks are transferred - you can<br>
also find a discussion on this in the DRBD mailing list archive, just<br>
search for max_hw_sectors_kb.<br>
<br>
Regards,<br>
<br>
Lutz Vieweg<div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/<u></u>mailman/listinfo/drbd-user</a><br>
</div></div></blockquote></div><br></div>