<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Arial; font-size: 12pt; color: #000000'><span style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; ">Hello All</span><div style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; "><span><br></span></div><div style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; "><span>I'm facing a strange problem after the reinstallation of our storage cluster. We upgraded the raid controller and the interconnect to 10GbE. The Storage is very fast now: </span></div><div style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; "><span><br></span></div><div style="color: rgb(0, 0, 0); font-family: Arial; "><div><font size="2"><b>dd if=/dev/zero of=/dev/sdb1 bs=512M count=12 oflag=direct</b></font></div><div><font size="2">12+0 Datensätze ein</font></div><div><font size="2">12+0 Datensätze aus</font></div><div><font size="2">6442450944 Bytes (6,4 GB) kopiert, 10,5372 s, 611 MB/s</font></div><div style="font-size: 12pt; "><br></div><div style="font-size: 12pt; ">As also the 10GbE interconnect: </div></div><div><div><font face="Arial" size="2"><b>netperf -H 10.0.2.1,ipv4</b></font></div><div><font face="Arial" size="2">TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.2.1 (10.0.2.1) port 0 AF_INET</font></div><div><font face="Arial" size="2">Recv Send Send </font></div><div><font face="Arial" size="2">Socket Socket Message Elapsed </font></div><div><font face="Arial" size="2">Size Size Size Time Throughput </font></div><div><font face="Arial" size="2">bytes bytes bytes secs. 10^6bits/sec </font></div><div><font face="Arial" size="2"><br></font></div><div><font face="Arial" size="2"> 87380 65536 65536 10.04 9335.75 </font></div><div><font face="Arial"><br></font></div><div><font face="Arial"><br></font></div><div><font face="Arial">Only the drbd is very slow. (Single node, just to test write perfomance on local block device.</font></div><div><font face="Arial"><br></font></div><div><font face="Arial"><div>dd if=/dev/zero of=/dev/drbd0 bs=512M count=8</div><div>4+0 Datensätze ein</div><div>4+0 Datensätze aus</div><div>2147483648 Bytes (2,1 GB) kopiert, 63,9839 s, 33,6 MB/s</div><div>Cancelled with CTRL+C</div></font></div><div><font face="Arial"><br></font></div><div>After long googleing and testing around with io scheduler nr_request etc. blockdev, I cannot see where the bottlenec lies. Had anybody mad similar experiences and can give me a hint. OS is Scientific Linux-6.4 x64 (CentOS) with drbd 8.4.1.</div><div><br></div><div>Regards</div><div><br></div><div>Roland</div><div><br></div><div><br></div><div><br></div><div>Here my drbd.conf</div><div><br></div><div><div>global { </div><div> usage-count no; </div><div>} </div><div>common { </div><div> protocol B; </div><div>} </div><div><br></div><div>resource target1 {</div><div> meta-disk internal;</div><div> device /dev/drbd0;</div><div> </div><div> syncer {</div><div> verify-alg sha1; </div><div> cpu-mask 0; </div><div> rate 850M; </div><div> al-extents 6433; </div><div> }</div><div> </div><div> net { </div><div> max-buffers 16384;</div><div> max-epoch-size 4096;</div><div> cram-hmac-alg sha1;</div><div> unplug-watermark 8192;</div><div> no-tcp-cork;</div><div> sndbuf-size 4096k;</div><div> rcvbuf-size 4096k;</div><div> }</div><div> </div><div> startup {</div><div> wfc-timeout 10;</div><div> degr-wfc-timeout 8;</div><div> }</div><div><br></div><div><br></div><div> disk {</div><div> resync-rate 4000M;</div><div> disk-flushes no;</div><div> disk-barrier no;</div><div> md-flushes no; </div><div> read-balancing 64K-striping;</div><div> }</div><div> </div><div> on storage-node1.ziilrz.ch {</div><div> disk /dev/sdb1;</div><div> address 10.0.1.1:7789;</div><div> }</div><div> on storage-node2.ziilrz.ch {</div><div> disk /dev/sdb1;</div><div> address 10.0.1.2:7789;</div><div> }</div><div>}</div><div><br></div><div>resource target2 {</div><div> meta-disk internal;</div><div> device /dev/drbd1;</div><div> syncer {</div><div> verify-alg sha1;</div><div> cpu-mask 0;</div><div> rate 850M;</div><div> al-extents 6433;</div><div> }</div><div> </div><div> net {</div><div> max-buffers 16384;</div><div> max-epoch-size 4096;</div><div> cram-hmac-alg sha1;</div><div> unplug-watermark 8192;</div><div> no-tcp-cork;</div><div> sndbuf-size 4096k;</div><div> rcvbuf-size 4096k;</div><div> }</div><div> startup {</div><div> wfc-timeout 10;</div><div> degr-wfc-timeout 8;</div><div> }</div><div><br></div><div> disk {</div><div> disk-flushes no;</div><div> disk-barrier no;</div><div> md-flushes no; </div><div> read-balancing 64K-striping;</div><div> resync-rate 4000M;</div><div> }</div><div><br></div><div> on storage-node1.ziilrz.ch {</div><div> disk /dev/sdb2;</div><div> address 10.0.2.1:7789;</div><div> }</div><div> </div><div> on storage-node2.ziilrz.ch {</div><div> disk /dev/sdb2;</div><div> address 10.0.2.2:7789;</div><div> }</div><div>}</div></div><div><br></div><div><br></div><div><br></div><br><font face="Arial" size="3">-- </font><br><font face="Arial" size="3"><span name="x"></span></font><div style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; "><meta http-equiv="content-type" content="text/html; charset=utf-8"><font size="2"><a href="http://www.ziil.ch">Ziil Informatiklösungen</a> GmbH</font><br><font size="2"><span style="font-weight: bold;">Roland Käser</span></font></div><div style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; "><font size="2"><span style="font-weight: bold;"></span>Leiter Technik<br>Romanshornerstr. 134, <br>CH-8280 Kreuzlingen</font><br><font size="2">Tel: +41 71 671 27 78</font></div><div style="color: rgb(0, 0, 0); font-family: Arial; font-size: 12pt; "><font size="2">Fax: +41 71 671 27 79<br>www.ziil.ch</font></div><font face="Arial" size="3"><span name="x"></span></font><br></div></div></body></html>