Hi Pascal<br><br>1) Both vdisks for /dev/sda and /dev/sdb are on the same datastore which is made of the entire RAID5 array capacity (7xHDs + 1 spare). Minus space used by the ESXi installation.<br>2) HD1 (/dev/sda) is SCSI (0:1) and HD2 (/dev/sdb) is SCSI (0:2). Both initialized with Thick Provisioning Eager Zeroed. The SCSI controller type is paravirtual.<br>
<br>Fred<br><br><div class="gmail_quote">On Wed, Feb 1, 2012 at 2:13 PM, Pascal BERTON (EURIALYS) <span dir="ltr"><<a href="mailto:pascal.berton@eurialys.fr">pascal.berton@eurialys.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Frederic,<br>
<br>
Let's take care of the virtualisation layer wich might induce significant<br>
side effects<br>
Are sda and sdb :<br>
1) vdisk files located on the same datastore ?<br>
2) vdisks plugged on the same virtual SCSI interface ? What type of SCSI<br>
interface ?<br>
<br>
Best regards,<br>
<br>
Pascal.<br>
<br>
-----Message d'origine-----<br>
De : <a href="mailto:drbd-user-bounces@lists.linbit.com">drbd-user-bounces@lists.linbit.com</a><br>
[mailto:<a href="mailto:drbd-user-bounces@lists.linbit.com">drbd-user-bounces@lists.linbit.com</a>] De la part de Frederic DeMarcy<br>
Envoyé : mercredi 1 février 2012 13:05<br>
À : <a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
Objet : Re: [DRBD-user] Slower disk throughput on DRBD partition<br>
<div class="HOEnZb"><div class="h5"><br>
Hi<br>
<br>
Note 1:<br>
Scientific Linux 6.1 with kernel 2.6.32-220.4.1.el6.x86_64<br>
DRBD 8.4.1 compiled from source<br>
<br>
Note 2:<br>
server1 and server2 are 2 VMware VMs on top of ESXi 5. However they reside<br>
on different physical 2U servers.<br>
The specs for the 2U servers are identical:<br>
- HP DL380 G7 (2U)<br>
- 2 x Six Core Intel Xeon X5680 (3.33GHz)<br>
- 24GB RAM<br>
- 8 x 146 GB SAS HD's (7xRAID5 + 1s)<br>
- Smart Array P410i with 512MB BBWC<br>
<br>
Note 3:<br>
I've tested the network throughput with iperf which yields close to 1Gb/s<br>
[root@server1 ~]# iperf -c 192.168.111.11 -f g<br>
------------------------------------------------------------<br>
Client connecting to 192.168.111.11, TCP port 5001<br>
TCP window size: 0.00 GByte (default)<br>
------------------------------------------------------------<br>
[ 3] local 192.168.111.10 port 54330 connected with 192.168.111.11 port<br>
5001<br>
[ ID] Interval Transfer Bandwidth<br>
[ 3] 0.0-10.0 sec 1.10 GBytes 0.94 Gbits/sec<br>
<br>
[root@server2 ~]# iperf -s -f g<br>
------------------------------------------------------------<br>
Server listening on TCP port 5001<br>
TCP window size: 0.00 GByte (default)<br>
------------------------------------------------------------<br>
[ 4] local 192.168.111.11 port 5001 connected with 192.168.111.10 port<br>
54330<br>
[ ID] Interval Transfer Bandwidth<br>
[ 4] 0.0-10.0 sec 1.10 GBytes 0.94 Gbits/sec<br>
<br>
Scp'ing a large file from server1 to server2 yields ~ 57MB/s but I guess<br>
it's due to the encryption overhead.<br>
<br>
Note 4:<br>
MySQL was not running.<br>
<br>
<br>
<br>
Base DRBD config:<br>
resource mysql {<br>
startup {<br>
wfc-timeout 3;<br>
degr-wfc-timeout 2;<br>
outdated-wfc-timeout 1;<br>
}<br>
net {<br>
protocol C;<br>
verify-alg sha1;<br>
csums-alg sha1;<br>
data-integrity-alg sha1;<br>
cram-hmac-alg sha1;<br>
shared-secret "MySecret123";<br>
}<br>
on server1 {<br>
device /dev/drbd0;<br>
disk /dev/sdb;<br>
address <a href="http://192.168.111.10:7789" target="_blank">192.168.111.10:7789</a>;<br>
meta-disk internal;<br>
}<br>
on server2 {<br>
device /dev/drbd0;<br>
disk /dev/sdb;<br>
address <a href="http://192.168.111.11:7789" target="_blank">192.168.111.11:7789</a>;<br>
meta-disk internal;<br>
}<br>
}<br>
<br>
<br>
After any change in the /etc/drbd.d/mysql.res file I issued a "drbdadm<br>
adjust mysql" on both nodes.<br>
<br>
Test #1<br>
DRBD partition on primary (secondary node disabled)<br>
Using Base DRBD config<br>
# dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096<br>
oflag=direct<br>
Throughput ~ 420MB/s<br>
<br>
Test #2<br>
DRBD partition on primary (secondary node enabled)<br>
Using Base DRBD config<br>
# dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096<br>
oflag=direct<br>
Throughput ~ 61MB/s<br>
<br>
Test #3<br>
DRBD partition on primary (secondary node enabled)<br>
Using Base DRBD config with:<br>
Protocol B;<br>
# dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096<br>
oflag=direct<br>
Throughput ~ 68MB/s<br>
<br>
Test #4<br>
DRBD partition on primary (secondary node enabled)<br>
Using Base DRBD config with:<br>
Protocol A;<br>
# dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096<br>
oflag=direct<br>
Throughput ~ 94MB/s<br>
<br>
Test #5<br>
DRBD partition on primary (secondary node enabled)<br>
Using Base DRBD config with:<br>
disk {<br>
disk-barrier no;<br>
disk-flushes no;<br>
md-flushes no;<br>
}<br>
# dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096<br>
oflag=direct<br>
Disk throughput ~ 62MB/s<br>
<br>
No difference from Test #2 really. Also cat /proc/drbd still shows wo:b in<br>
both cases so I'm not even sure<br>
these disk {..} parameters have been taken into account...<br>
<br>
Test #6<br>
DRBD partition on primary (secondary node enabled)<br>
Using Base DRBD config with:<br>
Protocol B;<br>
disk {<br>
disk-barrier no;<br>
disk-flushes no;<br>
md-flushes no;<br>
}<br>
# dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096<br>
oflag=direct<br>
Disk throughput ~ 68MB/s<br>
<br>
No difference from Test #3 really. Also cat /proc/drbd still shows wo:b in<br>
both cases so I'm not even sure<br>
these disk {..} parameters have been taken into account...<br>
<br>
<br>
What else can I try?<br>
Is it worth trying DRBD 8.3.x?<br>
<br>
Thx.<br>
<br>
Fred<br>
<br>
<br>
<br>
<br>
<br>
<br>
On 1 Feb 2012, at 08:35, James Harper wrote:<br>
<br>
>> Hi<br>
>><br>
>> I've configured DRBD with a view to use it with MySQL (and later on<br>
>> Pacemaker + Corosync) in a 2 nodes primary/secondary<br>
>> (master/slave) setup.<br>
>><br>
>> ...<br>
>><br>
>> No replication over the 1Gb/s crossover cable is taking place since the<br>
>> secondary node is down yet there's x2 lower disk performance.<br>
>><br>
>> I've tried to add:<br>
>> disk {<br>
>> disk-barrier no;<br>
>> disk-flushes no;<br>
>> md-flushes no;<br>
>> }<br>
>> to the config but it didn't seem to change anything.<br>
>><br>
>> Am I missing something here?<br>
>> On another note is 8.4.1 the right version to use?<br>
>><br>
><br>
> If you can do it just for testing, try changing to protocol B with one<br>
primary and one secondary and see how that impacts your performance, both<br>
with barrier/flushes on and off. I'm not sure if it will help but if<br>
protocol B makes things faster then it might hint as to where to start<br>
looking...<br>
><br>
> James<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br>
</div></div></blockquote></div><br>