Hi<br><br>Just trying to follow up on my post.<br>Is there an obvious explanation as to why 8.3.12 seems to perform better than 8.4.1?<br><br>Thx<br>Fred<br><br><div class="gmail_quote">On Fri, Feb 3, 2012 at 3:08 PM, Frederic DeMarcy <span dir="ltr"><<a href="mailto:fred.demarcy.ml@gmail.com">fred.demarcy.ml@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Lars<br>
<br>
Thanks for your answer.<br>
<br>
I restarted a bunch of tests using DRBD 8.3.12 and DRBD 8.4.1<br>
<br>
- Scientific Linux 64 bits with Kernel 2.6.32-220.4.1.el6.x86_64<br>
- DRBD 8.3.12 and 8.4.1 recompiled from source using<br>
./configure<br>
make rpm<br>
make km-rpm<br>
- Using the XFS filesystem (mkfs.xfs /dev/drbd0)<br>
- 1Gb/s crossover link between server1 and server2<br>
- Both server1 and server2 are configured the same way with 2 disks /dev/sda (all usual linux partitions) and /dev/sdb<br>
used only for DRBD<br>
- Both servers are VMs on top of VMware ESXi 5. /dev/sda and /dev/sdb are carved out of the same datastore. They use<br>
the same paravirtual SCSI adaptor.<br>
<br>
<br>
<br>
DRBD 8.3.12<br>
===========<br>
/etc/drbd.d/mysql.res<br>
<div class="im">resource mysql {<br>
startup {<br>
wfc-timeout 3;<br>
degr-wfc-timeout 2;<br>
outdated-wfc-timeout 1;<br>
}<br>
</div> syncer {<br>
verify-alg sha1;<br>
#Only interesting for WAN setups where you need to sync via a rather thin connection.<br>
#csums-alg sha1;<br>
rate 33M;<br>
al-extents 3389;<br>
}<br>
net {<br>
#Only if you have to assume there is buggy hardware on the way between your nodes ... like NICs pretending csums are ok while they are not!<br>
#data-integrity-alg sha1;<br>
cram-hmac-alg sha1;<br>
shared-secret "MySecret123";<br>
max-buffers 8000;<br>
max-epoch-size 8000;<br>
unplug-watermark 16;<br>
sndbuf-size 512k;<br>
sndbuf-size 0;<br>
}<br>
disk {<br>
no-disk-barrier;<br>
no-disk-flushes;<br>
no-md-flushes;<br>
<div class="im"> }<br>
on server1 {<br>
device /dev/drbd0;<br>
disk /dev/sdb;<br>
</div> address <a href="http://192.168.111.10:7789" target="_blank">192.168.111.10:7789</a>;<br>
<div class="im"> meta-disk internal;<br>
}<br>
on server2 {<br>
device /dev/drbd0;<br>
disk /dev/sdb;<br>
</div> address <a href="http://192.168.111.11:7789" target="_blank">192.168.111.11:7789</a>;<br>
meta-disk internal;<br>
}<br>
}<br>
<br>
<br>
DRBD 8.4.1<br>
==========<br>
/etc/drbd.d/mysql.res<br>
<div class="im">resource mysql {<br>
startup {<br>
wfc-timeout 3;<br>
degr-wfc-timeout 2;<br>
outdated-wfc-timeout 1;<br>
}<br>
net {<br>
protocol C;<br>
verify-alg sha1;<br>
</div> #Only interesting for WAN setups where you need to sync via a rather thin connection.<br>
#csums-alg sha1;<br>
#Only if you have to assume there is buggy hardware on the way between your nodes ... like NICs pretending csums are ok while they are not!<br>
#data-integrity-alg sha1;<br>
cram-hmac-alg sha1;<br>
shared-secret "MySecret123";<br>
}<br>
disk {<br>
resync-rate 33M;<br>
no-disk-barrier;<br>
no-disk-flushes;<br>
no-md-flushes;<br>
<div class="im"> }<br>
on server1 {<br>
device /dev/drbd0;<br>
disk /dev/sdb;<br>
</div> address <a href="http://192.168.111.10:7789" target="_blank">192.168.111.10:7789</a>;<br>
<div class="im"> meta-disk internal;<br>
}<br>
on server2 {<br>
device /dev/drbd0;<br>
disk /dev/sdb;<br>
</div> address <a href="http://192.168.111.11:7789" target="_blank">192.168.111.11:7789</a>;<br>
meta-disk internal;<br>
}<br>
}<br>
<br>
<br>
RESULTS<br>
=======<br>
In order to check the impact of DRBD I first compared the disk throughput writing a large 4GB file in /home/userxxx<br>
located on /dev/sda. Then I created a partition /dev/sdb1 formatted with XFS. Note that on /dev/sda all partitions are<br>
Ext4 so I only did this measurement to have a ballpark base figure.<br>
<br>
Test #1<br>
Note: Mysql is not running<br>
writing on /dev/sda<br>
<div class="im"> dd if=/dev/zero of=/home/userxxx/disk-test.xxx bs=1M count=4096 oflag=direct<br>
</div> 415 MB/s (Ext4)<br>
<br>
writing on /dev/sdb<br>
<div class="im"> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096 oflag=direct<br>
</div> 415 MB/s (XFS)<br>
<br>
Test #2<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with only error logs. Write 10000 rows with script running on same host.<br>
;general_log=1<br>
;general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
;slow_query_log=1<br>
;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
;innodb_flush_log_at_trx_commit=1<br>
;binlog-format=MIXED<br>
;sync_binlog=1<br>
;log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 3s<br>
<br>
Test #3<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script running on same host.<br>
general_log=1<br>
general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
slow_query_log=1<br>
slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
innodb_flush_log_at_trx_commit=1<br>
binlog-format=MIXED<br>
sync_binlog=1<br>
log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 6s<br>
<br>
<br>
Adding DRBD.<br>
<br>
<br>
DRBD 8.3.12<br>
~~~~~~~~~~~<br>
!With Secondary DRBD node down!<br>
Test #1<br>
writing on /dev/sdb<br>
<div class="im"> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096 oflag=direct<br>
</div> 194 MB/s (XFS) ~ x2 performance drop from ~ 415 MB/s<br>
<br>
Test #2<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with only error logs. Write 10000 rows with script running on same host.<br>
;general_log=1<br>
;general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
;slow_query_log=1<br>
;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
;innodb_flush_log_at_trx_commit=1<br>
;binlog-format=MIXED<br>
;sync_binlog=1<br>
;log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 3s ~ on par<br>
<br>
Test #3<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script running on same host.<br>
general_log=1<br>
general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
slow_query_log=1<br>
slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
innodb_flush_log_at_trx_commit=1<br>
binlog-format=MIXED<br>
sync_binlog=1<br>
log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 6s ~ on par<br>
<br>
!With Secondary DRBD node up!<br>
Test #1<br>
writing on /dev/sdb<br>
<div class="im"> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096 oflag=direct<br>
</div> 89 MB/s (XFS) ~ x4.5 performance dropped from 415 MB/s<br>
<br>
Test #2<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with only error logs. Write 10000 rows with script running on same host.<br>
;general_log=1<br>
;general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
;slow_query_log=1<br>
;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
;innodb_flush_log_at_trx_commit=1<br>
;binlog-format=MIXED<br>
;sync_binlog=1<br>
;log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 8s ~ x2.5 performance drop from ~ 3s<br>
<br>
Test #3<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script running on same host.<br>
general_log=1<br>
general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
slow_query_log=1<br>
slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
innodb_flush_log_at_trx_commit=1<br>
binlog-format=MIXED<br>
sync_binlog=1<br>
log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 25s ~ x4 performance drop from ~ 6s<br>
<br>
Test #4 (mysqlslap)<br>
mysqlslap --concurrency=1,5,10,20,30,50 --iterations=1 --engine=innodb --auto-generate-sql --auto-generate-sql-load-type=write --number-of-queries=10000 --password=<root_password><br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 24.737 seconds<br>
Minimum number of seconds to run all queries: 24.737 seconds<br>
Maximum number of seconds to run all queries: 24.737 seconds<br>
Number of clients running queries: 1<br>
Average number of queries per client: 10000<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 10.321 seconds<br>
Minimum number of seconds to run all queries: 10.321 seconds<br>
Maximum number of seconds to run all queries: 10.321 seconds<br>
Number of clients running queries: 5<br>
Average number of queries per client: 2000<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 9.752 seconds<br>
Minimum number of seconds to run all queries: 9.752 seconds<br>
Maximum number of seconds to run all queries: 9.752 seconds<br>
Number of clients running queries: 10<br>
Average number of queries per client: 1000<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 9.491 seconds<br>
Minimum number of seconds to run all queries: 9.491 seconds<br>
Maximum number of seconds to run all queries: 9.491 seconds<br>
Number of clients running queries: 20<br>
Average number of queries per client: 500<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 9.827 seconds<br>
Minimum number of seconds to run all queries: 9.827 seconds<br>
Maximum number of seconds to run all queries: 9.827 seconds<br>
Number of clients running queries: 30<br>
Average number of queries per client: 333<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 9.938 seconds<br>
Minimum number of seconds to run all queries: 9.938 seconds<br>
Maximum number of seconds to run all queries: 9.938 seconds<br>
Number of clients running queries: 50<br>
Average number of queries per client: 200<br>
<br>
<br>
<br>
DRBD 8.4.1<br>
~~~~~~~~~~~<br>
!With Secondary DRBD node down!<br>
Test #1<br>
writing on /dev/sdb<br>
<div class="im"> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096 oflag=direct<br>
</div> 194 MB/s (XFS) ~ x2 performance drop from ~ 415 MB/s<br>
<br>
Test #2<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with only error logs. Write 10000 rows with script running on same host.<br>
;general_log=1<br>
;general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
;slow_query_log=1<br>
;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
;innodb_flush_log_at_trx_commit=1<br>
;binlog-format=MIXED<br>
;sync_binlog=1<br>
;log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 24s ~ 8x performance drop from ~ 3s<br>
~ 8x performance drop from 8.3.12<br>
<br>
Test #3<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script running on same host.<br>
general_log=1<br>
general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
slow_query_log=1<br>
slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
innodb_flush_log_at_trx_commit=1<br>
binlog-format=MIXED<br>
sync_binlog=1<br>
log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 1m19s ~ x13 performance drop from ~ 6s<br>
~ x13 performance drop from 8.3.12<br>
<br>
!With Secondary DRBD node up!<br>
Test #1<br>
writing on /dev/sdb<br>
<div class="im"> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096 oflag=direct<br>
</div> 102 MB/s (XFS) ~ x4 performance dropped from 415 MB/s<br>
<br>
Test #2<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with only error logs. Write 10000 rows with script running on same host.<br>
;general_log=1<br>
;general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
;slow_query_log=1<br>
;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
;innodb_flush_log_at_trx_commit=1<br>
;binlog-format=MIXED<br>
;sync_binlog=1<br>
;log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 58s ~ x20 performance drop from ~ 3s<br>
~ x7 performance drop from 8.3.12<br>
<br>
Test #3<br>
MySQL is running with /var/lib/mysql on /dev/sdb.<br>
MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script running on same host.<br>
general_log=1<br>
general_log-file=/var/lib/mysql/var/log/mysql/general.log<br>
slow_query_log=1<br>
slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log<br>
innodb_flush_log_at_trx_commit=1<br>
binlog-format=MIXED<br>
sync_binlog=1<br>
log-bin=/var/lib/mysql/var/log/mysql/binlog<br>
Time ~ 2m50s ~ x28 performance drop from ~ 6s<br>
~ x7 performance drop from 8.3.12<br>
<br>
Test #4 (mysqlslap)<br>
mysqlslap --concurrency=1,5,10,20,30,50 --iterations=1 --engine=innodb --auto-generate-sql --auto-generate-sql-load-type=write --number-of-queries=10000 --password=<root_password><br>
<br>
Benchmark<br>
Running for engine innodb"<br>
Average number of seconds to run all queries: 169.607 seconds<br>
Minimum number of seconds to run all queries: 169.607 seconds<br>
Maximum number of seconds to run all queries: 169.607 seconds<br>
Number of clients running queries: 1<br>
Average number of queries per client: 10000<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 60.247 seconds<br>
Minimum number of seconds to run all queries: 60.247 seconds<br>
Maximum number of seconds to run all queries: 60.247 seconds<br>
Number of clients running queries: 5<br>
Average number of queries per client: 2000<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 51.656 seconds<br>
Minimum number of seconds to run all queries: 51.656 seconds<br>
Maximum number of seconds to run all queries: 51.656 seconds<br>
Number of clients running queries: 10<br>
Average number of queries per client: 1000<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 52.587 seconds<br>
Minimum number of seconds to run all queries: 52.587 seconds<br>
Maximum number of seconds to run all queries: 52.587 seconds<br>
Number of clients running queries: 20<br>
Average number of queries per client: 500<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 50.869 seconds<br>
Minimum number of seconds to run all queries: 50.869 seconds<br>
Maximum number of seconds to run all queries: 50.869 seconds<br>
Number of clients running queries: 30<br>
Average number of queries per client: 333<br>
<br>
Benchmark<br>
Running for engine innodb<br>
Average number of seconds to run all queries: 52.766 seconds<br>
Minimum number of seconds to run all queries: 52.766 seconds<br>
Maximum number of seconds to run all queries: 52.766 seconds<br>
Number of clients running queries: 50<br>
Average number of queries per client: 200<br>
<br>
Huge performance drop compared to 8.3.12 as well.<br>
<br>
Why is there such a big difference between 8.3.12 and 8.4.1?<br>
<br>
Thank you<br>
Fred<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On 2 Feb 2012, at 14:31, Lars Ellenberg wrote:<br>
<br>
> On Thu, Feb 02, 2012 at 02:24:08PM +0000, Frederic DeMarcy wrote:<br>
>> Hi Lars<br>
>><br>
>> I'm usinng 8.4.1<br>
>><br>
>> [root@server1 ~]# service drbd status<br>
>> drbd driver loaded OK; device status:<br>
>> version: 8.4.1 (api:1/proto:86-100)<br>
>><br>
><br>
> Is that so.<br>
> Apologies, I seem to have mixed a few threads here.<br>
><br>
> In that case, invalidate-remote when in StandAlone should be allowed.<br>
> Strange.<br>
><br>
> Maybe you should downgrade ;-)<br>
> and see where things go.<br>
><br>
> In fact, my guess is moving meta data to RAID1<br>
> will be most useful in your case.<br>
><br>
>>>><br>
>>>> Is the result the same if you execute a "drbdadm invalidate-remote<br>
>>>> mysql" on the primary before doing the "single node" test? .... that<br>
>>>> would disable activity log updates ...<br>
>>><br>
>>> ... with recent enough versions of DRBD.<br>
>>> Only the OP seems to be using 8.3.7.<br>
>>> Maybe you should consider an upgrade?<br>
><br>
><br>
> --<br>
> : Lars Ellenberg<br>
> : LINBIT | Your Way to High Availability<br>
> : DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
><br>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
> __<br>
> please don't Cc me, but send to list -- I'm subscribed<br>
> _______________________________________________<br>
> drbd-user mailing list<br>
> <a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
> <a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br>
</div></div></blockquote></div><br>