Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi
Just trying to follow up on my post.
Is there an obvious explanation as to why 8.3.12 seems to perform better
than 8.4.1?
Thx
Fred
On Fri, Feb 3, 2012 at 3:08 PM, Frederic DeMarcy
<fred.demarcy.ml at gmail.com>wrote:
> Hi Lars
>
> Thanks for your answer.
>
> I restarted a bunch of tests using DRBD 8.3.12 and DRBD 8.4.1
>
> - Scientific Linux 64 bits with Kernel 2.6.32-220.4.1.el6.x86_64
> - DRBD 8.3.12 and 8.4.1 recompiled from source using
> ./configure
> make rpm
> make km-rpm
> - Using the XFS filesystem (mkfs.xfs /dev/drbd0)
> - 1Gb/s crossover link between server1 and server2
> - Both server1 and server2 are configured the same way with 2 disks
> /dev/sda (all usual linux partitions) and /dev/sdb
> used only for DRBD
> - Both servers are VMs on top of VMware ESXi 5. /dev/sda and /dev/sdb are
> carved out of the same datastore. They use
> the same paravirtual SCSI adaptor.
>
>
>
> DRBD 8.3.12
> ===========
> /etc/drbd.d/mysql.res
> resource mysql {
> startup {
> wfc-timeout 3;
> degr-wfc-timeout 2;
> outdated-wfc-timeout 1;
> }
> syncer {
> verify-alg sha1;
> #Only interesting for WAN setups where you need to sync via a rather
> thin connection.
> #csums-alg sha1;
> rate 33M;
> al-extents 3389;
> }
> net {
> #Only if you have to assume there is buggy hardware on the way between
> your nodes ... like NICs pretending csums are ok while they are not!
> #data-integrity-alg sha1;
> cram-hmac-alg sha1;
> shared-secret "MySecret123";
> max-buffers 8000;
> max-epoch-size 8000;
> unplug-watermark 16;
> sndbuf-size 512k;
> sndbuf-size 0;
> }
> disk {
> no-disk-barrier;
> no-disk-flushes;
> no-md-flushes;
> }
> on server1 {
> device /dev/drbd0;
> disk /dev/sdb;
> address 192.168.111.10:7789;
> meta-disk internal;
> }
> on server2 {
> device /dev/drbd0;
> disk /dev/sdb;
> address 192.168.111.11:7789;
> meta-disk internal;
> }
> }
>
>
> DRBD 8.4.1
> ==========
> /etc/drbd.d/mysql.res
> resource mysql {
> startup {
> wfc-timeout 3;
> degr-wfc-timeout 2;
> outdated-wfc-timeout 1;
> }
> net {
> protocol C;
> verify-alg sha1;
> #Only interesting for WAN setups where you need to sync via a rather
> thin connection.
> #csums-alg sha1;
> #Only if you have to assume there is buggy hardware on the way between
> your nodes ... like NICs pretending csums are ok while they are not!
> #data-integrity-alg sha1;
> cram-hmac-alg sha1;
> shared-secret "MySecret123";
> }
> disk {
> resync-rate 33M;
> no-disk-barrier;
> no-disk-flushes;
> no-md-flushes;
> }
> on server1 {
> device /dev/drbd0;
> disk /dev/sdb;
> address 192.168.111.10:7789;
> meta-disk internal;
> }
> on server2 {
> device /dev/drbd0;
> disk /dev/sdb;
> address 192.168.111.11:7789;
> meta-disk internal;
> }
> }
>
>
> RESULTS
> =======
> In order to check the impact of DRBD I first compared the disk throughput
> writing a large 4GB file in /home/userxxx
> located on /dev/sda. Then I created a partition /dev/sdb1 formatted with
> XFS. Note that on /dev/sda all partitions are
> Ext4 so I only did this measurement to have a ballpark base figure.
>
> Test #1
> Note: Mysql is not running
> writing on /dev/sda
> dd if=/dev/zero of=/home/userxxx/disk-test.xxx bs=1M count=4096
> oflag=direct
> 415 MB/s (Ext4)
>
> writing on /dev/sdb
> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096
> oflag=direct
> 415 MB/s (XFS)
>
> Test #2
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with only error logs. Write 10000 rows with script running on
> same host.
> ;general_log=1
> ;general_log-file=/var/lib/mysql/var/log/mysql/general.log
> ;slow_query_log=1
> ;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> ;innodb_flush_log_at_trx_commit=1
> ;binlog-format=MIXED
> ;sync_binlog=1
> ;log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 3s
>
> Test #3
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script
> running on same host.
> general_log=1
> general_log-file=/var/lib/mysql/var/log/mysql/general.log
> slow_query_log=1
> slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> innodb_flush_log_at_trx_commit=1
> binlog-format=MIXED
> sync_binlog=1
> log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 6s
>
>
> Adding DRBD.
>
>
> DRBD 8.3.12
> ~~~~~~~~~~~
> !With Secondary DRBD node down!
> Test #1
> writing on /dev/sdb
> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096
> oflag=direct
> 194 MB/s (XFS) ~ x2 performance drop from ~ 415 MB/s
>
> Test #2
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with only error logs. Write 10000 rows with script running on
> same host.
> ;general_log=1
> ;general_log-file=/var/lib/mysql/var/log/mysql/general.log
> ;slow_query_log=1
> ;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> ;innodb_flush_log_at_trx_commit=1
> ;binlog-format=MIXED
> ;sync_binlog=1
> ;log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 3s ~ on par
>
> Test #3
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script
> running on same host.
> general_log=1
> general_log-file=/var/lib/mysql/var/log/mysql/general.log
> slow_query_log=1
> slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> innodb_flush_log_at_trx_commit=1
> binlog-format=MIXED
> sync_binlog=1
> log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 6s ~ on par
>
> !With Secondary DRBD node up!
> Test #1
> writing on /dev/sdb
> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096
> oflag=direct
> 89 MB/s (XFS) ~ x4.5 performance dropped from 415 MB/s
>
> Test #2
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with only error logs. Write 10000 rows with script running on
> same host.
> ;general_log=1
> ;general_log-file=/var/lib/mysql/var/log/mysql/general.log
> ;slow_query_log=1
> ;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> ;innodb_flush_log_at_trx_commit=1
> ;binlog-format=MIXED
> ;sync_binlog=1
> ;log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 8s ~ x2.5 performance drop from ~ 3s
>
> Test #3
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script
> running on same host.
> general_log=1
> general_log-file=/var/lib/mysql/var/log/mysql/general.log
> slow_query_log=1
> slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> innodb_flush_log_at_trx_commit=1
> binlog-format=MIXED
> sync_binlog=1
> log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 25s ~ x4 performance drop from ~ 6s
>
> Test #4 (mysqlslap)
> mysqlslap --concurrency=1,5,10,20,30,50 --iterations=1 --engine=innodb
> --auto-generate-sql --auto-generate-sql-load-type=write
> --number-of-queries=10000 --password=<root_password>
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 24.737 seconds
> Minimum number of seconds to run all queries: 24.737 seconds
> Maximum number of seconds to run all queries: 24.737 seconds
> Number of clients running queries: 1
> Average number of queries per client: 10000
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 10.321 seconds
> Minimum number of seconds to run all queries: 10.321 seconds
> Maximum number of seconds to run all queries: 10.321 seconds
> Number of clients running queries: 5
> Average number of queries per client: 2000
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 9.752 seconds
> Minimum number of seconds to run all queries: 9.752 seconds
> Maximum number of seconds to run all queries: 9.752 seconds
> Number of clients running queries: 10
> Average number of queries per client: 1000
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 9.491 seconds
> Minimum number of seconds to run all queries: 9.491 seconds
> Maximum number of seconds to run all queries: 9.491 seconds
> Number of clients running queries: 20
> Average number of queries per client: 500
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 9.827 seconds
> Minimum number of seconds to run all queries: 9.827 seconds
> Maximum number of seconds to run all queries: 9.827 seconds
> Number of clients running queries: 30
> Average number of queries per client: 333
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 9.938 seconds
> Minimum number of seconds to run all queries: 9.938 seconds
> Maximum number of seconds to run all queries: 9.938 seconds
> Number of clients running queries: 50
> Average number of queries per client: 200
>
>
>
> DRBD 8.4.1
> ~~~~~~~~~~~
> !With Secondary DRBD node down!
> Test #1
> writing on /dev/sdb
> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096
> oflag=direct
> 194 MB/s (XFS) ~ x2 performance drop from ~ 415 MB/s
>
> Test #2
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with only error logs. Write 10000 rows with script running on
> same host.
> ;general_log=1
> ;general_log-file=/var/lib/mysql/var/log/mysql/general.log
> ;slow_query_log=1
> ;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> ;innodb_flush_log_at_trx_commit=1
> ;binlog-format=MIXED
> ;sync_binlog=1
> ;log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 24s ~ 8x performance drop from ~ 3s
> ~ 8x performance drop from 8.3.12
>
> Test #3
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script
> running on same host.
> general_log=1
> general_log-file=/var/lib/mysql/var/log/mysql/general.log
> slow_query_log=1
> slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> innodb_flush_log_at_trx_commit=1
> binlog-format=MIXED
> sync_binlog=1
> log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 1m19s ~ x13 performance drop from ~ 6s
> ~ x13 performance drop from 8.3.12
>
> !With Secondary DRBD node up!
> Test #1
> writing on /dev/sdb
> dd if=/dev/zero of=/var/lib/mysql/disk-test.xxx bs=1M count=4096
> oflag=direct
> 102 MB/s (XFS) ~ x4 performance dropped from 415 MB/s
>
> Test #2
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with only error logs. Write 10000 rows with script running on
> same host.
> ;general_log=1
> ;general_log-file=/var/lib/mysql/var/log/mysql/general.log
> ;slow_query_log=1
> ;slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> ;innodb_flush_log_at_trx_commit=1
> ;binlog-format=MIXED
> ;sync_binlog=1
> ;log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 58s ~ x20 performance drop from ~ 3s
> ~ x7 performance drop from 8.3.12
>
> Test #3
> MySQL is running with /var/lib/mysql on /dev/sdb.
> MySQL setup with bin logs, sync, tx, etc... Write 10000 rows with script
> running on same host.
> general_log=1
> general_log-file=/var/lib/mysql/var/log/mysql/general.log
> slow_query_log=1
> slow_query_log_file=/var/lib/mysql/var/log/mysql/slow-query.log
> innodb_flush_log_at_trx_commit=1
> binlog-format=MIXED
> sync_binlog=1
> log-bin=/var/lib/mysql/var/log/mysql/binlog
> Time ~ 2m50s ~ x28 performance drop from ~ 6s
> ~ x7 performance drop from 8.3.12
>
> Test #4 (mysqlslap)
> mysqlslap --concurrency=1,5,10,20,30,50 --iterations=1 --engine=innodb
> --auto-generate-sql --auto-generate-sql-load-type=write
> --number-of-queries=10000 --password=<root_password>
>
> Benchmark
> Running for engine innodb"
> Average number of seconds to run all queries: 169.607 seconds
> Minimum number of seconds to run all queries: 169.607 seconds
> Maximum number of seconds to run all queries: 169.607 seconds
> Number of clients running queries: 1
> Average number of queries per client: 10000
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 60.247 seconds
> Minimum number of seconds to run all queries: 60.247 seconds
> Maximum number of seconds to run all queries: 60.247 seconds
> Number of clients running queries: 5
> Average number of queries per client: 2000
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 51.656 seconds
> Minimum number of seconds to run all queries: 51.656 seconds
> Maximum number of seconds to run all queries: 51.656 seconds
> Number of clients running queries: 10
> Average number of queries per client: 1000
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 52.587 seconds
> Minimum number of seconds to run all queries: 52.587 seconds
> Maximum number of seconds to run all queries: 52.587 seconds
> Number of clients running queries: 20
> Average number of queries per client: 500
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 50.869 seconds
> Minimum number of seconds to run all queries: 50.869 seconds
> Maximum number of seconds to run all queries: 50.869 seconds
> Number of clients running queries: 30
> Average number of queries per client: 333
>
> Benchmark
> Running for engine innodb
> Average number of seconds to run all queries: 52.766 seconds
> Minimum number of seconds to run all queries: 52.766 seconds
> Maximum number of seconds to run all queries: 52.766 seconds
> Number of clients running queries: 50
> Average number of queries per client: 200
>
> Huge performance drop compared to 8.3.12 as well.
>
> Why is there such a big difference between 8.3.12 and 8.4.1?
>
> Thank you
> Fred
>
>
>
> On 2 Feb 2012, at 14:31, Lars Ellenberg wrote:
>
> > On Thu, Feb 02, 2012 at 02:24:08PM +0000, Frederic DeMarcy wrote:
> >> Hi Lars
> >>
> >> I'm usinng 8.4.1
> >>
> >> [root at server1 ~]# service drbd status
> >> drbd driver loaded OK; device status:
> >> version: 8.4.1 (api:1/proto:86-100)
> >>
> >
> > Is that so.
> > Apologies, I seem to have mixed a few threads here.
> >
> > In that case, invalidate-remote when in StandAlone should be allowed.
> > Strange.
> >
> > Maybe you should downgrade ;-)
> > and see where things go.
> >
> > In fact, my guess is moving meta data to RAID1
> > will be most useful in your case.
> >
> >>>>
> >>>> Is the result the same if you execute a "drbdadm invalidate-remote
> >>>> mysql" on the primary before doing the "single node" test? .... that
> >>>> would disable activity log updates ...
> >>>
> >>> ... with recent enough versions of DRBD.
> >>> Only the OP seems to be using 8.3.7.
> >>> Maybe you should consider an upgrade?
> >
> >
> > --
> > : Lars Ellenberg
> > : LINBIT | Your Way to High Availability
> > : DRBD/HA support and consulting http://www.linbit.com
> >
> > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> > __
> > please don't Cc me, but send to list -- I'm subscribed
> > _______________________________________________
> > drbd-user mailing list
> > drbd-user at lists.linbit.com
> > http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20120209/e9896583/attachment.htm>