[DRBD-user] Slower disk throughput on DRBD partition

Frederic DeMarcy fred.demarcy.ml at gmail.com
Thu Feb 2 12:37:58 CET 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi Andreas

Thank you.

Couldn't try "drbdadm invalidate-remote mysql" because when both nodes are
connected it forces a sync (SyncSource > SyncTarget)...
If I disconnect the primary first then issue "drbdadm invalidate-remote
mysql" I get:

0: State change failed: (-15) Need a connection to start verify or resync
Command 'drbdsetup invalidate-remote 0' terminated with exit code 11

Kind regards,
Fred

On Wed, Feb 1, 2012 at 9:18 PM, Andreas Kurz <andreas at hastexo.com> wrote:

> On 02/01/2012 05:15 PM, Frederic DeMarcy wrote:
> > Hi Andrea
> >
> > Commenting out "csum-alg" doesn't seem to make any noticeable
> difference...
> > However commenting out "data-integrity-alg" and running Test #2 again
> > increases the throughput from ~ 61MB/s to ~ 97MB/s !
> > Note that I may well run into the 1Gb/s crossover link limit here since
> > my network tests showed ~ 0.94 Gb/s
> >
> > Also Test #1 was wrong in my email... It should have been split in 2:
> > Test #1
> > On non-DRBD device (/dev/sda)
> > # dd if=/dev/zero of=/home/userxxx/disk-test.xxx bs=1M count=4096
> > oflag=direct
> > Throughput ~ 420MB/s
> >
> > DRBD partition (/dev/sdb) on primary (secondary node disabled)
> > Using Base DRBD config
> > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M count=4096
> > oflag=direct
> > Throughput ~ 205MB/s
>
> Is the result the same if you execute a "drbdadm invalidate-remote
> mysql" on the primary before doing the "single node" test? .... that
> would disable activity log updates ...
>
> Regards,
> Andreas
>
> --
> Need help with DRBD?
> http://www.hastexo.com/services/remote
>
> >
> > With the above -alg commented out, disabling the secondary node and
> > running Test #1 again (correctly split this time) shows the same
> > throughputs of ~ 420MB/s and ~ 205MB/s
> >
> > Fred
> >
> > On Wed, Feb 1, 2012 at 1:48 PM, Andreas Kurz <andreas at hastexo.com
> > <mailto:andreas at hastexo.com>> wrote:
> >
> >     Hello,
> >
> >     On 02/01/2012 01:04 PM, Frederic DeMarcy wrote:
> >     > Hi
> >     >
> >     > Note 1:
> >     > Scientific Linux 6.1 with kernel 2.6.32-220.4.1.el6.x86_64
> >     > DRBD 8.4.1 compiled from source
> >     >
> >     > Note 2:
> >     > server1 and server2 are 2 VMware VMs on top of ESXi 5. However
> >     they reside on different physical 2U servers.
> >     > The specs for the 2U servers are identical:
> >     >   - HP DL380 G7 (2U)
> >     >   - 2 x Six Core Intel Xeon X5680 (3.33GHz)
> >     >   - 24GB RAM
> >     >   - 8 x 146 GB SAS HD's (7xRAID5 + 1s)
> >     >   - Smart Array P410i with 512MB BBWC
> >
> >     Have you tried to change the I/O scheduler to deadline or noop in
> >     the VMs?
> >
> >     ... see below ..
> >
> >     >
> >     > Note 3:
> >     > I've tested the network throughput with iperf which yields close
> >     to 1Gb/s
> >     > [root at server1 ~]# iperf -c 192.168.111.11 -f g
> >     > ------------------------------------------------------------
> >     > Client connecting to 192.168.111.11, TCP port 5001
> >     > TCP window size: 0.00 GByte (default)
> >     > ------------------------------------------------------------
> >     > [  3] local 192.168.111.10 port 54330 connected with
> >     192.168.111.11 port 5001
> >     > [ ID] Interval       Transfer     Bandwidth
> >     > [  3]  0.0-10.0 sec  1.10 GBytes  0.94 Gbits/sec
> >     >
> >     > [root at server2 ~]# iperf -s -f g
> >     > ------------------------------------------------------------
> >     > Server listening on TCP port 5001
> >     > TCP window size: 0.00 GByte (default)
> >     > ------------------------------------------------------------
> >     > [  4] local 192.168.111.11 port 5001 connected with 192.168.111.10
> >     port 54330
> >     > [ ID] Interval       Transfer     Bandwidth
> >     > [  4]  0.0-10.0 sec  1.10 GBytes  0.94 Gbits/sec
> >     >
> >     > Scp'ing a large file from server1 to server2 yields ~ 57MB/s but I
> >     guess it's due to the encryption overhead.
> >     >
> >     > Note 4:
> >     > MySQL was not running.
> >     >
> >     >
> >     >
> >     > Base DRBD config:
> >     > resource mysql {
> >     >   startup {
> >     >     wfc-timeout 3;
> >     >     degr-wfc-timeout 2;
> >     >     outdated-wfc-timeout 1;
> >     >   }
> >     >   net {
> >     >     protocol C;
> >     >     verify-alg sha1;
> >     >     csums-alg sha1;
> >
> >     using csums based resync is only interesting for WAN setups where you
> >     need to sync via a rather thin connection
> >
> >     >     data-integrity-alg sha1;
> >
> >     using data-integrity-alg is definitely not recommended (slow) for
> live
> >     setups, only if you have to assume there is buggy hardware on the way
> >     between your nodes ... like nics pretending csums are ok while they
> >     are not
> >
> >     and out of curiosity ... did you gave DRBD 8.3.12 already a try?
> >
> >     Regards,
> >     Andreas
> >
> >     --
> >     Need help with DRBD?
> >     http://www.hastexo.com/now
> >
> >
> >     >     cram-hmac-alg sha1;
> >     >     shared-secret "MySecret123";
> >     >   }
> >     >   on server1 {
> >     >     device    /dev/drbd0;
> >     >     disk      /dev/sdb;
> >     >     address   192.168.111.10:7789 <http://192.168.111.10:7789>;
> >     >     meta-disk internal;
> >     >   }
> >     >   on server2 {
> >     >     device    /dev/drbd0;
> >     >     disk      /dev/sdb;
> >     >     address   192.168.111.11:7789 <http://192.168.111.11:7789>;
> >     >     meta-disk internal;
> >     >   }
> >     > }
> >     >
> >     >
> >     > After any change in the /etc/drbd.d/mysql.res file I issued a
> >     "drbdadm adjust mysql" on both nodes.
> >     >
> >     > Test #1
> >     > DRBD partition on primary (secondary node disabled)
> >     > Using Base DRBD config
> >     > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M
> >     count=4096 oflag=direct
> >     > Throughput ~ 420MB/s
> >     >
> >     > Test #2
> >     > DRBD partition on primary (secondary node enabled)
> >     > Using Base DRBD config
> >     > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M
> >     count=4096 oflag=direct
> >     > Throughput ~ 61MB/s
> >     >
> >     > Test #3
> >     > DRBD partition on primary (secondary node enabled)
> >     > Using Base DRBD config with:
> >     >   Protocol B;
> >     > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M
> >     count=4096 oflag=direct
> >     > Throughput ~ 68MB/s
> >     >
> >     > Test #4
> >     > DRBD partition on primary (secondary node enabled)
> >     > Using Base DRBD config with:
> >     >   Protocol A;
> >     > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M
> >     count=4096 oflag=direct
> >     > Throughput ~ 94MB/s
> >     >
> >     > Test #5
> >     > DRBD partition on primary (secondary node enabled)
> >     > Using Base DRBD config with:
> >     >   disk {
> >     >     disk-barrier no;
> >     >     disk-flushes no;
> >     >     md-flushes no;
> >     >   }
> >     > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M
> >     count=4096 oflag=direct
> >     > Disk throughput ~ 62MB/s
> >     >
> >     > No difference from Test #2 really. Also cat /proc/drbd still shows
> >     wo:b in both cases so I'm not even sure
> >     > these disk {..} parameters have been taken into account...
> >     >
> >     > Test #6
> >     > DRBD partition on primary (secondary node enabled)
> >     > Using Base DRBD config with:
> >     >   Protocol B;
> >     >   disk {
> >     >     disk-barrier no;
> >     >     disk-flushes no;
> >     >     md-flushes no;
> >     >   }
> >     > # dd if=/dev/zero of=/var/lib/mysql/TMP/disk-test.xxx bs=1M
> >     count=4096 oflag=direct
> >     > Disk throughput ~ 68MB/s
> >     >
> >     > No difference from Test #3 really. Also cat /proc/drbd still shows
> >     wo:b in both cases so I'm not even sure
> >     > these disk {..} parameters have been taken into account...
> >     >
> >     >
> >     > What else can I try?
> >     > Is it worth trying DRBD 8.3.x?
> >     >
> >     > Thx.
> >     >
> >     > Fred
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > On 1 Feb 2012, at 08:35, James Harper wrote:
> >     >
> >     >>> Hi
> >     >>>
> >     >>> I've configured DRBD with a view to use it with MySQL (and later
> on
> >     >>> Pacemaker + Corosync) in a 2 nodes primary/secondary
> >     >>> (master/slave) setup.
> >     >>>
> >     >>> ...
> >     >>>
> >     >>> No replication over the 1Gb/s crossover cable is taking place
> >     since the
> >     >>> secondary node is down yet there's x2 lower disk performance.
> >     >>>
> >     >>> I've tried to add:
> >     >>>  disk {
> >     >>>    disk-barrier no;
> >     >>>    disk-flushes no;
> >     >>>    md-flushes no;
> >     >>>  }
> >     >>> to the config but it didn't seem to change anything.
> >     >>>
> >     >>> Am I missing something here?
> >     >>> On another note is 8.4.1 the right version to use?
> >     >>>
> >     >>
> >     >> If you can do it just for testing, try changing to protocol B
> >     with one primary and one secondary and see how that impacts your
> >     performance, both with barrier/flushes on and off. I'm not sure if
> >     it will help but if protocol B makes things faster then it might
> >     hint as to where to start looking...
> >     >>
> >     >> James
> >     >
> >     > _______________________________________________
> >     > drbd-user mailing list
> >     > drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
> >     > http://lists.linbit.com/mailman/listinfo/drbd-user
> >
> >
> >
> >
> >
> >     _______________________________________________
> >     drbd-user mailing list
> >     drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
> >     http://lists.linbit.com/mailman/listinfo/drbd-user
> >
> >
> >
> >
> > _______________________________________________
> > drbd-user mailing list
> > drbd-user at lists.linbit.com
> > http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
>
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20120202/9c9086ef/attachment.htm>


More information about the drbd-user mailing list