[DRBD-user] Slow Buffered Read Performance with DRBD 8.4.1

Lars Ellenberg lars.ellenberg at linbit.com
Fri Feb 24 10:47:16 CET 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Thu, Feb 23, 2012 at 05:33:01PM -0500, Chris Andrews wrote:
> Hey everyone. We're having a problem with DRBD 8.4.1 in our Test Environment.
> Buffered Reads operate 10x slower than unbuffered reads (60 MB/s vs 900 MB/s)
> through DRBD, while both (buffered/direct) perform at full speed if done
> against the raid controller directly without DRBD.

Does 8.3 behave the same?
How does dm-linear compare?

You did of course double-check the read-ahead settings?

Did you watch with "iostat -xk 1" while doing the test?
Ignore the utilization column.
Focus on differences in the sda row between tests.

> This situation is nearly identical to a previous posting to the list:
>   http://lists.linbit.com/pipermail/drbd-user/2012-January/017634.html
> 
> We have two high performance servers with 2x BBU raid controllers each and
> 24 disks each.
> 
> Big Picture + Dirty Stats
> 
> Host_A                                     Host_B
> --------------------------------------------------------------------------
> 12disks sda  drbd_a (priamry)    <10gigE>  drbd_a (secondary) sda  12disks
> 12disks sdb  drbd_b (secondary)  <10gigE>  drbd_b (primary)   sdb  12disks
> 
> Notes
> a) we've done these tests with and without the partner node connected; those
>    below are unconnected
> b) recreating the drbd device does does not resolve the problem
> 
> 
> Hardware
> 2x LSI Logic / Symbios Logic MegaRAID SAS 2108 (MegaRAID SAS 9260-16i)
> 
> root at host_a:~>uname -a
> Linux host_a 2.6.35.14 #1 SMP Mon Jan 23 22:12:58 UTC 2012 x86_64 GNU/Linux
> 
> root at host_a:~>cat /proc/drbd
> version: 8.4.1 (api:1/proto:86-100)
> GIT-hash: 91b4c048c1a0e06777b5f65d312b38d47abaea80
> 
> root at host_a:~>drbdadm -V
> DRBDADM_BUILDTAG=GIT-hash:\ 91b4c048c1a0e06777b5f65d312b38d47abaea80\ build\ by\ root at host_a\,\
> 2012-01-31\ 22:40:37
> DRBDADM_API_VERSION=1
> DRBD_KERNEL_VERSION_CODE=0x080401
> DRBDADM_VERSION_CODE=0x080401
> DRBDADM_VERSION=8.4.1
> 
> Clear Cache (run before every 'dd' test)
> root at host_a:~>echo 3 > /proc/sys/vm/drop_caches
> 
> Buffered
> --------
>   1. A Side A Resources
> slow:
> root at host_a:~>dd if=/dev/drbd_a of=/dev/null bs=1M count=1024
> 1073741824 bytes (1.1 GB) copied, 16.4465 s, 65.3 MB/s
> 
> fast:
> root at host_a:~>dd if=/dev/sda of=/dev/null bs=1M count=1024
> 1073741824 bytes (1.1 GB) copied, 1.20017 s, 895 MB/s
> 
>   2. A Side B Resources (After going Primary)
> slow:
> root at host_a:~>dd if=/dev/drbd_b of=/dev/null bs=1M count=1024
> 1073741824 bytes (1.1 GB) copied, 17.6546 s, 60.8 MB/s
> 
>   3. B Side B Resources
> slow:
> root at host_b:~>dd if=/dev/drbd_b of=/dev/null bs=1M count=1024
> 1073741824 bytes (1.1 GB) copied, 16.8178 s, 63.8 MB/s
> 
> fast:
> root at host_b:~>dd if=/dev/sdb of=/dev/null bs=1M count=1024
> 1073741824 bytes (1.1 GB) copied, 1.21691 s, 882 MB/s
> 
> Direct Unbuffered
> -----------------
>   1. A Side A Resources
> fast:
> root at host_a:~>dd if=/dev/drbd_a of=/dev/null bs=1M count=1024 iflag=direct
> 1073741824 bytes (1.1 GB) copied, 1.09098 s, 984 MB/s
> 
> fast:
> root at host_a:~>dd if=/dev/sda of=/dev/null bs=1M count=1024 iflag=direct
> 1073741824 bytes (1.1 GB) copied, 1.07937 s, 995 MB/s
> 
>   2. B Side B Resources
> fast:
> root at host_b:~>dd if=/dev/sdb of=/dev/null bs=1M count=1024 iflag=direct
> 1073741824 bytes (1.1 GB) copied, 1.23987 s, 866 MB/s
> 
> fast:
> root at host_b:~>dd if=/dev/drbd_b of=/dev/null bs=1M count=1024 iflag=direct
> 1073741824 bytes (1.1 GB) copied, 1.33096 s, 807 MB/s
> 
> 
> As you can see, if we access the data through DRBD, we get terrible performance,
> some ~14x slower.
> 
> If we access the raid device directly, or we force direct IO through DRBD,
> performance is fast and as expected.
> 
> 
> 
> DRBD Config
> 
> global {
>     usage-count no ;
> }
> 
> common {
>   protocol C ;
> 
>   syncer {
>     rate 4194304K ;
>   }
> 
>   net {
>     sndbuf-size 0 ;
>     timeout     120 ;
>     connect-int 20 ;
>     ping-int    20 ;
>     max-buffers         8192 ;
>     max-epoch-size      8192 ;
>     ko-count    30 ;
> 
>     cram-hmac-alg "sha1" ;
>     shared-secret "some-password" ;
>   }
> 
>   disk {
>     disk-barrier no ;
>     disk-flushes no ;
>     md-flushes no ;
>     al-extents  509 ;
>   }
> 
> }
> 
> resource drbd_a {
>   device                /dev/drbd_a minor 0 ;
>   disk                  /dev/sda ;
>   meta-disk             internal ;
> 
>   startup {
>     become-primary-on    host_a ;
>   }
> 
>   on host_a {
>     address     10.255.1.1:7788 ;
>   }
>   on host_b {
>     address     10.255.1.2:7788 ;
>   }
> }
> 
> resource drbd_b {
>   device                /dev/drbd_b minor 1 ;
>   disk                  /dev/sdb ;
>   meta-disk             internal ;
>   startup {
>     become-primary-on   host_b ;
>   }
> 
>   on host_a {
>     address     10.255.2.1:7788 ;
>   }
>   on host_b {
>     address     10.255.2.2:7788 ;
>   }
> }
> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed



More information about the drbd-user mailing list