[DRBD-user] Diagnosing DRBD Slowness

Ben Clewett ben at roadrunner.uk.com
Tue Oct 21 21:53:23 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.



Your bonnie is a bit broken up, but seems to show 78MB/sec sequential 
disk read.

I can see this in your config:

common {
   syncer {
     rate 33M;
     al-extents 257;
   }
}

This will restrict your sync to 33MB/sec, despite ability of disk.

Does this account for the problem?

Try changing to limit of network card (on both servers) and enter:

# drbdadm adjust

Does this fix your problem?

Ben




Robinson, Eric wrote:
> Can someone help please me diagnose a DRBD slow sync problem?
>  
> The servers have been syncing for 3-4 days and are now only 12.5% sync'd.
>  
> I'm getting conflicting results from benchmark tests.
>  
> Summary:
>  
> netperf is fast
> iperf is fast
> bonnie++ is fast
> dd|nc is extremely slow
> drbd sync is extremely slow
>  
> Details:
>  
>  
> Here's iperf... very fast...
>  
>  
> [root at ha03 download]# iperf -t TCP_STREAM -c 2.1.1.2
> ------------------------------------------------------------
> Client connecting to 2.1.1.2, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 2.1.1.1 port 59472 connected with 2.1.1.2 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-127.4 sec  13.9 GBytes    939 Mbits/sec
> 
>  
>  
> Now netperf... very fast...
>  
>  
>  
> [root at ha03 download]# netperf -H 2.1.1.2
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 2.1.1.2 
> (2.1.1.2) port 0 AF_INET
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>  87380  16384  16384    10.03     938.80
>  
>  
>  
> bonnie++ on one server... fast...
>  
> Version  1.03       ------Sequential Output------ --Sequential Input- 
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
> /sec %CP
> ha03.mycharts.md 6G 68212  90 187783  31 91553  17 73379  90 271470  28  
> 1099   1
>                     ------Sequential Create------ --------Random 
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
> /sec %CP
>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
> +++++ +++
> ha03.mycharts.md,6G,68212,90,187783,31,91553,17,73379,90,271470,28,1098.6,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++                       
> ++,+++
>  
> bonnie++ on the other server... fast...
>  
> Version  1.03       ------Sequential Output------ --Sequential Input- 
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
> /sec %CP
> ha04.mycharts.md 6G 69619  91 191459  31 83074  16 72740  89 265057  27 
> 964.9   1
>                     ------Sequential Create------ --------Random 
> Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
> /sec %CP
>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
> +++++ +++
> ha04.mycharts.md,6G,69619,91,191459,31,83074,16,72740,89,265057,27,964.9,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,++++
>  
>  
>  
> Now the dd command... very slow...
>  
>  
> [root at ha03 download]# dd if=/dev/zero bs=1M count=100 | nc 2.1.1.2 10000
> 2+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 12.7668 seconds, 82.1 kB/s
>  
>  
> drbd stats... very slow...
>  
> [root at ha04 download]# cat /proc/drbd
> version: 8.0.13 (api:86/proto:86)
> GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by 
> buildsvn at c5-i386-build <mailto:buildsvn at c5-i386-build>, 2008-08-07 13:42:56
>  0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
>     ns:0 nr:196 dw:196 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0
>         resync: used:0/61 hits:9 misses:1 starving:0 dirty:0 changed:1
>         act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
>  1: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:89232960 dw:89232960 dr:0 al:0 bm:5446 lo:0 pe:0 ua:0 ap:0
>         [=>..................] sync'ed: 13.8% (547719/634860)M
>         finish: 389:29:20 speed: 320 (316) K/sec
>         resync: used:0/61 hits:5571613 misses:5447 starving:0 dirty:0 
> changed:5447
>         act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
>  
> drbd.conf...
>  
> 
> [root at ha04 download]# cat /etc/drbd.conf
> # drbd.conf
>  
> global {
>     usage-count no;
> }
>  
> 
> common {
>   syncer {
>     rate 33M;
>     al-extents 257;
>   }
> }
>  
> resource mysql_ha {
>   protocol C;
>   handlers {
>     pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
>     pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
>     local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
>     outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
>     #pri-lost "echo pri-lost. Have a look at the log files. | mail -s 
> 'DRBD Alert' root";
>     # split-brain "echo split-brain. drbdadm -- --discard-my-data 
> connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' admin at pmcipa.com 
> <mailto:admin at pmcipa.com>";
>     #out-of-sync "echo out-of-sync. drbdadm down $DRBD_RESOURCE. drbdadm 
> ::::0 set-gi $DRBD_RESOURCE. drbdadm up $DRBD_RESOURCE. | mail -s 'DRBD 
> Alert' root";
>   }
>   startup {
>     wfc-timeout  0;          # infinite
>     degr-wfc-timeout 120;    # 2 minutes.
>   }
>   disk {
>     on-io-error   detach;
>   }
>   net {
>     cram-hmac-alg "sha1";
>     shared-secret "NerfBurger23!";
>     after-sb-0pri disconnect;
>     after-sb-1pri disconnect;
>     after-sb-2pri disconnect;
>     rr-conflict disconnect;
>     # data-integrity-alg "md5";
>   }
>   on ha03.mycharts.md {
>     device     /dev/drbd0;
>     disk       /dev/vg01/lv01;
>     address    2.1.1.1:7788;
>     meta-disk  internal;
>   }
>   on ha04.mycharts.md {
>     device     /dev/drbd0;
>     disk       /dev/vg01/lv01;
>     address   2.1.1.2:7788;
>     meta-disk  internal;
>   }
> }
>  
> resource ftp_ha {
>   protocol C;
>   handlers {
>     pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
>     pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
>     local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
>     outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
>     #pri-lost "echo pri-lost. Have a look at the log files. | mail -s 
> 'DRBD Alert' root";
>     # split-brain "echo split-brain. drbdadm -- --discard-my-data 
> connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' admin at pmcipa.com 
> <mailto:admin at pmcipa.com>";
>     #out-of-sync "echo out-of-sync. drbdadm down $DRBD_RESOURCE. drbdadm 
> ::::0 set-gi $DRBD_RESOURCE. drbdadm up $DRBD_RESOURCE. | mail -s 'DRBD 
> Alert' root";
>   }
>   startup {
>     wfc-timeout  0;          # infinite
>     degr-wfc-timeout 120;    # 2 minutes.
>   }
>   disk {
>     on-io-error   detach;
>   }
>   net {
>     cram-hmac-alg "sha1";
>     shared-secret "NerfBurger23!";
>     after-sb-0pri disconnect;
>     after-sb-1pri disconnect;
>     after-sb-2pri disconnect;
>     rr-conflict disconnect;
>     # data-integrity-alg "md5";
>   }
>   on ha03.mycharts.md {
>     device     /dev/drbd1;
>     disk       /dev/vg01/lv02;
>     address    2.1.1.1:7789;
>     meta-disk  internal;
>   }
>   on ha04.mycharts.md {
>     device     /dev/drbd1;
>     disk       /dev/vg01/lv02;
>     address   2.1.1.2:7789;
>     meta-disk  internal;
>   }
> }
>  
>  
>  
> 
> --
> Eric Robinson
> 
> 
> 
> Disclaimer - October 21, 2008
> 
> This email and any files transmitted with it are confidential and 
> intended solely for *drbd-user at lists.linbit.com*. If you are not the 
> named addressee you should not disseminate, distribute, copy or alter 
> this email. Any views or opinions presented in this email are solely 
> those of the author and might not represent those of . Warning: Although 
> has taken reasonable precautions to ensure no viruses are present in 
> this email, the company cannot accept responsibility for any loss or 
> damage arising from the use of this email or attachments.
> 
> This disclaimer was added by Policy Patrol: http://www.policypatrol.com 
> <http://www.policypatrol.com/>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user


*************************************************************************
This e-mail is confidential and may be legally privileged. It is intended
solely for the use of the individual(s) to whom it is addressed. Any
content in this message is not necessarily a view or statement from Road
Tech Computer Systems Limited but is that of the individual sender. If
you are not the intended recipient, be advised that you have received
this e-mail in error and that any use, dissemination, forwarding,
printing, or copying of this e-mail is strictly prohibited. We use
reasonable endeavours to virus scan all e-mails leaving the company but
no warranty is given that this e-mail and any attachments are virus free.
You should undertake your own virus checking. The right to monitor e-mail
communications through our networks is reserved by us

  Road Tech Computer Systems Ltd. Shenley Hall, Rectory Lane, Shenley,
  Radlett, Hertfordshire, WD7 9AN. - VAT Registration No GB 449 3582 17
  Registered in England No: 02017435, Registered Address: Charter Court, 
  Midland Road, Hemel Hempstead,  Hertfordshire, HP2 5GE. 
*************************************************************************



More information about the drbd-user mailing list