[DRBD-user] Sgisdela! Help! Horrible DRBD performance on CentOS 5. Where do I start looking?

Michael Joyner mjoyner at ewc.edu
Thu Jun 10 22:24:14 CEST 2010

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On 06/10/2010 11:39 AM, Andreas Kurz wrote:
> On Thursday 10 June 2010 17:19:53 Michael Joyner wrote:
>    
>> better, but still way below box2box bandwidth.... is fs type or fact
>> using lvm a factor? Here are my test results, will try w/o LVM next.
>>      
> I can really recommend  "Part V. Optimizing DRBD performance" in the DRBD
> Users Guide ... or invest in a "DRBD Healthcheck" offered by Linbit ;-)
>
> Regards,
> Andreas
>    

I split it up into 3 volumes, to prevent split brain issue with LVM. :)
I retest with EXT4, same horrible results.
I then tested with XFS, much, much better results:
Migrated VMs over to it. They all seem to be running much better with 
XFS and not EXT4.
As soon as I can get back into the COLO, I will reroute cabling to setup 
a RR bond between the two boxes directly to see if that helps.

Thanks for all the help.

=== *** === XFS Timing Test:

Large files test:

rsync box2box /var/tmp:

sent 6.96G bytes  received 164 bytes  47.16M bytes/sec

rsync local2/data w/ peer:

sent 6.96G bytes  received 164 bytes  41.53M bytes/sec

rsync local2/data no peer:

sent 6.96G bytes  received 164 bytes  114.99M bytes/sec

Iozone test (no peer):

real    0m4.983s
real    0m4.922s
real    0m4.920s
real    0m4.915s
real    0m4.917s

Iozone test (w/ peer);

real    0m29.644s
real    0m28.056s
real    0m27.634s
real    0m28.203s
real    0m28.209s

>    
>> On 06/10/2010 04:05 AM, Andreas Kurz wrote:
>>      
>>> .... LVM on your system supports barriers -->  drbd supports barriers -->
>>> barriers are used to enforce write-after-write dependencies per default
>>> ... use drbd config-param 'no-disk-barrier'
>>>        
>> === new layout ==============================
>>
>> using /dev/sdb as /dev/drbd0, non-spanned vg.
>>
>> === new drbd.conf ============================
>>
>> global {
>>     usage-count yes;
>> }
>> common {
>>       protocol C;
>>       syncer { rate 700000K; }
>>       net {
>>           sndbuf-size 0;
>>           after-sb-0pri discard-least-changes;
>>           after-sb-1pri discard-secondary;
>>       }
>>       startup {
>>           degr-wfc-timeout 30000;
>>       }
>>       disk {
>>           no-disk-flushes;
>>           no-md-flushes;
>>           no-disk-barrier;
>>       }
>>
>> }
>> resource sdb {
>>     on san-node-1.ewc.edu {
>>       device    /dev/drbd0;
>>       disk      /dev/sdb;
>>       address   192.168.75.201:7789;
>>       meta-disk internal;
>>     }
>>     on san-node-2.ewc.edu {
>>       device    /dev/drbd0;
>>       disk      /dev/sdb;
>>       address   192.168.75.202:7789;
>>       meta-disk internal;
>>     }
>> }
>>
>> *********
>> === *** mount /dev/vgdrbd0/nfs1 /data *** ===
>>
>> === iozone -a -e -g 4096 with peer down:
>> real    0m1.757s,  real    0m1.738s, real    0m1.753s, real
>> 0m1.762s,  real    0m1.745s
>>
>> === with peer up:
>> real    0m6.739s, real    0m6.758s, real    0m6.619s, real    0m6.653s,
>> real    0m6.636s
>>
>> === big files test =============================
>>
>> ===  DRBD === rm -rfv /data/ISOS; rsync -a --verbose --human-readable
>> --progress ~mjoyner/ISOS /data/ISOS
>>
>> building file list ...
>> 8 files to consider
>> created directory /data/ISOS
>> ISOS/
>> ISOS/ISOS/
>> ISOS/ISOS/.lck-5a00b81000000000
>>             84 100%    0.00kB/s    0:00:00 (xfer#1, to-check=5/8)
>> ISOS/ISOS/.lck-6500b81000000000
>>             84 100%   82.03kB/s    0:00:00 (xfer#2, to-check=4/8)
>> ISOS/ISOS/.lck-7700b81000000000
>>             84 100%   82.03kB/s    0:00:00 (xfer#3, to-check=3/8)
>> ISOS/ISOS/SW_DVD5_NTRL_SQL_Svr_2008_SP1_English_X15-51857.ISO
>>        943.07M 100%  189.42MB/s    0:00:04 (xfer#4, to-check=2/8)
>> ISOS/ISOS/SW_DVD5_SQL_Svr_Enterprise_Edtn_2008_English_MLF_X14-89207.ISO
>>          3.26G 100%  189.81MB/s    0:00:16 (xfer#5, to-check=1/8)
>> ISOS/ISOS/SW_DVD5_Windows_Svr_2008w_SP2_English__x64_DC_EE_SE_X15-41371.ISO
>>          2.76G 100%  162.03MB/s    0:00:16 (xfer#6, to-check=0/8)
>>
>> #1) sent 6.96G bytes  received 164 bytes  190.60M bytes/sec
>> #2) sent 6.96G bytes  received 164 bytes  244.10M bytes/sec
>> #3) sent 6.96G bytes  received 164 bytes  244.10M bytes/sec
>>
>> === LOCAL FS ===  rm -rfv /var/tmp/ISOS; rsync -a --verbose
>> --human-readable --progress ~mjoyner/ISOS /var/tmp/ISOS
>>
>> #1) sent 6.96G bytes  received 164 bytes  167.63M bytes/sec
>> #2) sent 6.96G bytes  received 164 bytes  185.51M bytes/sec
>> #3) sent 6.96G bytes  received 164 bytes  220.85M bytes/sec
>>
>> === time umount /data
>> real    2m11.090s
>> user    0m0.000s
>> sys    1m49.566s
>>
>>
>> *********
>> === *** mount -o sync,data=journal /dev/vgdrbd0/nfs1 /data *** ===
>>
>> === iozone test, no peer
>>
>> real    0m9.483s real    0m9.583s real    0m9.583s real    0m9.559s
>> real    0m9.491s
>>
>> === izone test, w/ peer up.
>>
>> real    0m41.558s real    0m41.386s real    0m41.456s real    0m41.448s
>>
>>
>> === BIG FILES TEST
>> no peer) sent 6.96G bytes  received 164 bytes  74.40M bytes/sec
>> w/ peer) sent 6.96G bytes  received 164 bytes  29.79M bytes/sec
>> rsync/var/tmp2/var/tmp) sent 6.96G bytes  received 164 bytes  47.49M
>> bytes/sec
>>
>> === time umount /data
>>
>> real    0m1.078s
>> user    0m0.000s
>> sys    0m1.075s
>>
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user at lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>>      
>    


-- 

Michael Joyner ᏩᏯ
System Administrator/Edward Waters College
1658 Kings Road, Jacksonville, FL 32209
904-470-8170 (V) / 904-470-8170 (F)
ᏩᏙ

========================================================================
          HOW TO SUBMIT A WORK REQUEST TO INFORMATION TECHNOLOGY
------------------------------------------------------------------------

Via the web: http://otrs.ewc.edu/otrs/customer.pl

Via email: support at ewc.edu

* It is imperative that you submit each problem as a new item.
* Please do NOT combine multiple problem reports in a single workorder.
* PLEASE DO NOT EMAIL SUPPORT STAFF REQUESTS DIRECTLY.
* PLEASE USE THE JOB TRACKING SYSTEM.

========================================================================





More information about the drbd-user mailing list