[DRBD-user] Antwort: Re: slow to stall when sync kicks in

Robert.Koeppl at knapp.com Robert.Koeppl at knapp.com
Wed Sep 7 16:25:38 CEST 2011

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Have a look at the net parameters. Autotuning does not seem to do a good 
job at high speeds. We had verifies starving the system to death with 2 10 
Gbit cards with 300M syncer. This was a active-passive setup, but the 
problem might be the same. Try send and recieve buffers at 10MB,  max 
buffers 128k (maximum value), adaptive syncer settings (introduced in 
8.3.10). We are now getting 550 Mb/s even during a ongoing verify.
Mit freundlichen Grüßen / Best Regards

Robert Köppl


KNAPP Systemintegration GmbH
Waltenbachstraße 9
8700 Leoben, Austria 
Phone: +43 3842 805-910
Fax: +43 3842 82930-500
robert.koeppl at knapp.com 

Commercial register number: FN 138870x
Commercial register court: Leoben

The information in this e-mail (including any attachment) is confidential 
and intended to be for the use of the addressee(s) only. If you have 
received the e-mail by mistake, any disclosure, copy, distribution or use 
of the contents of the e-mail is prohibited, and you must delete the 
e-mail from your system. As e-mail can be changed electronically KNAPP 
assumes no responsibility for any alteration to this e-mail or its 
attachments. KNAPP has taken every reasonable precaution to ensure that 
any attachment to this e-mail has been swept for virus. However, KNAPP 
does not accept any liability for damage sustained as a result of such 
attachment being virus infected and strongly recommend that you carry out 
your own virus check before opening any attachment.

Christophe Bouder <Christophe.Bouder at lip6.fr> 
Gesendet von: drbd-user-bounces at lists.linbit.com
07.09.2011 16:15
Bitte antworten an
Christophe.Bouder at lip6.fr

dsu at hemlock.com
drbd-user at lists.linbit.com
Re: [DRBD-user] slow to stall when sync kicks in

We have exactly the same problem with large files copy.
We're running on Debian 64 bit, with kernel 3.0.3, drbd 8.3.11 and ocfs2 
filesystem. We dedicated 2 10G ports to drbd.
Are you still encountering the same problem? Did you get some solutions 
from drbd team? Did you find some tricks on your own?

Thanks in advance,

Le 16/08/2011 01:24, Dennis Su a écrit : 
We have two servers built, different hardware and capacity, on 64 bit 
centos 5.6 with drbd 8.3.8, the backing devices are gfs2 devices layer on 
top of LVM, because different capacity on both nodes and that limits our 
ability to play with the barriers effectively. The server has two Gb 
ports, one connect to the network and the other connected with a crossover 
cable for dedicated drbd sync. We are running a number of tests to see how 
well they perform before putting into production. During the weeks long 
testing, one thing doesn't seem right and we've tried to play with the 
config to tweak it, but not gaining anything, so we go back to default for 
most settings as shown in the drbd show below. 
It is a duo primary setup on a 10TB device. The sync/resync operations 
between nodes, without other activities on the drbd devices, is 63MB 
(sustained) on both way. However, adding the write to either node during 
the sync can cause the both write and sync speed to drop to 2KB. The 
situation can happen on any new write operation as well, initially the 
writing, first 10 seconds, are at the expected speed, then drbd sync kicks 
in, can see it from the /proc/drbd, it slows to almost stall, once the 
sync stopped, observed from from both ifstat and /proc/drbd, the writing 
to the node resume to expected speed, but once the sync start again the 
writing slows down. The pattern repeats until the entire write operation 
is complete. We have tested with small size files, the effect is minimum 
so no problems in there, but we intent to use these serves to store large 
files, which can be few GByte each. Initially, we though that high IO 
might be the culprit, then we took drbd out of the picture and just runs 
simultaneous read and write tests and they were fine with large and small 
files. Now we think that drbd might need to locked the file to perform the 
sync, during locked time the continues write stream on the file is not 
permitted, then once the sync is done drbd releases it for writing again. 
Then we tried tweaking the buffer and bio, no help. Of course, this just a 
guess, but if it is true, is that any ways to tweak drbd to perform better 
with big files.
We also tried swapping a cross over with a straight cable no improvements, 

Thanks in advance,
###################drbdsetup /dev/drbd0 show ##############
disk {
        size                    0s _is_default; # bytes
        on-io-error             detach;
        fencing                 dont-care _is_default;
        max-bio-bvecs           0 _is_default;
net {
        timeout                 60 _is_default; # 1/10 seconds
        max-epoch-size          2048 _is_default;
        max-buffers             2048 _is_default;
        unplug-watermark        256;
        connect-int             10 _is_default; # seconds
        ping-int                10 _is_default; # seconds
        sndbuf-size             0 _is_default; # bytes
        rcvbuf-size             0 _is_default; # bytes
        ko-count                0 _is_default;
        cram-hmac-alg           "sha1";
        shared-secret           "U$eP at sswd";
        after-sb-0pri           discard-zero-changes;
        after-sb-1pri           discard-secondary;
        after-sb-2pri           disconnect _is_default;
        rr-conflict             disconnect _is_default;
        ping-timeout            5 _is_default; # 1/10 seconds
syncer {
        rate                    112640k; # bytes/second
        after                   -1 _is_default;
        al-extents              257;
        delay-probe-volume      16384k _is_default; # bytes
        delay-probe-interval    5 _is_default; # 1/10 seconds
        throttle-threshold      20 _is_default; # 1/10 seconds
        hold-off-threshold      100 _is_default; # 1/10 seconds
protocol C;
_this_host {
        device                  minor 0;
        disk                    "/dev/mapper/vg0-r0";
        meta-disk               internal;
        address                 ipv4;
_remote_host {
        address                 ipv4;
Hemlock Printers, Ltd.
(604) 439-5075

drbd-user mailing list
drbd-user at lists.linbit.com

Christophe Bouder,

drbd-user mailing list
drbd-user at lists.linbit.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110907/14937d56/attachment.htm>

More information about the drbd-user mailing list