Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Have a look at the net parameters. Autotuning does not seem to do a good
job at high speeds. We had verifies starving the system to death with 2 10
Gbit cards with 300M syncer. This was a active-passive setup, but the
problem might be the same. Try send and recieve buffers at 10MB, max
buffers 128k (maximum value), adaptive syncer settings (introduced in
8.3.10). We are now getting 550 Mb/s even during a ongoing verify.
Mit freundlichen Grüßen / Best Regards
Robert Köppl
Systemadministration
KNAPP Systemintegration GmbH
Waltenbachstraße 9
8700 Leoben, Austria
Phone: +43 3842 805-910
Fax: +43 3842 82930-500
robert.koeppl at knapp.com
www.KNAPP.com
Commercial register number: FN 138870x
Commercial register court: Leoben
The information in this e-mail (including any attachment) is confidential
and intended to be for the use of the addressee(s) only. If you have
received the e-mail by mistake, any disclosure, copy, distribution or use
of the contents of the e-mail is prohibited, and you must delete the
e-mail from your system. As e-mail can be changed electronically KNAPP
assumes no responsibility for any alteration to this e-mail or its
attachments. KNAPP has taken every reasonable precaution to ensure that
any attachment to this e-mail has been swept for virus. However, KNAPP
does not accept any liability for damage sustained as a result of such
attachment being virus infected and strongly recommend that you carry out
your own virus check before opening any attachment.
Christophe Bouder <Christophe.Bouder at lip6.fr>
Gesendet von: drbd-user-bounces at lists.linbit.com
07.09.2011 16:15
Bitte antworten an
Christophe.Bouder at lip6.fr
An
dsu at hemlock.com
Kopie
drbd-user at lists.linbit.com
Thema
Re: [DRBD-user] slow to stall when sync kicks in
Hi,
We have exactly the same problem with large files copy.
We're running on Debian 64 bit, with kernel 3.0.3, drbd 8.3.11 and ocfs2
filesystem. We dedicated 2 10G ports to drbd.
Are you still encountering the same problem? Did you get some solutions
from drbd team? Did you find some tricks on your own?
Thanks in advance,
Le 16/08/2011 01:24, Dennis Su a écrit :
Hi,
We have two servers built, different hardware and capacity, on 64 bit
centos 5.6 with drbd 8.3.8, the backing devices are gfs2 devices layer on
top of LVM, because different capacity on both nodes and that limits our
ability to play with the barriers effectively. The server has two Gb
ports, one connect to the network and the other connected with a crossover
cable for dedicated drbd sync. We are running a number of tests to see how
well they perform before putting into production. During the weeks long
testing, one thing doesn't seem right and we've tried to play with the
config to tweak it, but not gaining anything, so we go back to default for
most settings as shown in the drbd show below.
It is a duo primary setup on a 10TB device. The sync/resync operations
between nodes, without other activities on the drbd devices, is 63MB
(sustained) on both way. However, adding the write to either node during
the sync can cause the both write and sync speed to drop to 2KB. The
situation can happen on any new write operation as well, initially the
writing, first 10 seconds, are at the expected speed, then drbd sync kicks
in, can see it from the /proc/drbd, it slows to almost stall, once the
sync stopped, observed from from both ifstat and /proc/drbd, the writing
to the node resume to expected speed, but once the sync start again the
writing slows down. The pattern repeats until the entire write operation
is complete. We have tested with small size files, the effect is minimum
so no problems in there, but we intent to use these serves to store large
files, which can be few GByte each. Initially, we though that high IO
might be the culprit, then we took drbd out of the picture and just runs
simultaneous read and write tests and they were fine with large and small
files. Now we think that drbd might need to locked the file to perform the
sync, during locked time the continues write stream on the file is not
permitted, then once the sync is done drbd releases it for writing again.
Then we tried tweaking the buffer and bio, no help. Of course, this just a
guess, but if it is true, is that any ways to tweak drbd to perform better
with big files.
We also tried swapping a cross over with a straight cable no improvements,
Thanks in advance,
Dennis
###################drbdsetup /dev/drbd0 show ##############
disk {
size 0s _is_default; # bytes
on-io-error detach;
fencing dont-care _is_default;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 256;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 0 _is_default; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
allow-two-primaries;
cram-hmac-alg "sha1";
shared-secret "U$eP at sswd";
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect _is_default;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
}
syncer {
rate 112640k; # bytes/second
after -1 _is_default;
al-extents 257;
delay-probe-volume 16384k _is_default; # bytes
delay-probe-interval 5 _is_default; # 1/10 seconds
throttle-threshold 20 _is_default; # 1/10 seconds
hold-off-threshold 100 _is_default; # 1/10 seconds
}
protocol C;
_this_host {
device minor 0;
disk "/dev/mapper/vg0-r0";
meta-disk internal;
address ipv4 10.1.1.35:7788;
}
_remote_host {
address ipv4 10.1.1.29:7788;
}
##############################
Hemlock Printers, Ltd.
(604) 439-5075
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
--
Christophe Bouder,
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110907/14937d56/attachment.htm>