Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
We have two servers built, different hardware and capacity, on 64 bit
centos 5.6 with drbd 8.3.8, the backing devices are gfs2 devices layer
on top of LVM, because different capacity on both nodes and that limits
our ability to play with the barriers effectively. The server has two Gb
ports, one connect to the network and the other connected with a
crossover cable for dedicated drbd sync. We are running a number of
tests to see how well they perform before putting into production.
During the weeks long testing, one thing doesn't seem right and we've
tried to play with the config to tweak it, but not gaining anything, so
we go back to default for most settings as shown in the drbd show below.
It is a duo primary setup on a 10TB device. The sync/resync operations
between nodes, without other activities on the drbd devices, is 63MB
(sustained) on both way. However, adding the write to either node during
the sync can cause the both write and sync speed to drop to 2KB. The
situation can happen on any new write operation as well, initially the
writing, first 10 seconds, are at the expected speed, then drbd sync
kicks in, can see it from the /proc/drbd, it slows to almost stall, once
the sync stopped, observed from from both ifstat and /proc/drbd, the
writing to the node resume to expected speed, but once the sync start
again the writing slows down. The pattern repeats until the entire write
operation is complete. We have tested with small size files, the effect
is minimum so no problems in there, but we intent to use these serves to
store large files, which can be few GByte each. Initially, we though
that high IO might be the culprit, then we took drbd out of the picture
and just runs simultaneous read and write tests and they were fine with
large and small files. Now we think that drbd might need to locked the
file to perform the sync, during locked time the continues write stream
on the file is not permitted, then once the sync is done drbd releases
it for writing again. Then we tried tweaking the buffer and bio, no
help. Of course, this just a guess, but if it is true, is that any ways
to tweak drbd to perform better with big files.
We also tried swapping a cross over with a straight cable no
improvements,
Thanks in advance,
Dennis
###################drbdsetup /dev/drbd0 show ##############
disk {
size 0s _is_default; # bytes
on-io-error detach;
fencing dont-care _is_default;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 2048 _is_default;
max-buffers 2048 _is_default;
unplug-watermark 256;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 0 _is_default; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
allow-two-primaries;
cram-hmac-alg "sha1";
shared-secret "U$eP at sswd";
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect _is_default;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
}
syncer {
rate 112640k; # bytes/second
after -1 _is_default;
al-extents 257;
delay-probe-volume 16384k _is_default; # bytes
delay-probe-interval 5 _is_default; # 1/10 seconds
throttle-threshold 20 _is_default; # 1/10 seconds
hold-off-threshold 100 _is_default; # 1/10 seconds
}
protocol C;
_this_host {
device minor 0;
disk "/dev/mapper/vg0-r0";
meta-disk internal;
address ipv4 10.1.1.35:7788;
}
_remote_host {
address ipv4 10.1.1.29:7788;
}
##############################
Hemlock Printers, Ltd.
(604) 439-5075
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110815/02a9c0b6/attachment.htm>