Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
We're doing a new install of drbd 0.7.0 on fedora core 2 running the
standard 2.6.7 kernel. We've been using drbd for quite some time, but
we're goming from a very old version (0.5.8 cvs) to 0.7.0.
In testing out 0.7.0 we've been having a heck of a time getting any
decent sync speed out of it, and I don't seem to be able to figure out
where the bottleneck is. I think I've read the relevant man pages and
docs, but still can't seem to figure it out.
If we scp a large file from one machine to another, we get about 20-30
MB/sec through the dedicated gigabit network link, but our sync speed
seems stuck right around 2.8 MB/sec. We get slightly better performance
from protocol B than C, but other than that, we haven't found any real
speed increases. If we set rate down to 1M, we can see it slow down, but
when we increase it, we get stuck at the 2.8MB/sec average.
I've tried playing with buffers, and all sorts of stuff, but with no
luck. Our cpu utilization is quite low (both machines are Athlon 3000's
with 2gigs ram on primary, and 1 gig on secondary).
Below is some useful output to show our current configuration. I hope
that this question hasn't been beaten to death on this list, but I tried
to search the list, and didn't seem to be able to find anything recent
(anything at all, not just about this).
Thanks in advance for any help/ideas.. This is really driving me nuts
:)..
--------------------------------------------------------------
Configuration info:
--------------------------------------------------------------
The servers
--------------------------------------------------------------
athlon 3000, with 2 gigs ram on primary, 1 gig on secondary
nics: Intel Corp. 82540EM Gigabit Ethernet (3 per server, 1 of which is
for the dedicated link)
/dev/nb0 is a 1.4TB SATA Raid using a 3ware Inc 3ware 7000-series
ATA-RAID
---------------------------------------------------------------
DRBD Setup
---------------------------------------------------------------
[root at nupiter etc]# cat /proc/drbd
version: 0.7.0 svn $Rev: 1438 $ (api:74/proto:74)
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:15523840 nr:0 dw:0 dr:15523840 al:0 bm:947 lo:0 pe:1533 ua:0 ap:0
[===================>] sync'ed: 99.4% (641829/656983)M
finish: 65:12:06 speed: 2,796 (2,813) K/sec
[root at nupiter etc]# drbdsetup /dev/nb0 show
Lower device: 08:04 (sda4)
Meta device: internal
Disk options:
on-io-error = detach
Local address: 192.168.0.1:7788
Remote address: 192.168.0.2:7788
Wire protocol: B
Net options:
timeout = 6.0 sec (default)
connect-int = 10 sec (default)
ping-int = 10 sec (default)
max-epoch-size = 8192
max-buffers = 8192 # This is much larger than default, not sure
# ...how much is reasonable though
sndbuf-size = 4194304 # Tried all sorts of stuff, 512k, 256k etc
ko-count = 0 (default)
Syncer options:
rate = 20480 KB/sec
group = 1
al-extents = 7 #this was much larger before, didn't matter