[DRBD-user] slow sync speed of 12TB over 10GE

Lars Täuber taeuber at bbaw.de
Tue Jan 13 15:39:49 CET 2009

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hallo Lars.

Lars Ellenberg <lars.ellenberg at linbit.com> schrieb:
> On Mon, Jan 12, 2009 at 04:44:05PM +0100, Lars Täuber wrote:
> what does dmesg say about
>  raid5: using function 
>  raid6: using algorithm

# dmesg | grep -i "raid.: using"
raid6: using algorithm sse2x4 (3141 MB/s)

 
> what
>   if you only use ram disk, no md raid?

# cat /proc/drbd 
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root at monosan, 2009-01-09 15:15:29
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r---
    ns:0 nr:2021792 dw:2021760 dr:0 al:0 bm:123 lo:1 pe:816 ua:1 ap:0 ep:1 wo:d oos:26140
	[==================>.] sync'ed: 98.8% (26140/2047900)K
	finish: 0:00:00 speed: 137,808 (126,360) K/sec

# LANG=C dd if=/dev/zero of=/dev/drbd0 
dd: writing to `/dev/drbd0': No space left on device
4095801+0 records in
4095800+0 records out
2097049600 bytes (2.1 GB) copied, 18.8435 s, 111 MB/s

but:
# hdparm -t /dev/drbd0 

/dev/drbd0:
 Timing buffered disk reads:  980 MB in  3.00 seconds = 326.46 MB/sec


>   if you use dm-zero for the disk?

What is dm-zero? I don't undestand which device this should be.

 
> does your tcp do funny things, "optimizing" for throughput, thereby
> hurting latency? 

I didn't change anything from normal SuSE installation.

> have a look at packet timings with tcpdump,
> look out for gaps in packet timing.

I made pcap dump but couldn't find anything strange except some crc errors every now and then (600 or so packets).

> if you cannot get more than 130 MB/sec on top of a ram disk and 10GbE,
> there is something fishy going on,
> and I doubt that drbd can do much about it.

I'm afraid this is the situation. Something with my 10GE is not as it should be.

I try to investigate further.


Best regards
Lars



More information about the drbd-user mailing list