[DRBD-user] slow sync speed of 12TB over 10GE

[Lars Täuber]

Hallo Tomasz

Tomasz Chmielewski <mangoo at wpkg.org> schrieb:
> Lars Ellenberg wrote:
> 
> > if you cannot get more than 130 MB/sec on top of a ram disk and 10GbE,
> > there is something fishy going on,
> > and I doubt that drbd can do much about it.
> 
> I don't see anything fishy here.
> 
> I guess someone is confusing the _read_ speed of a device:

Good point.
 
> With its _write_ speed:
> 
> >         [==============>.....] sync'ed: 76.4% (242356/1023804)K
> >         finish: 0:00:01 speed: 130,240 (130,240) K/sec
> 
> How fast can you write on that RAID-6 device (with DRBD sync to the 
> other node disabled)?
> 
> i.e., do:
> 
> # sync
> # time dd if=/dev/zero of=/mnt/mountpint/bigfile bs=64k count=100000
> # time sync

 # time dd if=/dev/zero of=/dev/md4 bs=64k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
6553600000 Bytes (6,6 GB) kopiert, 25,2106 s, 260 MB/s

real	0m25.213s
user	0m0.020s
sys	0m11.673s

# dd if=/dev/zero of=/dev/md4 bs=64k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
6553600000 Bytes (6,6 GB) kopiert, 25,5979 s, 256 MB/s
# dd if=/dev/zero of=/dev/md4 bs=64k count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
6553600000 Bytes (6,6 GB) kopiert, 24,6065 s, 266 MB/s
# LANG=C dd if=/dev/zero of=/dev/md4 bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes (6.6 GB) copied, 25.4397 s, 258 MB/s


 
> Also, looking at "iostat -d 1 -k" output in the other terminal while the 
> writes are going would be interesting.

What about this results?
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdc             601,00       220,00     22360,00        220      22360
sdd             512,00       188,00     22000,00        188      22000
sde             543,00       228,00     22252,00        228      22252
sdf             629,00       196,00     22324,00        196      22324
sdg             616,00       236,00     22364,00        236      22364
sdh             575,00       264,00     22360,00        264      22360
sdi             657,00       160,00     22360,00        160      22360
sdj             648,00       180,00     22080,00        180      22080
sdk             688,00       212,00     22360,00        212      22360
sdl             629,00       168,00     22360,00        168      22360
sdm             675,00       228,00     22028,00        228      22028
sdn             700,00       240,00     22360,00        240      22360
sdo             589,00       236,00     22356,00        236      22356
sdp             680,00       256,00     22080,00        256      22080
sdq             581,00       280,00     22360,00        280      22360
sdr               0,00         0,00         0,00          0          0
md2               0,00         0,00         0,00          0          0
md3               0,00         0,00         0,00          0          0
md4           72164,00         0,00    288656,00          0     288656

The numbers are floating between ~18900 and ~23400.

Should I really mount the device with a filesystem on it? The last tests were made without a filesystem too.

Lars