[DRBD-user] Re: Huge max latency

Lars Ellenberg Lars.Ellenberg at linbit.com
Sat Jan 6 18:44:35 CET 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


/ 2007-01-05 15:48:15 -0600
\ Weilin Gong:
> Forgot to mention, DRBD version is 0.7.11.

> Hi,
> 
> I ran some drbd performance tests with 'tiobench':
> 
> tiobench.pl --identifier drbd-connected --size 5000 --numruns 1 --dir /
> secroot --block 4096 --threads 4
> 
> noticed the huge max latency numbers with Sequential Writes:

let me help you with the formating...

> Unit information
> ================
> File size            = megabytes
> Blk Size            = bytes
> Rate                 = megabytes per second
> CPU%             = percentage of CPU used during the test
> Latency            = milliseconds
> Lat%                = percent of requests that took longer than X seconds
> CPU Eff           = Rate divided by CPU% - throughput per cpu load
> 
> ┌─────────────────┬─────┬─────┬────┬─────┬───────┬───────┬─────────┬───────┬───────┬─────┐
> │Identifier       │File │Blk  │Num │Rate │Maximum│Avg    │Max      │Lat% > │Lat% > │CPU  │
> │                 │Size │Size │Thr │     │CPU%   │Latency│Latency  │2s     │10s    │Eff  │
> ├─────────────────┼─────┼─────┼────┼─────┼───────┼───────┼─────────┼───────┼───────┼─────┤
> │Raw-device       │5000 │4096 │4   │31.24│26.80% │0.364  │5635.06  │0.00242│0.00000│117  │
> ├─────────────────┼─────┼─────┼────┼─────┼───────┼───────┼─────────┼───────┼───────┼─────┤
> │Drbd-disconnected│5000 │4096 │4   │33.18│26.29% │0.319  │110394.58│0.00190│0.00016│126  │
> ├─────────────────┼─────┼─────┼────┼─────┼───────┼───────┼─────────┼───────┼───────┼─────┤
> │Drbd-connected   │5000 │4096 │4   │20.88│50.18% │0.457  │135712.09│0.00110│0.00008│42   │
> └─────────────────┴─────┴─────┴────┴─────┴───────┴───────┴─────────┴───────┴───────┴─────┘
> 
> My lab setting:
>   ● Two Linux nodes on 2.6.10_mvlcge401-pc_target-x86_pentium3-P3SMP.
>   ● Dual-core Intel Xeon 2GHz CPU, 4GB Memory, 1Gbits/s network interface.
>   ● 5.7GB partition on SCSI disk, ext2 file system with the "-T largefile4"
>     option.
>   ● DRBD rate=100M;  protocal C; sndbuf-size 1024k.
> I don't believe the disk (~50MB/s) or network may be the factors here. You
> can see the big number jump even with the DRBD disconnected mode.

of course, drbd housekeeping adds latency.
_especially_ in disconnected mode (activity-log and dirty-bitmap updates).

but I also see the _average_ latency _drop_ and the rate increase
for drbd-disconnected...

so, how do you like me to explain that?

the much higher average latency for drbd connected may be due to network latency.
overall throughput is minimum,
overall latency is _sum_ of the respective component values.

> Any help on analyzing/identifying the problem will be greatly appreciated.
> 
> Weilin

* try again with a higher numruns, to get better statistics

* try again with 0.7.22
* try again with drbd 8 (well, rcX, or svn trunk)
* try again with large "al-extends"
  and then again in a second run...

* what is the real storage device?  sdX ? mdX ? lvm something?
* what is the meta data storage device?

-- 
: Lars Ellenberg                            Tel +43-1-8178292-0  :
: LINBIT Information Technologies GmbH      Fax +43-1-8178292-82 :
: Vivenotgasse 48, A-1120 Vienna/Europe    http://www.linbit.com :
__
please use the "List-Reply" function of your email client.



More information about the drbd-user mailing list