[DRBD-user] DRBD performance (only uses 40% of a GE link)

Christian Balzer chibi at gol.com
Mon Mar 3 11:05:47 CET 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hello,

On Mon, 3 Mar 2008 09:34:30 +0100 Florian Haas wrote:
> And you are perfectly certain you know what you're doing in setting
> use-bmbv?
>
No, but a question from me about this last Thursday went unanswered so far
and the documentation suggests that it speeds things up.
So for the sake of determining the most effective setup...

> And you are also perfectly certain you want to set max-buffers != 
> max-epoch-size?
> 
Nothing I read suggested that they need to equal, but changing both to
2048 made no difference. Not that I expected any, given that a test with
protocol B gave pretty much the same results as C.

> And you have a good reason to disable the sndbuf-size 512k you had
> already set?
>
Yeah, since it did not have any noticeable effect on the performance.

It did test more than just the config posted, but the results did not vary
considerably. 
Right now I'm doing a run with use-bmbv off, max buffers and epoch at 8192
and sndbuf at 1MB. ethstats hovers around:
---
eth1:  446.41 Mb/s In     6.09 Mb/s Out -   9123.4 p/s In    4778.3 p/s Out
---
Which suggest the results will be slightly better, but still it's just
using half of the available bandwidth. 
What is the most throughput anybody has achieved on writes with Protocol
C and what settings (and network hardware) did they use? 

> Just my €.02.
> 
Just my 2yen. ^_^

Christian

P.S.
The results just came back:
---
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
borg00a      50000M           54917  17 47675  15           410592  55 392.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                128 79514  95 +++++ +++ 65768  76 78165  94 +++++ +++ 81634 100
---
So yes, slightly better, but still nothing to write home about.
Which of the changes above you think brought this about and will cranking
up things to even more insane levels get me to about 80MB/s (which is what
I could settle for)?

> 
> On Saturday 01 March 2008 06:49:30 Christian Balzer wrote:
> > Hello,
> >
> > I'm trying to build a HA cluster here. Each node has 8 2.66GHz cpu
> > cores, 24GB RAM and 8 1TB SATA drives behind a LSI (Fusion MPT) SAS
> > 1068E controller. Interconnection is via one of 4 1GE interfaces,
> > directly. Kernel is 2.6.22.18 and DRBD is 8.0.11, the storage device
> > in question is a 3TB MD RAID5 spread across all 8 drives. The native
> > results for this device using ext3 and bonnie for benchmarking are:
> > ---
> >  Version  1.03       ------Sequential Output------ --Sequential Input-
> > --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> > Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> > %CP  /sec %CP borg00a      50000M           120486  36 87998
> > 17           535665  44 390.9   1 ------Sequential Create------
> > --------Random Create-------- -Create-- --Read--- -Delete-- -Create--
> > --Read--- -Delete-- files  /sec %CP /sec %CP  /sec %CP  /sec %CP  /sec
> > %CP  /sec %CP 128 74265  90 +++++ +++ 83659 100 71540  88 +++++ +++
> > 81619  99 ---
> >
> > The same test done on the resulting (UpToDate) drbd device:
> > ---
> > Version  1.03       ------Sequential Output------ --Sequential Input-
> > --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> > Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> > %CP  /sec %CP borg00a      50000M           41801  13 39659
> > 11           413367  37 397.7   1 ------Sequential Create------
> > --------Random Create-------- -Create-- --Read--- -Delete-- -Create--
> > --Read--- -Delete-- files  /sec %CP /sec %CP  /sec %CP  /sec %CP  /sec
> > %CP  /sec %CP 128 78847  95 +++++ +++ 86936  99 78722  95 +++++ +++
> > 63054  76 ---
> 


-- 
Christian Balzer        Network/Systems Engineer                NOC
chibi at gol.com   	Global OnLine Japan/Fusion Network Services
http://www.gol.com/



More information about the drbd-user mailing list