[DRBD-user] Directly connected GigE ports bonded together no switch

Jake Smith jsmith at argotec.com
Fri Aug 12 18:03:48 CEST 2011


----- Original Message -----
> From: "Christian Balzer" <chibi at gol.com>
> To: drbd-user at lists.linbit.com
> Sent: Friday, August 12, 2011 12:58:11 AM
> Subject: Re: [DRBD-user] Directly connected GigE ports bonded together no switch
> 
> 
> On Wed, 10 Aug 2011 17:20:12 -0400 (EDT) Jake Smith wrote:
> 
> [Huge snip]
> > 
> > 
> > I tuned my MTU setting on the direct link bond to 9000 and saw a
> > 10%
> > improvement on throughput. Negligible on latency though.
> > 
> > I was getting consistent 180-185MB/s using the throughput testing
> > script
> > in the DRBD Users guide with mtu 1500. Iperf was 1.75-1.85Gb/s.
> > After
> > changing MTU I get 198-99MB/s consistently and highs at
> > 209-215MB/s.
> > Without DRBD my storage controller is delivering 225MB/s so now
> > there's
> > almost no cost on the throughput side.  Iperf was rock solid at
> > 1.97-1.98Gb/s repeatedly.
> > 
> 
> These numbers match my similar (dual GigE balance-RR replication
> link)
> setup.
> And if you look back in the archives you can find my numbers for quad
> GigE
> balance-RR link).
> 
> What is more than puzzling to me are these write speeds:
> 
> - Initial resync happens at near wire-speed (rate was set to 200MB/s,
>   ethstats output confirms this speed).
> - A makefs (ext4) happens at about the same speed (staring at
> ethstats).
> - A bonnie++ run on the mounted ext4 fs of the DRBD device clocks in
> at
>   about 130-150MB/s depending on trailing winds and phase of the
>   moon.
>   This bonnie result matches what I see from ethstats.
> - The same bonnie++ run on the underlying backing device delivers
> about
>   350MB/s.

I've not used bonnie++ before but if I'm reading it right I got 198...
What parameters did you run bonnie++ with?  Oh and here's my output so you can make sure I'm intrepreting it correctly!

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Condor       24048M   861  99 198487  19 80007  10  3715  78 259450  17 585.5  18
Latency              9474us     263ms    1524ms     160ms     279ms     122ms
Version  1.96       ------Sequential Create------ --------Random Create--------
Condor              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 28774  35 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               464us     396us     405us    1127us      18us      47us
1.96,1.96,Condor,1,1313173265,24048M,,861,99,198487,19,80007,10,3715,78,259450,17,585.5,18,16,,,,,28774,35,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,9474us,263ms,1524ms,160ms,279ms,122ms,464us,396us,405us,1127us,18us,47us

> 
> So what is different when a FS is mounted as opposed to the raw
> (DRBD) device? Where are those 50MB/s hiding or getting lost?
> 
> Regards,
> 
> Christian
> --
> Christian Balzer        Network/Systems Engineer
> chibi at gol.com   	Global OnLine Japan/Fusion Communications
> http://www.gol.com/
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
> 
> 



More information about the drbd-user mailing list