[DRBD-user] Directly connected GigE ports bonded together no switch

[Christian Balzer]

On Wed, 10 Aug 2011 17:20:12 -0400 (EDT) Jake Smith wrote:

[Huge snip]
> 
> 
> I tuned my MTU setting on the direct link bond to 9000 and saw a 10%
> improvement on throughput. Negligible on latency though.
> 
> I was getting consistent 180-185MB/s using the throughput testing script
> in the DRBD Users guide with mtu 1500. Iperf was 1.75-1.85Gb/s. After
> changing MTU I get 198-99MB/s consistently and highs at 209-215MB/s.
> Without DRBD my storage controller is delivering 225MB/s so now there's
> almost no cost on the throughput side.  Iperf was rock solid at
> 1.97-1.98Gb/s repeatedly.
> 

These numbers match my similar (dual GigE balance-RR replication link)
setup.
And if you look back in the archives you can find my numbers for quad GigE
balance-RR link).

What is more than puzzling to me are these write speeds:

- Initial resync happens at near wire-speed (rate was set to 200MB/s,
  ethstats output confirms this speed).
- A makefs (ext4) happens at about the same speed (staring at ethstats).
- A bonnie++ run on the mounted ext4 fs of the DRBD device clocks in at
  about 130-150MB/s depending on trailing winds and phase of the moon.
  This bonnie result matches what I see from ethstats. 
- The same bonnie++ run on the underlying backing device delivers about
  350MB/s.

So what is different when a FS is mounted as opposed to the raw
(DRBD) device? Where are those 50MB/s hiding or getting lost?

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/