[DRBD-user] drbd resource write performance and sync speeds

Christian Balzer chibi at gol.com
Tue Apr 8 05:27:25 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hello Ingard,

On Fri, 04 Apr 2008 11:29:46 +0200 Ingard Mevåg wrote:
> I've tried to tune the parameters of the config file and i achieve 
> somewhat different results:
> 
> The best read speeds i've seen is with the default config, I got close 
> to 60000K/sec vs bare metal ~100000K/sec
> 
> The best write speeds i've seen is with the following config:
> common {
>          protocol C;
>          syncer {
>                  rate 160M;
>                  al-extents 1801;
>          }
> }
> 
>          net {
>                  allow-two-primaries;
>                  sndbuf-size 2M;
>                  max-buffers 16000;
>                  max-epoch-size 16000;
> 		unplug-watermark 16000;
>
These are (except for the 2 primaries) nearly exactly the same settings I
came up with after a lot of testing.

In my case it's a dual quad core box with 24GB RAM and 8 1TB SATA drives
in a RAID-5 that give me about 120MB/s writes on the "bare" MD device.
The first try with default values on DRBD gave me about 41MB/s writes and
a big "WTH?" feeling.
After tuning all the above parameters I wound up with about 60MB/s.
Taking the advice from this ML I moved the meta-data from internal to an
external RAID-10 partition (since writes on RAID-5 are sucky by nature) 
and got to a much more satisfying 75MB/s. This is also roughly the speed 
I see for syncs.
Since you are already on a RAID-10, no need for that. Of course if you
have additional drives not in that raid, putting the meta-data there
might help. 

In addition to that I turned on "use-bmbv" since nobody here actually
managed to give me a good reason (other than cargo-cult quoting the manual) 
not to with my particular setup (identical disks and all on Linux MD) and 
it made about a 5MB/s difference in writes...

> where i've gotten ~ 65000K/sec as opposed to ~ 75000K/sec bare metal.
> 
> Anyone have any suggestions to what parameter settings I should try in 
> order to achieve a better read performance from the drbd resource?
>
I don't think there is much more you can do here. I would reckon that
being able to crank up the buffers even more (like doubling them) might
help, but the max value of 20k for unplug-watermark puts a stop on that. 
I have not seen anybody quoting higher numbers combined with actual setups
and configs here either.

My personal conclusion is that if one wants to build a high-speed 
(writes) DRBD setup where speed is definitely more important than 
storage capacity, go for a RAID-10 (preferably the Linux MD RAID-10
with far or offset replication) with as many and as fast drives you
can fit. For extra oomph, consider a battery backed ramdisk or solid
state drive to hold the meta-data.

Regards,

Christian
-- 
Christian Balzer        Network/Systems Engineer                NOC
chibi at gol.com   	Global OnLine Japan/Fusion Network Services
http://www.gol.com/



More information about the drbd-user mailing list