[DRBD-user] degraded write performace

Todd Denniston Todd.Denniston at ssa.crane.navy.mil
Thu Mar 29 21:20:41 CEST 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Duane Cox wrote:
>> OK, I haven't been paying attention which means this is probably a
>> stupid question, but is there a reason you're using the raw device
>> instead of the drbd device when drbd is loaded?
> Is it bad to test write performace to the raw device when drbd is loaded?
> at this point I have no valid data, so I don't care about anything
> destructive...

A basic rule with DRBD:
_NEVER_ access (read or write) the underlying block device while drbd is 
supposed to be in control of it.

A general rule is that you should not use the underlying block device that 
DRBD has been setup for after you setup DRBD, with out all traffic passing 
through DRBD.  The exception to this rule is when you are in full disaster 
recovery, i.e., the drive has physically been moved to another machine and you 
NEED  the data off of it because some one put .45 rounds through the original 

to test DRBD speed you should be running your test tool against /dev/drbdX for 
the device. Also you should be aware that by writing to the lower level 
device, if you wrote enough data you may have munged DRBD's meta-data, but 
most certainly have munged DRBD's understanding of what data needs synced 
between the machines when they reconnect.

> I was trying to determine where/when/how the problem (bottleneck) is
> happening.
> I have tried various tests, reading archived posts, and this kinda stuck out
> as something peculiar and suspecious.
> (drbd loaded in disconnected state)
> initially I get ~70 MB/s write througput to /dev/sdb
> eventually I get ~30 MB/s write througput to /dev/sdb
> possible even gets worse, I just stopped testing at this point.
> Then I STOP/UNLOAD drbd
> now I get ~70 MB/s write throughpout to /dev/sdb again.
>> ...and does the drbd device show the same degradation?
> Yes and no, meaning it's not the same... it starts out at a lower MB/s and
> gets even worse.

A) the BM tool is issuing an appropriate Sync command, so their is no extra 
data hanging around in the block device's buffer.
B) you DON'T get the SAME/SIMILAR degradation when drbd is not running during 
the tests.
C) nothing else is happening on the machine at the time.
	(if I read the data correctly your runs
	take < 2 minutes, if you start backing up
	applications like X and sendmail you can
	see big effects in that short a period,
	and they can take a while to clear.)
D) the BM tool writes to the same physical location on the disk each time.
E) the disk is not doing any dynamic relocation of the data (ala Disk On a Chip)
F) your test set is big enough that a) the kernel block drivers are not 
buffering a significant amount b) the physical drive is not buffering a 
significant amount, i.e., try ~10-20GB.
G) your test set does not create a (big) memory buffer itself to then dump to 
disk, i.e, cause other apps to swap out and then swap back in while you are 
running the later tests.

Tis a bit strange (that it should go down when writing to the same physical 
location on the disk).

Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter

More information about the drbd-user mailing list