Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Duane Cox wrote: >> OK, I haven't been paying attention which means this is probably a >> stupid question, but is there a reason you're using the raw device >> instead of the drbd device when drbd is loaded? > > Is it bad to test write performace to the raw device when drbd is loaded? > at this point I have no valid data, so I don't care about anything > destructive... > A basic rule with DRBD: _NEVER_ access (read or write) the underlying block device while drbd is supposed to be in control of it. A general rule is that you should not use the underlying block device that DRBD has been setup for after you setup DRBD, with out all traffic passing through DRBD. The exception to this rule is when you are in full disaster recovery, i.e., the drive has physically been moved to another machine and you NEED the data off of it because some one put .45 rounds through the original machines. to test DRBD speed you should be running your test tool against /dev/drbdX for the device. Also you should be aware that by writing to the lower level device, if you wrote enough data you may have munged DRBD's meta-data, but most certainly have munged DRBD's understanding of what data needs synced between the machines when they reconnect. > I was trying to determine where/when/how the problem (bottleneck) is > happening. > I have tried various tests, reading archived posts, and this kinda stuck out > as something peculiar and suspecious. > > (drbd loaded in disconnected state) > initially I get ~70 MB/s write througput to /dev/sdb > eventually I get ~30 MB/s write througput to /dev/sdb > possible even gets worse, I just stopped testing at this point. > Then I STOP/UNLOAD drbd > now I get ~70 MB/s write throughpout to /dev/sdb again. > > >> ...and does the drbd device show the same degradation? > > Yes and no, meaning it's not the same... it starts out at a lower MB/s and > gets even worse. > > Assumptions: A) the BM tool is issuing an appropriate Sync command, so their is no extra data hanging around in the block device's buffer. B) you DON'T get the SAME/SIMILAR degradation when drbd is not running during the tests. C) nothing else is happening on the machine at the time. (if I read the data correctly your runs take < 2 minutes, if you start backing up applications like X and sendmail you can see big effects in that short a period, and they can take a while to clear.) D) the BM tool writes to the same physical location on the disk each time. E) the disk is not doing any dynamic relocation of the data (ala Disk On a Chip) F) your test set is big enough that a) the kernel block drivers are not buffering a significant amount b) the physical drive is not buffering a significant amount, i.e., try ~10-20GB. G) your test set does not create a (big) memory buffer itself to then dump to disk, i.e, cause other apps to swap out and then swap back in while you are running the later tests. Tis a bit strange (that it should go down when writing to the same physical location on the disk). -- Todd Denniston Crane Division, Naval Surface Warfare Center (NSWC Crane) Harnessing the Power of Technology for the Warfighter