[DRBD-user] High io-wait

Arnold Krille arnold at arnoldarts.de
Fri Aug 17 23:16:59 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Thu, 16 Aug 2012 09:47:25 -0400
Phil Frost <phil at macprofessionals.com> wrote:
> On 08/15/2012 10:26 PM, Dennis Jacobfeuerborn wrote:
> > On 08/15/2012 03:54 AM, Phil Frost wrote:
> >> I'd guess (and this is
> >> just a guess, I've never examined DRBD internals beyond what's in
> >> the manual) that the unusually high %util is due to the activity
> >> log [1] or perhaps some other housekeeping function.
> >> [1] http://www.drbd.org/users-guide/s-activity-log.html
> > Thanks for the detailed explanation. I'm wondering though why
> > something like this wouldn't be common knowledge and/or explained
> > in the FAQ if this is a generic symptom of DRBD. Tomorrow I'm going
> > to do some performance tests to see if this is a real problem or
> > just a phantom issue.
> It would probably be very educational to set up a DRBD device with 
> external metadata, and then examine the disk activity during your 
> benchmark with blktrace.

I did that: I didn't watch blktrace, but I compared various
configurations for vm-images (non-cloudy applications). I haven't
decided on how to make the results public, but some things to share:

I used a virtual machine with minimal debian, gave the test-sample-disk
as vdb, formated it with ext3, mounted it and ran "dbench" (*) with
various sync-options and with 2^n clients with n ranging from 0 to 6 or
7. Two runs each, with sync and buffer-flushes in the vm in between each
The different disks and images I gave to it had our traditional
disk-image on a nfs-share on a different machine, but also a local
drbd-volume with replication A and C, with single Gigabit-link and with
a bonded Gigabit-link and with internal and external meta-data.

The result you might want to hear:
The improvement from single-gigabit to bonded dual gigabit wasn't
really visible, which lead us to suspect (and prove) that the disk is
limiting the rates by seeking on the disk between writing data and
meta-data. (IO-Utilization of atop also reported figures near 80% for
the single disk.)
Using external meta-data improved the performance a lot! We are now
thinking of using several normal hdd per server for the normal-usage
drbd volumes and the server os. And add one or two ssd for the
meta-data and some important machines (database-server for example)
that need fast seek but not real big space.

But the next steps are to be realized only after our holidays and if
time permits.

Have a nice weekend,


(*) dbench is a nice test for us as we and our clients aren't
interested in reading/writing sequentially in some big files, but in
reading/writing randomly in many, many small files. Normal office
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20120817/a6c5dfa4/attachment.pgp>

More information about the drbd-user mailing list