[DRBD-user] DRBD very slow because of 4 KiB IO request size

Sebastian Riemer sebastian.riemer at profitbricks.com
Mon Jul 9 18:04:27 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Hi all,

we at ProfitBricks do virtual data centers and we've got a customer who
has built-up a DRBD 8.3.11 system inside his virtual data center.
Storage is connected via InfiniBand and SCSI RDMA Protocol (SRP),
presented as local storage in the VMs (VirtIO blk). Like with iSCSI and
iSER such connections have trouble with 4k synchronous reads. The
connection between VMs is done via VirtIO net but is also IB-based on
the physical layer.

The IO request size detection worked - the customer can do up to 128 KiB
IO requests (not more because of the 128 KiB hashing functions) on
regular data. But everything which is related to sync or resync is at
only 4 KiB IO request size, verified by "blktrace". It is not a matter
of the syncer rate limit. He has put it to 1000 MB/s. He can be lucky
that he has 40 MB/s, because synchronous 4 KiB reads on 1 Gbit/s
Ethernet with iSCSI are at only 10 MB/s.

Why does DRBD syncs and resyncs in only slow 4 KiB IO requests?

Btw.: The IO request size detection also doesn't work above MD RAID
devices. In this configuration it always remains at 4 KiB both syncer
stuff and regular data, although the connection to the other host is
established. This IO request stuff is seriously broken IMHO! A regular
HDD can handle IO request sizes up to 512 KiB. Why not DRBD?

You limit this stuff to 4 KiB even if there is no other host. The
request size detection is only performed when there is a connection to
the other host and set back to 4 KiB if the connection gets lost. This
kills the performance completely.

What about an option to disable this IO request size detection and
setting the limit directly to the limits from the devices below (use
kernel 3.3 function "blk_set_stacking_limits()") ???

We've done this by patching the kernel code and it works. But seriously,
a 128 KiB limit above a 512 KiB chunk default MD RAID 0 doesn't make
sense. This kills all performance! So forget about using DRBD above MD
RAID 0, for now!


Sebastian Riemer
Linux Kernel Developer

ProfitBricks GmbH
Greifswalder Str. 207
10405 Berlin, Germany

Tel.:  +49 - 30 - 60 98 56 991 - 303
Fax:   +49 - 30 - 51 64 09 22
Email: sebastian.riemer at profitbricks.com
Web:   http://www.profitbricks.com/

Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Andreas Gauger, Achim Weiss

More information about the drbd-user mailing list