Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi all, we at ProfitBricks do virtual data centers and we've got a customer who has built-up a DRBD 8.3.11 system inside his virtual data center. Storage is connected via InfiniBand and SCSI RDMA Protocol (SRP), presented as local storage in the VMs (VirtIO blk). Like with iSCSI and iSER such connections have trouble with 4k synchronous reads. The connection between VMs is done via VirtIO net but is also IB-based on the physical layer. The IO request size detection worked - the customer can do up to 128 KiB IO requests (not more because of the 128 KiB hashing functions) on regular data. But everything which is related to sync or resync is at only 4 KiB IO request size, verified by "blktrace". It is not a matter of the syncer rate limit. He has put it to 1000 MB/s. He can be lucky that he has 40 MB/s, because synchronous 4 KiB reads on 1 Gbit/s Ethernet with iSCSI are at only 10 MB/s. Why does DRBD syncs and resyncs in only slow 4 KiB IO requests? Btw.: The IO request size detection also doesn't work above MD RAID devices. In this configuration it always remains at 4 KiB both syncer stuff and regular data, although the connection to the other host is established. This IO request stuff is seriously broken IMHO! A regular HDD can handle IO request sizes up to 512 KiB. Why not DRBD? You limit this stuff to 4 KiB even if there is no other host. The request size detection is only performed when there is a connection to the other host and set back to 4 KiB if the connection gets lost. This kills the performance completely. What about an option to disable this IO request size detection and setting the limit directly to the limits from the devices below (use kernel 3.3 function "blk_set_stacking_limits()") ??? We've done this by patching the kernel code and it works. But seriously, a 128 KiB limit above a 512 KiB chunk default MD RAID 0 doesn't make sense. This kills all performance! So forget about using DRBD above MD RAID 0, for now! Cheers, Sebastian -- Sebastian Riemer Linux Kernel Developer ProfitBricks GmbH Greifswalder Str. 207 10405 Berlin, Germany Tel.: +49 - 30 - 60 98 56 991 - 303 Fax: +49 - 30 - 51 64 09 22 Email: sebastian.riemer at profitbricks.com Web: http://www.profitbricks.com/ Sitz der Gesellschaft: Berlin Registergericht: Amtsgericht Charlottenburg, HRB 125506 B Geschäftsführer: Andreas Gauger, Achim Weiss