[DRBD-user] Unusual Performance. Faster one way.

Richard Walmsley rich.sbs at gmail.com
Mon Apr 27 06:52:52 CEST 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Hi there,

I'm doing some testing with Ganeti, a cluster-based VM manager, which
uses DRBD as the default syncing system between nodes. My test cluster
has two nodes, Alpha & Beta. Ganeti provides a way to migrate from one
to another, handling DRBD automatically to sync to the node that isn't
running the VM.

I'm running into an issue that I'm having difficultly understanding.
When benchmarking from within the VM, I'm getting drastically
different results depending on the node that the VM is running from.

The DRBD benchmarks are below;
Sequential Read: 3760.788 MB/s
Sequential Write: 13.102 MB/s
Random Read 512KB: 2564.699 MB/s
Random Write 512KB: 24.064 MB/s
Random Read 4KB (QD=1): 46.476 MB/s [11346.8 IOPS]
Random Write 4KB (QD=1): 3.397 MB/s [829.4 IOPS]
Random Read 4KB (QD=32): 206.652 MB/s [50452.2 IOPS]
Random Write 4KB (QD=32): 5.617 MB/s [1371.4 IOPS]

Sequential Read: 3211.824 MB/s
Sequential Write: 342.400 MB/s
Random Read 512KB: 2111.121 MB/s
Random Write 512KB: 206.603 MB/s
Random Read 4KB (QD=1): 32.109 MB/s [7839.2 IOPS]
Random Write 4KB (QD=1): 3.858 MB/s [941.8 IOPS]
Random Read 4KB (QD=32): 174.499 MB/s [42602.4 IOPS]
Random Write 4KB (QD=32): 4.508 MB/s [1100.6 IOPS]

For comparison, I've done benchmarks with DRBD turned off. These are below;
Sequential Read: 3720.154 MB/s
Sequential Write: 206.250 MB/s
Random Read 512KB: 2621.003 MB/s
Random Write 512KB: 237.334 MB/s
Random Read 4KB (QD=1): 46.905 MB/s [11451.4 IOPS]
Random Write 4KB (QD=1): 5.571 MB/s [1360.0 IOPS]
Random Read 4KB (QD=32): 204.227 MB/s [49860.0 IOPS]
Random Write 4KB (QD=32): 9.771 MB/s [2385.4 IOPS]

Sequential Read: 3097.205 MB/s
Sequential Write: 207.551 MB/
Random Read 512KB: 2030.797 MB/s
Random Write 512KB: 212.068 MB/s
Random Read 4KB (QD=1): 28.565 MB/s [6973.8 IOPS]
Random Write 4KB (QD=1): 3.437 MB/s [839.2 IOPS]
Random Read 4KB (QD=32): 179.701 MB/s [43872.4 IOPS]
Random Write 4KB (QD=32): 12.433 MB/s [3035.4 IOPS]

I'm getting great performance when running DRBD from Beta, somehow
even better than native performance for sequential writes, but the
opposite direction is poor. I can't understand why this would be. I
would have expected them to match.

The VM used is Windows 2008 server, using CrystalDiskMark as the
benchmark utility with the default settings. It's using VirtIO v1.0

The two nodes are IBM System x3500 M4 servers. Both are running an up
to date Debian Jessie with DRBD having a dedicated network interface
which is made up of three gigabit Ethernet NICs bonded together in
802.3ad mode. All of its NICs are in use and they're going directly to
each server, no switches in between.

The only difference between the two is their storage setup. For where
the VM resides, Alpha is using a ServeRAID M1115 hardware RAID using
two disks (I believe in a RAID-0 configuration, but can't confirm at
present as I don't have physical access to the server), while Beta has
a ServeRAID M5110 hardware RAID using three disks in RAID-5.

>From /proc/drbd, it's version is; version: 8.4.3 (api:1/proto:86-101)
srcversion: 1A9F77B1CA5FF92235C2213, and according to it, it's syncing
fine during these benchmarks. Ganeti doesn't seem to expose its DRBD
configuration file, at least not in /etc, otherwise I would have
included it.

If you need any further information, please let me know, and thanks to
any replies.

More information about the drbd-user mailing list