Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi there, I'm doing some testing with Ganeti, a cluster-based VM manager, which uses DRBD as the default syncing system between nodes. My test cluster has two nodes, Alpha & Beta. Ganeti provides a way to migrate from one to another, handling DRBD automatically to sync to the node that isn't running the VM. I'm running into an issue that I'm having difficultly understanding. When benchmarking from within the VM, I'm getting drastically different results depending on the node that the VM is running from. The DRBD benchmarks are below; Alpha; Sequential Read: 3760.788 MB/s Sequential Write: 13.102 MB/s Random Read 512KB: 2564.699 MB/s Random Write 512KB: 24.064 MB/s Random Read 4KB (QD=1): 46.476 MB/s [11346.8 IOPS] Random Write 4KB (QD=1): 3.397 MB/s [829.4 IOPS] Random Read 4KB (QD=32): 206.652 MB/s [50452.2 IOPS] Random Write 4KB (QD=32): 5.617 MB/s [1371.4 IOPS] Beta; Sequential Read: 3211.824 MB/s Sequential Write: 342.400 MB/s Random Read 512KB: 2111.121 MB/s Random Write 512KB: 206.603 MB/s Random Read 4KB (QD=1): 32.109 MB/s [7839.2 IOPS] Random Write 4KB (QD=1): 3.858 MB/s [941.8 IOPS] Random Read 4KB (QD=32): 174.499 MB/s [42602.4 IOPS] Random Write 4KB (QD=32): 4.508 MB/s [1100.6 IOPS] For comparison, I've done benchmarks with DRBD turned off. These are below; Alpha; Sequential Read: 3720.154 MB/s Sequential Write: 206.250 MB/s Random Read 512KB: 2621.003 MB/s Random Write 512KB: 237.334 MB/s Random Read 4KB (QD=1): 46.905 MB/s [11451.4 IOPS] Random Write 4KB (QD=1): 5.571 MB/s [1360.0 IOPS] Random Read 4KB (QD=32): 204.227 MB/s [49860.0 IOPS] Random Write 4KB (QD=32): 9.771 MB/s [2385.4 IOPS] Beta; Sequential Read: 3097.205 MB/s Sequential Write: 207.551 MB/ Random Read 512KB: 2030.797 MB/s Random Write 512KB: 212.068 MB/s Random Read 4KB (QD=1): 28.565 MB/s [6973.8 IOPS] Random Write 4KB (QD=1): 3.437 MB/s [839.2 IOPS] Random Read 4KB (QD=32): 179.701 MB/s [43872.4 IOPS] Random Write 4KB (QD=32): 12.433 MB/s [3035.4 IOPS] I'm getting great performance when running DRBD from Beta, somehow even better than native performance for sequential writes, but the opposite direction is poor. I can't understand why this would be. I would have expected them to match. The VM used is Windows 2008 server, using CrystalDiskMark as the benchmark utility with the default settings. It's using VirtIO v1.0 drivers. The two nodes are IBM System x3500 M4 servers. Both are running an up to date Debian Jessie with DRBD having a dedicated network interface which is made up of three gigabit Ethernet NICs bonded together in 802.3ad mode. All of its NICs are in use and they're going directly to each server, no switches in between. The only difference between the two is their storage setup. For where the VM resides, Alpha is using a ServeRAID M1115 hardware RAID using two disks (I believe in a RAID-0 configuration, but can't confirm at present as I don't have physical access to the server), while Beta has a ServeRAID M5110 hardware RAID using three disks in RAID-5. >From /proc/drbd, it's version is; version: 8.4.3 (api:1/proto:86-101) srcversion: 1A9F77B1CA5FF92235C2213, and according to it, it's syncing fine during these benchmarks. Ganeti doesn't seem to expose its DRBD configuration file, at least not in /etc, otherwise I would have included it. If you need any further information, please let me know, and thanks to any replies.