[DRBD-user] Drastically Higher Disk Utilization on DRBD Node in Secondary Mode

Andrew Martin amartin at xes-inc.com
Tue Apr 30 07:16:16 CEST 2013

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hello,

I am running a 2-node active/passive DRBD cluster with DRBD running on top of LVM which itself is on top of a 10x2TB software RAID of 7200RPM SATAII HDDs. Both nodes are running Ubuntu 12.04 amd64. The DRBD replication link is a direct 10Gb SFP connection with the MTU increased to 9000. Here is a link to the DRBD configuration:
http://pastebin.com/AQuNYN5s

I am concerned in general about the poor write performance that I am seeing, however in particular I've recently noticed odd behavior where the secondary (passive) node's disk utilization is much higher than the active node (using iostat):

# iostat -dxmN 5:

on primary node:
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
storage-drives     0.00     0.00   12.40  447.80     0.05     1.78     8.17     2.51    5.46   20.26    5.05   0.54  25.04

on secondary node:
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
storage-drives     0.00     0.00    0.00  524.60     0.00     1.80     7.02     3.17    6.05    0.00    6.05   1.76  92.08

I started iostat on both nodes at approximately the same time (hence the slight variation in columns like wMB/s), however the utilization variation by around 50% between the nodes. They are both currently UpToDate so a resync cannot be the explanation for this difference. 

Write speeds on small files are very slow. Here are some tests I did with dd while adjusting various settings recommended by the DRBD Users Guide (e.g. deadline scheduler with front_merges, read_expire, and write_expire adjusted):
http://pastebin.com/64xn2ptg

Writing out 100KB files results in around 7MB/s write performance. Writing 10MB files results in better performance, around 40MB/s. What can I do to increase performance, particularly with small files?

Thanks,

Andrew



More information about the drbd-user mailing list