Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
I'm currently testing DRBD and am having write performance problems.
On the local raid array I achieve 124MB/s, but with DRBD I get only 41MB/s
out of it, or if the secondary node is down (to rule out network issues)
then 53MB/s at best.
Tried protocol A / B / C, with and without no-disk-barriers and
no-disk-flushes, but this didn't change much (only +/- 2MB/s difference).
I'm on Debian 7.1, which ships with drbd version 8.3.11 (kernel) / 8.3.13
(userspace).
Any help would be greatly appreciated.
Setup & test results below. If you are missing anything then just let me know.
SETUP:
* two nodes (Primary [st1]/Secondary [st2])
* two-disk RAID1
* /localraid = RAID1
* /data = RAID1 + DRBD
* Both /localraid and /data are arrays using the same two harddisks
* 10GbE link between primary and secondary
### HDD/RAID LAYOUT
root at st1:/data# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdd2[1] sdc2[0] <==== /localraid
976431936 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdc1[0] sdd1[1] <==== /data (through DRBD)
976428864 blocks super 1.2 [2/2] [UU]
### LOCAL WRITE SPEED
root at st1:/localraid# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 4.23937 s, 124 MB/s
real 0m4.241s
user 0m0.004s
sys 0m0.104s
### DRBD WRITE SPEED
root at st1:/localraid# cd /data
root at st1:/data# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 13.0039 s, 40.3 MB/s
real 0m13.020s
user 0m0.000s
sys 0m0.152s
root at st1:/data# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 12.7954 s, 41.0 MB/s
real 0m12.797s
user 0m0.000s
sys 0m0.152s
### DRBD WRITE SPEED WHEN SECONDARY NODE IS OFFLINE
root at st1:/data# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 9.97073 s, 52.6 MB/s
real 0m9.972s
user 0m0.000s
sys 0m0.132s
root at st1:/data# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 9.92215 s, 52.8 MB/s
real 0m9.924s
user 0m0.000s
sys 0m0.136s
### NETWORK SPEED BETWEEN PRIMARY AND SECONDARY
# netperf -D -H 192.168.220.2
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 9898.55
# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:3492760 nr:0 dw:6188592 dr:922197 al:1385 bm:75 lo:0 pe:0 ua:0 ap:0
ep:1 wo:f oos:0
## on secondary:
# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:905608 dw:905608 dr:0 al:0 bm:59 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root at st1:~# drbdadm dump
# /etc/drbd.conf
common {
protocol C;
startup {
degr-wfc-timeout 120;
wfc-timeout 120;
}
}
# resource data on st1: not ignored, not stacked
resource data {
on st1 {
device /dev/drbd1 minor 1;
disk /dev/md2;
address ipv4 192.168.220.1:7789;
meta-disk internal;
}
on st2 {
device /dev/drbd1 minor 1;
disk /dev/md2;
address ipv4 192.168.220.2:7789;
meta-disk internal;
}
net {
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512k;
}
syncer {
rate 250M;
}
}
root at st2:/# drbdadm dump
# /etc/drbd.conf
common {
protocol C;
startup {
degr-wfc-timeout 120;
wfc-timeout 120;
}
}
# resource data on st2: not ignored, not stacked
resource data {
on st1 {
device /dev/drbd1 minor 1;
disk /dev/md2;
address ipv4 192.168.220.1:7789;
meta-disk internal;
}
on st2 {
device /dev/drbd1 minor 1;
disk /dev/md2;
address ipv4 192.168.220.2:7789;
meta-disk internal;
}
net {
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 512k;
}
syncer {
rate 250M;
}
}
# no-disk-barriers and no-disk-flushes did not change anything:
root at st1:/data# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 13.0782 s, 40.1 MB/s
real 0m13.080s
user 0m0.016s
sys 0m0.160s
### WITH SECONDARY NODE DOWN:
root at st1:/data# rm -f test; sync; time dd if=/dev/zero of=test bs=1M
count=500 conv=fdatasync oflag=direct; rm -f test
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 10.5591 s, 49.7 MB/s
real 0m10.561s
user 0m0.000s
sys 0m0.140s
Thanks again,
Bram.
--
Bram Matthys
Software developer/IT consultant syzop at vulnscan.org
Website: www.vulnscan.org
PGP key: www.vulnscan.org/pubkey.asc
PGP fp: EBCA 8977 FCA6 0AB0 6EDB 04A7 6E67 6D45 7FE1 99A6