Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I am going currently going nuts because of slow DRBD performance and after
many hours trying different parameters I cannot find the problem here. I
hope someone is able to help.
Regardless of how I tune the parameters I only achieve between 40-80 MB/s
when writing to the drbd device from the active drbd node.
Even stranger is that I export some folder of the drbd with NFS and with
some NFS client I receive basically a 100% write speed (110 MB/s) to drbd
mounted via NFS. So writing to drbd via NFS is faster than writing to drbd
from the active drbd node?
May the problem lie in RAID 6 of the subsystems? Now I am kinda helpless.
There are probably other problems with this configuration too. However, for
now I would like to focus on getting the speed problem solved, so please
don't get too offtopic. Thank you!
Here is all information of the system:
## Disk Througput
root at stor02 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
bs=1G count=1 oflag=direct; done
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 1,32608 s, 810 MB/s
real 0m1.391s
user 0m0.000s
sys 0m0.428s
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 1,60039 s, 671 MB/s
real 0m1.678s
user 0m0.000s
sys 0m0.424s
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 1,24388 s, 863 MB/s
real 0m1.334s
user 0m0.000s
sys 0m0.448s
---
root at stor01 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
bs=1G count=1 oflag=direct; done
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 1,22048 s, 880 MB/s
real 0m1.265s
user 0m0.000s
sys 0m0.312s
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 1,2497 s, 859 MB/s
real 0m1.309s
user 0m0.000s
sys 0m0.328s
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 1,22641 s, 876 MB/s
real 0m1.290s
user 0m0.000s
sys 0m0.328s
---
## Disk Latency
root at stor01 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
bs=512 count=1000 oflag=direct; done
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,555512 s, 922 kB/s
real 0m0.557s
user 0m0.000s
sys 0m0.028s
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,463406 s, 1,1 MB/s
real 0m0.465s
user 0m0.004s
sys 0m0.028s
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,621884 s, 823 kB/s
real 0m0.623s
user 0m0.000s
sys 0m0.028s
---
t at stor02 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
bs=512 count=1000 oflag=direct; done
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,485988 s, 1,1 MB/s
real 0m0.488s
user 0m0.000s
sys 0m0.036s
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,560012 s, 914 kB/s
real 0m0.562s
user 0m0.000s
sys 0m0.032s
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,542077 s, 945 kB/s
real 0m0.544s
user 0m0.000s
sys 0m0.032s
---
## Network Throughput
Tests with iperf showed almost 10 Gbps.
Both DRBD machines are sharing a dedicated 10 Gbps cable with MTU 9192.
root at stor02 ~ $ ifconfig eth2
eth2 Link encap:Ethernet HWaddr 90:e2:ba:48:2c:dc
inet addr:10.0.0.2 Bcast:10.0.0.3 Mask:255.255.255.252
inet6 addr: fe80::92e2:baff:fe48:2cdc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9192 Metric:1
RX packets:1481638371 errors:0 dropped:0 overruns:0 frame:0
TX packets:988570639 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10338057837352 (9.4 TiB) TX bytes:3222573497308
(2.9 TiB)
## Network Latency
root at stor02 ~ $ fping -l 10.0.0.1
10.0.0.1 : [0], 96 bytes, 0.12 ms (0.12 avg, 0% loss)
10.0.0.1 : [1], 96 bytes, 0.13 ms (0.12 avg, 0% loss)
10.0.0.1 : [2], 96 bytes, 0.12 ms (0.12 avg, 0% loss)
# DRBD Version and Status
root at stor02 ~ $ cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:17378172 nr:0 dw:177485633 dr:1946127117 al:3708575 bm:557 lo:0
pe:0 ua:0 ap:0 ep:1 wo:f oos:0
# DRBD Config
root at stor02 ~ $ drbdsetup /dev/drbd0 show
disk {
size 0s _is_default; # bytes
on-io-error detach;
fencing dont-care _is_default;
max-bio-bvecs 0 _is_default;
}
net {
timeout 60 _is_default; # 1/10 seconds
max-epoch-size 19997;
max-buffers 19997;
unplug-watermark 64;
connect-int 10 _is_default; # seconds
ping-int 10 _is_default; # seconds
sndbuf-size 524288; # bytes
rcvbuf-size 0 _is_default; # bytes
ko-count 0 _is_default;
cram-hmac-alg "sha1";
shared-secret "...";
after-sb-0pri disconnect _is_default;
after-sb-1pri disconnect _is_default;
after-sb-2pri disconnect _is_default;
rr-conflict disconnect _is_default;
ping-timeout 5 _is_default; # 1/10 seconds
on-congestion block _is_default;
congestion-fill 0s _is_default; # byte
congestion-extents 127 _is_default;
}
syncer {
rate 2097152k; # bytes/second
after -1 _is_default;
al-extents 3833;
on-no-data-accessible io-error _is_default;
c-plan-ahead 0 _is_default; # 1/10 seconds
c-delay-target 10 _is_default; # 1/10 seconds
c-fill-target 0s _is_default; # bytes
c-max-rate 102400k _is_default; # bytes/second
c-min-rate 4096k _is_default; # bytes/second
}
protocol C;
_this_host {
device minor 0;
disk "/dev/vgRAPTOR/lvol0";
meta-disk internal;
address ipv4 10.0.0.2:7788;
}
_remote_host {
address ipv4 10.0.0.1:7788;
}
# Hardware Raid
root at stor02 ~ $ arcconf getconfig 1 ld
Controllers found: 1
----------------------------------------------------------------------
Logical device information
----------------------------------------------------------------------
Logical device number 0
Logical device name : SYSTEM
RAID level : 6 Reed-Solomon
Status of logical device : Optimal
Size : 51188 MB
Stripe-unit size : 256 KB
Read-cache setting : Enabled
Read-cache status : On
Write-cache setting : Enabled
Write-cache status : Off
Partitioned : Yes
Protected by Hot-Spare : No
Bootable : Yes
Failed stripes : No
Power settings : Disabled
--------------------------------------------------------
Logical device segment information
--------------------------------------------------------
Segment 0 : Present
(Controller:1,Enclosure:0,Slot:0) WD-WXH1...
Segment 1 : Present
(Controller:1,Enclosure:0,Slot:1) WD-WXD1...
Segment 2 : Present
(Controller:1,Enclosure:0,Slot:2) WD-WXD1...
Segment 3 : Present
(Controller:1,Enclosure:0,Slot:3) WD-WXH1...
Segment 4 : Present
(Controller:1,Enclosure:0,Slot:4) WD-WXJ1...
Segment 5 : Present
(Controller:1,Enclosure:0,Slot:5) WD-WXD1...
Segment 6 : Present
(Controller:1,Enclosure:0,Slot:6) WD-WXH1...
Segment 7 : Present
(Controller:1,Enclosure:0,Slot:7) WD-WXH1...
Logical device number 1
Logical device name : RAPTOR
RAID level : 6 Reed-Solomon
Status of logical device : Optimal
Size : 5668850 MB
Stripe-unit size : 256 KB
Read-cache setting : Enabled
Read-cache status : On
Write-cache setting : Enabled
Write-cache status : Off (Does this mean
write cache is off?)
Partitioned : No
Protected by Hot-Spare : No
Bootable : No
Failed stripes : No
Power settings : Disabled
--------------------------------------------------------
Logical device segment information
--------------------------------------------------------
Segment 0 : Present
(Controller:1,Enclosure:0,Slot:0) WD-WXH1...
Segment 1 : Present
(Controller:1,Enclosure:0,Slot:1) WD-WXD1...
Segment 2 : Present
(Controller:1,Enclosure:0,Slot:2) WD-WXD1...
Segment 3 : Present
(Controller:1,Enclosure:0,Slot:3) WD-WXH1...
Segment 4 : Present
(Controller:1,Enclosure:0,Slot:4) WD-WXJ1...
Segment 5 : Present
(Controller:1,Enclosure:0,Slot:5) WD-WXD1...
Segment 6 : Present
(Controller:1,Enclosure:0,Slot:6) WD-WXH1...
Segment 7 : Present
(Controller:1,Enclosure:0,Slot:7) WD-WXH1...
# Results
## Throughput
root at stor02 /mnt/drbd_r0 $ for i in $(seq 3); do time dd if=/dev/zero
of=testfile bs=1G count=1 oflag=direct; done
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 24,4773 s, 43,9 MB/s
real 0m24.540s
user 0m0.000s
sys 0m0.492s
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 25,4277 s, 42,2 MB/s
real 0m25.514s
user 0m0.000s
sys 0m0.504s
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB) copied, 24,7216 s, 43,4 MB/s
real 0m24.795s
user 0m0.000s
sys 0m0.508s
## Latency
root at stor02 /mnt/drbd_r0 $ for i in $(seq 3); do time dd if=/dev/zero
of=testfile bs=512 count=1000 oflag=direct; done
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 1,13063 s, 453 kB/s
real 0m1.133s
user 0m0.000s
sys 0m0.044s
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 1,18928 s, 431 kB/s
real 0m1.191s
user 0m0.004s
sys 0m0.040s
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0,763934 s, 670 kB/s
real 0m0.766s
user 0m0.000s
sys 0m0.048s
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20150630/b7ea7b69/attachment.htm>