Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Tue, Jun 30, 2015 at 10:53:02PM +0200, Leon Kramer wrote:
> I am going currently going nuts because of slow DRBD performance and after
> many hours trying different parameters I cannot find the problem here. I
> hope someone is able to help.
>
> Regardless of how I tune the parameters I only achieve between 40-80 MB/s
> when writing to the drbd device from the active drbd node.
>
> Even stranger is that I export some folder of the drbd with NFS and with
> some NFS client I receive basically a 100% write speed (110 MB/s) to drbd
> mounted via NFS. So writing to drbd via NFS is faster than writing to drbd
> from the active drbd node?
>
> May the problem lie in RAID 6 of the subsystems? Now I am kinda helpless.
It is a *very* good idea to upgrade to 8.4.
It is a good idea to double check your controller write cache settings,
make sure it is "crash-safe" (has a healthy BBU, or some flash +
supercap or whatever your controller uses).
It may be a good idea to put DRBD meta data on RAID 1 instead.
And *then* we can look into tuning DRBD ;-)
> There are probably other problems with this configuration too. However, for
> now I would like to focus on getting the speed problem solved, so please
> don't get too offtopic. Thank you!
>
>
> Here is all information of the system:
>
> ## Disk Througput
>
> root at stor02 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
> bs=1G count=1 oflag=direct; done
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,32608 s, 810 MB/s
>
> real 0m1.391s
> user 0m0.000s
> sys 0m0.428s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,60039 s, 671 MB/s
>
> real 0m1.678s
> user 0m0.000s
> sys 0m0.424s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,24388 s, 863 MB/s
>
> real 0m1.334s
> user 0m0.000s
> sys 0m0.448s
>
> ---
>
> root at stor01 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
> bs=1G count=1 oflag=direct; done
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,22048 s, 880 MB/s
>
> real 0m1.265s
> user 0m0.000s
> sys 0m0.312s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,2497 s, 859 MB/s
>
> real 0m1.309s
> user 0m0.000s
> sys 0m0.328s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,22641 s, 876 MB/s
>
> real 0m1.290s
> user 0m0.000s
> sys 0m0.328s
>
> ---
>
>
>
>
>
> ## Disk Latency
>
> root at stor01 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
> bs=512 count=1000 oflag=direct; done
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,555512 s, 922 kB/s
>
> real 0m0.557s
> user 0m0.000s
> sys 0m0.028s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,463406 s, 1,1 MB/s
>
> real 0m0.465s
> user 0m0.004s
> sys 0m0.028s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,621884 s, 823 kB/s
>
> real 0m0.623s
> user 0m0.000s
> sys 0m0.028s
>
> ---
>
>
> t at stor02 ~ $ for i in $(seq 3); do time dd if=/dev/zero of=testfile
> bs=512 count=1000 oflag=direct; done
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,485988 s, 1,1 MB/s
>
> real 0m0.488s
> user 0m0.000s
> sys 0m0.036s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,560012 s, 914 kB/s
>
> real 0m0.562s
> user 0m0.000s
> sys 0m0.032s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,542077 s, 945 kB/s
>
> real 0m0.544s
> user 0m0.000s
> sys 0m0.032s
>
> ---
>
>
>
> ## Network Throughput
>
> Tests with iperf showed almost 10 Gbps.
> Both DRBD machines are sharing a dedicated 10 Gbps cable with MTU 9192.
>
> root at stor02 ~ $ ifconfig eth2
> eth2 Link encap:Ethernet HWaddr 90:e2:ba:48:2c:dc
> inet addr:10.0.0.2 Bcast:10.0.0.3 Mask:255.255.255.252
> inet6 addr: fe80::92e2:baff:fe48:2cdc/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:9192 Metric:1
> RX packets:1481638371 errors:0 dropped:0 overruns:0 frame:0
> TX packets:988570639 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:10338057837352 (9.4 TiB) TX bytes:3222573497308
> (2.9 TiB)
>
>
> ## Network Latency
>
> root at stor02 ~ $ fping -l 10.0.0.1
> 10.0.0.1 : [0], 96 bytes, 0.12 ms (0.12 avg, 0% loss)
> 10.0.0.1 : [1], 96 bytes, 0.13 ms (0.12 avg, 0% loss)
> 10.0.0.1 : [2], 96 bytes, 0.12 ms (0.12 avg, 0% loss)
>
> # DRBD Version and Status
>
> root at stor02 ~ $ cat /proc/drbd
> version: 8.3.11 (api:88/proto:86-96)
> srcversion: F937DCB2E5D83C6CCE4A6C9
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
> ns:17378172 nr:0 dw:177485633 dr:1946127117 al:3708575 bm:557 lo:0
> pe:0 ua:0 ap:0 ep:1 wo:f oos:0
>
>
>
> # DRBD Config
>
> root at stor02 ~ $ drbdsetup /dev/drbd0 show
> disk {
> size 0s _is_default; # bytes
> on-io-error detach;
> fencing dont-care _is_default;
> max-bio-bvecs 0 _is_default;
> }
> net {
> timeout 60 _is_default; # 1/10 seconds
> max-epoch-size 19997;
> max-buffers 19997;
> unplug-watermark 64;
> connect-int 10 _is_default; # seconds
> ping-int 10 _is_default; # seconds
> sndbuf-size 524288; # bytes
> rcvbuf-size 0 _is_default; # bytes
> ko-count 0 _is_default;
> cram-hmac-alg "sha1";
> shared-secret "...";
> after-sb-0pri disconnect _is_default;
> after-sb-1pri disconnect _is_default;
> after-sb-2pri disconnect _is_default;
> rr-conflict disconnect _is_default;
> ping-timeout 5 _is_default; # 1/10 seconds
> on-congestion block _is_default;
> congestion-fill 0s _is_default; # byte
> congestion-extents 127 _is_default;
> }
> syncer {
> rate 2097152k; # bytes/second
> after -1 _is_default;
> al-extents 3833;
> on-no-data-accessible io-error _is_default;
> c-plan-ahead 0 _is_default; # 1/10 seconds
> c-delay-target 10 _is_default; # 1/10 seconds
> c-fill-target 0s _is_default; # bytes
> c-max-rate 102400k _is_default; # bytes/second
> c-min-rate 4096k _is_default; # bytes/second
> }
> protocol C;
> _this_host {
> device minor 0;
> disk "/dev/vgRAPTOR/lvol0";
> meta-disk internal;
> address ipv4 10.0.0.2:7788;
> }
> _remote_host {
> address ipv4 10.0.0.1:7788;
> }
>
>
> # Hardware Raid
>
> root at stor02 ~ $ arcconf getconfig 1 ld
> Controllers found: 1
> ----------------------------------------------------------------------
> Logical device information
> ----------------------------------------------------------------------
> Logical device number 0
> Logical device name : SYSTEM
> RAID level : 6 Reed-Solomon
> Status of logical device : Optimal
> Size : 51188 MB
> Stripe-unit size : 256 KB
> Read-cache setting : Enabled
> Read-cache status : On
> Write-cache setting : Enabled
> Write-cache status : Off
> Partitioned : Yes
> Protected by Hot-Spare : No
> Bootable : Yes
> Failed stripes : No
> Power settings : Disabled
> --------------------------------------------------------
> Logical device segment information
> --------------------------------------------------------
> Segment 0 : Present
> (Controller:1,Enclosure:0,Slot:0) WD-WXH1...
> Segment 1 : Present
> (Controller:1,Enclosure:0,Slot:1) WD-WXD1...
> Segment 2 : Present
> (Controller:1,Enclosure:0,Slot:2) WD-WXD1...
> Segment 3 : Present
> (Controller:1,Enclosure:0,Slot:3) WD-WXH1...
> Segment 4 : Present
> (Controller:1,Enclosure:0,Slot:4) WD-WXJ1...
> Segment 5 : Present
> (Controller:1,Enclosure:0,Slot:5) WD-WXD1...
> Segment 6 : Present
> (Controller:1,Enclosure:0,Slot:6) WD-WXH1...
> Segment 7 : Present
> (Controller:1,Enclosure:0,Slot:7) WD-WXH1...
>
>
> Logical device number 1
>
> Logical device name : RAPTOR
>
> RAID level : 6 Reed-Solomon
>
> Status of logical device : Optimal
>
> Size : 5668850 MB
>
> Stripe-unit size : 256 KB
>
> Read-cache setting : Enabled
>
> Read-cache status : On
>
> Write-cache setting : Enabled
>
> Write-cache status : Off (Does this mean
> write cache is off?)
> Partitioned : No
>
> Protected by Hot-Spare : No
>
> Bootable : No
>
> Failed stripes : No
>
> Power settings : Disabled
>
> --------------------------------------------------------
>
> Logical device segment information
>
> --------------------------------------------------------
>
> Segment 0 : Present
> (Controller:1,Enclosure:0,Slot:0) WD-WXH1...
> Segment 1 : Present
> (Controller:1,Enclosure:0,Slot:1) WD-WXD1...
> Segment 2 : Present
> (Controller:1,Enclosure:0,Slot:2) WD-WXD1...
> Segment 3 : Present
> (Controller:1,Enclosure:0,Slot:3) WD-WXH1...
> Segment 4 : Present
> (Controller:1,Enclosure:0,Slot:4) WD-WXJ1...
> Segment 5 : Present
> (Controller:1,Enclosure:0,Slot:5) WD-WXD1...
> Segment 6 : Present
> (Controller:1,Enclosure:0,Slot:6) WD-WXH1...
> Segment 7 : Present
> (Controller:1,Enclosure:0,Slot:7) WD-WXH1...
>
>
> # Results
> ## Throughput
> root at stor02 /mnt/drbd_r0 $ for i in $(seq 3); do time dd if=/dev/zero
> of=testfile bs=1G count=1 oflag=direct; done
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 24,4773 s, 43,9 MB/s
>
> real 0m24.540s
> user 0m0.000s
> sys 0m0.492s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 25,4277 s, 42,2 MB/s
>
> real 0m25.514s
> user 0m0.000s
> sys 0m0.504s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 24,7216 s, 43,4 MB/s
>
> real 0m24.795s
> user 0m0.000s
> sys 0m0.508s
>
>
> ## Latency
> root at stor02 /mnt/drbd_r0 $ for i in $(seq 3); do time dd if=/dev/zero
> of=testfile bs=512 count=1000 oflag=direct; done
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1,13063 s, 453 kB/s
>
> real 0m1.133s
> user 0m0.000s
> sys 0m0.044s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1,18928 s, 431 kB/s
>
> real 0m1.191s
> user 0m0.004s
> sys 0m0.040s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,763934 s, 670 kB/s
>
> real 0m0.766s
> user 0m0.000s
> sys 0m0.048s
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
--
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA and Pacemaker support and consulting
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed