Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Not sure if this is your bottleneck, but if your node interconnect link is 10Gbps then your c-max-rate number is low by a factor of 10. (See https://drbd.linbit.com/users-guide/s-configure-sync-rate.html). YMMV, but we found the following resource settings work well for a low latency 10Gbps primary <-----> secondary link:
disk {
c-plan-ahead 1;
c-max-rate 1000M;
c-fill-target 1M;
read-balancing least-pending;
}
Side note, I believe read-balancing is an 8.4 feature. You should upgrade! =) Good luck!
-OLW
-----Original Message-----
From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Lars Ellenberg
Sent: Thursday, August 06, 2015 12:21 PM
To: drbd-user at lists.linbit.com
Subject: Re: [DRBD-user] Bad DRBD Performance
On Tue, Jun 30, 2015 at 10:53:02PM +0200, Leon Kramer wrote:
> I am going currently going nuts because of slow DRBD performance and
> after many hours trying different parameters I cannot find the problem
> here. I hope someone is able to help.
>
> Regardless of how I tune the parameters I only achieve between 40-80
> MB/s when writing to the drbd device from the active drbd node.
>
> Even stranger is that I export some folder of the drbd with NFS and
> with some NFS client I receive basically a 100% write speed (110 MB/s)
> to drbd mounted via NFS. So writing to drbd via NFS is faster than
> writing to drbd from the active drbd node?
>
> May the problem lie in RAID 6 of the subsystems? Now I am kinda helpless.
It is a *very* good idea to upgrade to 8.4.
It is a good idea to double check your controller write cache settings, make sure it is "crash-safe" (has a healthy BBU, or some flash + supercap or whatever your controller uses).
It may be a good idea to put DRBD meta data on RAID 1 instead.
And *then* we can look into tuning DRBD ;-)
> There are probably other problems with this configuration too.
> However, for now I would like to focus on getting the speed problem
> solved, so please don't get too offtopic. Thank you!
>
>
> Here is all information of the system:
>
> ## Disk Througput
>
> root at stor02 ~ $ for i in $(seq 3); do time dd if=/dev/zero
> of=testfile bs=1G count=1 oflag=direct; done
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,32608 s, 810 MB/s
>
> real 0m1.391s
> user 0m0.000s
> sys 0m0.428s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,60039 s, 671 MB/s
>
> real 0m1.678s
> user 0m0.000s
> sys 0m0.424s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,24388 s, 863 MB/s
>
> real 0m1.334s
> user 0m0.000s
> sys 0m0.448s
>
> ---
>
> root at stor01 ~ $ for i in $(seq 3); do time dd if=/dev/zero
> of=testfile bs=1G count=1 oflag=direct; done
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,22048 s, 880 MB/s
>
> real 0m1.265s
> user 0m0.000s
> sys 0m0.312s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,2497 s, 859 MB/s
>
> real 0m1.309s
> user 0m0.000s
> sys 0m0.328s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 1,22641 s, 876 MB/s
>
> real 0m1.290s
> user 0m0.000s
> sys 0m0.328s
>
> ---
>
>
>
>
>
> ## Disk Latency
>
> root at stor01 ~ $ for i in $(seq 3); do time dd if=/dev/zero
> of=testfile
> bs=512 count=1000 oflag=direct; done
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,555512 s, 922 kB/s
>
> real 0m0.557s
> user 0m0.000s
> sys 0m0.028s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,463406 s, 1,1 MB/s
>
> real 0m0.465s
> user 0m0.004s
> sys 0m0.028s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,621884 s, 823 kB/s
>
> real 0m0.623s
> user 0m0.000s
> sys 0m0.028s
>
> ---
>
>
> t at stor02 ~ $ for i in $(seq 3); do time dd if=/dev/zero
> of=testfile
> bs=512 count=1000 oflag=direct; done
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,485988 s, 1,1 MB/s
>
> real 0m0.488s
> user 0m0.000s
> sys 0m0.036s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,560012 s, 914 kB/s
>
> real 0m0.562s
> user 0m0.000s
> sys 0m0.032s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,542077 s, 945 kB/s
>
> real 0m0.544s
> user 0m0.000s
> sys 0m0.032s
>
> ---
>
>
>
> ## Network Throughput
>
> Tests with iperf showed almost 10 Gbps.
> Both DRBD machines are sharing a dedicated 10 Gbps cable with MTU 9192.
>
> root at stor02 ~ $ ifconfig eth2
> eth2 Link encap:Ethernet HWaddr 90:e2:ba:48:2c:dc
> inet addr:10.0.0.2 Bcast:10.0.0.3 Mask:255.255.255.252
> inet6 addr: fe80::92e2:baff:fe48:2cdc/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:9192 Metric:1
> RX packets:1481638371 errors:0 dropped:0 overruns:0 frame:0
> TX packets:988570639 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:10338057837352 (9.4 TiB) TX
> bytes:3222573497308
> (2.9 TiB)
>
>
> ## Network Latency
>
> root at stor02 ~ $ fping -l 10.0.0.1
> 10.0.0.1 : [0], 96 bytes, 0.12 ms (0.12 avg, 0% loss)
> 10.0.0.1 : [1], 96 bytes, 0.13 ms (0.12 avg, 0% loss)
> 10.0.0.1 : [2], 96 bytes, 0.12 ms (0.12 avg, 0% loss)
>
> # DRBD Version and Status
>
> root at stor02 ~ $ cat /proc/drbd
> version: 8.3.11 (api:88/proto:86-96)
> srcversion: F937DCB2E5D83C6CCE4A6C9
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
> ns:17378172 nr:0 dw:177485633 dr:1946127117 al:3708575 bm:557
> lo:0
> pe:0 ua:0 ap:0 ep:1 wo:f oos:0
>
>
>
> # DRBD Config
>
> root at stor02 ~ $ drbdsetup /dev/drbd0 show
> disk {
> size 0s _is_default; # bytes
> on-io-error detach;
> fencing dont-care _is_default;
> max-bio-bvecs 0 _is_default;
> }
> net {
> timeout 60 _is_default; # 1/10 seconds
> max-epoch-size 19997;
> max-buffers 19997;
> unplug-watermark 64;
> connect-int 10 _is_default; # seconds
> ping-int 10 _is_default; # seconds
> sndbuf-size 524288; # bytes
> rcvbuf-size 0 _is_default; # bytes
> ko-count 0 _is_default;
> cram-hmac-alg "sha1";
> shared-secret "...";
> after-sb-0pri disconnect _is_default;
> after-sb-1pri disconnect _is_default;
> after-sb-2pri disconnect _is_default;
> rr-conflict disconnect _is_default;
> ping-timeout 5 _is_default; # 1/10 seconds
> on-congestion block _is_default;
> congestion-fill 0s _is_default; # byte
> congestion-extents 127 _is_default;
> }
> syncer {
> rate 2097152k; # bytes/second
> after -1 _is_default;
> al-extents 3833;
> on-no-data-accessible io-error _is_default;
> c-plan-ahead 0 _is_default; # 1/10 seconds
> c-delay-target 10 _is_default; # 1/10 seconds
> c-fill-target 0s _is_default; # bytes
> c-max-rate 102400k _is_default; # bytes/second
> c-min-rate 4096k _is_default; # bytes/second
> }
> protocol C;
> _this_host {
> device minor 0;
> disk "/dev/vgRAPTOR/lvol0";
> meta-disk internal;
> address ipv4 10.0.0.2:7788;
> }
> _remote_host {
> address ipv4 10.0.0.1:7788;
> }
>
>
> # Hardware Raid
>
> root at stor02 ~ $ arcconf getconfig 1 ld
> Controllers found: 1
> ----------------------------------------------------------------------
> Logical device information
> ----------------------------------------------------------------------
> Logical device number 0
> Logical device name : SYSTEM
> RAID level : 6 Reed-Solomon
> Status of logical device : Optimal
> Size : 51188 MB
> Stripe-unit size : 256 KB
> Read-cache setting : Enabled
> Read-cache status : On
> Write-cache setting : Enabled
> Write-cache status : Off
> Partitioned : Yes
> Protected by Hot-Spare : No
> Bootable : Yes
> Failed stripes : No
> Power settings : Disabled
> --------------------------------------------------------
> Logical device segment information
> --------------------------------------------------------
> Segment 0 : Present
> (Controller:1,Enclosure:0,Slot:0) WD-WXH1...
> Segment 1 : Present
> (Controller:1,Enclosure:0,Slot:1) WD-WXD1...
> Segment 2 : Present
> (Controller:1,Enclosure:0,Slot:2) WD-WXD1...
> Segment 3 : Present
> (Controller:1,Enclosure:0,Slot:3) WD-WXH1...
> Segment 4 : Present
> (Controller:1,Enclosure:0,Slot:4) WD-WXJ1...
> Segment 5 : Present
> (Controller:1,Enclosure:0,Slot:5) WD-WXD1...
> Segment 6 : Present
> (Controller:1,Enclosure:0,Slot:6) WD-WXH1...
> Segment 7 : Present
> (Controller:1,Enclosure:0,Slot:7) WD-WXH1...
>
>
> Logical device number 1
>
> Logical device name : RAPTOR
>
> RAID level : 6 Reed-Solomon
>
> Status of logical device : Optimal
>
> Size : 5668850 MB
>
> Stripe-unit size : 256 KB
>
> Read-cache setting : Enabled
>
> Read-cache status : On
>
> Write-cache setting : Enabled
>
> Write-cache status : Off (Does this mean
> write cache is off?)
> Partitioned : No
>
> Protected by Hot-Spare : No
>
> Bootable : No
>
> Failed stripes : No
>
> Power settings : Disabled
>
> --------------------------------------------------------
>
> Logical device segment information
>
> --------------------------------------------------------
>
> Segment 0 : Present
> (Controller:1,Enclosure:0,Slot:0) WD-WXH1...
> Segment 1 : Present
> (Controller:1,Enclosure:0,Slot:1) WD-WXD1...
> Segment 2 : Present
> (Controller:1,Enclosure:0,Slot:2) WD-WXD1...
> Segment 3 : Present
> (Controller:1,Enclosure:0,Slot:3) WD-WXH1...
> Segment 4 : Present
> (Controller:1,Enclosure:0,Slot:4) WD-WXJ1...
> Segment 5 : Present
> (Controller:1,Enclosure:0,Slot:5) WD-WXD1...
> Segment 6 : Present
> (Controller:1,Enclosure:0,Slot:6) WD-WXH1...
> Segment 7 : Present
> (Controller:1,Enclosure:0,Slot:7) WD-WXH1...
>
>
> # Results
> ## Throughput
> root at stor02 /mnt/drbd_r0 $ for i in $(seq 3); do time dd
> if=/dev/zero of=testfile bs=1G count=1 oflag=direct; done
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 24,4773 s, 43,9 MB/s
>
> real 0m24.540s
> user 0m0.000s
> sys 0m0.492s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 25,4277 s, 42,2 MB/s
>
> real 0m25.514s
> user 0m0.000s
> sys 0m0.504s
> 1+0 records in
> 1+0 records out
> 1073741824 bytes (1,1 GB) copied, 24,7216 s, 43,4 MB/s
>
> real 0m24.795s
> user 0m0.000s
> sys 0m0.508s
>
>
> ## Latency
> root at stor02 /mnt/drbd_r0 $ for i in $(seq 3); do time dd
> if=/dev/zero of=testfile bs=512 count=1000 oflag=direct; done
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1,13063 s, 453 kB/s
>
> real 0m1.133s
> user 0m0.000s
> sys 0m0.044s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1,18928 s, 431 kB/s
>
> real 0m1.191s
> user 0m0.004s
> sys 0m0.040s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 0,763934 s, 670 kB/s
>
> real 0m0.766s
> user 0m0.000s
> sys 0m0.048s
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
--
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA and Pacemaker support and consulting
DRBD(r) and LINBIT(r) are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed
_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
--------------------------------------------------------------------------------------------------------------------------------------------------------------
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those
of Ricardo (save for reports and other documentation formally approved and signed for release to the intended recipient). Only Ricardo's
authorized representatives may enter into legally binding obligations on behalf of Ricardo. Ricardo may monitor outgoing and incoming e-mails and
other telecommunications systems. By replying to this e-mail you give consent to such monitoring. The recipient should check e-mail and
any attachments for the presence of viruses. Ricardo accepts no liability for any damage caused by any virus transmitted by this e-mail.
"Ricardo" means Ricardo Inc. and its affiliated companies.
--------------------------------------------------------------------------------------------------------------------------------------------------------------'.