Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Also I triggered the sync by "drbdadm invalidate-remote wandk0" on vmA.
Seeing same bandwidth limit around 30MB/s.
[root at vmC ~]# cat /proc/drbd
version: 8.4.7-1 (api:1/proto:86-101)
GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by phil at Build64R7,
2016-01-12 14:29:40
1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate A r-----
ns:0 nr:8636123 dw:8635099 dr:0 al:0 bm:0 lo:1 pe:69 ua:1 ap:0 ep:1
wo:d oos:44851772
[>...................] sync'ed: 5.0% (43800/46076)M
finish: 0:24:52 speed: 30,052 (32,852) want: 4,194,304 K/sec
[root at vmC ~]#
On Fri, Jul 1, 2016 at 11:18 AM, T.J. Yang <tjyang2001 at gmail.com> wrote:
> Thanks Igor for another tip by switching over to asynchronous A protocol,
> for WAN network.
> There was no improvement as suggested by other email thread
> <http://www.gossamer-threads.com/lists/drbd/users/27510> that we should
> see 2x write bandwidth increase.
>
> [root at vmA2vmC ~]# ./drbdtest.bash
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.854164 s, 629 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.578418 s, 928 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.631658 s, 850 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.577289 s, 930 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.773748 s, 694 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.635504 s, 845 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.63825 s, 841 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.602557 s, 891 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.583016 s, 921 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 0.606236 s, 886 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 19.3083 s, 27.8 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 17.7925 s, 30.2 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 16.5809 s, 32.4 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 17.4038 s, 30.8 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 16.5223 s, 32.5 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 16.7767 s, 32.0 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 15.1224 s, 35.5 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 16.668 s, 32.2 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 16.5105 s, 32.5 MB/s
> 1+0 records in
> 1+0 records out
> 536870912 bytes (537 MB) copied, 16.6164 s, 32.3 MB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.39062 s, 368 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.40759 s, 364 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.6254 s, 315 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.61219 s, 318 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 2.18549 s, 234 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.22317 s, 419 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.42974 s, 358 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.7516 s, 292 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.72105 s, 297 kB/s
> 1000+0 records in
> 1000+0 records out
> 512000 bytes (512 kB) copied, 1.57352 s, 325 kB/s
> [root at vmA2vmC ~]# cat ./drbdtest.bash
> TEST_FILE=drbd-write-test.img
> # write to local disk
> for i in $(seq 10); do
> dd if=/dev/zero of=/root/$TEST_FILE bs=512M count=1 oflag=direct
> sleep 5
> done
> rm -f /root/$TEST_FILE
>
> # write to /dev/drbd1 mounted as /pub
> for i in $(seq 10); do
> dd if=/dev/zero of=/pub/$TEST_FILE bs=512M count=1 oflag=direct
> sleep 5
> done
> rm -f /pub/$TEST_FILE
>
>
> for i in $(seq 10); do
> dd if=/dev/zero of=/pub/$TEST_FILE bs=512 count=1000 oflag=direct
> sleep 5
> done
> rm -f /pub/$TEST_FILE
>
>
> [root at vmA2vmC ~]# cat /proc/drbd
> version: 8.4.7-1 (api:1/proto:86-101)
> GIT-hash: 3a6a769340ef93b1ba2792c6461250790795db49 build by phil at Build64R7,
> 2016-01-12 14:29:40
>
> 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----
> ns:5253671 nr:0 dw:5253671 dr:1577 al:51 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
> wo:d oos:0
> [root at vmA2vmC ~]#
>
>
> On Fri, Jul 1, 2016 at 10:46 AM, Igor Cicimov <icicimov at gmail.com> wrote:
>
>>
>> On 1 Jul 2016 3:48 pm, "T.J. Yang" <tjyang2001 at gmail.com> wrote:
>> >
>> > Hi All
>> >
>> > I am new to drbd performance turning and I have been study (R0).
>> > Aso I am browsing others effort in drbd-user archive (R1).
>> > I was able to get 350MB/s rsync rate (R2) for two Centos 7.2 VMs(A and
>> B) when they are on same LAN with turning from (R1) thread.
>> >
>> > My goal is to have C Centos 7.2 VM paired with A VM that go over a fast
>> WAN pipe(R3). but when I reuse B's drbd config and change the IP info. The
>> rsync rate back to 30-40M rsync rate(See R4).
>> >
>> > I tried the jumbo frame turning to raise MTU from 1500 to 8000(RH
>> Support article recommend 8000, not 9000). But this change on VM A and C
>> doesn't not improve rsync rate.
>> >
>> >
>> > Does networking team need do any change on their swith/router for
>> drbd case ?
>> >
>> You need jumbo frames enabled on the switch too.
>>
>> >
>> > References:
>> > R0: https://www.drbd.org/en/doc/users-guide-84/p-performance
>> > R1:
>> http://lists.linbit.com/pipermail/drbd-user/2016-January/022611.html
>> > R2:
>> >
>> > #this is between vmA(10.65.184.1) and vmB(10.65.184.3), same subnet.
>> >
>> > [root at vmA ~]# ./drbd-pm-test.bash wandk0 # script from R0
>> >
>> > testing wandk0 on /dev/drbd1
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 8.00925 s, 67.0 MB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 1.72338 s, 312 MB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 1.84181 s, 291 MB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 1.66079 s, 323 MB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 1.69359 s, 317 MB/s
>> >
>> > testing wandk0 on backing device:/dev/centos/wandk0
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.504366 s, 1.1 GB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.550144 s, 976 MB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.502675 s, 1.1 GB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.473032 s, 1.1 GB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.470139 s, 1.1 GB/s
>> >
>> > [root at vmA ~]#
>> >
>> >
>> >
>> > R3:
>> >
>> > [root at vmC ~]# iperf3 -s -p 5900
>> >
>> > warning: this system does not seem to support IPv6 - trying IPv4
>> >
>> > -----------------------------------------------------------
>> >
>> > Server listening on 5900
>> >
>> > -----------------------------------------------------------
>> >
>> > Accepted connection from 10.65.184.1(vmA), port 56750
>> >
>> > [ 5] local 10.64.5.245 port 5900 connected to 10.65.184.1 port 56754
>> >
>> > [ ID] Interval Transfer Bandwidth
>> >
>> > [ 5] 0.00-1.00 sec 73.9 MBytes 620 Mbits/sec
>> >
>> > [ 5] 1.00-2.00 sec 100 MBytes 842 Mbits/sec
>> >
>> > [ 5] 2.00-3.00 sec 107 MBytes 895 Mbits/sec
>> >
>> > [ 5] 3.00-4.00 sec 113 MBytes 947 Mbits/sec
>> >
>> > [ 5] 4.00-5.00 sec 117 MBytes 984 Mbits/sec
>> >
>> > [ 5] 5.00-6.00 sec 120 MBytes 1.01 Gbits/sec
>> >
>> > [ 5] 6.00-7.00 sec 123 MBytes 1.03 Gbits/sec
>> >
>> > [ 5] 7.00-8.00 sec 124 MBytes 1.04 Gbits/sec
>> >
>> > [ 5] 8.00-9.00 sec 124 MBytes 1.04 Gbits/sec
>> >
>> > [ 5] 9.00-10.00 sec 125 MBytes 1.04 Gbits/sec
>> >
>> > [ 5] 10.00-10.04 sec 5.25 MBytes 1.25 Gbits/sec
>> >
>> > - - - - - - - - - - - - - - - - - - - - - - - - -
>> >
>> > [ ID] Interval Transfer Bandwidth
>> >
>> > [ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec
>> sender
>> >
>> > [ 5] 0.00-10.04 sec 1.11 GBytes 947 Mbits/sec
>> receiver
>> >
>> > -----------------------------------------------------------
>> >
>> > Server listening on 5900
>> >
>> > -----------------------------------------------------------
>> >
>> > ^Ciperf3: interrupt - the server has terminated
>> >
>> > [root at vmC ~]# date
>> >
>> > Wed Jun 29 12:26:37 EDT 2016
>> >
>> > [root at vmC ~]#
>> >
>> >
>> >
>> > R4: only getting 35MB/s when cross WAN network.
>> >
>> > [root at vmA ~]# ./scratch-test.bash wandk0 # from R1
>> >
>> > 1: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
>> >
>> > Writing via wandk0 on /dev/drbd1
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 15.1859 s, 35.4 MB/s
>> >
>> > <snipped>
>> >
>> > 536870912 bytes (537 MB) copied, 15.598 s, 34.4 MB/s
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 16.9145 s, 31.7 MB/s
>> >
>> > Writing directly into backing device:/dev/centos/wandk0
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.625202 s, 859 MB/s
>> >
>> > <snipped>
>> >
>> > 1+0 records in
>> >
>> > 1+0 records out
>> >
>> > 536870912 bytes (537 MB) copied, 0.566128 s, 948 MB/s
>> >
>> > [root at vmA ~]# date
>> >
>> > Thu Jun 30 13:07:16 EDT 2016
>> >
>> > [root at vmA ~]#
>> >
>> >
>> >
>> > --
>> > T.J. Yang
>> >
>> > _______________________________________________
>> > drbd-user mailing list
>> > drbd-user at lists.linbit.com
>> > http://lists.linbit.com/mailman/listinfo/drbd-user
>> >
>>
>>
>
>
> --
> T.J. Yang
>
--
T.J. Yang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20160701/3ccc9815/attachment.htm>