Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
So,
I played during hours with the dynamic resync rate controller.
Here are my settings :
c-plan-ahead 10; //10ms between my 2 nodes, but minimum 1 second
recommended here : https://blogs.linbit.com/p/128/drbd-sync-rate-controller/
resync-rate 680M; //mainly ignored
c-min-rate 400M;
c-max-rate 680M;
c-fill-target 20M; //my BDP is 6.8M, guides say to use a value between 1x
and 3x BDP
Resync can achieve up to 680M when there are no application IOs on the
source.
However, as soon as there are application IOs (writes with dd in my tests),
resync slows down to some MB/s...
I played with c-plan-ahead and c-fill-target without success.
I also tested c-delay-target.
I tried to set unplug-watermark to 16.
My IO scheduler is already the deadline one...
Well, I'm a little bit lost, I can't achieve to get resync with a minimum
rate of 400M when there are application IOs...
Here, Lars says :
http://lists.linbit.com/pipermail/drbd-user/2011-August/016739.html
The dynamic resync rate controller basically tries to use as much as
c-max-rate bandwidth, but will automatically throttle, if
- application IO on the device is detected (READ or WRITE), AND the
estimated current resync speed is above c-min-rate
- the amount of in-flight resync requests exceeds c-fill-target
However, does DRBD throttle application IOs when resync rate is lower than
c-min-rate ?
According to my tests I'm not so sure.
Thank you very much for your support,
Ben
2015-05-26 15:06 GMT+02:00 A.Rubio <aurusa at etsii.upv.es>:
> Have you test these values ?
>
> https://drbd.linbit.com/users-guide/s-throughput-tuning.html
>
>
> El 26/05/15 a las 13:16, Ben RUBSON escribió:
>
> RAID controller is OK yes.
>
> Here is a 4 steps example of the issue :
>
>
>
> ### 1 - initial state :
>
> Source :
> - sdb read MB/s : 0
> - sdb write MB/s : 0
> - eth1 incoming MB/s : 0
> - eth1 outgoing MB/s : 0
> Target :
> - sdb read MB/s : 0
> - sdb write MB/s : 0
> - eth1 incoming MB/s : 0
> - eth1 outgoing MB/s : 0
>
>
>
> ### 2 - dd if=/dev/zero of=bigfile :
>
> Source :
> - sdb read MB/s : 0
> - sdb write MB/s : 670
> - eth1 incoming MB/s : 1
> - eth1 outgoing MB/s : 670
> Target :
> - sdb read MB/s : 0
> - sdb write MB/s : 670
> - eth1 incoming MB/s : 670
> - eth1 outgoing MB/s : 1
>
>
>
> ### 3 - disable the link between the 2 nodes :
>
> Source :
> - sdb read MB/s : 0
> - sdb write MB/s : 670
> - eth1 incoming MB/s : 0
> - eth1 outgoing MB/s : 0
> Target :
> - sdb read MB/s : 0
> - sdb write MB/s : 0
> - eth1 incoming MB/s : 0
> - eth1 outgoing MB/s : 0
>
>
>
> ### 4 - re-enable the link between the 2 nodes :
>
> Source :
> - sdb read MB/s : ~20
> - sdb write MB/s : ~670
> - eth1 incoming MB/s : 1
> - eth1 outgoing MB/s : 670
> Target :
> - sdb read MB/s : 0
> - sdb write MB/s : 670
> - eth1 incoming MB/s : 670
> - eth1 outgoing MB/s : 1
> DRBD resource :
> 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
> ns:62950732 nr:1143320132 dw:1206271712 dr:1379744185 al:9869 bm:6499
> lo:2 pe:681 ua:1 ap:0 ep:1 wo:d oos:11883000
> [>...................] sync'ed: 6.9% (11604/12448)M
> finish: 0:34:22 speed: 5,756 (6,568) want: 696,320 K/sec
>
>
>
> ### values I would have expected in step 4 :
>
> Source :
> - sdb read MB/s : ~400 (because of c-min-rate 400M)
> - sdb write MB/s : ~370
> - eth1 incoming MB/s : 1
> - eth1 outgoing MB/s : 670
> Target :
> - sdb read MB/s : 0
> - sdb write MB/s : 670
> - eth1 incoming MB/s : 670
> - eth1 outgoing MB/s : 1
>
> Why resync is totally ignored and application (dd here in the example)
> still consumes all available IOs / bandwidth ?
>
>
>
> Thank you,
>
> Ben
>
>
>
> 2015-05-25 16:50 GMT+02:00 A.Rubio <aurusa at etsii.upv.es>:
>
>> Cache settings an I/O in RAID controller is optimal ??? Write-back,
>> write-through, cache enablad, I/O direct, ...
>>
>> El 25/05/15 a las 11:50, Ben RUBSON escribió:
>>
>>> The link between nodes is a 10Gb/s link.
>>> The DRBD resource is a RAID-10 array which is able to resync at up to
>>> 800M (as you can see I have lowered it to 680M in my configuration file).
>>>
>>> The "issue" here seems to be a prioritization "issue" between
>>> application IOs and resync IOs.
>>> Perhaps I miss-configured something ?
>>> Goal is to have resync rate up to 680M, with a minimum of 400M, even if
>>> there are application IOs.
>>> This in order to be sure to complete the resync even if there are a lot
>>> of write IOs from the application.
>>>
>>> With my simple test below, this is not the case, dd still writes at its
>>> best throughput, lowering resync rate which can’t reach 400M at all.
>>>
>>> Thank you !
>>>
>>> Le 25 mai 2015 à 11:18, A.Rubio <aurusa at etsii.upv.es> a écrit :
>>>>
>>>> the link between nodes is ??? 1Gb/s ??? , 10Gb/s ??? ...
>>>>
>>>> the Hard Disks are ??? SATA 7200rpm ???, 10000rpm ???, SAS ???,
>>>> SSD ???...
>>>>
>>>> 400M to 680M with a 10Gb/s link and SAS 15.000 rpm is OK but less ...
>>>>
>>>> Le 12 avr. 2014 à 17:23, Ben RUBSON <ben.rubson at gmail.com> a écrit :
>>>>>
>>>>> Hello,
>>>>>
>>>>> Let's assume the following configuration :
>>>>> disk {
>>>>> c-plan-ahead 0;
>>>>> resync-rate 680M;
>>>>> c-min-rate 400M;
>>>>> }
>>>>>
>>>>> Both nodes are uptodate, and on the primary, I have a test IO burst
>>>>> running, using dd.
>>>>>
>>>>> I then cut replication link for a few minutes so that secondary node
>>>>> will be several GB behind primary node.
>>>>>
>>>>> I then re-enable replication link.
>>>>> What I expect here according to the configuration is that secondary
>>>>> node will fetch missing GB at a 400 MB/s throughput.
>>>>> DRBD should then prefer resync IOs over application (dd here) IOs.
>>>>>
>>>>> However, it does not seems to work.
>>>>> dd still writes at its best throughput, meanwhile reads are made from
>>>>> the primary disk between 30 and 60 MB/s to complete the resync.
>>>>> Of course this is not the expected behaviour.
>>>>>
>>>>> Did I miss something ?
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20150528/6a3dc64c/attachment.htm>