[DRBD-user] DRBD9 sync speed

Vladimír Bartoš bartos at jadro.org
Tue Jun 21 16:20:36 CEST 2016

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


hi,

try this:

drbdmanage peer-device-options --common --c-max-rate ...
this command update configuration on all hosts for you.

drbdmanage help peer-device-options

--
Vlado Bartos
bartos at jadro.org




> On 21. 6. 2016, at 14:10, Tytus Rogalewski <tytanick at gmail.com> wrote:
> 
> almost solved.
> After reboot, file /var/lib/drbd.d/drbdmanage_global_common.conf is overwritten and empty.
> So settings are not working after rebot.
> Anybody know how to make it permanent ?
> 
> Writing settings in another file like this doesnt work !
> 
> 
> root at node2:/etc/drbd.d# cat global_common.conf
> # DRBD is the result of over a decade of development by LINBIT.
> # In case you need professional services for DRBD or have
> # feature requests visit http://www.linbit.com <http://www.linbit.com/>
> 
> global {
>         usage-count no;
>         # minor-count dialog-refresh disable-ip-verification
>         # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;
> }
> 
> common {
>         handlers {
>                 # These are EXAMPLE handlers only.
>                 # They may have severe implications,
>                 # like hard resetting the node under certain circumstances.
>                 # Be careful when chosing your poison.
> 
>                 # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
>                 # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
>                 # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
>                 # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
>                 # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
>                 # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
>                 # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
>                 # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
>         }
> 
>         startup {
>                 # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
>         }
> 
>         options {
>                 # cpu-mask on-no-data-accessible
>         }
> 
>         disk {
>       on-io-error             detach;
>         no-disk-flushes ;
>         no-disk-barrier;
>         c-plan-ahead 10;
>         c-fill-target 24M;
>         c-min-rate 10M;
>         c-max-rate 100M;
>                 # size on-io-error fencing disk-barrier disk-flushes
>                 # disk-drain md-flushes resync-rate resync-after al-extents
>                 # c-plan-ahead c-delay-target c-fill-target c-max-rate
>                 # c-min-rate disk-timeout
>         }
> 
>         net {
>                 protocol C;
>         # max-epoch-size          20000;
>         max-buffers             36k;
>         sndbuf-size            1024k ;
>         rcvbuf-size            2048k;
> 
>                 # allow-two-primaries yes;
>                 # protocol timeout max-epoch-size max-buffers unplug-watermark
>                 # connect-int ping-int sndbuf-size rcvbuf-size ko-count
>                 # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
>                 # after-sb-1pri after-sb-2pri always-asbp rr-conflict
>                 # ping-timeout data-integrity-alg tcp-cork on-congestion
>                 # congestion-fill congestion-extents csums-alg verify-alg
>                 # use-rle
>         }
> }
> 
> 
> 
> --
> sky-link.net <http://sky-link.net/> Tytus Rogalewski
> eMail: tytanick at gmail.com <mailto:tytanick at gmail.com>
> strona: http://sky-link.net <http://sky-link.net/>
> kom. 790 202 300
> gg: 210533
> skype: tytanick
> NIP: 9570976234
> REGON: 221197388
> 
> 
> 2016-06-21 11:01 GMT+02:00 Tytus Rogalewski <tytanick at gmail.com <mailto:tytanick at gmail.com>>:
> Lol i made it :)
> 
> 
> Ok so i thought that it was buggy drbd9 but after 3 days of lost time i finally made it
> Before that fix i had 20mbit/s syncing but after that i have 800mbit/s on 1GBit
> 
> Code:
> NET | bond0    80% | pcki   13298 | pcko  133286 | si 3879 Kbps | so  807 Mbps | coll       0 | mlti      12 | erri       0 | erro       0 | drpi       0 | drpo       0 |
> NET | eth1     80% | pcki   13298 | pcko  133286 | si 3879 Kbps | so  807 Mbps | coll       0 | mlti      12 | erri       0 | erro       0 | drpi       0 | drpo       0 |
> NET | tap0      0% | pcki       4 | pcko       4 | si    0 Kbps | so   12 Kbps | coll       0 | mlti       0 | erri       0 | erro       0 | drpi       0 | drpo       0 |
> NET | eth2
> it was all about stupid config file ..... and accually you need to edit proper config ....
> 
> First of all if you have working drbd9 cluster first stop all VMs and then all resources on both drbd nodes:
> Code:
> drbdadm disconnect all
> Then you need to edit config (and this is what i was looking for so long) /var/lib/drbd.d/drbdmanage_global_common.conf because editing /etc/drbd.d/global_common.conf doesnt work at all
> 
> Enter those settings to /var/lib/drbd.d/drbdmanage_global_common.conf ON BOTH NODES
> Code:
> # it must be content of /var/lib/drbd.d/drbdmanage_global_common.conf !!!!
> common {
> disk {
>         on-io-error             detach;
>         no-disk-flushes ;
>         no-disk-barrier;
>         c-plan-ahead 10;
>         c-fill-target 24M;
>         c-min-rate 10M;
>         c-max-rate 100M;
> }.
> net {
>         # max-epoch-size          20000;
>         max-buffers             36k;
>         sndbuf-size            1024k ;
>         rcvbuf-size            2048k;
> }
> }
> 
> Then simply restart drbd by:
> Code:
> /etc/init.d/drbd restart
> 
> And thats it. Syncing should be at 800mbits
> In you have 10GBit just change c-max-rate to 1000M
> Play with those settings
> 
> --
> sky-link.net <http://sky-link.net/> Tytus Rogalewski
> eMail: tytanick at gmail.com <mailto:tytanick at gmail.com>
> strona: http://sky-link.net <http://sky-link.net/>
> kom. 790 202 300
> gg: 210533
> skype: tytanick
> NIP: 9570976234
> REGON: 221197388
> 
> 
> 2016-06-21 11:00 GMT+02:00 Tytus Rogalewski <tytanick at gmail.com <mailto:tytanick at gmail.com>>:
> Lol i made it :)
> 
> 
> Ok so i thought that it was buggy drbd9 but after 3 days of lost time i finally made it
> Before that fix i had 20mbit/s syncing but after that i have 800mbit/s on 1GBit
> 
> Code:
> NET | bond0    80% | pcki   13298 | pcko  133286 | si 3879 Kbps | so  807 Mbps | coll       0 | mlti      12 | erri       0 | erro       0 | drpi       0 | drpo       0 |
> NET | eth1     80% | pcki   13298 | pcko  133286 | si 3879 Kbps | so  807 Mbps | coll       0 | mlti      12 | erri       0 | erro       0 | drpi       0 | drpo       0 |
> NET | tap0      0% | pcki       4 | pcko       4 | si    0 Kbps | so   12 Kbps | coll       0 | mlti       0 | erri       0 | erro       0 | drpi       0 | drpo       0 |
> NET | eth2
> it was all about stupid config file ..... and accually you need to edit proper config ....
> 
> First of all if you have working drbd9 cluster first stop all VMs and then all resources on both drbd nodes:
> Code:
> drbdadm disconnect all
> Then you need to edit config (and this is what i was looking for so long) /var/lib/drbd.d/drbdmanage_global_common.conf because editing /etc/drbd.d/global_common.conf doesnt work at all
> 
> Enter those settings to /var/lib/drbd.d/drbdmanage_global_common.conf ON BOTH NODES
> Code:
> # it must be content of /var/lib/drbd.d/drbdmanage_global_common.conf !!!!
> common {
> disk {
>         on-io-error             detach;
>         no-disk-flushes ;
>         no-disk-barrier;
>         c-plan-ahead 10;
>         c-fill-target 24M;
>         c-min-rate 10M;
>         c-max-rate 100M;
> }.
> net {
>         # max-epoch-size          20000;
>         max-buffers             36k;
>         sndbuf-size            1024k ;
>         rcvbuf-size            2048k;
> }
> }
> 
> Then simply restart drbd by:
> Code:
> /etc/init.d/drbd restart
> 
> And thats it. Syncing should be at 800mbits
> In you have 10GBit just change c-max-rate to 1000M
> Play with those settings
> 
> --
> sky-link.net <http://sky-link.net/> Tytus Rogalewski
> eMail: tytanick at gmail.com <mailto:tytanick at gmail.com>
> strona: http://sky-link.net <http://sky-link.net/>
> kom. 790 202 300
> gg: 210533
> skype: tytanick
> NIP: 9570976234
> REGON: 221197388
> 
> 
> 2016-06-21 8:02 GMT+02:00 Tytus Rogalewski <tytanick at gmail.com <mailto:tytanick at gmail.com>>:
> Drbd 8.x was totally fine.
> 9.0 in proxmox 4.2 is bugged i suppose.
> I will try 8.x in 4.2
> 
> 21 cze 2016 01:00 "Adam Goryachev" <mailinglists at websitemanagers.com.au <mailto:mailinglists at websitemanagers.com.au>> napisał(a):
> On 20/06/16 22:47, Emmanuel Florac wrote:
> Le Sun, 19 Jun 2016 08:56:18 +0200
> Piero Baudino <pbaudino at formatnet.it <mailto:pbaudino at formatnet.it>> écrivait:
> 
> Hi all,
> 
> I have a cluster with 2 PVE servers with DRBD9.
> The replicated volume is about 7 Tb, the disks are 7200 rpm near line
> SAS. The communications channel for DRBD9 is done via 4 Gbit
> dedicated network cards in bond configuration (round robbin).
> I have created a VM with 1 Tb disk for test purpuse in the DRBD
> volume. After that, DRBD starts sync...
> The sync speed is about 10 Mbyte/sec
> The effective network speed misured with IPERF is about 3,6 Gbit/sec
> 
> Is there a way to increse sync speed ?
> All the docs about this topic are related to 8.x.y DRBD version....
> 
> See this thread maybe:
> http://www.gossamer-threads.com/lists/drbd/users/27675 <http://www.gossamer-threads.com/lists/drbd/users/27675>
> 
> 
> Stupid question perhaps, but doesn't that thread also relate to 8.4.x rather than 9.0.x ?
> 
> 
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au <http://www.websitemanagers.com.au/>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
> http://lists.linbit.com/mailman/listinfo/drbd-user <http://lists.linbit.com/mailman/listinfo/drbd-user>
> 
> 
> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20160621/cbf02886/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20160621/cbf02886/attachment.pgp>


More information about the drbd-user mailing list