[DRBD-user] drbd re-sync performance

[Alexey Obitotskiy]

Hello,

We are using drbd on testbench with primary/secondary setup between two 
hosts. Resource volume is a lvm volume on disk. Resource uses internal 
meta-disk. Hosts are connected via 10Gb Ethernet link. Disk performance 
is about 500M/s. Hosts are real hardware (not under any kind of 
supervisor) using ubuntu with 2.6.x kernel (also tested on 3.x kernels 
with the same results).

We need to get maximum performance during full re-sync one of the nodes. 
If we use drbd versions either from 8.3.10 to 8.3.15 or from 8.4.0 to 
8.4.3 we get ~360-380 M/s resync rate (which is OK, because it is near 
disk I/O limit). But if we switch to 8.3.16 or 8.4.4 performance is 
dropped approximately twice, i.e. ~160-180M/s. Changing max-buffers, 
rate, c-plan-ahead or al-extents parameters do not help to increase 
re-sync performance.

These cases work identically in both modes with static re-sync rate or 
with using c-plan-ahead.

Values of some parameters affecting performance:

protocol C;
max-buffers 8000;
max-epoch-size 8000;
unplug-watermark 128;
sndbuf-size 0;

al-extents 3389;

Old way:

c-plan-ahead 0;
rate 600M;

Or new way:

c-plan-ahead 100;
c-max-rate 600M;
c-min-rate 0;
c-fill-target 0;

Could you please comment, is this expected performance change or not? If 
this is expected, please comment what is expected benefits of a such 
approach in latest DRBD versions (8.3.16, 8.4.4). Thank you in advance.


global {
         disable-ip-verification;
         usage-count no;
}

resource "r0" {
         protocol                C;

         device          minor 0;


         on ubuntu1 {
                 disk                "/dev/vg0/lv0";
                 address          172.16.0.1:12000;
                 meta-disk       "/dev/vg0/lv0_meta"[0];
         }

         net {
                 max-buffers 12000;
                 max-epoch-size 12000;
                 unplug-watermark 128;
                 sndbuf-size 0;
                 ping-int 4;
                 timeout 30;
         }

         handlers {
         }

         on peer {
                 disk /dev/fake_disk_d64ixM-66iU-16R3-ArYM-EFe1-dmIo-a5yDJx;
                 address            172.16.0.2:12000;
                 meta-disk 
/dev/fake_metadisk_d64ixM-66iU-16R3-ArYM-EFe1-dmIo-a5yDJx[0];
         }

         syncer {
                 al-extents 3389;
                 c-plan-ahead 100;
                 c-max-rate 600M;
                 c-min-rate 0;
                 c-fill-target 0;
                 rate 600M;
         }

         disk {
                 on-io-error     call-local-io-error;
                 fencing         resource-only;
         }
}

-- 
Alexey