[DRBD-user] Dual-primary/ Very slow synchronization

[Daniel Meszaros]

Hi there,

I had a split-brain situation and tried to resync both machines (running 
Citrix Xenserver 5.6/ CentOS). But I figured out that the sync speed was 
below normal - I remember sync rates between 100 and 250 MB/s. After a 
few attempts I decided to have both machines reboot. But no improvement.

The current sync speed is around 5 kB/s.

I tried downloading a file to both machines which happened with > 500 KB/s.

Then I did a benchmark of the throughput between both machines with 
"netio". The results are always > 200 MB/s.

As I said: Nothing has changed on the machines. Only the 10GbE switch 
got four additional Mini GBICs and needed to be switched off for a few 
minutes - this is when I wanted to turn one primary to secondary but 
recognized that they where out of sync already.

Another thing worth to mention: When running the sync my virtual guests 
on the Xenserver respond slowly. When executing "top" in dom0 CPU usage 
seldomly reaches values higher than 3%.

Playing around with other syncer rates did not lead to an improvement.

MTU size is 9000. I played around with a few network "tweak" settings 
when configuring the machines and had good results before. These values 
have been stored in a script that I manually run at system start before 
invoking DRBD.

Did anyone possibly have a similar problem already and could you suggest 
where I could tweak the system to run the way it ran before?

DRBD is version 8.3.8.1 . Currently the sync is paused.

My "drbd.conf":
-------------------------------------------------------------
global { usage-count no; }

common {
         protocol C;

         net {
                 allow-two-primaries;
                 shared-secret "drbd";
                 after-sb-0pri discard-zero-changes;
                 after-sb-1pri consensus;
                 after-sb-2pri disconnect;
                 sndbuf-size 0;
                 no-tcp-cork;
                 unplug-watermark 8000;
                 max-buffers 8000;
                 max-epoch-size 8000;
         }

         disk {
             max-bio-bvecs 1;
             no-disk-flushes;
             #no-md-flushes;
             no-disk-barrier;
         }

         startup {
             become-primary-on both;
         }

         handlers {
             split-brain "/usr/lib/drbd/notify-split-brain.sh";
         }

         syncer {
             al-extents 511;
             rate 2048M;
         }
}

resource drbd-sr1 {
         device /dev/drbd0;
         meta-disk internal;
         on server1 {
             address 192.168.30.201:7788 ;
             disk /dev/disk/by-id/scsi-SAdaptec_data_84A1F62C ;
         }
         on server2 {
             address 192.168.30.202:7788 ;
             disk /dev/disk/by-id/scsi-SAdaptec_data_2548D82A ;
         }
}

-------------------------------------------------------------

CU,
Mészi.