[DRBD-user] Stacked resource high CPU during sync

[Miles Lott]

We have a 3 node setup nearing deployment into production use.  We're 
running 8.4.2 on Centos 6.3.

The 3rd node is at the other end of a point to point 60Mb/S fiber.  
Whenever we start copying large amounts of data, say several hundred 
gigabytes, the third node falls out of sync.  That wouldn't be so 
terrible except that during the resync process, the load average on the 
current primary server runs up to about 8 or 10.

All 3 nodes are dual quad core Xeon E5606 servers with 24G RAM each.  
All 3 are using Supermicro motherboards, 3ware 24i4e cards with 32T RAID 
6 configured and BBU write cache enabled.  The two main nodes are tied 
together with two bonded 1G interfaces.  I haven't noticed them falling 
out of sync, fortunately.  The issue is probably buffers filling up due 
to the slow connection to the offsite third node.  Any thoughts on how 
we might improve this?  Proxy?  More buffering?  I'd love to bump up the 
connection speed of course.

Global config:
global {
     usage-count yes;
}

... handlers...

disk {
         read-balancing when-congested-remote;
}

Resource config:

resource main
{
     startup
     {
         wfc-timeout 60;
         degr-wfc-timeout 30;
     }

     disk
     {
         disk-barrier no;
         disk-flushes no;
         on-io-error detach;
         fencing resource-only;
         resync-rate 66M;

         al-extents 1801;
     }

     net
     {
         protocol C;
         after-sb-0pri discard-zero-changes;
         after-sb-1pri consensus;
         after-sb-2pri disconnect;
         max-buffers 8000;
         max-epoch-size 8000;
         sndbuf-size 512k;
     }

     syncer
     {
         cpu-mask E0;
     }

     on fs01
     {
         device /dev/drbd0;
         disk /dev/vg00/lv00;
         meta-disk internal;
         address 10.1.2.1:7788;
     }

     on fs02
     {
         device /dev/drbd0;
         disk /dev/vg00/lv00;
         meta-disk internal;
         address 10.1.2.2:7788;
     }
}

resource main-U
{
     options {
         cpu-mask E0;
     }

     net
     {
         protocol A;
         max-buffers 8000;
         max-epoch-size 8000;
         sndbuf-size 512k;
     }

     disk
     {
         c-plan-ahead 0;
         c-min-rate 500K;
         c-max-rate 3500K;
         resync-rate 3500K;
     }

     stacked-on-top-of main
     {
         device     /dev/drbd10;
         address    172.22.2.147:7789;
     }

     on colodisk02
     {
         device     /dev/drbd10;
         disk       /dev/vg00/lv00;
         address    172.16.1.10:7789;
         meta-disk  internal;
     }
}





--------------------------------------------------------------------------------
Privileged, Proprietary and/or Confidential Information may be contained in
this electronic message.  If you are not the intended recipient, you should
not copy it, re-transmit it, use it or disclose its contents, but should
kindly return to the sender immediately and delete your copy from your system.
Gulf Interstate Engineering Company does not accept responsibility for
changes made to this electronic message or its attachments.