[DRBD-user] Speeding up sync rate on fast links and storage

[Parak]

Hi all,

I'm currently playing with DRBD (8.2.7) on 20Gb/s Infiniband, and it seems
that I'm running at the sync rate as the limiting speed factor. The local
storage on both nodes is identical (SAS array), and has been benchmarked at
about 650MB/s (or higher, depending on benchmark) to native disk, and about
550MB/s when writing to it through a disconnected DRBD device. The network
link for DRBD is Infiniband as well (IPoIB), which has been benchmarked with
netperf at ~800MB/s.

The fastest speed that I'm able to get from the DRBD sync with this
configuration is ~340MB/s, which limits the speed from my initiator to that
as well. Interestingly, I was also able to benchmark DRBD sync speed over
10Gbe, which despite my repeated attempts to tweak drbd.conf, mtu, and tcp
kernel parameters, has produced the same speed as the aformentioned 340MB/s
over IPoIB.

Here's the drbd.conf:

global {
    usage-count yes;
}

common {
  syncer {
     rate 900M;
         }
}

resource drbd0 {

  protocol C;

  handlers {
  }

  startup {
    degr-wfc-timeout 30;
  }

  disk {
    on-io-error   detach;
    fencing dont-care;
    no-disk-flushes;
    no-md-flushes;
    no-disk-drain;
    no-disk-barrier;
  }

  net {
    ko-count 2;
    after-sb-1pri discard-secondary;
    sndbuf-size 1M;
  }

  on srpt1 {
    device     /dev/drbd0;
    disk       /dev/sdb;
    address    10.0.0.2:7789;
    flexible-meta-disk  internal;
  }

  on srpt2 {
    device     /dev/drbd0;
    disk       /dev/sdb;
    address    10.0.0.3:7789;
    flexible-meta-disk  internal;
  }
}

Any advice/thoughts would be highly appreciated; thanks!

-Gennadiy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20081217/f73f1622/attachment.htm>