[DRBD-user] Protocol A vs Protocol C performance issue?

[Lars Ellenberg]

On Tue, Oct 20, 2015 at 01:07:48PM +0200, Joeri Casteels wrote:
> Dear All,
> 
> 
> Doing a new set-up of DRBD on Ubuntu LTS 14.04 with stock DRBD 
> When Setting to protocol A i get double the bandwidth as protocol C anybody an idea? not changing anything else of the settings except the protocol
> 
> PROTOCOL C:
> d if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 46.3583 s, 452 MB/s
> 
> PROTOCOL A:
> dd if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct
> 20000+0 records in
> 20000+0 records out
> 20971520000 bytes (21 GB) copied, 23.8356 s, 880 MB/s

dd does:
  write one block size # which takes some time
  write next bs # which again takes some time
  ...

Without oflag=direct,
you write into the linux page cache, the write itself takes microseconds
(at most), and you get many such writes per second, resulting usually in
a throughput of several gigabytes per second (until you hit the
boundaries of the page cache and are throttled down to the rate of the
background write-out).

With oflag=direct,
it bypasses the linux page cache,
the single write call takes longer,
and you do no longer get that many writes per second.

With protocol B and C, the single write latency even includes
the network latency and remote disk latency, possibly flush latencies,
and you get even less writes per second.

Throughput is concurrency times request size by request completion latency.

dd has no concurrency, in the given example, request size is 1M,
if you get 450 MB/s, your latency apparently is in the order of 2.3 ms.

If you want more throughput,
you need to decrease latency, or increase concurrency.


-- 
: Lars Ellenberg
: http://www.LINBIT.com | Your Way to High Availability
: DRBD, Linux-HA  and  Pacemaker support and consulting

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list   --   I'm subscribed