Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
20.10.2015 21:52, Joeri Casteels wrote: > > >> On 20 Oct 2015, at 20:47, Lars Ellenberg <lars.ellenberg at linbit.com> wrote: >> >> On Tue, Oct 20, 2015 at 01:07:48PM +0200, Joeri Casteels wrote: >>> Dear All, >>> >>> >>> Doing a new set-up of DRBD on Ubuntu LTS 14.04 with stock DRBD >>> When Setting to protocol A i get double the bandwidth as protocol C anybody an idea? not changing anything else of the settings except the protocol >>> >>> PROTOCOL C: >>> d if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct >>> 20000+0 records in >>> 20000+0 records out >>> 20971520000 bytes (21 GB) copied, 46.3583 s, 452 MB/s >>> >>> PROTOCOL A: >>> dd if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct >>> 20000+0 records in >>> 20000+0 records out >>> 20971520000 bytes (21 GB) copied, 23.8356 s, 880 MB/s >> >> dd does: >> write one block size # which takes some time >> write next bs # which again takes some time >> ... >> >> Without oflag=direct, >> you write into the linux page cache, the write itself takes microseconds >> (at most), and you get many such writes per second, resulting usually in >> a throughput of several gigabytes per second (until you hit the >> boundaries of the page cache and are throttled down to the rate of the >> background write-out). >> >> With oflag=direct, >> it bypasses the linux page cache, >> the single write call takes longer, >> and you do no longer get that many writes per second. >> >> With protocol B and C, the single write latency even includes >> the network latency and remote disk latency, possibly flush latencies, >> and you get even less writes per second. >> >> Throughput is concurrency times request size by request completion latency. >> >> dd has no concurrency, in the given example, request size is 1M, >> if you get 450 MB/s, your latency apparently is in the order of 2.3 ms. >> >> If you want more throughput, >> you need to decrease latency, or increase concurrency. > > Any suggestions on how to improve that then? There was advise on some of related lists (may be even this) to tune following parameters: net/max-buffers=80k net/max-epoch-size=20000 I additionally use disk/c-max-rate=1500m disk/c-fill-target=0 and get >800MB/s writes with protocol C over 10Gbps Ethernet (ConnectX-3 in ethernet or IPoIB mode) with IO depth >=32 and IO size 512k (I use disktest from LTP project). That is what 6 Intel DC S3610 can provide in HW RAID-10 with disk cache turned on. And, of course, Lars is absolutely correct that it is merely impossible to get high IO speeds with just one IO thread. Best, Vladislav