Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi Lars, 22.10.2015 17:57, Lars Ellenberg wrote: > On Wed, Oct 21, 2015 at 11:42:52AM +0300, Vladislav Bogdanov wrote: >>>> Throughput is concurrency times request size by request completion latency. >>>> >>>> dd has no concurrency, in the given example, request size is 1M, >>>> if you get 450 MB/s, your latency apparently is in the order of 2.3 ms. >>>> >>>> If you want more throughput, >>>> you need to decrease latency, or increase concurrency. >>> >>> Any suggestions on how to improve that then? >> >> There was advise on some of related lists (may be even this) to tune >> following parameters: >> net/max-buffers=80k >> net/max-epoch-size=20000 > > Tuning max-buffers is certainly needed if you have a capable backend and > replication link. > > Epoch size as only a very minor effect, typical epoch sizes will be > rather in the small hundreds at most, anyways, so the > new-epoch-because-we-exceeded-the-configured-maximum will very > likely never be hit. I can't recall details right now, but I tried several settings and this one was chosen. I'll do more testing (on a capable HW) in next few months. > >> I additionally use >> disk/c-max-rate=1500m >> disk/c-fill-target=0 > > I strongly recommend using c-fill-target, and not c-delay target, > but you may need to tune it to a value suitable for your setup. The point was to provide/restore redundancy as fast as possible after second node is powered on in the price of app speed. I played several days with different settings, but this one was optimal for my task. I got more than 1GB/s on initial sync - more than disks can provide on writes (because of read-read verification with SSDs?). > > c-max-rate is by default at 100M, so yes, if you want more than that, > you need to change it. > > To clarify, > both are meaningful only in the context of a *resync*, > so are meaningless for *replication*, *cannot* have any effect > on the appliaction throughput during normal operation. Sure. > >> and get >800MB/s writes with protocol C over 10Gbps Ethernet >> (ConnectX-3 in ethernet or IPoIB mode) with IO depth >=32 and IO >> size 512k (I use disktest from LTP project). That is what 6 Intel DC >> S3610 can provide in HW RAID-10 with disk cache turned on. >> >> And, of course, Lars is absolutely correct that it is merely >> impossible to get high IO speeds with just one IO thread. > Best, Vladislav