Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, May 01, 2009 at 01:17:41PM -0400, Gennadiy Nerubayev wrote: > > I did a number of dd runs; the results are attached. The 32k direct writes > > are the worst when connected. yes, that is expected... the resync does 32K requests, but it has thousands of them in flight, while dd always has only one. the 4M odirect would be what I expect to be best (assuming your cpu cache is >= 4M), which is why I suggested that in the explicitly given dd line. > >> you can also start pinning your "dd" to a single cpu, > >> preferably the same your DRBD kernel threads are running on. > >> or allow only the first two cores, or whatever. > > > > > > The cpu (single) in both boxes is dual non-hyperthreaded core. I can repeat > > the benchmarks on one core - would passing nosmp and maxcpus=1 to the kernel > > be sufficient for this test case? man taskset ;) but, yes. > Woops, would be nice if I actually attached a file. > Connected, 4M: > > dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 > 4194304000 bytes (4.2 GB) copied, 18.3088 seconds, 229 MB/s nice. you are going into page cache here, mostly (how much RAM did you say you have?) and still "only" 230. > dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 oflag=direct > 4194304000 bytes (4.2 GB) copied, 17.6633 seconds, 237 MB/s do variations. use 512k, 1M, 2M, 100M. don't just use two samle points. cover the range. find the "optimum". repeat measurements. see how reproducable they are. also read up about the "al-extents" drbd.conf parameter, "max-buffers", and the rest of the usual tuning stuff. the activity log updates will slow you down on streaming writes. > dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 conv=fsync > 4194304000 bytes (4.2 GB) copied, 10.7923 seconds, 389 MB/s so this is what your box can handle. what is your rtt, small packets, large packets? ping -w 5 -f -s 64 other-node ping -w 5 -f -s 5000 other-node > Disconnected, 32k: > dd if=/dev/zero of=/dev/sdb bs=4M count=1000 oflag=direct > 4194304000 bytes (4.2 GB) copied, 9.93983 seconds, 422 MB/s > > dd if=/dev/zero of=/dev/sdb bs=4M count=1000 oflag=dsync > 4194304000 bytes (4.2 GB) copied, 14.3305 seconds, 293 MB/s > > dd if=/dev/zero of=/dev/sdb bs=4M count=1000 conv=fsync > 4194304000 bytes (4.2 GB) copied, 9.46309 seconds, 443 MB/s so we would expect the "conv=fsync" to be best for connected mode as well, even though (or because?) it goes through the buffer cache... -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed