Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, May 1, 2009 at 1:16 PM, Gennadiy Nerubayev <parakie at gmail.com>wrote: > On Fri, May 1, 2009 at 10:34 AM, Lars Ellenberg <lars.ellenberg at linbit.com > > wrote: > >> what is your micro benchmark? > > > iometer.. which is not particularly micro :p > > for sequential write throughput micro benchmark, >> I suggest >> >> dd if=/dev/zero of=/dev/drbdX bs=4M count=1000 oflag=direct >> >> do variations in bs= and count= (to reveal possibly issues >> with cpu cache sizes). >> >> also do variations of >> oflag=direct >> no page cache/buffer cache involved, >> oflag=dsync >> completely through buffer cache/page cache, >> but does the equivalent of "fsync" for every "bs" >> no oflag, but conv=fsync >> completely through buffer cache/page cache, >> and does a real fsync only once all count * bs >> blocks are written >> >> smalish bs (< the size of your cpu cache), say bs=32k, high count, >> and oflag=direct is what is most like what the resync is doing. > > > I did a number of dd runs; the results are attached. The 32k direct writes > are the worst when connected. > > >> you can also start pinning your "dd" to a single cpu, >> preferably the same your DRBD kernel threads are running on. >> or allow only the first two cores, or whatever. > > > The cpu (single) in both boxes is dual non-hyperthreaded core. I can repeat > the benchmarks on one core - would passing nosmp and maxcpus=1 to the kernel > be sufficient for this test case? > Woops, would be nice if I actually attached a file. -Gennadiy -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20090501/22c3250c/attachment.htm> -------------- next part -------------- Connected, 4M: dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 18.3088 seconds, 229 MB/s dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 oflag=direct 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 17.6633 seconds, 237 MB/s dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 oflag=dsync 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 24.503 seconds, 171 MB/s dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 conv=fsync 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 18.5762 seconds, 226 MB/s Connected, 32k: dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 12.6497 seconds, 259 MB/s dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 oflag=direct 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 35.4416 seconds, 92.5 MB/s dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 oflag=dsync 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 54.9502 seconds, 59.6 MB/s dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 conv=fsync 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 11.9722 seconds, 274 MB/s Disconnected, 4M: dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 10.7446 seconds, 390 MB/s dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 oflag=direct 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 11.7017 seconds, 358 MB/s dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 oflag=dsync 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 15.0324 seconds, 279 MB/s dd if=/dev/zero of=/dev/drbd0 bs=4M count=1000 conv=fsync 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 10.7923 seconds, 389 MB/s Disconnected, 32k: dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 7.56056 seconds, 433 MB/s dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 oflag=direct 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 16.8983 seconds, 194 MB/s dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 oflag=dsync 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 21.2287 seconds, 154 MB/s dd if=/dev/zero of=/dev/drbd0 bs=32k count=100000 conv=fsync 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 7.6951 seconds, 426 MB/s Just in case on the underlying device: dd if=/dev/zero of=/dev/sdb bs=4M count=1000 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 9.48327 seconds, 442 MB/s dd if=/dev/zero of=/dev/sdb bs=4M count=1000 oflag=direct 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 9.93983 seconds, 422 MB/s dd if=/dev/zero of=/dev/sdb bs=4M count=1000 oflag=dsync 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 14.3305 seconds, 293 MB/s dd if=/dev/zero of=/dev/sdb bs=4M count=1000 conv=fsync 1000+0 records in 1000+0 records out 4194304000 bytes (4.2 GB) copied, 9.46309 seconds, 443 MB/s dd if=/dev/zero of=/dev/sdb bs=32k count=100000 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 7.1826 seconds, 456 MB/s dd if=/dev/zero of=/dev/sdb bs=32k count=100000 oflag=direct 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 16.1541 seconds, 203 MB/s dd if=/dev/zero of=/dev/sdb bs=32k count=100000 oflag=dsync 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 19.1514 seconds, 171 MB/s dd if=/dev/zero of=/dev/sdb bs=32k count=100000 conv=fsync 100000+0 records in 100000+0 records out 3276800000 bytes (3.3 GB) copied, 7.46252 seconds, 439 MB/s