Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
> Does 8.3 behave the same?
No. See below.
> How does dm-linear compare?
For this I'll show stats with just straight up lvm on sda below.
In short, it does not experience the issue.
> You did of course double-check the read-ahead settings?
We hadn't touched the read-ahead settings, but doing so has some
interesting results.
via:
blockdev --setra ${READ_AHEAD} ${device}
echo 3 > /proc/sys/vm/drop_caches
dd if=${device} of=/dev/null bs=1M count=10240
(lvm on drbd on sda)
(drbd on sda)
(lvm on sda)
(sda)
Note: all values in MBytes/s
READ_AHEAD LVM_DRBD_8.4.1 DRBD_8.4.1 DRBD_8.3.12 LVM_SDA SDA
0 1 1 110 110 120
256 (default) 60 60 830 850 900
512 130 130 930 930 930
727 160 --> 190 940 930 930
728 690 --> 800 930 940 930
1024 820 850 930 930 930
2048 840 850 940 950 950
iflag=direct 770 900 820 920 900
Going from 727 read-ahead to 728 read-ahead is a magic tipping point
resulting in 4.3x performance on DRBD 8.4.1, while DRBD 8.3.12 is fast for
all read-ahead values.
Small File Reads
----------------
While initially this suggests a solution to our performance issue on 8.4.1,
it unfortunately does not resolve the same problem with small-file reads, where
read-ahead has a much smaller effect.
ie. small-file test data (1.4 gb, 1472 dirs, 6942 files, 200k avg file size)
via:
blockdev --setra ${READ_AHEAD} ${device}
echo 3 > /proc/sys/vm/drop_caches
time find /mnt -type f | while read a; do cat "$a" > /dev/null; done
(ext4 on lvm on drbd on sda)
(ext4 on drbd on sda)
(ext4 on lvm on sda)
(ext4 on sda)
Note: all values in seconds, lower is better
READ_AHEAD LVM_DRBD_8.4.1 DRBD_8.4.1 DRBD_8.3.12 LVM_SDA SDA
0 2725.13 2712.05 46.78 32.21 32.56
256 (default) 58.72 --> 58.35 --> 23.62 20.54 23.12
512 46.40 45.94 24.09 21.18 23.66
727 45.21 47.29 24.86 21.84 24.08
728 39.58 45.99 24.99 21.53 23.86
1024 40.70 --> 41.96 --> 23.60 19.74 21.63
DRBD 8.3.12 is around ~2x faster than 8.4.1 for most read-ahead values.
Also, of note, some particularly bad mojo occurs in 8.4.1 with a read-ahead
of zero.
/ca
On Fri, Feb 24, 2012 at 10:47:18AM +0100, drbd-user-request at lists.linbit.com's all...
>
>
> Did you watch with "iostat -xk 1" while doing the test?
> Ignore the utilization column.
> Focus on differences in the sda row between tests.