<div class="gmail_quote">On Sat, May 9, 2009 at 6:14 AM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On Fri, May 08, 2009 at 06:27:36PM -0400, Gennadiy Nerubayev wrote:<br>
> Possibly related to my earlier post, but this time I focused on two things<br>
> primarily: SSDs and metadata.<br>
><br>
> Hardware: DRBD on top of 1 X25-e in each of the DRBD nodes, configured as<br>
> simple volume on a hardware raid controller, and connected via IP over<br>
> infiniband.<br>
> Workload: 100% random io writes, 8KB block size<br>
><br>
> Direct to the disk: 77.78MB/s, average latency 2ms<br>
> Disconnected DRBD, metadata on ramdisk: 75.64MB/s, average latency 2ms<br>
> Connected DRBD, metadata on ramdisk: 50.87MB/s, average latency 4ms<br>
> Disconnected DRBD, metadata internal: 6.25MB/s, average latency 39ms<br>
> Connected DRBD, metadata internal: 6.20MB/s, average latency 39ms<br>
> Disconnected DRBD, metadata on a single 15K sas disk: 43.46MB/s, average<br>
> latency 5ms<br>
> Connected DRBD, metadata on a single 15K sas disk: 39.32MB/s, average<br>
> latency 5ms<br>
<br>
</div>Could you add a "dm linear" to the table?<br>
i.e. just checking if just one small layer of "virtual" block device<br>
has any effect on throughput and or latency.<br>
dmsetup create experiment <<<"0 $[20 <<21] linear /dev/sdX"<br>
then do your benchmark against /dev/mapper/experiment<br>
(just to see if that performs any different than "raw" /dev/sdx)</blockquote><div> <br>Doesn't seem to work:<br>dmsetup create experiment <<< "0 $[20 <<21] linear /dev/sdb"<br>device-mapper: reload ioctl failed: Invalid argument<br>
Command failed<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">> Full resync speeds for all tests were 180-200MB/s (about what's expected<br>
<div class="im">
> from a single x25-e). There is no difference between flexible and regular<br>
> metadata for internal or external usage (metadata was recreated for those<br>
> tests). Interestingly, ~6MBs is the same speed that I got when testing a 7<br>
> disk raid 0 15K sas array with internal metadata (everything else is the<br>
> same) and putting metadata on ramdisk moved that up to ~35MB/s.<br>
><br>
> So for some reason even in the case of a very fast SSD, internal metadata<br>
> performance for random writes is really bad. Putting it on any kind of<br>
> external disk brings an immediate exponential performance increase..<br>
<br>
</div>IO scheduler? try deadline and noop.<br>
echo deadline > /sys/block/sdX/queue/scheduler</blockquote><div><br>Oooh, aaah:<br><br>Disconnected DRBD, metadata internal, deadline: 42.59MB/s, average latency 5ms<br>Connected DRBD, metadata internal, deadline: 39.34MB/s, average latency 5ms<br>
Disconnected DRBD, metadata internal, noop: 44.06MB/s, average latency 5ms<br>Connected DRBD, metadata internal, noop: 38.61MB/s, average latency 5ms<br><br>After running the above deadline and noop tests, I went back to cfq and got identical results as in the original test (~6MB/s). Kernel for reference is 2.6.29.1.<br>
<br>-Gennadiy<br></div></div>