Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, May 08, 2009 at 06:27:36PM -0400, Gennadiy Nerubayev wrote: > Possibly related to my earlier post, but this time I focused on two things > primarily: SSDs and metadata. > > Hardware: DRBD on top of 1 X25-e in each of the DRBD nodes, configured as > simple volume on a hardware raid controller, and connected via IP over > infiniband. > Workload: 100% random io writes, 8KB block size > > Direct to the disk: 77.78MB/s, average latency 2ms > Disconnected DRBD, metadata on ramdisk: 75.64MB/s, average latency 2ms > Connected DRBD, metadata on ramdisk: 50.87MB/s, average latency 4ms > Disconnected DRBD, metadata internal: 6.25MB/s, average latency 39ms > Connected DRBD, metadata internal: 6.20MB/s, average latency 39ms > Disconnected DRBD, metadata on a single 15K sas disk: 43.46MB/s, average > latency 5ms > Connected DRBD, metadata on a single 15K sas disk: 39.32MB/s, average > latency 5ms Could you add a "dm linear" to the table? i.e. just checking if just one small layer of "virtual" block device has any effect on throughput and or latency. dmsetup create experiment <<<"0 $[20 <<21] linear /dev/sdX" then do your benchmark against /dev/mapper/experiment (just to see if that performs any different than "raw" /dev/sdx) > Full resync speeds for all tests were 180-200MB/s (about what's expected > from a single x25-e). There is no difference between flexible and regular > metadata for internal or external usage (metadata was recreated for those > tests). Interestingly, ~6MBs is the same speed that I got when testing a 7 > disk raid 0 15K sas array with internal metadata (everything else is the > same) and putting metadata on ramdisk moved that up to ~35MB/s. > > So for some reason even in the case of a very fast SSD, internal metadata > performance for random writes is really bad. Putting it on any kind of > external disk brings an immediate exponential performance increase.. IO scheduler? try deadline and noop. echo deadline > /sys/block/sdX/queue/scheduler thanks for feedback. -- : Lars Ellenberg : LINBIT HA-Solutions GmbH : DRBD®/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed