Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Wed, Sep 24, 2008 at 07:05:21PM -0500, eBoundHost: Artur wrote: > ----- "Trent Lloyd" wrote: > On 25/09/2008, at 7:10 AM, eBoundHost: Artur wrote: > | Putting together a system to compete with netapp and bluearc. I say 90% > | of it can be done with drbd and supermicro + LVM + ext3. > | > | Problem: slow write when server1 and server2 are connected. > | > | Here are my benchmarks: > | dd if=/dev/zero bs=4096 count=10000 oflag=dsync of=/data/file1 > | > | DRBD Active and connected: > | 40960000 bytes (41 MB) copied, 24.8387 seconds, 1.6 MB/s > | > | DRBD Active and NOT connected: > | 40960000 bytes (41 MB) copied, 10.8259 seconds, 3.8 MB/s > | > | File system partition, NON DRBD, same disk: > | 40960000 bytes (41 MB) copied, 10.3686 seconds, 4.0 MB/s > | > | These are very consistent and repeatable. so you have latencies [in ms per 4k sync IO] of 1.3 ms on "raw disk" (lv?) 1.8 ms on drbd, unconnected "same disk": other lv, other offset we have seen large differences on the same disk but at different offsets, the "inner" and "outer" "cylinders" can have very different latencies and throughput. but yes, drbd itself also incurs some overhead. should not be 40 % though, in my experience it is typically < 2%. 2.4 ms on drbd, connected which means you have a network roundtrip time of ~ 0.6 ms. you have (2.4 - 1.3)/1.3, ca. 85% degradation against "local disk only" so there is much room for improvement. first, get a decent battery backed non-volatile write cache on the controller, and enable it. (and disable the volatile write caches on the disks themselves) that should get your local latency down to 70µs (factor 20 against what you have now). typical direct link ethernet rtt should be ~ 0.2 ms or less (factor 3 better than what you apparently have now; if you get down to 0.05, which should be possible for a direct link, that is a factore of 12). so tune there as well. should get you down to about 0.2 to 0.5 ms on DRBD connected, depending on controller and interconnect type/topology. which is a lot better than what you have now "local disk only". of course, when staying with the same interconnect, keeping the _absolute_ network latency _constant_, the _relative_ latency overhead of the network will _increase_ as the _absolute_ latency of the local storages _decrease_. but the total latency will decrease as well, obviously. and, please DO NOT measure latency in MB/s. that unit should be used for the throughput aspect. the unit for the latency aspect is time per synchronous IO-operation, so ms or µs per synchronous small io. right now I exercise drbd on a combination of various interconnects (dolphin DX, 1GbE, bonded 2x 1GbE, 10GbE) with different adaptec raid controllers. what I see is latency for synchronous single sector writes (depending on controller and probably MoBo/CPU frequency/bus bandwidth) of 120 µs or 65µs to local raid controller, and 130 µs to 300 µs network penalty (measured with "null-io", drbd on top of dm-zero target ;->, meta data on ramdisk), resulting in 200 µs to 380 µs on connected drbd, depending on controller and interconnect combination. the worst value for connected drbd there is still a factor 4 better than your "local disk". your local disk alone is 350 % worse than my connected drbd. (650% if I use my best combination). what I wanted to say: if you think drbd is too slow, latency with drbd is too high, that is NOT a limitation of drbd. but your combination of hardware and network topology is the limiting factor. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed