Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
I was a day at an IBM site here in Vienna and had the chance
to measure DRBD's performance on two new ppc64 boxes.
The hardware I had was one OpenPower 720
4 CPUs 1.5 GHz, SMT enabled -> 8 logic CPUs; 4GB Ram
Disks: IBM VSBPD4E1 U4SCSI (10000rpm U320 SCSI 73.4GB)
HUS103073FL3800 part no:26K5191 Feat: 3274
An a p520
2 CPUs 1.65 GHz; 6GB Ram. For the tests a micro
partition was used: 0.3 CE uncapped; 200MB Ram later 3GB Ram
Software: SLES9 ( -> Linux 2.6.something )
DRBD-0.7.10
The network used for mirroring was a direct cable, attached
by two intel 82546EB gigabit controller to the machines.
The raw throughput measured by dm -a 0 -b 1M -sXX -y -p | netcat ...
100.62 MB/sec (536870912 B / 00:05.088569)
100.73 MB/sec (10737418240 B / 01:41.661196)
I created two partitions on the 73 GB SCSI disks, so I was
able to measure the throughput on the inner cylinders and the
middle cylinders of the disk:
command ./dm -a 0 -b 1M -o /dev/sdb2 -s 2G -y -p
inner, /dev/sdX:
59.51 MB/sec (1073741824 B / 00:17.206547)
60.52 MB/sec (2147483648 B / 00:33.840951)
inner, /dev/drbdX, disconnected:
60.08 MB/sec (1073741824 B / 00:17.043908)
60.68 MB/sec (2147483648 B / 00:33.748231)
middle, /dev/sdX:
51.60 MB/sec (2147483648 B / 00:39.687503)
51.98 MB/sec (3221225472 B / 00:59.098640)
middle, /dev/drbdX, disconnected:
50.44 MB/sec (1073741824 B / 00:20.302781)
50.21 MB/sec (1073741824 B / 00:20.394660)
and on the other machine (the micro partition):
inner, /dev/sdX:
59.12 MB/sec (1073741824 B / 00:17.319759)
59.09 MB/sec (2147483648 B / 00:34.658179)
middle, /dev/sdX:
54.60 MB/sec (2147483648 B / 00:37.509743)
54.83 MB/sec (4294967296 B / 01:14.709334)
So we have two machines with about the same throughput to
the disks. The partitioned p520 also has virtual disks, which
are provided by a so called VIO - Server, which is basically
an AIX instance.
virtual disk: [200M]
29.26 MB/sec (1073741824 B / 00:34.992719)
29.11 MB/sec (1073741824 B / 00:35.171781)
other virtual disk: (via FS) [200M]
32.86 MB/sec (536870912 B / 00:15.583470)
30.74 MB/sec (536870912 B / 00:16.657706)
virtual disk: [3GB]
74.26 MB/sec (1073741824 B / 00:13.789230)
76.71 MB/sec (2147483648 B / 00:26.698629)
virtual disk, disconnected: [3GB]
71.79 MB/sec (536870912 B / 00:07.132181)
70.31 MB/sec (1073741824 B / 00:14.563278)
73.32 MB/sec (1073741824 B / 00:13.966345)
Interestingly the IO throughput of the virtual disks depended
heavily on the RAM that was allocated to that micro partition,
whereas IO throughput to the physical disks was independent of
how mach RAM was allocated to Linux.
The first values we got with DRBD where rather poor, but soon
it turned out that this IBM storage controllers needs like to be
overwhelmed with IO requests to deliver good performance.
I guess that these storage controllers are doing some disk
scheduling by themselves. In order to get them to write immediately
you need to fill them up with requests ]
special DRBD configuration:
max-buffers 16384;
max-epoch-size 14336;
Usually drbdadm allows only up to 10000 buffers, this limit
was raised on site using gcc :) And it will be raised in subsequent
DRBD releases.
inner (mirrored with protocol C):
54.81 MB/sec (1073741824 B / 00:18.682203)
55.37 MB/sec (1073741824 B / 00:18.493835)
Resync: total 40 sec; paused 0 sec; 52428 K/sec
middle (mirrored with protocol C):
49.69 MB/sec (1073741824 B / 00:20.607737)
49.85 MB/sec (1073741824 B / 00:20.542222)
Resync: total 62 sec; paused 40 sec; 47660 K/sec
This are the highest numbers I have ever seen. They are
only about 8% below raw disk performance. While the measured
inhomogeneity of the disk was over 8% on both machines!
It needs to be noted that the mirroring performance using one of
the virtual disks on the p520 was in the area of 12MB/sec. We
were not able to find the reason for this unexpected low performance
in the limited time.
-philipp
--
: Dipl-Ing Philipp Reisner Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schönbrunnerstr 244, 1120 Vienna, Austria http://www.linbit.com :