[DRBD-user] MySQL-over-DRBD Performance

[Lars Ellenberg]

On Mon, Dec 17, 2007 at 11:28:41PM -0800, Art Age Software wrote:
> > About the performance issues you mentioned. The performance tests I've
> > conducted were run using mysqlslap, and backed up using some lower-level
> > latency tests, but test results were similar initially. Going back to the
> > test results you've reported:
> >
> > > Old Servers (2 single-core CPUs):
> > >       Database on Local Storage
> > >               insert test:  212.00 sec.
> > >       Database on DRBD Device
> > >               insert test:  998.00 sec.
> > >       ----------------------------
> > >       DRBD Overhead:  786 sec. = 370%
> >
> > Eliminate competition for CPU resources between DRBD an MySQL, by pinning
> > DRBD's kernel threads to one CPU core and mysqld on the other, like so
> > (assuming drbd0 is your device you're running your MySQL databases on):
> >
> > for thread in worker asender receiver; do
> >   taskset -p 0x01 $(pidof drbd0_$thread)
> > done
> > taskset -p 0x02 $(pidof mysqld)
> >
> > If you're unfamiliar with taskset, be sure to read its man page to understand
> > what those CPU affinity masks mean.
> >
> > Do that, then re-run your tests (on your old server) and share your results.
> >
> > That tweak alone reduced my DRBD overhead for mysqlslap from 224% to 57% on
> > the test system I have at my disposal.
> >
> > > New Servers (2 quad-core CPUs):
> > >       Database on Local Storage
> > >               insert test:  164.00 sec.       (22% better than old servers)
> > >       Database on DRBD Device
> > >               insert test:  1137.00 sec.      (14% *worse* than old servers)
> > >       ----------------------------
> > >       DRBD Overhead:  973 sec. = 590%
> >
> > Yup, now given the CPU resource competition issues described earlier, these
> > are probably being exacerbated by the fact that there are now 8 logical
> > CPUs (cores) available, versus 2 on your old server. You can do one of two
> > things here:
> >
> > 1. Pin the DRBD threads to one core, and allocate the others to MySQL.
> > 2. Pin the DRBD threads to one core, and allocate only a few of the others to
> > MySQL. I've heard some SMP issues are present in InnoDB; more cores doesn't
> > necessarily mean better performance.
> >
> > There are other settings that can be tweaked (I can come down to about 26%
> > overhead on my system, which is about as low as the network will let me), but
> > I'd be interested to learn whether you can confirm my results with regard to
> > CPU affinity tweaking.
> 
> OK, so I have re-run the benchmarks after pinning the DRBD threads to
> one core in the first CPU and the mysqld process to 4 cores in the
> second CPU like so:
> 
> taskset -p 0x80 (pids of drbd threads)
> taskset -p 0x0f (pid of mysqld)
> 
> The benchmark result improved, but not as dramatically as I had hoped:
> 
>        Database on DRBD Device
>               insert test:  938.00 sec.
> 
> Should I have expected a more dramatic improvement? What else can I do
> to get to the bottom of the poor performance of DRBD in my setup?
> Thoughts?

please do
  one-node# ping -w 10 -f -s 4100 replication-link-ip-of-other-node
and show me the output.

-- 
: Lars Ellenberg                           http://www.linbit.com :
: DRBD/HA support and consulting             sales at linbit.com :
: LINBIT Information Technologies GmbH      Tel +43-1-8178292-0  :
: Vivenotgasse 48, A-1120 Vienna/Europe     Fax +43-1-8178292-82 :
__
please use the "List-Reply" function of your email client.