Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Mon, Dec 17, 2007 at 11:28:41PM -0800, Art Age Software wrote: > > About the performance issues you mentioned. The performance tests I've > > conducted were run using mysqlslap, and backed up using some lower-level > > latency tests, but test results were similar initially. Going back to the > > test results you've reported: > > > > > Old Servers (2 single-core CPUs): > > > Database on Local Storage > > > insert test: 212.00 sec. > > > Database on DRBD Device > > > insert test: 998.00 sec. > > > ---------------------------- > > > DRBD Overhead: 786 sec. = 370% > > > > Eliminate competition for CPU resources between DRBD an MySQL, by pinning > > DRBD's kernel threads to one CPU core and mysqld on the other, like so > > (assuming drbd0 is your device you're running your MySQL databases on): > > > > for thread in worker asender receiver; do > > taskset -p 0x01 $(pidof drbd0_$thread) > > done > > taskset -p 0x02 $(pidof mysqld) > > > > If you're unfamiliar with taskset, be sure to read its man page to understand > > what those CPU affinity masks mean. > > > > Do that, then re-run your tests (on your old server) and share your results. > > > > That tweak alone reduced my DRBD overhead for mysqlslap from 224% to 57% on > > the test system I have at my disposal. > > > > > New Servers (2 quad-core CPUs): > > > Database on Local Storage > > > insert test: 164.00 sec. (22% better than old servers) > > > Database on DRBD Device > > > insert test: 1137.00 sec. (14% *worse* than old servers) > > > ---------------------------- > > > DRBD Overhead: 973 sec. = 590% > > > > Yup, now given the CPU resource competition issues described earlier, these > > are probably being exacerbated by the fact that there are now 8 logical > > CPUs (cores) available, versus 2 on your old server. You can do one of two > > things here: > > > > 1. Pin the DRBD threads to one core, and allocate the others to MySQL. > > 2. Pin the DRBD threads to one core, and allocate only a few of the others to > > MySQL. I've heard some SMP issues are present in InnoDB; more cores doesn't > > necessarily mean better performance. > > > > There are other settings that can be tweaked (I can come down to about 26% > > overhead on my system, which is about as low as the network will let me), but > > I'd be interested to learn whether you can confirm my results with regard to > > CPU affinity tweaking. > > OK, so I have re-run the benchmarks after pinning the DRBD threads to > one core in the first CPU and the mysqld process to 4 cores in the > second CPU like so: > > taskset -p 0x80 (pids of drbd threads) > taskset -p 0x0f (pid of mysqld) > > The benchmark result improved, but not as dramatically as I had hoped: > > Database on DRBD Device > insert test: 938.00 sec. > > Should I have expected a more dramatic improvement? What else can I do > to get to the bottom of the poor performance of DRBD in my setup? > Thoughts? please do one-node# ping -w 10 -f -s 4100 replication-link-ip-of-other-node and show me the output. -- : Lars Ellenberg http://www.linbit.com : : DRBD/HA support and consulting sales at linbit.com : : LINBIT Information Technologies GmbH Tel +43-1-8178292-0 : : Vivenotgasse 48, A-1120 Vienna/Europe Fax +43-1-8178292-82 : __ please use the "List-Reply" function of your email client.