Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Art Age Software wrote, On 12/18/2007 02:28 AM: >> About the performance issues you mentioned. The performance tests I've >> conducted were run using mysqlslap, and backed up using some lower-level >> latency tests, but test results were similar initially. Going back to the >> test results you've reported: >> >>> Old Servers (2 single-core CPUs): >>> Database on Local Storage >>> insert test: 212.00 sec. >>> Database on DRBD Device >>> insert test: 998.00 sec. >>> ---------------------------- >>> DRBD Overhead: 786 sec. = 370% >> Eliminate competition for CPU resources between DRBD an MySQL, by pinning >> DRBD's kernel threads to one CPU core and mysqld on the other, like so >> (assuming drbd0 is your device you're running your MySQL databases on): >> >> for thread in worker asender receiver; do >> taskset -p 0x01 $(pidof drbd0_$thread) >> done >> taskset -p 0x02 $(pidof mysqld) >> >> If you're unfamiliar with taskset, be sure to read its man page to understand >> what those CPU affinity masks mean. >> >> Do that, then re-run your tests (on your old server) and share your results. >> >> That tweak alone reduced my DRBD overhead for mysqlslap from 224% to 57% on >> the test system I have at my disposal. >> >>> New Servers (2 quad-core CPUs): >>> Database on Local Storage >>> insert test: 164.00 sec. (22% better than old servers) >>> Database on DRBD Device >>> insert test: 1137.00 sec. (14% *worse* than old servers) >>> ---------------------------- >>> DRBD Overhead: 973 sec. = 590% >> Yup, now given the CPU resource competition issues described earlier, these >> are probably being exacerbated by the fact that there are now 8 logical >> CPUs (cores) available, versus 2 on your old server. You can do one of two >> things here: >> >> 1. Pin the DRBD threads to one core, and allocate the others to MySQL. >> 2. Pin the DRBD threads to one core, and allocate only a few of the others to >> MySQL. I've heard some SMP issues are present in InnoDB; more cores doesn't >> necessarily mean better performance. >> >> There are other settings that can be tweaked (I can come down to about 26% >> overhead on my system, which is about as low as the network will let me), but >> I'd be interested to learn whether you can confirm my results with regard to >> CPU affinity tweaking. > > OK, so I have re-run the benchmarks after pinning the DRBD threads to > one core in the first CPU and the mysqld process to 4 cores in the > second CPU like so: > > taskset -p 0x80 (pids of drbd threads) > taskset -p 0x0f (pid of mysqld) > > The benchmark result improved, but not as dramatically as I had hoped: > > Database on DRBD Device > insert test: 938.00 sec. > > Should I have expected a more dramatic improvement? What else can I do > to get to the bottom of the poor performance of DRBD in my setup? > Thoughts? > > Sam out of curiosity, does spreading pining the drbd threads to multiple cores in the first CPU work better or worse than pinning them all to just one core in that CPU? i.e., (_assuming_ CPU0 holds cores 0x80, 0x40, 0x20 & 0x10) taskset -p 0x80 drbd_pid_1 taskset -p 0x40 drbd_pid_2 taskset -p 0x20 drbd_pid_3 taskset -p 0x10 drbd_pid_4 taskset -p 0x80 drbd_pid_5 taskset -p 0x40 drbd_pid_6 taskset -p 0x20 drbd_pid_7 taskset -p 0x10 drbd_pid_8 ... and did you try Florian's second suggestion of pinning all of the MySQLs to _one_core_ in _one_CPU_? taskset -p 0x02 (pid of mysqld) That is would you be kind enough to run two or three more test sets? -- Todd Denniston Crane Division, Naval Surface Warfare Center (NSWC Crane) Harnessing the Power of Technology for the Warfighter