Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi Sam, Im sorry for being late once again, I´m still onvacation. ----- Original Message ----- From: "Art Age Software" <artagesw at gmail.com> To: "Carlos Xavier" <cbastos at connection.com.br> Sent: Friday, January 25, 2008 6:28 PM Subject: Re: [DRBD-user] MySQL-over-DRBD Performance > Hi Carlos, > > Do your Dell PowerEdge SC 1435 servers use the Dell-supplied SAS 5/iR > Adapter adapter? That is what I am using, and my benchmark results are > abysmal. > The system do not have any optional controller. The controller in use is provided by the cipset of the mainboard. The controler is a Broadcom BCM5785 [HT1000] > The following results are direct to disk (no DRBD). > > Without the "oflag=dsync" option, things look fine: > > # dd if=/dev/zero bs=4096 count=10000 of=/tmp/testfile > 10000+0 records in > 10000+0 records out > 40960000 bytes (41 MB) copied, 0.178766 seconds, 229 MB/s > > But look what happens when I add the "oflag=dsync" option: > > # dd if=/dev/zero bs=4096 count=10000 of=/tmp/testfile oflag=dsync > 10000+0 records in > 10000+0 records out > 40960000 bytes (41 MB) copied, 275.086 seconds, 149 kB/s > > What do you make of that? > I cant reproduce your tests righ now, since the system is on production, but soon I´ll be able to. We are migrating the system. > Thanks, > > Sam > Regards, Carlos. > On Jan 23, 2008 11:00 AM, Carlos Xavier <cbastos at connection.com.br> wrote: >> Im sorry for the big delay on the answer, I was on vacation. >> >> I got 2 clusters running, one with Dell PowerEdge SC 1435 and controller >> BCM5785 and another on Dell PowerEdge 1900 with controller SAS1068. >> On both systems the disks used are sata disk WDC WD2500JS. >> >> >> ----- Original Message ----- >> From: "Art Age Software" <artagesw at gmail.com> >> To: <drbd-user at linbit.com> >> >> Sent: Friday, December 21, 2007 6:05 PM >> Subject: Re: [DRBD-user] MySQL-over-DRBD Performance >> >> >> > Well, at least you are getting much better performance than I am >> > getting. >> > >> > I don't understand why even my local write performance is so much >> > worse than yours. What sort of disk subsystem are you using? >> > >> > On Dec 21, 2007 11:52 AM, Carlos Xavier <cbastos at connection.com.br> >> > wrote: >> >> Hi, >> >> I have been following this thread since i want to do a very similar >> >> configuration. >> >> >> >> The system is running on Dell 1435SC each one with 2 dual core AMD >> >> Opteron >> >> and 4GB of ram. >> >> the network cards are: >> >> 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 >> >> Gigabit >> >> Ethernet PCI Express (rev 21) >> >> 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 >> >> Gigabit >> >> Ethernet PCI Express (rev 21) >> >> 06:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit >> >> Ethernet >> >> Controller (Copper) (rev 06) >> >> >> >> Right now it is running a OCFS2 over DRBD and we dont have Myqld >> >> database >> >> over it yet. I run the commands to see the throughput of the write on >> >> the >> >> disk. As you can see bellow is that when the DRBD is up and connected >> >> the >> >> througput fall a litle below the middle of the value we got with it >> >> disconnected. >> >> >> >> DRBD and OCFS2 cluster connected >> >> >> >> root at apolo1:~# dd if=/dev/zero bs=4096 count=10000 >> >> of=/clusterdisk/testfile >> >> oflag=dsync >> >> 10000+0 records in >> >> 10000+0 records out >> >> 40960000 bytes (41 MB) copied, 3.89017 s, 10.5 MB/s >> >> >> >> >> >> DRBD connected and OCFS2 remote disconnected >> >> root at apolo1:~# dd if=/dev/zero bs=4096 count=10000 >> >> of=/clusterdisk/testfile >> >> oflag=dsync >> >> 10000+0 records in >> >> 10000+0 records out >> >> 40960000 bytes (41 MB) copied, 3.65195 s, 11.2 MB/s >> >> >> >> DRBD remote stopped and OCFS2 local mounted >> >> root at apolo1:~# dd if=/dev/zero bs=4096 count=10000 >> >> of=/clusterdisk/testfile >> >> oflag=dsync >> >> 10000+0 records in >> >> 10000+0 records out >> >> 40960000 bytes (41 MB) copied, 1.50187 s, 27.3 MB/s >> >> >> >> Regards, >> >> Carlos. >> >> >> >> >> >> >> >> ----- Original Message ----- >> >> From: "Art Age Software" <artagesw at gmail.com> >> >> To: <drbd-user at linbit.com> >> >> Sent: Thursday, December 20, 2007 7:35 PM >> >> Subject: Re: [DRBD-user] MySQL-over-DRBD Performance >> >> >> >> >> >> > On Dec 20, 2007 1:01 PM, Lars Ellenberg <lars.ellenberg at linbit.com> >> >> > wrote: >> >> >> >> >> >> On Thu, Dec 20, 2007 at 11:08:56AM -0800, Art Age Software wrote: >> >> >> > On Dec 20, 2007 3:05 AM, Lars Ellenberg >> >> >> > <lars.ellenberg at linbit.com> >> >> >> > wrote: >> >> >> > > On Wed, Dec 19, 2007 at 04:41:37PM -0800, Art Age Software >> >> >> > > wrote: >> >> >> > > > I have run some additional tests: >> >> >> > > > >> >> >> > > > 1) Disabled bonding on the network interfaces (both nodes). >> >> >> > > > No >> >> >> > > > significant change. >> >> >> > > > >> >> >> > > > 2) Changed the DRBD communication interface. Was using a >> >> >> > > > direct >> >> >> > > > crossover connection between the on-board NICs of the >> >> >> > > > servers. I >> >> >> > > > switched to Intel Gigabit NIC cards in both machines, >> >> >> > > > connecting >> >> >> > > > through a Gigabit switch. No significant change. >> >> >> > > > >> >> >> > > > 3) Ran a file copy from node1 to node2 via scp. Even with the >> >> >> > > > additional overhead of scp, I get a solid 65 MB/sec. >> >> >> > > > throughput. >> >> >> > > >> >> >> > > this is streaming. >> >> >> > > completely different than what we measured below. >> >> >> > > >> >> >> > > > So, at this stage I have seemingly ruled out: >> >> >> > > > >> >> >> > > > 1) Slow IO subsystem (both machines measured and check out >> >> >> > > > fine). >> >> >> > > > >> >> >> > > > 2) Bonding driver (additional latency) >> >> >> > > > >> >> >> > > > 3) On-board NICs (hardware/firmware problem) >> >> >> > > > >> >> >> > > > 4) Network copy speed. >> >> >> > > > >> >> >> > > > What's left? I'm stumped as to why DRBD can only do about >> >> >> > > > 3.5 >> >> >> > > > BM/sec. >> >> >> > > > on this very fast hardware. >> >> >> > > >> >> >> > > doing one-by-one synchronous 4k writes, which are latency >> >> >> > > bound. >> >> >> > > if you do streaming writes, it probably get up to your 65 >> >> >> > > MB/sec >> >> >> > > again. >> >> >> > >> >> >> > Ok, but we have tested that with and without DRBD by the dd >> >> >> > command, >> >> >> > right? So at this point, by all tests performed so far, it looks >> >> >> > like >> >> >> > DRBD is the bottleneck. What other tests can I perform that can >> >> >> > say >> >> >> > otherwise? >> >> >> >> >> >> sure. >> >> >> but comparing 3.5 (with drbd) against 13.5 (without drbd) is bad >> >> >> enough, >> >> >> no need to now compare it with some streaming number (65) to make >> >> >> it >> >> >> look _really_ bad ;-) >> >> > >> >> > Sorry, my intent was not to make DRBD look bad. I think DRBD is >> >> > **fantastic** and I just want to get it working properly. My point >> >> > in >> >> > trying the streaming test was simply to make sure that there was >> >> > nothing totally broken on the network side. I suppose I should also >> >> > try a streaming test to the DRBD device and compare that to the raw >> >> > streaming number. And, back to my last question: What other tests >> >> > can >> >> > I perform at this point to narrow down the source of the (latency?) >> >> > problem? >> >> >> >> > _______________________________________________ >> >> > drbd-user mailing list >> >> > drbd-user at lists.linbit.com >> >> > http://lists.linbit.com/mailman/listinfo/drbd-user >> >> > >> >> >> >> _______________________________________________ >> >> drbd-user mailing list >> >> drbd-user at lists.linbit.com >> >> http://lists.linbit.com/mailman/listinfo/drbd-user >> >> >> > _______________________________________________ >> > drbd-user mailing list >> > drbd-user at lists.linbit.com >> > http://lists.linbit.com/mailman/listinfo/drbd-user >> > >> >> >> _______________________________________________ >> drbd-user mailing list >> drbd-user at lists.linbit.com >> http://lists.linbit.com/mailman/listinfo/drbd-user >> >