Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Found out what was wrong, or at least we've identified the problem, but not yet fixed it. One of the storage servers has a habit of booting up and delivering just over half the I/O performance it should. We rebooted both boxes 10 times and ran the same command on each : "dd if=/dev/zero of=/dev/VolGroup0/LogVol0 bs=1G count=10 oflag=direct" One of the boxes delivered 340MB/s each time. The other had 5 instances of 340MB/s and 5 instances of 185MB/s. Very odd indeed but it explains things. We tested drbd sync performance on occasions when the box delivers 340MB/s and found it to be as before. So - we'll fix that and then probably add the second CPUs to reduce CPU contention. -----Original Message----- From: drbd-user-bounces at lists.linbit.com [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Lee Christie Sent: 03 July 2008 15:19 To: drbd-user at lists.linbit.com Subject: RE: [DRBD-user] Odd observation when sync'ing Well yes clearly 300MB/s is the maximum writes, however my point was that two of the drbd resources were sync'ing very slowly compared to the others. When we looked at top, we noticed these two down the bottom of the receiver processes. Using renice I bumped them up and they increased. So we suspected CPU contention in as much that the writes weren't evenly spread. Now they are spread evenly, even using a single CPU, but we've somehow lost a whole load of performance. On the first run I also found (using sar) that both system and iowait were quite high (>40%) and the system load on the box doing the writes was over 6 ! it was a box under high IO make no mistake. However now on a resync the %system is 15%, there is no iowait and the load is 0.5. So definitely, something in the first scenario made the storage subsystem go as fast and hard as it could, and now its not. Baffling. > -----Original Message----- > From: drbd-user-bounces at lists.linbit.com > [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Florian Haas > Sent: 03 July 2008 15:12 > To: drbd-user at lists.linbit.com > Subject: Re: [DRBD-user] Odd observation when sync'ing > > Lee Christie wrote: > > Whats even odder is that looking at top we suspected CPU > contention (the > > boxes have a single CPU each) > > CPU contention? You got a cumulative syncer rate of over 300 MB/s, and > your disk stats confirmed this. Since I doubt your storage is capable > of pulling more than 300 MB/s streaming writes, it seems like you > simply saturated your storage bandwidth, doesn't it? > > Florian > > -- > : Florian G. Haas > : LINBIT Information Technologies GmbH > : Vivenotgasse 48, A-1120 Vienna, Austria > > Enterprise consultancy and support for DRBD is available from LINBIT. > If you are interested, Please go to http://www.linbit.com/en/contact > and leave your contact details. > > When replying, there is no need to CC my personal address. I monitor > the list on a daily basis. Thank you. > _______________________________________________ > drbd-user mailing list > drbd-user at lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > ------------------------------------------------------------------------ ------- This email may contain legally privileged and/or confidential information. It is solely for and is confidential for use by the addressee. Unauthorised recipients must preserve, observe and respect this confidentiality. If you have received it in error please notify us and delete it from your computer. Do not discuss, distribute or otherwise copy it. Unless expressly stated to the contrary this e-mail is not intended to, and shall not, have any contractually binding effect on the Company and its clients. We accept no liability for any reliance placed on this e-mail other than to the intended recipient. If the content is not about the business of this Company or its clients then the message is neither from nor sanctioned by the Company. We accept no liability or responsibility for any changes made to this e-mail after it was sent or any viruses transmitted through this e-mail or any attachment. It is your responsibility to satisfy yourself that this e-mail or any attachment is free from viruses and can be opened without harm to your systems. _______________________________________________ drbd-user mailing list drbd-user at lists.linbit.com http://lists.linbit.com/mailman/listinfo/drbd-user