[DRBD-user] Odd observation when sync'ing

Lee Christie Lee at titaninternet.co.uk
Thu Jul 3 21:10:43 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Found out what was wrong, or at least we've identified the problem, but
not yet fixed it.

One of the storage servers has a habit of booting up and delivering just
over half the I/O performance it should. We rebooted both boxes 10 times
and ran the same command on each :

"dd if=/dev/zero of=/dev/VolGroup0/LogVol0 bs=1G count=10 oflag=direct"

One of the boxes delivered 340MB/s each time. The other had 5 instances
of 340MB/s and 5 instances of 185MB/s.

Very odd indeed but it explains things. We tested drbd sync performance
on occasions when the box delivers 340MB/s and found it to be as before.

So - we'll fix that and then probably add the second CPUs to reduce CPU
contention.

-----Original Message-----
From: drbd-user-bounces at lists.linbit.com
[mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Lee Christie
Sent: 03 July 2008 15:19
To: drbd-user at lists.linbit.com
Subject: RE: [DRBD-user] Odd observation when sync'ing

Well yes clearly 300MB/s is the maximum writes, however my point was
that two of the drbd resources were sync'ing very slowly compared to the
others. When we looked at top, we noticed these two down the bottom of
the receiver processes. Using renice I bumped them up and they
increased.

So we suspected CPU contention in as much that the writes weren't evenly
spread.

Now they are spread evenly, even using a single CPU, but we've somehow
lost a whole load of performance.


On the first run I also found (using sar) that both system and iowait
were quite high (>40%) and the system load on the box doing the writes
was over 6 ! it was a box under high IO make no mistake.

However now on a resync the %system is 15%, there is no iowait and the
load is 0.5.


So definitely, something in the first scenario made the storage
subsystem go as fast and hard as it could, and now its not. Baffling.

> -----Original Message-----
> From: drbd-user-bounces at lists.linbit.com
> [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Florian Haas
> Sent: 03 July 2008 15:12
> To: drbd-user at lists.linbit.com
> Subject: Re: [DRBD-user] Odd observation when sync'ing
> 
> Lee Christie wrote:
> > Whats even odder is that looking at top we suspected CPU
> contention (the
> > boxes have a single CPU each)
> 
> CPU contention? You got a cumulative syncer rate of over 300 MB/s, and

> your disk stats confirmed this. Since I doubt your storage is capable 
> of pulling more than 300 MB/s streaming writes, it seems like you 
> simply saturated your storage bandwidth, doesn't it?
> 
> Florian
> 
> --
> : Florian G. Haas
> : LINBIT Information Technologies GmbH
> : Vivenotgasse 48, A-1120 Vienna, Austria
> 
> Enterprise consultancy and support for DRBD is available from LINBIT. 
> If you are interested, Please go to http://www.linbit.com/en/contact 
> and leave your contact details.
> 
> When replying, there is no need to CC my personal address. I monitor 
> the list on a daily basis. Thank you.
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
> 

------------------------------------------------------------------------
-------

This email may contain legally privileged and/or confidential
information. It is solely for and is confidential for use by the
addressee. Unauthorised recipients must preserve, observe and respect
this confidentiality. If you have received it in error please notify us
and delete it from your computer. Do not discuss, distribute or
otherwise copy it. 

Unless expressly stated to the contrary this e-mail is not intended to,
and shall not, have any contractually binding effect on the Company and
its clients. We accept no liability for any reliance placed on this
e-mail other than to the intended recipient. If the content is not about
the business of this Company or its clients then the message is neither
from nor sanctioned by the Company. 

We accept no liability or responsibility for any changes made to this
e-mail after it was sent or any viruses transmitted through this e-mail
or any attachment. It is your responsibility to satisfy yourself that
this e-mail or any attachment is free from viruses and can be opened
without harm to your systems.

_______________________________________________
drbd-user mailing list
drbd-user at lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user



More information about the drbd-user mailing list