[DRBD-user] Protocol A Pending

J.R. Lillard jlillard at ghfllc.com
Tue Jul 3 16:13:59 CEST 2012

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Tue, Jul 3, 2012 at 2:09 AM, Lars Ellenberg <lars.ellenberg at linbit.com>wrote:

> On Mon, Jul 02, 2012 at 05:00:51PM -0500, J.R. Lillard wrote:
> > On Mon, Jun 25, 2012 at 2:10 PM, Lars Ellenberg
> > <lars.ellenberg at linbit.com>wrote:
> >
> > > On Fri, Jun 22, 2012 at 02:38:07PM -0500, J.R. Lillard wrote:
> > > > Witnessed another bandwidth spike that slowed my stacked layer down.
> > > >
> > > > 10: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----
> > > >     ns:192538032 nr:0 dw:599650316 dr:1701817080 al:4481613 bm:43214
> lo:1
> > > > pe:2050 ua:0 ap:2049 ep:1 wo:f oos:0
> > >
> > > You want to increase your max-buffers.
> > >
> >
> > I set max-buffers and max-epoch-size to 8000 but I'm still having issues.
> >  r1-U filled up almost 75% of it's 200M proxy buffer.  During that time
> the
> > pe for r1-U kept climbing and all the disk writes were put on hold until
> it
> > could start catching up.  I can tell this because my virtual machines all
> > freeze up and write latency reported from ESXi skyrockets as well.  I'm
> > still wondering if I'm not understanding some fundamental part of DRBD or
> > Proxy.  I am expecting my third node to never affect the performance of
> my
> > primary nodes unless the Proxy buffers filled up completely.  Is that not
> > the case?
>
> "never" certainly not.
> "much less", more like.
> depending on a lot of things.
>

Do you have a whitepaper that explains how things operate in more detail?


> > > And have a look at the other hints on drbd performance tuning.
> > >
> >
> > I did but nothing else seems to apply.
>
> We will likely be able to help tune that thing.
> There is no "for best performance just use these settings".
> If there was, we'd made "those settings" the default.
>

Understood.  I wouldn't expect a one-size-fits-all config for something
this specialized.


> And proper tuning need understanding,
> understanding may need some time and careful observation.
>

Also understood.  I've been working at this for several months on my own
before finally turning to the list for help.


> If you are not satisfied with the proxy,
> or need help with performance tuning,
> why not contact your LINBIT support rep directly?
>

We purchased Proxy without support so that's not an option.

In my test environment I have the ability to rate limit my third node to
simulate it's final, offsite resting place.  When I have it limited is when
I run into the above problems.  To make sure things don't run a muck
overnight I put it back on its gigabit port.  When I checked on things this
morning one of the stacked resources was in a WFConnection state on both
sides and the Proxy shows it as Half-up. This is another issue I've had
only when using the Proxy.  DRBD on its own has been rock solid but Proxy
seems to be less stable over time.  The first thing I tried was
disconnecting and reconnecting the resource from the third node.  The
primary node didn't even acknowledge that anything was happening.  So I
restarted the Proxy on the third node to get things to start moving again.

Whenever I start Proxy my resources don't just come up quickly.  They
bounce between Down, Half-Up and Up for several minutes before they finally
connect and start to sync.  And they don't do it at the same time either.
 The two that were still connected came up first with the disconnected one
taking about 10 or 15 minutes before it decided to sync.  Here's what my
logs from the primary node look like with the resource disconnected
overnight.

http://pastebin.com/ed8bbeAz

And here's my third node at the same time.

http://pastebin.com/VWeztttY

As I was typing up this email my other stacked resources disconnected for
no reason.  One of them started a sync but is back to disconnected now.
 The one I had problems with overnight had to start its sync over.
 Everything seems very unstable for some reason.  I never had any problems
like this without Proxy.  I'm running DRBD 8.3.11 with Proxy 3.0.6 on
Ubuntu 12.04.  Are there known issues between DRBD 8.3.11 and Proxy 3.0.6?

-- 
J.R. Lillard
System / Network Admin
Web Programmer
Golden Heritage Foods
120 Santa Fe St.
Hillsboro, KS  67063
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20120703/0eee3f0f/attachment.htm>


More information about the drbd-user mailing list