<div class="gmail_quote">On Tue, Jul 3, 2012 at 2:09 AM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Jul 02, 2012 at 05:00:51PM -0500, J.R. Lillard wrote:<br>
> On Mon, Jun 25, 2012 at 2:10 PM, Lars Ellenberg<br>
> <<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>>wrote:<br>
><br>
> > On Fri, Jun 22, 2012 at 02:38:07PM -0500, J.R. Lillard wrote:<br>
> > > Witnessed another bandwidth spike that slowed my stacked layer down.<br>
> > ><br>
> > > 10: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----<br>
> > > ns:192538032 nr:0 dw:599650316 dr:1701817080 al:4481613 bm:43214 lo:1<br>
> > > pe:2050 ua:0 ap:2049 ep:1 wo:f oos:0<br>
> ><br>
> > You want to increase your max-buffers.<br>
> ><br>
><br>
> I set max-buffers and max-epoch-size to 8000 but I'm still having issues.<br>
> r1-U filled up almost 75% of it's 200M proxy buffer. During that time the<br>
> pe for r1-U kept climbing and all the disk writes were put on hold until it<br>
> could start catching up. I can tell this because my virtual machines all<br>
> freeze up and write latency reported from ESXi skyrockets as well. I'm<br>
> still wondering if I'm not understanding some fundamental part of DRBD or<br>
> Proxy. I am expecting my third node to never affect the performance of my<br>
> primary nodes unless the Proxy buffers filled up completely. Is that not<br>
> the case?<br>
<br>
</div>"never" certainly not.<br>
"much less", more like.<br>
depending on a lot of things.<br></blockquote><div><br></div><div>Do you have a whitepaper that explains how things operate in more detail?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">> > And have a look at the other hints on drbd performance tuning.<br>
> ><br>
><br>
> I did but nothing else seems to apply.<br>
<br>
</div>We will likely be able to help tune that thing.<br>
There is no "for best performance just use these settings".<br>
If there was, we'd made "those settings" the default.<br></blockquote><div><br></div><div>Understood. I wouldn't expect a one-size-fits-all config for something this specialized.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
And proper tuning need understanding,<br>
understanding may need some time and careful observation.<br></blockquote><div><br></div><div>Also understood. I've been working at this for several months on my own before finally turning to the list for help.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">If you are not satisfied with the proxy,<br>
or need help with performance tuning,<br>
why not contact your LINBIT support rep directly?<br></blockquote><div><br></div><div>We purchased Proxy without support so that's not an option.</div><div><br></div><div>In my test environment I have the ability to rate limit my third node to simulate it's final, offsite resting place. When I have it limited is when I run into the above problems. To make sure things don't run a muck overnight I put it back on its gigabit port. When I checked on things this morning one of the stacked resources was in a WFConnection state on both sides and the Proxy shows it as Half-up. This is another issue I've had only when using the Proxy. DRBD on its own has been rock solid but Proxy seems to be less stable over time. The first thing I tried was disconnecting and reconnecting the resource from the third node. The primary node didn't even acknowledge that anything was happening. So I restarted the Proxy on the third node to get things to start moving again.</div>
<div><br></div><div>Whenever I start Proxy my resources don't just come up quickly. They bounce between Down, Half-Up and Up for several minutes before they finally connect and start to sync. And they don't do it at the same time either. The two that were still connected came up first with the disconnected one taking about 10 or 15 minutes before it decided to sync. Here's what my logs from the primary node look like with the resource disconnected overnight.</div>
<div><br></div><div><div><a href="http://pastebin.com/ed8bbeAz">http://pastebin.com/ed8bbeAz</a>
</div></div><div><br></div><div>And here's my third node at the same time.</div><div><br></div><div><a href="http://pastebin.com/VWeztttY">http://pastebin.com/VWeztttY</a>
</div><div><br></div><div>As I was typing up this email my other stacked resources disconnected for no reason. One of them started a sync but is back to disconnected now. The one I had problems with overnight had to start its sync over. Everything seems very unstable for some reason. I never had any problems like this without Proxy. I'm running DRBD 8.3.11 with Proxy 3.0.6 on Ubuntu 12.04. Are there known issues between DRBD 8.3.11 and Proxy 3.0.6?</div>
<div><br></div></div>-- <br>J.R. Lillard<div>System / Network Admin</div><div>Web Programmer</div><div>Golden Heritage Foods</div><div>120 Santa Fe St.</div><div>Hillsboro, KS 67063</div><br>