<div dir="ltr">I didn't explain myself well enough, I meant like would a stop/start, verify or some other command cause the OOS to be pushed to the secondary?<div><br></div><div>The link speed when performing a verify is about 75MB/s I would think that would be fast enough and the disk is a cloud block volume which is potentially much faster.</div><div>It just seems that once bytes are in OOS that they're not pushed again until something else happens, maybe it's some idle state or something else?</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 21, 2019 at 7:52 PM Digimer <<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Without knowing your setup or what is limiting you; I can suggest two<br>
options;<br>
<br>
1. Faster hardware (links speed / peer disk)<br>
2. Switch to Protocol C<br>
<br>
digimer<br>
<br>
On 2019-10-21 4:36 p.m., G C wrote:<br>
> Is there anything that will force the OOS to push what is out of sync?<br>
> <br>
> <br>
> On Mon, Oct 21, 2019 at 11:00 AM Digimer <<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a><br>
> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>>> wrote:<br>
> <br>
> Tuning is quite instance-specific. I would always suggest starting by<br>
> commenting out all tuning, see how it behaves, then tune. Premature<br>
> optimization never is.<br>
> <br>
> digimer<br>
> <br>
> On 2019-10-21 1:31 p.m., G C wrote:<br>
> > Would any of these values being changed help or would it need to<br>
> be the<br>
> > actual speed between the two nodes that needs to be increased?<br>
> ><br>
> > disk {<br>
> > on-io-error detach;<br>
> > c-plan-ahead 10;<br>
> > c-fill-target 24M;<br>
> > c-min-rate 80M;<br>
> > c-max-rate 720M;<br>
> > }<br>
> > net {<br>
> > protocol A;<br>
> > max-buffers 36k;<br>
> > sndbuf-size 1024k;<br>
> > rcvbuf-size 2048k;<br>
> > }<br>
> ><br>
> > Thank you<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Oct 21, 2019 at 10:10 AM Digimer <<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a><br>
> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>><br>
> > <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>>>> wrote:<br>
> ><br>
> > I assumed it wasn't paused, but that confirms it.<br>
> ><br>
> > Protocol A allows for out of sync to grow. It says "when the<br>
> data in on<br>
> > the network buffer to send to the peer, consider the write<br>
> complete". As<br>
> > such, data that hasn't made it over to the peer causes oos to<br>
> climb. If<br>
> > you have a steady write rate that is faster than your transmit<br>
> > bandwidth, then seeing fairly steady OOS makes sense.<br>
> ><br>
> > To "fix" it, you need to increase the connection speed to the<br>
> peer node.<br>
> > Or, less likely, if the peer's disk is slower than the bandwidth<br>
> > connecting it, speed up the disk write speed.<br>
> ><br>
> > In either case, what you are seeing is not a surprise, and<br>
> it's not a<br>
> > problem with DRBD. The only other option is to use protocol C,<br>
> so that a<br>
> > write isn't complete until it reaches the peer, but that will<br>
> slow down<br>
> > the write performance of the primary node to be whatever speed<br>
> you have<br>
> > to the peer. That's likely unacceptable.<br>
> ><br>
> > In short, you have a hardware/resource issue.<br>
> ><br>
> > digimer<br>
> ><br>
> > On 2019-10-21 12:19 p.m., G C wrote:<br>
> > > version: 8.4.10<br>
> > > Ran the resume-sync all and received:<br>
> > > 0: Failure: (135) Sync-pause flag is already cleared<br>
> > > Command 'drbdsetup-84 resume-sync 0' terminated with exit<br>
> code 10<br>
> > ><br>
> > > Protocol used is 'A', our systems are running on a cloud<br>
> environment.<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On Mon, Oct 21, 2019 at 9:09 AM Digimer <<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a><br>
> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>><br>
> > <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>>><br>
> > > <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>><br>
> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a> <mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>>>>> wrote:<br>
> > ><br>
> > > 8.9.2 is the utils version, what is the kernel module<br>
> version?<br>
> > > (8.3.x/8.4.x/9.0.x)?<br>
> > ><br>
> > > It's possible something paused sync, but I doubt it. You<br>
> can try<br>
> > > 'drbdadm resume-sync all'. The oos number should change<br>
> > constantly, any<br>
> > > time a block changes it should go up and every time a block<br>
> > syncs it<br>
> > > should go down.<br>
> > ><br>
> > > What protocol are you using? A, B or C?<br>
> > ><br>
> > > digimer<br>
> ><br>
> ><br>
> > --<br>
> > Digimer<br>
> > Papers and Projects: <a href="https://alteeve.com/w/" rel="noreferrer" target="_blank">https://alteeve.com/w/</a><br>
> > "I am, somehow, less interested in the weight and convolutions of<br>
> > Einstein’s brain than in the near certainty that people of<br>
> equal talent<br>
> > have lived and died in cotton fields and sweatshops." -<br>
> Stephen Jay<br>
> > Gould<br>
> ><br>
> <br>
> <br>
> -- <br>
> Digimer<br>
> Papers and Projects: <a href="https://alteeve.com/w/" rel="noreferrer" target="_blank">https://alteeve.com/w/</a><br>
> "I am, somehow, less interested in the weight and convolutions of<br>
> Einstein’s brain than in the near certainty that people of equal talent<br>
> have lived and died in cotton fields and sweatshops." - Stephen Jay<br>
> Gould<br>
> <br>
<br>
<br>
-- <br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.com/w/" rel="noreferrer" target="_blank">https://alteeve.com/w/</a><br>
"I am, somehow, less interested in the weight and convolutions of<br>
Einstein’s brain than in the near certainty that people of equal talent<br>
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould<br>
</blockquote></div>