Hello Dan,<div><br></div><div>Thank you very much for your reply and very informative message.</div><div><br></div><div>Yes, It is high availability setup as those 3 servers are each other load balanced. I have a failover IP assigned to one of the servers which play role as main server and in case that this server fail, then failover IP is switching to the other servers. No DNS round robin setup done.</div>
<div><br></div><div>For now I decided to put the cache in NFS and put it outside the sync until I learn better csync2 :)</div><div><br></div><div>Everything else seems to be working fine for now. Except one more issue I just reported. It is related to corrupting files that are just uploading to the system.</div>
<div><br></div><div>I will do local tests on setup you have described and see how all will go.</div><div><br></div><div>Thank you once again.</div><div><br></div><div>Regards,</div><div>Ali</div><div><br></div><div><br><div class="gmail_quote">
On Mon, Dec 10, 2012 at 7:18 PM, Dan Brown <span dir="ltr"><<a href="mailto:dan.brown@zu.com" target="_blank">dan.brown@zu.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is this a high availability setup? Load balanced? Fail over? DNS round robin?<div><br></div><div>We do something similar on pairs of servers which are load balanced and failed over via both LVS and BGP. LVS is sticky so when a client connects to a site it will continue to use that server for time period X until it is idle > X or the server is booted due to health checks, etc. Given that csync2 is not synchronous and we are using sticky LVS rather than round robin or weighted connections for a single client connection, this is the best way for us to adequately handle it. csync2 runs every other minute alternately via cron.</div>
<div><br></div><div>What happens with us occasionally is that developers will deploy code which wipes out directories and recreates and repopulates them while csync2 is in the middle of indexing a particular set of directories and files it will register a couple of directories as missing, but then see the files it indexed earlier as still there. So rather than marking everything under the missing directory dirty, it only marks the directory and a couple of files dirty (to delete) and will then find the rest of the files as updated and mark them dirty (to update). This will cause csync2 to fail continuously until the problem is fixed because it can't update files to a directory which doesn't exist.</div>
<div>Since csync2 pushes updates rather than pulling updates, fatally dying by default seems like an appropriate thing to do, but I agree that the option to ignore errors such as this and make them warnings should be available (as the directory will then be recreated and the files will update) so that further files which are to be synched (and not in the problem directory) do so.</div>
<div><br></div><div>So to get around this issue what we do is this.</div><div><br></div><div>SERVER A:</div><div><br></div><div># usual sync</div><div>csync2 -crv / -G servergroup1 -N <a href="http://serverA.domain.com" target="_blank">serverA.domain.com</a></div>
<div># We then do a check here so see if any fatal errors have occurred, and email an admin if so</div><div># Nothing here is ever fatal however so usually it's ignored.</div><div>csync2 -xv -G servergroup1 -N <a href="http://serverA.domain.com" target="_blank">serverA.domain.com</a></div>
<div>syncreturn=$?</div><div><br></div><div>if [ $syncreturn -ne 0 ]; then</div><div> # mail system admins with an alarm so they can double check it has self corrected</div><div> mailCSync2Alarm</div><div> # create a directory list for the other server to check</div>
<div> find ${site}/www/htdocs -d type d > /tmp/sitedirectories.log</div><div>fi</div><div><br></div><div><br></div><div>Then on SERVER B</div><div><br></div><div># look for site directory error log file</div><div>rsync -e ssh syncuser@serverA.domain.com:/tmp/sitedirectories.log /tmp/sitedirectories.log</div>
<div>if [ $? -eq 0 ]; then</div><div> # auto recovery file found</div><div> for line in `cat /tmp/sitedirectories.log`</div><div> do</div><div> path="`clean_path $line`"</div><div> mkdir -p "${sitesdirectory}/${path}"</div>
<div> rsync -pogt -e ssh syncuser@serverA.domain.com:${sitesdirectory}/${path} ${sitesdirectory}/${path}</div><div> # check status and mail success or failed rebuild</div><div> done</div><div>fi</div><div># continue the sync from server B to server A</div>
<div><br></div>
<div>Then on SERVER A the next time around it will update all of the files it wasn't able to update before after the fatal error, either until it encounters the next error, or finishes. So while unfortunately it can take 3 minutes for an error to fix itself, it's less costly for developers to deploy since they don't require a system administrator to babysit their code deployment.</div>
<div><br></div><div>We've been using csync2 for 5+ years now and the self-correction implementation has been in it this way for at least the last 2-3.</div><div><br clear="all"><p><font color="#B2B2B2"><span style="font-size:14px"></span></font></p>
<p><font color="#B2B2B2">
<span style="font-size:10.5pt;line-height:12pt;color:rgb(178,178,178)">_________________________________</span><br>
<font>
<span>Dan Brown</span>
</font><br>
<span style="font-size:9pt;color:gray">Manager, IT</span>
<span style="font-size:9pt;color:rgb(247,0,132)">| </span>
<b><span style="font-size:9pt">zu</span></b>
</font></p><font color="#B2B2B2">
<p>
<span style="font-size:7.5pt;color:gray">Twitter @zutweets</span>
<span style="font-size:7.5pt;color:rgb(247,0,132)">| </span>
<span style="font-size:7.5pt;color:gray">Facebook <a href="http://www.facebook.com/zulive" target="_blank">www.facebook.com/zulive</a></span>
<br>
<span style="font-size:7.5pt;color:gray">Phone <a href="tel:306%20653%204747" value="+13066534747" target="_blank">306 653 4747</a></span>
<span style="font-size:7.5pt;color:rgb(247,0,132)">| </span>
<span style="font-size:7.5pt;color:gray">Fax <a href="tel:306%20653%204774" value="+13066534774" target="_blank">306 653 4774</a></span>
<br>
<span style="font-size:7.5pt;color:gray">303 Pacific Ave</span>
<span style="font-size:7.5pt;color:rgb(247,0,132)"> | </span>
<span style="font-size:7.5pt;color:gray">Saskatoon SK</span>
<span style="font-size:7.5pt;color:rgb(247,0,132)">| </span>
<span style="font-size:7.5pt;color:gray">S7K 1P2</span>
<br>
<span style="font-size:8.5pt;color:rgb(255,0,102)"><a href="http://www.zu.com/" target="_blank">www.zu.com</a>
</span>
</p></font><p></p>
<br>
<br><br><div class="gmail_quote"><div><div class="h5">On Thu, Nov 15, 2012 at 4:50 AM, Ali Nebi <span dir="ltr"><<a href="mailto:anebi@iguanait.com" target="_blank">anebi@iguanait.com</a>></span> wrote:<br></div></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi,<div><br></div><div>I have 3 servers and i have installed csync2 on each. I have set conflict resolution to be younger. I have a very busy web site and its cache change very often.</div><div><br></div><div>When i run csync -xv on one of the servers i get following error:</div>
<div><br></div><div>This should not happen: Can't stat /var/www/var/ezflow_site/cache/ezmutex/575cf8b73520dd064648e178b78bbae0_meta.</div><div><br></div><div>Same happens also for few dirs. I checked in file system and I see that files and folders that csync2 complains for do not exists. I suppose these folders/files were deleted during sync time.</div>
<div><br></div><div>It is ok to complain, but problem is that csync2 stop sync process because of the errors. Shouldn't be better if it just mark this error (do nothing about it) and continue its sync process instead of stopping to work?</div>
<div><br></div><div>I use lsyncd together with csync2 and lsyncd stops to work, because csync2 stop to work when such error occurs.</div><div><br></div><div>What is the best solution for such situations?</div><div><br></div>
<div>I'm sorry if I do it wrong way. I'm still newbie with csync2 ;)</div><div><br></div><div>Awaiting for your reply.</div><div><br></div><div>Regards,</div><div>Ali</div><div><br></div><div><br clear="all"><div>
<br></div>-- <br><div><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Iguana Information Technologies, SL</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Calle López de Hoyos 35, 1º</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">28002 Madrid</span>, España (Spain)</div><a href="tel:%2B34%20915569100" value="+34915569100" target="_blank">+34 915569100</a><br><a href="tel:%2B34%20649336286" value="+34649336286" target="_blank">+34 649336286</a><br>
<a href="http://www.iguanait.com/" target="_blank">http://www.iguanait.com/</a><br>
<br>Advertencia<br>-----------<br>Este mensaje contiene información privada y confidencial. Si usted<br>no es el destinatario, no está autorizado a leer, imprimir, retener,<br>copiar o difundir este mensaje o parte de él. En caso de que usted<br>
reciba este mensaje por error debe borrarlo. Gracias.<br><br>Confidentiality notice<br>----------------------<br>This message contains private and confidential information. If you<br>are not the named addressee, you are not authorized to read, print,<br>
retain, copy or disseminate this message or any part of it. In case<br>you receive this message by mistake you should delete it. Thanks.<br>
</div>
<br></div></div>_______________________________________________<br>
Csync2 mailing list<br>
<a href="mailto:Csync2@lists.linbit.com" target="_blank">Csync2@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/csync2" target="_blank">http://lists.linbit.com/mailman/listinfo/csync2</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Iguana Information Technologies, SL</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Calle López de Hoyos 35, 1º</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">28002 Madrid</span>, España (Spain)</div>+34 915569100<br>+34 649336286<br><a href="http://www.iguanait.com/" target="_blank">http://www.iguanait.com/</a><br>
<br>Advertencia<br>-----------<br>Este mensaje contiene información privada y confidencial. Si usted<br>no es el destinatario, no está autorizado a leer, imprimir, retener,<br>copiar o difundir este mensaje o parte de él. En caso de que usted<br>
reciba este mensaje por error debe borrarlo. Gracias.<br><br>Confidentiality notice<br>----------------------<br>This message contains private and confidential information. If you<br>are not the named addressee, you are not authorized to read, print,<br>
retain, copy or disseminate this message or any part of it. In case<br>you receive this message by mistake you should delete it. Thanks.<br>
</div>