[Csync2] csync2 stop if cannot stat file/dir

[Ali Nebi]

Hello Dan,

Thank you very much for your reply and very informative message.

Yes, It is high availability setup as those 3 servers are each other load
balanced. I have a failover IP assigned to one of the servers which play
role as main server and in case that this server fail, then failover IP is
switching to the other servers. No DNS round robin setup done.

For now I decided to put the cache in NFS and put it outside the sync until
I learn better csync2 :)

Everything else seems to be working fine for now. Except one more issue I
just reported. It is related to corrupting files that are just uploading to
the system.

I will do local tests on setup you have described and see how all will go.

Thank you once again.

Regards,
Ali


On Mon, Dec 10, 2012 at 7:18 PM, Dan Brown <dan.brown at zu.com> wrote:

> Is this a high availability setup? Load balanced? Fail over? DNS round
> robin?
>
> We do something similar on pairs of servers which are load balanced and
> failed over via both LVS and BGP.  LVS is sticky so when a client connects
> to a site it will continue to use that server for time period X until it is
> idle > X or the server is booted due to health checks, etc.  Given that
> csync2 is not synchronous and we are using sticky LVS rather than round
> robin or weighted connections for a single client connection, this is the
> best way for us to adequately handle it. csync2 runs every other minute
> alternately via cron.
>
> What happens with us occasionally is that developers will deploy code
> which wipes out directories and recreates and repopulates them while csync2
> is in the middle of indexing a particular set of directories and files it
> will register a couple of directories as missing, but then see the files it
> indexed earlier as still there.  So rather than marking everything under
> the missing directory dirty, it only marks the directory and a couple of
> files dirty (to delete) and will then find the rest of the files as updated
> and mark them dirty (to update).  This will cause csync2 to fail
> continuously until the problem is fixed because it can't update files to a
> directory which doesn't exist.
> Since csync2 pushes updates rather than pulling updates, fatally dying by
> default seems like an appropriate thing to do, but I agree that the option
> to ignore errors such as this and make them warnings should be available
> (as the directory will then be recreated and the files will update) so that
> further files which are to be synched (and not in the problem directory) do
> so.
>
> So to get around this issue what we do is this.
>
> SERVER A:
>
> # usual sync
> csync2 -crv / -G servergroup1 -N serverA.domain.com
> # We then do a check here so see if any fatal errors have occurred, and
> email an admin if so
> # Nothing here is ever fatal however so usually it's ignored.
> csync2 -xv -G servergroup1 -N serverA.domain.com
> syncreturn=$?
>
> if [ $syncreturn -ne 0 ]; then
>   # mail system admins with an alarm so they can double check it has self
> corrected
>   mailCSync2Alarm
>   # create a directory list for the other server to check
>   find ${site}/www/htdocs -d type d > /tmp/sitedirectories.log
> fi
>
>
> Then on SERVER B
>
> # look for site directory error log file
> rsync -e ssh syncuser at serverA.domain.com:/tmp/sitedirectories.log
> /tmp/sitedirectories.log
> if [ $? -eq 0 ]; then
>   # auto recovery file found
>   for line in `cat /tmp/sitedirectories.log`
>   do
>     path="`clean_path $line`"
>     mkdir -p "${sitesdirectory}/${path}"
>     rsync -pogt -e ssh syncuser at serverA.domain.com:${sitesdirectory}/${path}
> ${sitesdirectory}/${path}
>     # check status and mail success or failed rebuild
>   done
> fi
> # continue the sync from server B to server A
>
> Then on SERVER A the next time around it will update all of the files it
> wasn't able to update before after the fatal error, either until it
> encounters the next error, or finishes.  So while unfortunately it can take
> 3 minutes for an error to fix itself, it's less costly for developers to
> deploy since they don't require a system administrator to babysit their
> code deployment.
>
> We've been using csync2 for 5+ years now and the self-correction
> implementation has been in it this way for at least the last 2-3.
>
>  _________________________________
>  Dan Brown
> Manager, IT | *zu*
>
> Twitter @zutweets | Facebook www.facebook.com/zulive
> Phone 306 653 4747 | Fax 306 653 4774
> 303 Pacific Ave | Saskatoon SK | S7K 1P2
> www.zu.com
>
>
>
>
> On Thu, Nov 15, 2012 at 4:50 AM, Ali Nebi <anebi at iguanait.com> wrote:
>
>> Hi,
>>
>> I have 3 servers and i have installed csync2 on each. I have set conflict
>> resolution to be younger. I have a very busy web site and its cache change
>> very often.
>>
>> When i run csync -xv on one of the servers i get following error:
>>
>> This should not happen: Can't stat
>> /var/www/var/ezflow_site/cache/ezmutex/575cf8b73520dd064648e178b78bbae0_meta.
>>
>> Same happens also for few dirs. I checked in file system and I see that
>> files and folders that csync2 complains for do not exists. I suppose these
>> folders/files were deleted during sync time.
>>
>> It is ok to complain, but problem is that csync2 stop sync process
>> because of the errors. Shouldn't be better if it just mark this error (do
>> nothing about it) and continue its sync process instead of stopping to work?
>>
>> I use lsyncd together with csync2 and lsyncd stops to work, because
>> csync2 stop to work when such error occurs.
>>
>> What is the best solution for such situations?
>>
>> I'm sorry if I do it wrong way. I'm still newbie with csync2 ;)
>>
>> Awaiting for your reply.
>>
>> Regards,
>> Ali
>>
>>
>>
>> --
>> Iguana Information Technologies, SL
>> Calle López de Hoyos 35, 1º
>> 28002 Madrid, España (Spain)
>> +34 915569100
>> +34 649336286
>> http://www.iguanait.com/
>>
>> Advertencia
>> -----------
>> Este mensaje contiene información privada y confidencial. Si usted
>> no es el destinatario, no está autorizado a leer, imprimir, retener,
>> copiar o difundir este mensaje o parte de él. En caso de que usted
>> reciba este mensaje por error debe borrarlo. Gracias.
>>
>> Confidentiality notice
>> ----------------------
>> This message contains private and confidential information. If you
>> are not the named addressee, you are not authorized to read, print,
>> retain, copy or disseminate this message or any part of it. In case
>> you receive this message by mistake you should delete it. Thanks.
>>
>> _______________________________________________
>> Csync2 mailing list
>> Csync2 at lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/csync2
>>
>>
>


-- 
Iguana Information Technologies, SL
Calle López de Hoyos 35, 1º
28002 Madrid, España (Spain)
+34 915569100
+34 649336286
http://www.iguanait.com/

Advertencia
-----------
Este mensaje contiene información privada y confidencial. Si usted
no es el destinatario, no está autorizado a leer, imprimir, retener,
copiar o difundir este mensaje o parte de él. En caso de que usted
reciba este mensaje por error debe borrarlo. Gracias.

Confidentiality notice
----------------------
This message contains private and confidential information. If you
are not the named addressee, you are not authorized to read, print,
retain, copy or disseminate this message or any part of it. In case
you receive this message by mistake you should delete it. Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/csync2/attachments/20121217/6e538cfd/attachment-0001.htm>