[Csync2] symlinks preventing file updates?

[Dan Brown]

I've got a bunch of servers running in LVS which are being updated via
csync2.  Recently we decided to move them over to a couple of DRBD/NFS
fileservers because the sites are too dynamic and the csync2 lag too long
(servers are running in master/master/master/master/master mode (5 servers),
younger file wins conflicts) so we yanked two out of the cluster and created
a new csync2 config for that pair.  Syncs from any given server only take
(on average) 30 seconds so all servers were up to date once every five
minutes.  

One of them still gets updates from the other three as well so four of them
were running together, and the other one just got updates but didn't serve
out any sites.  This ran for a little while and all was okay.  So now after
ensuring that still works, we yanked the update-only server and shrank it's
filesystems so that we could move our site directories onto a DRBD/NFS
shared filesystem (these two servers becoming the fileservers).  
Now our filesystem for the sites looks like this:

/dev/sda1  /var                - contains mysql, site mail, etc
/dev/sda2  /opt/mnt/drbd-meta  - drbd meta-disk
/dev/sda3  /opt/mnt/data       - actual site files

The directory structure changed from this:
/sites/<client>/www/htdocs
                   /logs
                   /conf

To this:
/sites/<client>/www/htdocs -> /opt/mnt/data/sites/<client>/htdocs/
                   /logs -> /opt/mnt/data/sites/<client>/<server>-logs/
                   /conf

Now I'm syncing the files one more time before we lock the servers to be all
read only (ie. no one can update them), bring them down one by one,
reconfigure them, and bring up the first of the two DRBD hosts.
Unfortunately for some files (some 415 of them at the moment), they aren't
synching and I don't see why.  Conflict resolution wins on the server which
the file has been updated on, but the update still doesn't occur.  The
update looks like this:

Updating /sites/VOL/www/htdocs/events.html on seahawk.csync2 ...
While syncing file /clients/VOL/www/htdocs/events.html:
ERROR from peer seahawk.csync2: File is also marked dirty here!
Auto-resolving conflict: Won 'younger/older' test.
Updating /sites/VOL/www/htdocs/events.html on seahawk.csync2 ...
While syncing file /clients/VOL/www/htdocs/events.html:
ERROR from peer seahawk.csync2: File is also marked dirty here!
ERROR: Auto-resolving failed. Giving up.

Other than moving the directories to a different partition, the only other
thing done to the site directories is symlink them from their original
locations to the new location.  
This is an example of a file trying to sync:

[root at nitehawk htdocs]# l /clients/VOL/www/htdocs/events.html
-rw-r--r--  1 drillbit69 noftp 9139 Oct 23 00:08
/clients/VOL/www/htdocs/events.html

And the one being updated:

[root at seahawk ssh]# l /clients/VOL/www/htdocs/events.html
-rw-r--r--  1 drillbit69 noftp 7483 Oct  6 12:55
/clients/VOL/www/htdocs/events.html

I've tried touching a couple of files to see if they will update but it
still errors out.   By the looks of it, the entire file structure on the
modified server (even though 99% of times have not changed, just the symlink
to the DocumentRoot directory) is being marked as dirty.  Is there any way
to run a check on the non-updated host which will unmark files being dirty
if a) they are in sync, or b) they would lose the conflict resolution
process and be updated from the other host?

__________________________________________________________
Dan Brown
danb at zu.com