[DRBD-user] 3 simple questions for a nifty setup

paddy paddy at panici.net
Wed Aug 24 19:34:25 CEST 2005

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Thu, Aug 25, 2005 at 12:46:30AM +0930, David wrote:
> 
> Thanks paddy for your reply. I think I am better understanding the
> differences

Glad to help.  Hope I've not mislead you :)

> - I am not after the full-scale mysql clustering solution; I
> don't need it/have the resources.

I'd love to know what its for.
 
> What do you mean when you say you have mysql running on top of drbd? Have
> you got both PC's mysql data directories separate, or do they share the same
> data files? (ie. /mnt/drbd0/var/lib/mysql)

yes.  I have an ext3 filesystem which I mount on /var/lib/mysql.

It is mounted on the primary side and my haresources looks like:

node1  drbddisk::r0  Filesystem::/dev/drbd0::/var/lib/mysql::ext3  mysql
node2

(removing some extraneous noise for clarity)

heartbeat takes care of the dependencies, there is a drbd init.d script 
but neither the filesystem mount nor mysql are started by the regular init. 

The resources.di/mysql script is really no more than:

	op=$1;
	
	case $op in
	  start)
	        /etc/init.d/mysql start;;
	  stop)
	        /etc/init.d/mysql stop;;
	  status|monitor)
	        if mysqld_status check_alive nowarn; then
	          echo "running"
	        else
	          echo "stopped"
	        fi
	        ;;
	  *)
	        usage
	        exit 1;;
	esac
	exit $?
	
where mysqld_status() is just cribbed from the code in the debian init.d script.
The debian init.d script seems to handle the mysql fsck :)

I've cobbled together a nasty trick with bind to fail over a dns name, so 
that a writer can access the db from the secondary node. 

I've seen mysql fsck and complain a couple of times so far, 
without any apparent harm :)  Not seen any problem at the filesystem level
yet (nor do I expect to, it seems to be well tested).

For my application (simply logging - unrelated inserts a row at a time)
its even possible for the whole setup to go split brain (as it has done
a couple of times) with minimal pain to me.  I just dump the extra records
on one side, resync and then load the dump in.   It's so absurd that I'm
almost ashamed to admit it, but it does what I want. I'm also fortunate
that there are various secondary systems which would enable me to rebuild
the database without loss if it all went horribly wrong.

Not exactly your average database problem ;)

To be honest, I'm not even watching it terribly closely right now.
It could be going wrong and I just haven't noticed :)
(actually, I'm aware of bugs elsewhere, so it's not just a question of
whether it goes wrong, but what goes wrong and how often, which is why
its more dificult for me to be certain that there isn't a problem
with the mysql over drbd part of iti. That and I'm working on other 
things :).

> I was wondering if such a setup as the latter is possible, since it seems
> logical to me and pretty straight-forward; if a server running mysql
> crashes, it must reboot and recover. If the primary server of a two-server
> heartbeat "cluster" crashes, same thing - it's just a different machine
> which must recover from the cluster, it is not privy to any extra/different
> information... or is it?

This is pretty much my reasoning too.

Its not too much of a stretch to imagine that drbd might introduce a reordering
of writes that could perhaps violate assumptions in the filesystem fsck or
in the mysql fsck, or even just tickle unused codepaths and shake out 
interesting bugs, even if drbd is correct.  Its clear that the authors have
taken considerable care about this aspect of drbd, but did I read something
on this list about fscking a journalled filesystem here recently ?

There is also the issue of whether a commit propagates correctly.  My assumption
would be that mysql would return from a commit only after some reasonably 
conservative effort has been made to write the necessary data to disk and 
that drbd protocol A will thus guarantee that the commit is written both sides.
It is an assumption based entirely on imagination, without foundation in research.

By comparison, my best understanding of mysql replication was that a commit
could get lost if the master crashed before it propagated to a slave.
(I could be wrong on this. I'd be happy to be corrected).

Other possible pros and cons include: 

There may be some efficiencies to be had by doing the replication in mysql
where there is more knowledge of the data.  I don't actually care about this.

Something like drbd protocol A is likely to add latency to certain operations.
Even if performance is not a consideration, this may challenge the assumptions
that clients make about timeouts, etc.

It is probably easier to find support and experience for the built-in 
facilities of mysql.

Even with the possibility of losing a commit, mysql style replication should
give a better chance that you'll at least have something running.  I've seen
too many filesystems go south over the years to imagine that the risk that
the secondary can't automatically fsck the failed-over block device, or
the same but for mysql, is zero.  Clearly its not, and a having a shared
resource may be less robust for certain applications.  Actually this is 
one of the reasons I plan to look at mysql replication long term: for me, 
losing a commit temporarily is better than losing availability.  But right
now admin time is more precious :)

By a similar token, drbd doesn't know about the applications above it.
If it replicates data in good faith, but the data is bad, there is no filtering
at the secondary node to check that the data is good from an application
point of view.  Conversely having a single well tested service rather than
having every application reinvent the wheel can help with the bug count.

The bottom line is that, as far as I know, this setup is essentially untested,
and all hope of it working are based purely on my fevered imagining :)
Since I like the look of drbd for other problems, and this problem is 
non-critcal for me, this is a freebie for me, otherwise I couldn't jusitfy 
it.  Looked at another way, its a chance for me to try out and get to know 
drbd before commiting more deeply.  So far drbd does well.

> Does anybody see any danger with this proposed setup? 

yes :)

> Anybody actually doing this?

yes :)

> What do you mean by mysql on drbd - could you say (briefly) how you have
> mysql running on drbd (ie. which files are available to which servers?)

I hope I covered this above.  Feel free to enquire further on any details.

Sorry, I missed the bit earlier about having multiple secondaries in drbd.

As you know, I had assumed that this is not possible with 0.7, but I don't
recall exactly why.  I'm not in a position to try it out right now, but I
am interested to hear whether it works.

Regards,
Paddy
-- 
Perl 6 will give you the big knob. -- Larry Wall



More information about the drbd-user mailing list