[DRBD-user] Auto and controled clvmd and gfs2 services start

[Theophanis Kontogiannis]

Dear All,

After my previous request for food for thoughts, I decided to move on with
GFS2 for a number of reasons.

I had an issue however.  If for any reason drbd has not turned all resources
in Primary/Primary, then clvmd starts without knowledge of the clustered
LVs. In that case, if after manual intervention, I brought up all resources
as Primary/Primary, then I had to run lvscan again so that clvmd will notice
the DRBD based LV, then manually mount the gfs2 LVs.

Obviously this is not automatic.

So I did the following. I removed autostart of clvmd and gfs2 (chkconfig
--del clvmd & gfs2).

Created two failover restricted domains, one for each node, containing only
one node each.

Then using any cluster managing interface (i.e. either CONGA or
system-config-cluster), created two script resources, one is the clvmd
script and the second is the gfs2 script (the /etc/init.d/ scripts that is),
named CLVMD and GFS2 respectively.

Then created two services, one per node (so that the gfs2 file system will
be controlled independently on each node) that have the following
characteristics:

1.	Top level resource is the CLVMD script resource and attached
underneath as related, is the GFS2 script resource.
2.	Check if ALL drbd resources are running Primary/Primary (because
anything else might imply a non recovered Split Brain)
3.	If and only if all DRBD resources then and only then the cluster
manager tries to mount the gfs2 on a per node basis.

Extract from my /etc/cluster/cluster.conf

                <failoverdomains>
                        <failoverdomain name="LVs-tweety1" restricted="1">
                                <failoverdomainnode name="tweety-1"
priority="1"/>
                        </failoverdomain>
                        <failoverdomain name="LVs-tweety2" restricted="1">
                                <failoverdomainnode name="tweety-2"
priority="1"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <script file="/etc/init.d/clvmd" name="clvmd"/>
                        <script file="/etc/init.d/gfs2" name="GFS2"/>
                </resources>
                <service autostart="1" domain="LVs-tweety1"
name="LV-tweety1">
                        <script ref="clvmd">
                                <script ref="GFS2"/>
                        </script>
                </service>
                <service autostart="1" domain="LVs-tweety2"
name="LV-tweety2">
                        <script ref="clvmd">
                                <script ref="GFS2"/>
                        </script>


Now the issue was how to manage the condition "if and only if ALL drbd
resources are running Primary/Primary then and only then proceed to starting
clvmd and if successful then start gfs2", so that it will find all LVs on
drbd, make them active and then mount the gfs2 file system that will be in
tune.

I did the following 3 lines (underlined) addition on the /etc/init.d/clvmd
script:


# See how we were called.
case "$1" in
  start)
        if drbdadm state all | grep -v "Primary/Primary"
        then exit 1
        fi
        start
        rtrn=$?
        [ $rtrn = 0 ] && touch $LOCK_FILE
        ;;


That way the cluster service will start if and only if ALL drbd resources
are Primary/Primary. 
The modified script actually looks if the outcome of "drbdadm state all"
gives any non Primary/Primary results.
Otherwise it will keep on trying forever. 

So if we have a split brain or something went wrong, then we will solve it
manually. But the moment cman finds out that all resources are
Primary/Primary it will automatically start the clvmd and gfs2 services (and
mount the file system).

Hope it helps someone.

Theophanis Kontogiannis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20080320/99828120/attachment.htm>