[DRBD-user] Fw: Where is How to DRBDB ?

Diego Julian Remolina dijuremo at ibb.gatech.edu
Wed Aug 3 16:28:03 CEST 2005

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


there you go...

Diego

On Wednesday 03 August 2005 10:20, Felippe Beling wrote:
> Hi Mrs,
>     I need of a how to complete to implement DRBD. I found something in the
> internet but this not helped me. Anyone can send me ?
>
> Bye, Tks.

-- 
Diego Julian Remolina
System Administrator - Systems Support Specialist II
Institute for Bioengineering and Bioscience
Georgia Institute of Technology
Phone (404) 385-0127
-------------- next part --------------
This is how DRBD was installed and configured on IBB Staff systems:
arwen is the primary
aragorn is the slave

1. Obtaining and Installing drbd.

As a regular user
cd /tmp
wget http://oss.linbit.com/drbd/0.7/drbd-0.7.10.tar.gz

Move the file to the server where you want to run drbd
scp drbd-0.7.10.tar.gz root at arwen:/tmp

ssh root at arwen
cd /usr/src

tar -xzf /tmp/drbd-0.7.10.tar.gz
cd drbd-0.7.10
make rpm

The rpms built just fine in RHEL4.  They are located in the folder /usr/src/drbd-0.7.10/dist/RPMS/i386/
cd /usr/src/drbd-0.7.10/dist/RPMS/i386/
rpm -ivh drbd-0.7.10-1.i386.rpm
rpm -ivh drbd-debuginfo-0.7.10-1.i386.rpm
rpm -ivh drbd-km-2.6.9_5.0.3.ELsmp-0.7.10-1.i386.rpm

Copy those files to the second server and install them there.
scp drbd*rpm aragorn2:/tmp
ssh aragorn2 'rpm -ivh /tmp/drbd\*.rpm'

2. Configuration of drbd

Make sure you have one partition for your data and a separate one for your metadata.
It is usually better to have a separate partition for your metadata.

In this setup we use /dev/sda6 for data and /dev/sda7 for metadata.
The default size  of metadata used is 128MB per drbd partition.  Make sure /dev/sda7 is big
enough if you have more then one drbd partition; e.g. 256MB for 2 drbd partitions, 384MB for
3 drbd partitions, and so on.

Load the kernel module:
modprobe drbd

######################### /etc/drbd.conf file ####################
Create the configuration file /etc/drbd.conf
cat << EOF > /etc/drbd.conf.sample
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd.conf
#
#
resource export {
  protocol C;
  incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
  startup {
    wfc-timeout         0;  ## Infinite!
    degr-wfc-timeout  120;  ## 2 minutes.
  }
  disk {
    on-io-error detach;
  }
  net {
    # timeout           60;
    # connect-int       10;
    # ping-int          10;
    # max-buffers     2048;
    # max-epoch-size  2048;
  }
  syncer {
    rate   60M;
    group   1;
    al-extents 257;
  }

  on arwen.ibb.gatech.edu {
    device      /dev/drbd0;
    disk        /dev/sda6;
    address     10.0.0.16:7789;
    meta-disk   /dev/sda7[0];
  }

  on aragorn.ibb.gatech.edu {
    device     /dev/drbd0;
    disk       /dev/sda6;
    address    10.0.0.17:7789;
    meta-disk  /dev/sda7[0];
  }
}
EOF
################## end drbd.conf ##########################
Note:  If you have more than one drbd device, then you need to make
sure you adjust the meta-disk entry accordingly.  If you had /dev/drbd1,
then the metadisk entries would be:
     meta-disk  /dev/sda7[1];
so that drbd knows that the metadata for /dev/drbd1 should also go in 
/dev/sda7 but be different to the metadata for /deb/drbd0.

Since the drbd documentation does not explain how to use
drbdsetup correctly, we first setup the drbd.conf file and then
run the drbdadm with the -d (dry run method):

[root at arwen /]# drbdadm -d adjust export
/sbin/drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda7 0 --on-io-error=detach
/sbin/drbdsetup /dev/drbd0 syncer --rate=60M --group=1 --al-extents=257
/sbin/drbdsetup /dev/drbd0 net 10.0.0.16:7789 10.0.0.17:7789 C

After confirming that is what you want run the command again without -d

Copy the drbd.conf file to the slave and run the same command
Notice that on the secondary, the IPs get inverted

[root at aragorn ~]# drbdadm -d adjust export
/sbin/drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda7 0 --on-io-error=detach
/sbin/drbdsetup /dev/drbd0 syncer --rate=60M --group=1 --al-extents=257
/sbin/drbdsetup /dev/drbd0 net 10.0.0.17:7789 10.0.0.16:7789 C

Now back on the primary set /dev/drbd0 as the primary with the command:
drbdsetup /dev/drbd0 primary --do-what-I-say

On the primary machine:
Create the filesystem and disable automatic fsck after certain number of mounts
or by time intervals:
Mare sure you create the filesystem on /dev/drbd and NOT on /dev/da6 or you
will find problems later with things like quotas...

mkfs -j /dev/drbd0
tune2fs -c -1 -i 0 /dev/drbd0

3. Starting drbd

Firewall Considerations
If you are running a firewall, make sure the ports you used for each drbd set 
on /etc/drbd.conf are open.  A more relaxed setting will be to simply allow
all trafic by adding the following line to /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -i ethX -j ACCEPT
where ethX refers to whichever interface is dedicated to replicate the data
over the network. This iptables rule is excellent if you are using a 
cross-over cable.

Now start drbd on both systems:
root at arwen /]# /etc/init.d/drbd start
root at aragorn /]# /etc/init.d/drbd start

You can see what is going on with:
[root at arwen /]# cat /proc/drbd
or 
[root at arwen /]# /etc/init.d/drbd status
version: 0.7.10 (api:77/proto:74)
SVN Revision: 1743 build by root at arwen.ibb.gatech.edu, 2005-02-22 09:05:46
 0: cs:SyncSource st:Primary/Secondary ld:Consistent
    ns:3221512 nr:0 dw:8 dr:3221649 al:1 bm:33108 lo:0 pe:0 ua:0 ap:0
        [>...................] sync'ed:  0.6% (523443/526589)M
        finish: 14:30:08 speed: 10,236 (10,096) K/sec
 1: cs:Unconfigured

4. Loading the module on reboot:
You will need to edit the file /etc/rc.modules and add:
modprobe drbd
so that drbd is loaded everytime the system boots up.
chmod 744 /etc/rc.modules or it will not run.


APPENDIX
In cases where you have more than one drbd device, the drbd.conf file will
have 2 resource entries. The considerations are:
- Make sure you do not use the same drbd device for each resource.  This
  means that the first resource will use /dev/drbd0, the second /dev/drbd1
  and so on. Also make sure you are not using the same partition for more
  than one drbd device, meaning, if you used /dev/hda3 for /dev/drbd0, you
  can no longer use /dev/hda3 for /dev/drbd1.

- You can use the same interface to sync both resources, as long as you use
  a different port for each resource. For example, use port 7789 for resource
  one and port 7790 for resource two.

- The metadata partition can be allocated on the same physical device, but 
  make sure that you change the metadata id.  Resouce one will use [0], resource
  two will use [1], and so on.

Here is an example of a drbd.conf with two drbd devices:
######################### /etc/drbd.conf file ####################
cat << EOF > /etc/drbd.conf.sample
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd.conf
#
#
resource export {
  protocol C;
  incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
  startup {
    wfc-timeout         0;  ## Infinite!
    degr-wfc-timeout   60;  ## 2 minutes.
  }
  disk {
    on-io-error detach;
  }
  net {
    # timeout           60;
    # connect-int       10;
    # ping-int          10;
    # max-buffers     2048;
    # max-epoch-size  2048;
  }
  syncer {
    rate   60M;
    group   1;
    al-extents 257;
  }

  on sauron.ibb.gatech.edu {
    device      /dev/drbd0;
    disk        /dev/sdb1;
    address     10.0.0.21:7789;
    meta-disk   /dev/sdb3[0];
  }

  on saruman.ibb.gatech.edu {
    device     /dev/drbd0;
    disk       /dev/sdb1;
    address    10.0.0.22:7789;
    meta-disk  /dev/sdb3[0];
  }
}

resource web {
  protocol C;
  incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
  startup {
    wfc-timeout         0;  ## Infinite!
    degr-wfc-timeout   60;  ## 2 minutes.
  }
  disk {
    on-io-error detach;
  }
  net {
    # timeout           60;
    # connect-int       10;
    # ping-int          10;
    # max-buffers     2048;
    # max-epoch-size  2048;
  }
  syncer {
    rate   60M;
    group   1;
    al-extents 257;
  }

  on sauron.ibb.gatech.edu {
    device      /dev/drbd1;
    disk        /dev/sdb2;
    address     10.0.0.21:7790;
    meta-disk   /dev/sdb3[1];
  }

  on saruman.ibb.gatech.edu {
    device     /dev/drbd1;
    disk       /dev/sdb2;
    address    10.0.0.22:7790;
    meta-disk  /dev/sdb3[1];
  }
}
EOF
################## end drbd.conf ##########################
-------------- next part --------------
cd /home/sysadm/src
mkdir -p heartbeat/1.2.3-1
cd !$ 
wget http://linux-ha.org/download/heartbeat-1.2.3-1.src.rpm (DOES NOT BUILD!!!)
wget http://www.ultramonkey.org/download/heartbeat/stable.latest/redhat_el_3.0/heartbeat-1.2.3-2.rh.el.3.0.src.rpm

scp heartbeat-1.2.3-2.rh.el.3.0.src.rpm root at arwen:/tmp/
ssh arwen
cd /tmp

We need to first install the dependencies:
up2date -i glib-devel
up2date -i net-snmp-devel
up2date -i elfutils-libelf-devel 

libnet is not available from RedHat, it was built with a spec
file I downloaded from: http://dag.wieers.com/apt/
Download libnet from:
http://www.packetfactory.net/libnet/
cd /usr/src/redhat/SPECS
rpmbuild -ba libnet.spec
cd /usr/src/redhat/RPMS/i386/
rpm -ivh libnet-*

Download and install ipvsadm:
wget http://people.redhat.com/cfeist/cluster/RHEL4/alpha/latest_rpms/ipvsadm-1.24-6.i386.rpm
wget http://mirror.linux.duke.edu/pub/fedora/linux/core/4/x86_64/os/SRPMS/ipvsadm-1.24-7.src.rpm
rpmbuild --rebuild ipvsadm-1.24-7.src.rpm

In order to get the heartbeat rpms built, you need a few perl
modules.  Make sure your CPAN shell is configured and working.

cat <<EOF > /usr/lib/perl5/5.8.5/CPAN/Config.pm
# This is CPAN.pm's systemwide configuration file. This file provides
# defaults for users, and the values can be changed in a per-user
# configuration file. The user-config file is being looked for as
# ~/.cpan/CPAN/MyConfig.pm.

$CPAN::Config = {
  'build_cache' => q[10],
  'build_dir' => q[/var/cpan/build],
  'cache_metadata' => q[1],
  'cpan_home' => q[/var/cpan],
  'dontload_hash' => {  },
  'ftp' => q[/usr/kerberos/bin/ftp],
  'ftp_proxy' => q[],
  'getcwd' => q[cwd],
  'gpg' => q[/usr/bin/gpg],
  'gzip' => q[/bin/gzip],
  'histfile' => q[/var/cpan/histfile],
  'histsize' => q[5000],
  'http_proxy' => q[],
  'inactivity_timeout' => q[0],
  'index_expire' => q[1],
  'inhibit_startup_message' => q[0],
  'keep_source_where' => q[/var/cpan/sources],
  'links' => q[/usr/bin/links],
  'make' => q[/usr/bin/make],
  'make_arg' => q[],
  'make_install_arg' => q[],
  'makepl_arg' => q[],
  'ncftp' => q[],
  'ncftpget' => q[],
  'no_proxy' => q[],
  'pager' => q[/usr/bin/less],
  'prerequisites_policy' => q[ask],
  'scan_cache' => q[atstart],
  'shell' => q[/bin/bash],
  'tar' => q[/bin/tar],
  'term_is_latin' => q[1],
  'unzip' => q[/usr/bin/unzip],
  'urllist' => [q[ftp://cpan.pair.com/pub/CPAN/], q[ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/], q[ftp://ftp.duke.edu/pub/perl/], q[ftp://ftp.ibiblio.org/pub/languages/perl/CPAN], q[ftp://ftp.sunsite.utk.edu/pub/CPAN/], q[ftp://csociety-ftp.ecn.purdue.edu/pub/CPAN], q[ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/]],
  'wget' => q[/usr/bin/wget],
};
1;
__END__
EOF

Now run:
perl -MCPAN -e shell
and install the following packages:
cpan> install Bundle::CPAN  
cpan> install Mail::IMAPClient
cpan> install Net::SSLeay
cpan> install Net::LDAP

Now make sure the packages are correctly installed and available:
perl -e 'use Mail::IMAPClient'
If you see an error message saying: Can't locate Mail/IMAPClient.... then
there is something wrong.  If you see no output then it means the package
was installed correctly.

Now proceed to build heartbeat (uncompress the package and replace)
cd /usr/src/redhat/SPEC
Edit heartbeat.spec as follows:
Release:        2.rh.el.3.0
by
Release:        2.rh.el.4.0

rpmbuild --rebuild heartbeat.spec
Now install all the rpms
rpm -ivh heartbeat-*
The package heartbeat-ldirectord will bitch about perl-Mail-IMAPClient, 
perl-Net-SSLeay and perl-ldap not being installed.  They were not installed
as rpms, but you installed them and tested them, so force the install:
rpm -ivh --nodeps heartbeat-*

-------------------- Heartbeat Configuration ------------------------
You should look at: http://www.linux-ha.org/download/GettingStarted.html
if you have questions.

Make sure your systems have the network interfaces configured.
I usually give the private interfaces 10.0.0.X addresses where X is
the same number of the real ip of the server.  So if eth0 has IP
128.61.131.16 then eth1 gets 10.0.0.16.

Test that your serial connection is working.  Connect the cable
between the two machines and then on one machine type:
cat </dev/ttyS0

On the other machine you should type:
echo "wasssssssup" >/dev/ttyS0

On machine one you will see:
[root at biolab1 ha.d]# cat </dev/ttyS0
wasssssssup

OK, so your serial connection works, keep going

You will need to create the files: authkeys, ha.cf, haresources in
/etc/ha.d
cd /etc/ha.d
cat <<EOF > authkeys
uth 1
1 sha1 12345abcd98766zyxwv
EOF

Note:  The key can be and alphanumeric sequence.  There is a limit in length.
       I do not know what it is, but do not make it too long.

Adjust as necessary for your configuration
cat <<EOF > ha.cf
serial          /dev/ttyS0
bcast           eth1
keepalive       2
warntime        10
deadtime        20
initdead        30
baud            19200
udpport         694
auto_failback   on
node arwen.ibb.gatech.edu
node aragorn.ibb.gatech.edu
respawn hacluster /usr/lib/heartbeat/ipfail
ping 128.61.131.1 128.61.132.1 128.61.133.1
EOF

cat <<EOF > haresources
arwen.ibb.gatech.edu 128.61.131.15 drbddisk Filesystem::/dev/drbd0::/export::ext3::rw,usrquota,grpquota,acl,user_xattr smb nfs restart-rpcidmapd
EOF

NOTE
If you had more than one drbd device and you wanted to run different services on 
each one of the nodes so that you would not have one node sitting there doing nothing,
your haresources will look like:

cat <<EOF > haresources
sauron.ibb.gatech.edu 128.61.131.20 drbddisk::export Filesystem::/dev/drbd0::/export::ext3::rw,usrquota,grpquota,acl,user_xattr smb
saruman.ibb.gatech.edu 128.61.131.19 drbddisk::web Filesystem::/dev/drbd1::/web::ext3::rw,usrquota,grpquota,acl,user_xattr httpd
EOF

The example above sets the resource called export as primary on sauron and starts smb 
on that server, while it also sets the resource web as primary on saruman and runs a
web server on that node.

The section above is for drbd and heartbeat.  It serves the virtual IP:
128.61.131.15 with the services:
drbddisk sets the drbd device as primary
Filesystem::/dev/drbd0::/export::ext3::rw,usrquota,grpquota,acl,user_xattr mounts the
file system on IP takeover
smb starts samba
nfs starts nfs server
restart-rpcidmapd - Restarts the rpcidmapd service after nfs is started.  This is
                    required to avoid nfs client error: 
			kernel: decode_getfattr: xdr error 10008!
cat <<EOF > /etc/ha.d/resource.d/restart-rpcidmapd
#!/bin/bash
#
# This script is inteded to be used as resource script by heartbeat
#
# Mar 2005 by Diego Remolina.
#
###

case "$1" in
    start)
        echo "start"
        /etc/init.d/rpcidmapd restart;;
    stop)
        echo "stop"
        /etc/init.d/rpcidmapd restart;;
    status)
        /etc/init.d/rpcidmapd status;;
    *)
        echo "Usage: restart-rpcidmapd {start|stop|status}"
        exit 1
        ;;
esac

exit 0
EOF

Now you can actually start hearbeat
/etc/init.d/drbd start  (If you are using drbd)
/etc/init.d/heartbeat start

Note that heartbeat should start all the shared services: e.g. smb, nfs, etc.
You MUST NOT mount your drbd device from /etc/fstab.
[root at arwen ha.d]# chkconfig --list | grep drbd
drbd            0:off   1:off   2:off   3:on    4:on    5:on    6:off
[root at arwen ha.d]# chkconfig --list | grep heartbeat
heartbeat       0:off   1:off   2:off   3:on    4:on    5:on    6:off
ot at arwen ha.d]# chkconfig --list | grep smb
smb             0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root at arwen ha.d]# chkconfig --list | grep nfs
nfs             0:off   1:off   2:off   3:off   4:off   5:off   6:off
nfslock         0:off   1:off   2:off   3:on    4:on    5:on    6:off
-------------- next part --------------
drbd needs to be updated (re-compiled) everytime a new kernel is installed.
In order to update drbd you need to follow these steps:

1. Make sure there were not protocol or api changes so that you can do changes
   in the secondary, then let it sync and then bring the primary down.

ON THE SECONDARY (aragorn):

2. Update the system: up2date -u -f
   so that the new kernels get installed
3. Extract the sources in /usr/src:
   cd /usr/src
   tar -xzf /path/to/drbd-0.7.11.tar.gz
4. Find out what the new kernel is so that you can build drbd for that kernel:
   root at aragorn drbd-0.7.11]# ls /usr/src/kernels/
   2.6.9-11.EL-hugemem-i686  2.6.9-11.EL-i686  2.6.9-11.EL-smp-i686  2.6.9-5.0.5.EL-i686
   Since 2.6.9-11 is the latest kernel, then we want to build the package for the smp
   kernel.
5. cd /usr/src/drbd-0.7.11
   Version 0.7.11 has a catch on RHEL4.  Since redhat modified the kernel, you need to
   apply the following patch:
   
-----------------patch begins-----------------------------
--- drbd/drbd_int.h     2005-06-07 13:21:52.000000000 -0400
+++ drbd/drbd_int.h.FIXED       2005-06-30 07:52:25.186292464 -0400
@@ -1554,7 +1554,7 @@
 static inline void drbd_suicide(void)
 {
 #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10)
-       set_current_state(TASK_ZOMBIE);
+       set_current_state(EXIT_ZOMBIE);
 #else
        current->exit_state = EXIT_ZOMBIE;
 #endif
----------------patch ends-------------------------------
   
6. Now you are ready to build for your new kernel:
   make rpm KDIR=/usr/src/kernels/2.6.9-11.EL-smp-i686
7. Install the rpms located in dist/RPMS/{arch}/
   rpm -Uvh dist/RPMS/i386/drbd-*

   NOTE: This will stop drbd !!!!!!!!!!   

8. Reboot the server.

NOW THE PRIMARY

9. Copy the rpms from the secondary to the primary. DO NOT INSTALL THEM
   YET. /tmp is a good location.

10. Stop heartbeat on the primary and make sure the secondary took over
    Usually nfs4 needs the /etc/init.d/rpcidmapd restart for the clients
    to keep working.

11. Update the primary:  up2date -u -f

12. Install the rpms:
    cd /tmp
    rpm -Uvh drbd-*.rpm

13. Reboot the primary and make sure it comes back up and takes over
    correctly.


More information about the drbd-user mailing list