Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
there you go...
Diego
On Wednesday 03 August 2005 10:20, Felippe Beling wrote:
> Hi Mrs,
> I need of a how to complete to implement DRBD. I found something in the
> internet but this not helped me. Anyone can send me ?
>
> Bye, Tks.
--
Diego Julian Remolina
System Administrator - Systems Support Specialist II
Institute for Bioengineering and Bioscience
Georgia Institute of Technology
Phone (404) 385-0127
-------------- next part --------------
This is how DRBD was installed and configured on IBB Staff systems:
arwen is the primary
aragorn is the slave
1. Obtaining and Installing drbd.
As a regular user
cd /tmp
wget http://oss.linbit.com/drbd/0.7/drbd-0.7.10.tar.gz
Move the file to the server where you want to run drbd
scp drbd-0.7.10.tar.gz root at arwen:/tmp
ssh root at arwen
cd /usr/src
tar -xzf /tmp/drbd-0.7.10.tar.gz
cd drbd-0.7.10
make rpm
The rpms built just fine in RHEL4. They are located in the folder /usr/src/drbd-0.7.10/dist/RPMS/i386/
cd /usr/src/drbd-0.7.10/dist/RPMS/i386/
rpm -ivh drbd-0.7.10-1.i386.rpm
rpm -ivh drbd-debuginfo-0.7.10-1.i386.rpm
rpm -ivh drbd-km-2.6.9_5.0.3.ELsmp-0.7.10-1.i386.rpm
Copy those files to the second server and install them there.
scp drbd*rpm aragorn2:/tmp
ssh aragorn2 'rpm -ivh /tmp/drbd\*.rpm'
2. Configuration of drbd
Make sure you have one partition for your data and a separate one for your metadata.
It is usually better to have a separate partition for your metadata.
In this setup we use /dev/sda6 for data and /dev/sda7 for metadata.
The default size of metadata used is 128MB per drbd partition. Make sure /dev/sda7 is big
enough if you have more then one drbd partition; e.g. 256MB for 2 drbd partitions, 384MB for
3 drbd partitions, and so on.
Load the kernel module:
modprobe drbd
######################### /etc/drbd.conf file ####################
Create the configuration file /etc/drbd.conf
cat << EOF > /etc/drbd.conf.sample
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd.conf
#
#
resource export {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
startup {
wfc-timeout 0; ## Infinite!
degr-wfc-timeout 120; ## 2 minutes.
}
disk {
on-io-error detach;
}
net {
# timeout 60;
# connect-int 10;
# ping-int 10;
# max-buffers 2048;
# max-epoch-size 2048;
}
syncer {
rate 60M;
group 1;
al-extents 257;
}
on arwen.ibb.gatech.edu {
device /dev/drbd0;
disk /dev/sda6;
address 10.0.0.16:7789;
meta-disk /dev/sda7[0];
}
on aragorn.ibb.gatech.edu {
device /dev/drbd0;
disk /dev/sda6;
address 10.0.0.17:7789;
meta-disk /dev/sda7[0];
}
}
EOF
################## end drbd.conf ##########################
Note: If you have more than one drbd device, then you need to make
sure you adjust the meta-disk entry accordingly. If you had /dev/drbd1,
then the metadisk entries would be:
meta-disk /dev/sda7[1];
so that drbd knows that the metadata for /dev/drbd1 should also go in
/dev/sda7 but be different to the metadata for /deb/drbd0.
Since the drbd documentation does not explain how to use
drbdsetup correctly, we first setup the drbd.conf file and then
run the drbdadm with the -d (dry run method):
[root at arwen /]# drbdadm -d adjust export
/sbin/drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda7 0 --on-io-error=detach
/sbin/drbdsetup /dev/drbd0 syncer --rate=60M --group=1 --al-extents=257
/sbin/drbdsetup /dev/drbd0 net 10.0.0.16:7789 10.0.0.17:7789 C
After confirming that is what you want run the command again without -d
Copy the drbd.conf file to the slave and run the same command
Notice that on the secondary, the IPs get inverted
[root at aragorn ~]# drbdadm -d adjust export
/sbin/drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda7 0 --on-io-error=detach
/sbin/drbdsetup /dev/drbd0 syncer --rate=60M --group=1 --al-extents=257
/sbin/drbdsetup /dev/drbd0 net 10.0.0.17:7789 10.0.0.16:7789 C
Now back on the primary set /dev/drbd0 as the primary with the command:
drbdsetup /dev/drbd0 primary --do-what-I-say
On the primary machine:
Create the filesystem and disable automatic fsck after certain number of mounts
or by time intervals:
Mare sure you create the filesystem on /dev/drbd and NOT on /dev/da6 or you
will find problems later with things like quotas...
mkfs -j /dev/drbd0
tune2fs -c -1 -i 0 /dev/drbd0
3. Starting drbd
Firewall Considerations
If you are running a firewall, make sure the ports you used for each drbd set
on /etc/drbd.conf are open. A more relaxed setting will be to simply allow
all trafic by adding the following line to /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -i ethX -j ACCEPT
where ethX refers to whichever interface is dedicated to replicate the data
over the network. This iptables rule is excellent if you are using a
cross-over cable.
Now start drbd on both systems:
root at arwen /]# /etc/init.d/drbd start
root at aragorn /]# /etc/init.d/drbd start
You can see what is going on with:
[root at arwen /]# cat /proc/drbd
or
[root at arwen /]# /etc/init.d/drbd status
version: 0.7.10 (api:77/proto:74)
SVN Revision: 1743 build by root at arwen.ibb.gatech.edu, 2005-02-22 09:05:46
0: cs:SyncSource st:Primary/Secondary ld:Consistent
ns:3221512 nr:0 dw:8 dr:3221649 al:1 bm:33108 lo:0 pe:0 ua:0 ap:0
[>...................] sync'ed: 0.6% (523443/526589)M
finish: 14:30:08 speed: 10,236 (10,096) K/sec
1: cs:Unconfigured
4. Loading the module on reboot:
You will need to edit the file /etc/rc.modules and add:
modprobe drbd
so that drbd is loaded everytime the system boots up.
chmod 744 /etc/rc.modules or it will not run.
APPENDIX
In cases where you have more than one drbd device, the drbd.conf file will
have 2 resource entries. The considerations are:
- Make sure you do not use the same drbd device for each resource. This
means that the first resource will use /dev/drbd0, the second /dev/drbd1
and so on. Also make sure you are not using the same partition for more
than one drbd device, meaning, if you used /dev/hda3 for /dev/drbd0, you
can no longer use /dev/hda3 for /dev/drbd1.
- You can use the same interface to sync both resources, as long as you use
a different port for each resource. For example, use port 7789 for resource
one and port 7790 for resource two.
- The metadata partition can be allocated on the same physical device, but
make sure that you change the metadata id. Resouce one will use [0], resource
two will use [1], and so on.
Here is an example of a drbd.conf with two drbd devices:
######################### /etc/drbd.conf file ####################
cat << EOF > /etc/drbd.conf.sample
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd.conf
#
#
resource export {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
startup {
wfc-timeout 0; ## Infinite!
degr-wfc-timeout 60; ## 2 minutes.
}
disk {
on-io-error detach;
}
net {
# timeout 60;
# connect-int 10;
# ping-int 10;
# max-buffers 2048;
# max-epoch-size 2048;
}
syncer {
rate 60M;
group 1;
al-extents 257;
}
on sauron.ibb.gatech.edu {
device /dev/drbd0;
disk /dev/sdb1;
address 10.0.0.21:7789;
meta-disk /dev/sdb3[0];
}
on saruman.ibb.gatech.edu {
device /dev/drbd0;
disk /dev/sdb1;
address 10.0.0.22:7789;
meta-disk /dev/sdb3[0];
}
}
resource web {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
startup {
wfc-timeout 0; ## Infinite!
degr-wfc-timeout 60; ## 2 minutes.
}
disk {
on-io-error detach;
}
net {
# timeout 60;
# connect-int 10;
# ping-int 10;
# max-buffers 2048;
# max-epoch-size 2048;
}
syncer {
rate 60M;
group 1;
al-extents 257;
}
on sauron.ibb.gatech.edu {
device /dev/drbd1;
disk /dev/sdb2;
address 10.0.0.21:7790;
meta-disk /dev/sdb3[1];
}
on saruman.ibb.gatech.edu {
device /dev/drbd1;
disk /dev/sdb2;
address 10.0.0.22:7790;
meta-disk /dev/sdb3[1];
}
}
EOF
################## end drbd.conf ##########################
-------------- next part --------------
cd /home/sysadm/src
mkdir -p heartbeat/1.2.3-1
cd !$
wget http://linux-ha.org/download/heartbeat-1.2.3-1.src.rpm (DOES NOT BUILD!!!)
wget http://www.ultramonkey.org/download/heartbeat/stable.latest/redhat_el_3.0/heartbeat-1.2.3-2.rh.el.3.0.src.rpm
scp heartbeat-1.2.3-2.rh.el.3.0.src.rpm root at arwen:/tmp/
ssh arwen
cd /tmp
We need to first install the dependencies:
up2date -i glib-devel
up2date -i net-snmp-devel
up2date -i elfutils-libelf-devel
libnet is not available from RedHat, it was built with a spec
file I downloaded from: http://dag.wieers.com/apt/
Download libnet from:
http://www.packetfactory.net/libnet/
cd /usr/src/redhat/SPECS
rpmbuild -ba libnet.spec
cd /usr/src/redhat/RPMS/i386/
rpm -ivh libnet-*
Download and install ipvsadm:
wget http://people.redhat.com/cfeist/cluster/RHEL4/alpha/latest_rpms/ipvsadm-1.24-6.i386.rpm
wget http://mirror.linux.duke.edu/pub/fedora/linux/core/4/x86_64/os/SRPMS/ipvsadm-1.24-7.src.rpm
rpmbuild --rebuild ipvsadm-1.24-7.src.rpm
In order to get the heartbeat rpms built, you need a few perl
modules. Make sure your CPAN shell is configured and working.
cat <<EOF > /usr/lib/perl5/5.8.5/CPAN/Config.pm
# This is CPAN.pm's systemwide configuration file. This file provides
# defaults for users, and the values can be changed in a per-user
# configuration file. The user-config file is being looked for as
# ~/.cpan/CPAN/MyConfig.pm.
$CPAN::Config = {
'build_cache' => q[10],
'build_dir' => q[/var/cpan/build],
'cache_metadata' => q[1],
'cpan_home' => q[/var/cpan],
'dontload_hash' => { },
'ftp' => q[/usr/kerberos/bin/ftp],
'ftp_proxy' => q[],
'getcwd' => q[cwd],
'gpg' => q[/usr/bin/gpg],
'gzip' => q[/bin/gzip],
'histfile' => q[/var/cpan/histfile],
'histsize' => q[5000],
'http_proxy' => q[],
'inactivity_timeout' => q[0],
'index_expire' => q[1],
'inhibit_startup_message' => q[0],
'keep_source_where' => q[/var/cpan/sources],
'links' => q[/usr/bin/links],
'make' => q[/usr/bin/make],
'make_arg' => q[],
'make_install_arg' => q[],
'makepl_arg' => q[],
'ncftp' => q[],
'ncftpget' => q[],
'no_proxy' => q[],
'pager' => q[/usr/bin/less],
'prerequisites_policy' => q[ask],
'scan_cache' => q[atstart],
'shell' => q[/bin/bash],
'tar' => q[/bin/tar],
'term_is_latin' => q[1],
'unzip' => q[/usr/bin/unzip],
'urllist' => [q[ftp://cpan.pair.com/pub/CPAN/], q[ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/], q[ftp://ftp.duke.edu/pub/perl/], q[ftp://ftp.ibiblio.org/pub/languages/perl/CPAN], q[ftp://ftp.sunsite.utk.edu/pub/CPAN/], q[ftp://csociety-ftp.ecn.purdue.edu/pub/CPAN], q[ftp://ftp.cise.ufl.edu/pub/mirrors/CPAN/]],
'wget' => q[/usr/bin/wget],
};
1;
__END__
EOF
Now run:
perl -MCPAN -e shell
and install the following packages:
cpan> install Bundle::CPAN
cpan> install Mail::IMAPClient
cpan> install Net::SSLeay
cpan> install Net::LDAP
Now make sure the packages are correctly installed and available:
perl -e 'use Mail::IMAPClient'
If you see an error message saying: Can't locate Mail/IMAPClient.... then
there is something wrong. If you see no output then it means the package
was installed correctly.
Now proceed to build heartbeat (uncompress the package and replace)
cd /usr/src/redhat/SPEC
Edit heartbeat.spec as follows:
Release: 2.rh.el.3.0
by
Release: 2.rh.el.4.0
rpmbuild --rebuild heartbeat.spec
Now install all the rpms
rpm -ivh heartbeat-*
The package heartbeat-ldirectord will bitch about perl-Mail-IMAPClient,
perl-Net-SSLeay and perl-ldap not being installed. They were not installed
as rpms, but you installed them and tested them, so force the install:
rpm -ivh --nodeps heartbeat-*
-------------------- Heartbeat Configuration ------------------------
You should look at: http://www.linux-ha.org/download/GettingStarted.html
if you have questions.
Make sure your systems have the network interfaces configured.
I usually give the private interfaces 10.0.0.X addresses where X is
the same number of the real ip of the server. So if eth0 has IP
128.61.131.16 then eth1 gets 10.0.0.16.
Test that your serial connection is working. Connect the cable
between the two machines and then on one machine type:
cat </dev/ttyS0
On the other machine you should type:
echo "wasssssssup" >/dev/ttyS0
On machine one you will see:
[root at biolab1 ha.d]# cat </dev/ttyS0
wasssssssup
OK, so your serial connection works, keep going
You will need to create the files: authkeys, ha.cf, haresources in
/etc/ha.d
cd /etc/ha.d
cat <<EOF > authkeys
uth 1
1 sha1 12345abcd98766zyxwv
EOF
Note: The key can be and alphanumeric sequence. There is a limit in length.
I do not know what it is, but do not make it too long.
Adjust as necessary for your configuration
cat <<EOF > ha.cf
serial /dev/ttyS0
bcast eth1
keepalive 2
warntime 10
deadtime 20
initdead 30
baud 19200
udpport 694
auto_failback on
node arwen.ibb.gatech.edu
node aragorn.ibb.gatech.edu
respawn hacluster /usr/lib/heartbeat/ipfail
ping 128.61.131.1 128.61.132.1 128.61.133.1
EOF
cat <<EOF > haresources
arwen.ibb.gatech.edu 128.61.131.15 drbddisk Filesystem::/dev/drbd0::/export::ext3::rw,usrquota,grpquota,acl,user_xattr smb nfs restart-rpcidmapd
EOF
NOTE
If you had more than one drbd device and you wanted to run different services on
each one of the nodes so that you would not have one node sitting there doing nothing,
your haresources will look like:
cat <<EOF > haresources
sauron.ibb.gatech.edu 128.61.131.20 drbddisk::export Filesystem::/dev/drbd0::/export::ext3::rw,usrquota,grpquota,acl,user_xattr smb
saruman.ibb.gatech.edu 128.61.131.19 drbddisk::web Filesystem::/dev/drbd1::/web::ext3::rw,usrquota,grpquota,acl,user_xattr httpd
EOF
The example above sets the resource called export as primary on sauron and starts smb
on that server, while it also sets the resource web as primary on saruman and runs a
web server on that node.
The section above is for drbd and heartbeat. It serves the virtual IP:
128.61.131.15 with the services:
drbddisk sets the drbd device as primary
Filesystem::/dev/drbd0::/export::ext3::rw,usrquota,grpquota,acl,user_xattr mounts the
file system on IP takeover
smb starts samba
nfs starts nfs server
restart-rpcidmapd - Restarts the rpcidmapd service after nfs is started. This is
required to avoid nfs client error:
kernel: decode_getfattr: xdr error 10008!
cat <<EOF > /etc/ha.d/resource.d/restart-rpcidmapd
#!/bin/bash
#
# This script is inteded to be used as resource script by heartbeat
#
# Mar 2005 by Diego Remolina.
#
###
case "$1" in
start)
echo "start"
/etc/init.d/rpcidmapd restart;;
stop)
echo "stop"
/etc/init.d/rpcidmapd restart;;
status)
/etc/init.d/rpcidmapd status;;
*)
echo "Usage: restart-rpcidmapd {start|stop|status}"
exit 1
;;
esac
exit 0
EOF
Now you can actually start hearbeat
/etc/init.d/drbd start (If you are using drbd)
/etc/init.d/heartbeat start
Note that heartbeat should start all the shared services: e.g. smb, nfs, etc.
You MUST NOT mount your drbd device from /etc/fstab.
[root at arwen ha.d]# chkconfig --list | grep drbd
drbd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
[root at arwen ha.d]# chkconfig --list | grep heartbeat
heartbeat 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ot at arwen ha.d]# chkconfig --list | grep smb
smb 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root at arwen ha.d]# chkconfig --list | grep nfs
nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
-------------- next part --------------
drbd needs to be updated (re-compiled) everytime a new kernel is installed.
In order to update drbd you need to follow these steps:
1. Make sure there were not protocol or api changes so that you can do changes
in the secondary, then let it sync and then bring the primary down.
ON THE SECONDARY (aragorn):
2. Update the system: up2date -u -f
so that the new kernels get installed
3. Extract the sources in /usr/src:
cd /usr/src
tar -xzf /path/to/drbd-0.7.11.tar.gz
4. Find out what the new kernel is so that you can build drbd for that kernel:
root at aragorn drbd-0.7.11]# ls /usr/src/kernels/
2.6.9-11.EL-hugemem-i686 2.6.9-11.EL-i686 2.6.9-11.EL-smp-i686 2.6.9-5.0.5.EL-i686
Since 2.6.9-11 is the latest kernel, then we want to build the package for the smp
kernel.
5. cd /usr/src/drbd-0.7.11
Version 0.7.11 has a catch on RHEL4. Since redhat modified the kernel, you need to
apply the following patch:
-----------------patch begins-----------------------------
--- drbd/drbd_int.h 2005-06-07 13:21:52.000000000 -0400
+++ drbd/drbd_int.h.FIXED 2005-06-30 07:52:25.186292464 -0400
@@ -1554,7 +1554,7 @@
static inline void drbd_suicide(void)
{
#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10)
- set_current_state(TASK_ZOMBIE);
+ set_current_state(EXIT_ZOMBIE);
#else
current->exit_state = EXIT_ZOMBIE;
#endif
----------------patch ends-------------------------------
6. Now you are ready to build for your new kernel:
make rpm KDIR=/usr/src/kernels/2.6.9-11.EL-smp-i686
7. Install the rpms located in dist/RPMS/{arch}/
rpm -Uvh dist/RPMS/i386/drbd-*
NOTE: This will stop drbd !!!!!!!!!!
8. Reboot the server.
NOW THE PRIMARY
9. Copy the rpms from the secondary to the primary. DO NOT INSTALL THEM
YET. /tmp is a good location.
10. Stop heartbeat on the primary and make sure the secondary took over
Usually nfs4 needs the /etc/init.d/rpcidmapd restart for the clients
to keep working.
11. Update the primary: up2date -u -f
12. Install the rpms:
cd /tmp
rpm -Uvh drbd-*.rpm
13. Reboot the primary and make sure it comes back up and takes over
correctly.