<br>
<br><font size=2><tt> Following on my Three node setup questions I
have attempted to</tt></font>
<br><font size=2><tt> setup a three node cluster. See the drbd.conf
at the end of this</tt></font>
<br><font size=2><tt> message and the following background.</tt></font>
<br>
<br><font size=2><tt> I get to the point of executing "drbdadm
--stacked primary drbd-U"</tt></font>
<br><font size=2><tt> and I get the following error. Any help would
be appreciated.</tt></font>
<br>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm --stacked primary drbd-U<br>
/dev/drbd1: State change failed: (-2) Refusing to be Primary without at
least one UpToDate disk<br>
Command 'drbdsetup /dev/drbd1 primary' terminated with exit code 17</tt></font>
<br>
<br>
<br><font size=2><tt> HW and SW Background :</tt></font>
<br>
<br><font size=2><tt> All three machines :<br>
- HW : - Proliant DL380G4<br>
- SW : - Latest RHEL5 and kernel-2.6.18-128.1.1.el5</tt></font>
<br><font size=2><tt>- drbd : drbd-8.3.0-3, drbd-km-2.6.18_128.1.1.el5-8.3.0-3<br>
</tt></font>
<br>
<br><font size=2><tt> Here is what the game plan was :</tt></font>
<br>
<br><font size=2><tt>1. Bring up the underlying resource "drbd"
:<br>
On hatest1/2:<br>
- drbdadm create-md drbd<br>
- modprobe drbd<br>
- drbdadm up drbd<br>
On hatest1:<br>
- drbdadm -- --overwrite-data-of-peer primary drbd</tt></font>
<br><font size=2><tt>2. Manually on the "primary" bring up the
165.115.204.197(hatest)<br>
address specified for the stacked resource - it is this<br>
address which talks to the nlstor1 backup.</tt></font>
<br>
<br><font size=2><tt>On hatest1:<br>
ifconfig eth1:0 165.115.204.197 up</tt></font>
<br>
<br><font size=2><tt>3. Start the stacked drbd-U resource :<br>
</tt></font>
<br><font size=2><tt>As with unstacked resources, you must create DRBD
meta data on the stacked resources. This is done using the following command:<br>
</tt></font>
<br><font size=2><tt>On hatest1:<br>
drbdadm --stacked create-md drbd-U</tt></font>
<br>
<br><font size=2><tt>Then, you may enable the stacked resource:<br>
</tt></font>
<br><font size=2><tt>On hatest1:<br>
drbdadm --stacked up drbd-U<br>
drbdadm --stacked primary drbd-U</tt></font>
<br>
<br><font size=2><tt>After this, you may bring up the resource on the backup
node, enabling three-node replication:<br>
</tt></font>
<br><font size=2><tt>On nlstor1:<br>
drbdadm create-md drbd-U<br>
drbdadm up drbd-U</tt></font>
<br>
<br><font size=2><tt>Here is the command history from the machines as they
were executed :<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm create-md drbd<br>
md_offset 1572859904<br>
al_offset 1572827136<br>
bm_offset 1572777984<br>
</tt></font>
<br><font size=2><tt>Found some data<br>
==> This might destroy existing data! <==</tt></font>
<br>
<br><font size=2><tt>Do you want to proceed?<br>
[need to type 'yes' to confirm] yes<br>
</tt></font>
<br><font size=2><tt>Writing meta data...<br>
initialising activity log<br>
NOT initialized bitmap<br>
New drbd meta data block sucessfully created.<br>
[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest2 etc]# drbdadm create-md drbd<br>
md_offset 1572859904<br>
al_offset 1572827136<br>
bm_offset 1572777984<br>
</tt></font>
<br><font size=2><tt>Found some data<br>
==> This might destroy existing data! <==</tt></font>
<br>
<br><font size=2><tt>Do you want to proceed?<br>
[need to type 'yes' to confirm] yes<br>
</tt></font>
<br><font size=2><tt>Writing meta data...<br>
initialising activity log<br>
NOT initialized bitmap<br>
New drbd meta data block sucessfully created.<br>
[root@hatest2 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# modprobe drbd<br>
[root@hatest1 etc]# lsmod|grep drbd<br>
drbd 220696
0<br>
[root@hatest1 etc]# service drbd status<br>
drbd driver loaded OK; device status:<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest1,
2009-02-13 09:00:29<br>
m:res cs ro ds p mounted fstype<br>
[root@hatest1 etc]# drbdadm up drbd<br>
[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest2 etc]# modprobe drbd<br>
[root@hatest2 etc]# service drbd status<br>
drbd driver loaded OK; device status:<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest2,
2009-02-13 09:59:05<br>
m:res cs ro ds p mounted fstype<br>
[root@hatest2 etc]# drbdadm up drbd<br>
[root@hatest2 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest1,
2009-02-13 09:00:29</tt></font>
<br><font size=2><tt>0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent
C r---<br>
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1535916</tt></font>
<br><font size=2><tt>[root@hatest1 etc]#<br>
[root@hatest2 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest2,
2009-02-13 09:59:05</tt></font>
<br><font size=2><tt>0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent
C r---<br>
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1535916</tt></font>
<br><font size=2><tt>[root@hatest2 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm -- --overwrite-data-of-peer
primary drbd<br>
[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest1,
2009-02-13 09:00:29</tt></font>
<br><font size=2><tt>0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent
C r---<br>
ns:1239832 nr:0 dw:0 dr:1241792 al:0 bm:74 lo:161 pe:175 ua:223 ap:0 ep:1
wo:b oos:301676</tt></font>
<br><font size=2><tt>[===============>....] sync'ed: 80.6% (301676/1535916)K<br>
finish: 0:00:06 speed: 48,812 (49,368) K/sec</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest1,
2009-02-13 09:00:29</tt></font>
<br><font size=2><tt>0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate
C r---<br>
ns:1535916 nr:0 dw:0 dr:1535916 al:0 bm:94 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0</tt></font>
<br><font size=2><tt>[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>root@hatest2 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest2,
2009-02-13 09:59:05</tt></font>
<br><font size=2><tt>0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate
C r---<br>
ns:0 nr:1535916 dw:1535916 dr:0 al:0 bm:94 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0</tt></font>
<br><font size=2><tt>[root@hatest2 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm --stacked create-md drbd-U<br>
md_offset 1572773888<br>
al_offset 1572741120<br>
bm_offset 1572691968<br>
</tt></font>
<br><font size=2><tt>Found some data<br>
==> This might destroy existing data! <==</tt></font>
<br>
<br><font size=2><tt>Do you want to proceed?<br>
[need to type 'yes' to confirm] yes<br>
</tt></font>
<br><font size=2><tt>Writing meta data...<br>
initialising activity log<br>
NOT initialized bitmap<br>
New drbd meta data block sucessfully created.<br>
[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm --stacked up drbd-U<br>
[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm --stacked primary drbd-U<br>
/dev/drbd1: State change failed: (-2) Refusing to be Primary without at
least one UpToDate disk<br>
Command 'drbdsetup /dev/drbd1 primary' terminated with exit code 17<br>
[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest1,
2009-02-13 09:00:29</tt></font>
<br><font size=2><tt>0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate
C r---<br>
ns:1536005 nr:0 dw:89 dr:1536107 al:1 bm:94 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0</tt></font>
<br><font size=2><tt>1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown
A r---<br>
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1535832</tt></font>
<br><font size=2><tt>[root@hatest1 etc]#<br>
</tt></font>
<br><font size=2><tt>root@nlstor1 etc]# drbdadm create-md drbd-U<br>
md_offset 2147479552<br>
al_offset 2147446784<br>
bm_offset 2147381248<br>
</tt></font>
<br><font size=2><tt>Found some data<br>
==> This might destroy existing data! <==</tt></font>
<br>
<br><font size=2><tt>Do you want to proceed?<br>
[need to type 'yes' to confirm] yes<br>
</tt></font>
<br><font size=2><tt>Writing meta data...<br>
initialising activity log<br>
NOT initialized bitmap<br>
New drbd meta data block sucessfully created.<br>
[root@nlstor1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@nlstor1 etc]# drbdadm up drbd-U<br>
[root@nlstor1 etc]#<br>
[root@hatest1 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@hatest1,
2009-02-13 09:00:29</tt></font>
<br><font size=2><tt>0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate
C r---<br>
ns:1536006 nr:0 dw:90 dr:1536158 al:1 bm:94 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:0</tt></font>
<br><font size=2><tt>1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent
A r---<br>
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1535832</tt></font>
<br><font size=2><tt>[root@hatest1 etc]#<br>
[root@nlstor1 etc]# cat /proc/drbd<br>
version: 8.3.0 (api:88/proto:86-89)<br>
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@nlstor1,
2009-02-19 11:56:15<br>
</tt></font>
<br><font size=2><tt>1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent
A r---<br>
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:1535832</tt></font>
<br><font size=2><tt>[root@nlstor1 etc]#<br>
</tt></font>
<br><font size=2><tt>[root@hatest1 etc]# drbdadm --stacked primary drbd-U<br>
/dev/drbd1: State change failed: (-2) Refusing to be Primary without at
least one UpToDate disk<br>
Command 'drbdsetup /dev/drbd1 primary' terminated with exit code 17<br>
[root@hatest1 etc]#<br>
</tt></font>
<br>
<br><font size=2><tt>Here is the drbd.conf :<br>
</tt></font>
<br>
<br>
<br><font size=2><tt>global {<br>
usage-count no;<br>
}<br>
</tt></font>
<br><font size=2><tt>common {<br>
disk {<br>
on-io-error detach; # What to do when the lower level device errors.<br>
}<br>
</tt></font>
<br><font size=2><tt>net {<br>
max-buffers 2048; #datablock buffers used before writing to disk.<br>
ko-count 4; # Peer is dead if this count is exceeded.<br>
#on-disconnect reconnect; # Peer disconnected, try to reconnect.<br>
}<br>
</tt></font>
<br><font size=2><tt>startup {<br>
wfc-timeout 0; # drbd init script will wait infinitely on resources.<br>
degr-wfc-timeout 120; # 2 minutes.<br>
}<br>
} # End of common<br>
</tt></font>
<br><font size=2><tt>resource drbd {<br>
protocol C;<br>
syncer {<br>
rate 100M; # Synchronization rate, in megebytes. Good for 100Mb network.<br>
al-extents 257; # Must be prime, number of active sets.<br>
}<br>
on hatest1 {<br>
device /dev/drbd0; # The name of our drbd device.<br>
disk /dev/hatest_vg/vol01; # Partition we wish drbd to use.<br>
address 192.168.36.130:7788; # node0 IP address and port number.<br>
meta-disk internal; # Stores meta-data in lower portion of disk.<br>
}<br>
on hatest2 {<br>
device /dev/drbd0; # Our drbd device, must match node0.<br>
disk /dev/hatest_vg/vol01; # Partition we wish drbd to use.<br>
address 192.168.36.129:7788; # node0 IP address and port number.<br>
meta-disk internal; # Stores meta-data in lower portion of disk.<br>
}<br>
} # End of resource drbd<br>
</tt></font>
<br><font size=2><tt>resource drbd-U {<br>
protocol A;<br>
</tt></font>
<br><font size=2><tt>syncer {<br>
rate 30M;<br>
al-extents 257;</tt></font>
<br><font size=2><tt>}<br>
</tt></font>
<br><font size=2><tt>stacked-on-top-of drbd {<br>
device /dev/drbd1;<br>
address 165.115.204.197:7788;<br>
}</tt></font>
<br>
<br><font size=2><tt>on nlstor1 {<br>
device /dev/drbd1; # The name of our drbd device.<br>
disk /dev/VolGroup00/LogVol02; # Partition we wish drbd to
use.<br>
address 165.115.163.52:7788; # node0 IP address and port number.<br>
meta-disk internal; # Stores meta-data in lower portion of disk<br>
}</tt></font>
<br><font size=2><tt>} # End of resource drbd-U<br>
</tt></font>
<br>
<br>
<br>