<div dir="ltr">
<div>Hello all, my first post.<br>it is quite long, but I try to give details.<br><br>I have 2xcentos5.2 servers with heartbeat+drbd<br>two eth channels and no stonith at the moment (suggestions in this respect are welcome, yet to analyze in deep)<br>
packages versions are<br>heartbeat-2.1.3-3.el5.centos<br>drbd82-8.2.6-1.el5.centos<br>kmod-drbd82-8.2.6-1.2.6.18_92.el5<br>kernel-2.6.18-92.el5<br><br>I intend to provide nfs services in HA and consulted many docs, arriving at a specific config (see below)<br>
At the moment I'm planning to give only a primary/slave service on one nfs resource<br><br>I'm trying to consider and simulate various planned/unplanned scenarios and at the moment I have this with doubts about heartbeat and/or drbd config and relative "service" behaviour.<br>
Excuse if it could be offtopic in case of a misconfiguration of heartbeat <br><br>I have the cluster running:<br>nfsnode1 active and master<br>nfsnode2 active and slave<br><br>heartbeat resource is<br>nfsnode2 drbddisk::drbd-resource-0 \<br>
Filesystem::/dev/drbd0::/drbd0::ext3 \<br> killnfsd \<br> nfslock \<br> nfs \<br> Delay::3::0 \<br> IPaddr::<a href="http://10.4.5.103/24/eth0">10.4.5.103/24/eth0</a><br><br>and config is<br>
<br>keepalive 1<br>deadtime 10<br>warntime 2<br>ucast eth0 <a href="http://10.4.5.102">10.4.5.102</a><br>ucast eth1 <a href="http://10.4.192.242">10.4.192.242</a><br>auto_failback off<br>node nfsnode1<br>node nfsnode2<br>
respawn hacluster /usr/lib/heartbeat/dopd<br>apiauth dopd gid=haclient uid=hacluster<br>use_logd yes<br><br>actions<br>1) nfs service provided and write operations on drbd device active from clients<br>2) shutdown nfsnode2 (so heartbeat and drbd cleanly stop)<br>
<br>3) write operations continue against drbd device (on nfsnode1) while nfsnode2 is powered off<br><br>4) shutdown nfsnode1 (so heartbeat and drbd cleanly stop)<br><br>5) restart nfsnode2 with nfsnode1 yet powered down<br>
(suppose for example wrong action done by an operator during maintenance activities..)<br>===> I think it should not start the services as it was slave when shut down<br>and so it is .. Good<br><br>6) start nfsnode1<br>
===> I would expect now nfsnode1 carrying on the service as it was the latest master, while the other was slave and both shutdown operations were clean, and infact sync correctly happens between the twos<br>But both drbd resources remain Secondary... Bad (in my opinion)<br>
<br>0:drbd-resource-0 Connected Secondary/Secondary UpToDate/UpToDate C<br>So that heartbeat chain doesn't start and nfs service is not provided<br>my drbd.conf at the moment is:<br>resource "drbd-resource-0" {<br>
protocol C;<br> handlers {<br> pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";<br> pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";<br> local-io-error "echo o > /proc/sysrq-trigger ; halt -f";<br>
outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";<br> }<br> startup {<br># wfc-timeout 0; ## Infinite!<br> degr-wfc-timeout 120; ## 2 minutes.<br> }<br> disk {<br> on-io-error detach;<br>
fencing resource-only;<br> }<br> net {<br> after-sb-0pri disconnect;<br> after-sb-1pri disconnect;<br> after-sb-2pri disconnect;<br> rr-conflict disconnect;<br> }<br> syncer {<br> rate 60M;<br> al-extents 257;<br>
}<br><br> # It is valid to move device, disk and meta-disk to the<br> # resource level.<br> device /dev/drbd0;<br> disk /dev/sdb1;<br> meta-disk internal;<br><br> on nfsnode1 {<br> address <a href="http://10.4.192.241:7789">10.4.192.241:7789</a>;<br>
}<br><br> on nfsnode2 {<br> address <a href="http://10.4.192.242:7789">10.4.192.242:7789</a>;<br> }<br>}<br><br>Some log data:<br><br>at shutdown of nfsnode2 at step 2), nfsnode1 becomes<br><br>0:drbd-resource-0 WFConnection Primary/Unknown UpToDate/Outdated C /drbd0<br>
<br>and in messages:<br>Aug 22 12:42:04 nfsnode1 kernel: drbd0: State change failed: Refusing to be Primary while peer is not outdated<br>Aug 22 12:42:04 nfsnode1 kernel: drbd0: state = { cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate r--- }<br>
Aug 22 12:42:04 nfsnode1 kernel: drbd0: wanted = { cs:TearDown st:Primary/Unknown ds:UpToDate/DUnknown r--- }<br>Aug 22 12:42:04 nfsnode1 kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> Outdated )<br>
<br>at restart of nfsnode2 in step 5) it will wait forever connection with nfsnode1 in os console (wfc-timeout=0 )<br>And this is a safe (?) step because it is not a correct situation<br>I stop the wait writing in console yes+<INVIO> (I simulate the perfect operator... I have been an operator too in the past, so I can tell... ;-)<br>
<br>drbd status in nfsnode2 is and remains<br>0:drbd-resource-0 WFConnection Secondary/Unknown Outdated/DUnknown C<br><br>and in the log 6 times:<br>Aug 22 12:57:37 nfsnode2 kernel: drbd0: State change failed: Refusing to be Primary without at least one UpToDate disk<br>
Aug 22 12:57:37 nfsnode2 kernel: drbd0: state = { cs:WFConnection st:Secondary/Unknown ds:Outdated/DUnknown r--- }<br>Aug 22 12:57:37 nfsnode2 kernel: drbd0: wanted = { cs:WFConnection st:Primary/Unknown ds:Outdated/DUnknown r--- }<br>
...<br>Aug 22 12:57:43 nfsnode2 ResourceManager[2610]: [2683]: ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk<br>Aug 22 12:57:43 nfsnode2 ResourceManager[2610]: [2684]: CRIT: Giving up resources due to failure of drbddisk::drbd-resource-0<br>
Aug 22 12:57:43 nfsnode2 ResourceManager[2610]: [2685]: info: Releasing resource group: nfsnode2 drbddisk::drbd-resource-0 Filesystem::/dev/drbd0::/drbd0::ext3 killnfsd nfslock nfs Delay::3::0 IPaddr::<a href="http://10.4.5.103/24/eth0">10.4.5.103/24/eth0</a><br>
...<br>Aug 22 12:58:13 nfsnode2 hb_standby[3047]: [3053]: Going standby [foreign].<br>Aug 22 12:58:14 nfsnode2 heartbeat: [2204]: info: nfsnode2 wants to go standby [foreign]<br>Aug 22 12:58:24 nfsnode2 heartbeat: [2204]: WARN: No reply to standby request. Standby request cancelled.<br>
<br>Now startup nfsnode1 step 6):<br>the node nfsnode1 correctly executes sync of drbd data versus nfsnode2 that was not aligned<br><br>Aug 22 13:06:11 nfsnode1 kernel: drbd0: Began resync as SyncSource (will sync 304 KB [76 bits set]).<br>
Aug 22 13:06:11 nfsnode1 kernel: drbd0: Writing meta data super block now.<br>Aug 22 13:06:11 nfsnode1 kernel: drbd0: Resync done (total 1 sec; paused 0 sec; 304 K/sec)<br>Aug 22 13:06:11 nfsnode1 kernel: drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<br>
<br>at the end both nodes are in state Secondary<br><br>0:drbd-resource-0 Connected Secondary/Secondary UpToDate/UpToDate C<br><br>and so heartbeat doesn't activate nfs service and relative virtual ip<br><br>heartbeat logs give:<br>
Aug 22 13:06:12 nfsnode1 heartbeat: [2040]: info: Local status now set to: 'up'<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Link nfsnode2:eth0 up.<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Status update for node nfsnode2: status active<br>
Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: WARN: G_CH_dispatch_int: Dispatch function for read child took to<br>o long to execute: 100 ms (> 50 ms) (GSource: 0x83b8940)<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Link nfsnode2:eth1 up.<br>
Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Comm_now_up(): updating status to active<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Local status now set to: 'active'<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Starting child client "/usr/lib/heartbeat/dopd" (498,496)<br>
Aug 22 13:06:13 nfsnode1 heartbeat: [2188]: info: Starting "/usr/lib/heartbeat/dopd" as uid 498 gid 496 (pid<br> 2188)<br>Aug 22 13:06:13 nfsnode1 harc[2186]: [2196]: info: Running /etc/ha.d/rc.d/status status<br>
Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: remote resource transition completed.<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: remote resource transition completed.<br>Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Local Resource acquisition completed. (none)<br>
Aug 22 13:06:13 nfsnode1 heartbeat: [2040]: info: Initial resource acquisition complete (T_RESOURCES(them))<br><br>Any hints and suggestions? <br>Thanks in advance<br><br>Gianluca<br></div>
</div>