Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I'll try not to speculate in an effort to save face and not appear [too]
foolish.
I stood up drbd 8.3.8 on 2 Gentoo VM's. Replication was working between the
nodes & everything was super quick. During a ~100GB transfer to the primary
I stopped the service on the secondary to see how quickly it would catch up.
After a few minutes I restarted the drbd service on the secondary node and
it replicated just fine. Sometime after that (after it caught up and wasn't
oos) I again stopped the service on the secondary node but this time left it
off until the job completed (a couple of hours). When I restarted it, they
wouldn't sync. The connection states would change but the secondary wasn't
'catching up.'
I tried collecting as much information as possible but it's too much to
share here; dare I say too much to be useful.? Instead, if that's ok, I'd
like to refer to my pastebin where I've stored this information:
http://pastebin.com/VEs6YYgR
It includes /var/log/messages, the config file dmesg, /proc/drbd and other
output.
DRBD wasn't built as a module; its baked into the kernel; lsmod doesn't show
the drbd module loaded. (i.e.: find /lib/modules -name "drbd*" and find
/lib/modules/2.6.*gentoo-*/ -type f -iname '*.o' -or -iname '*.ko' | grep
drbd return no results)
Linux primary 2.6.38-gentoo-r6 #1 SMP Mon Jun 27 07:35:00 EDT 2011 x86_64
Intel(R) Xeon(R) CPU E5504 @ 2.00GHz GenuineIntel GNU/Linux
Linux secondary 2.6.38-gentoo-r6 #1 SMP Sun Jun 26 15:57:21 EDT 2011 x86_64
Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz GenuineIntel GNU/Linux
Using 'sys-cluster/drbd' emerge build (v 8.3.8.1) on both nodes but I did
not emerge 'sys-cluster/drbd-kernel' (v 8.0.16).
Other steps performed:
restarted services on both nodes
`drbdadm {disconnect,connect} resource` on both nodes
`drbdadm role resource` on both nodes and primary reports primary, secondary
reports secondary
`drbdadm -- --discard-my-data connect resource` on the secondary
I tried trolling through a number of forums & mailing lists, and I feel it's
very similar to what was outlined here:
http://copilotco.com/mail-archives/drbd.2009/msg00449.html. Instead of a
bad link, or link with latency I simply stopped the DRBD service on the
secondary to simulate an outage or disconnect scenario. I left it off for
hours just to get an idea of how well it replicated & how long it would take
but to my dismay it hasn't reconnected.
I'm not sure what I may have done wrong. I feel I've run into this issue
before and I couldn't get the boxes to sync unless I restarted them both.
This time I would prefer to troubleshoot this a bit more while I have the
problem to see if I can learn from this & avoid these types of problems in
the future.
Many thanks to all for reviewing this message as well as your time & effort.
Cheers
J
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110630/a54b6ab1/attachment.htm>