<div dir="ltr">Hi, thanks for reply.<div>Here's the output of drbdsetup status</div><div><br></div><div>node1:</div><div><div><font face="monospace, monospace">root@deb1:~# drbdsetup status</font></div><div><font face="monospace, monospace">.drbdctrl role:Primary</font></div><div><font face="monospace, monospace"> volume:0 disk:UpToDate</font></div><div><font face="monospace, monospace"> volume:1 disk:UpToDate</font></div></div><div><br></div><div>node2:</div><div><span style="font-family:monospace,monospace">root@deb2:~# drbdsetup status</span><br></div><div><div><font face="monospace, monospace">.drbdctrl role:Secondary</font></div><div><font face="monospace, monospace"> volume:0 disk:Inconsistent</font></div><div><font face="monospace, monospace"> volume:1 disk:Inconsistent</font></div><div><font face="monospace, monospace"> deb1 connection:Connecting</font></div></div><div><br></div><div><br></div><div>I figured out, that this problem only occurs when using dedicated interfaces for drbd.</div><div>In a testsetup it's not important to use just one nic, but I want to run drbd for productional use cases.<br></div><div><br></div><div>Here's the complete setup:</div><div><br></div><div><br></div><div><u><b>Node1:</b></u></div><div><br></div><div><u>nic1</u>: </div><div><ul><li>ip: 192.168.2.103 </li><li>netmask: 255.255.255.0<br></li><li>gateway: 192.168.2.1</li></ul></div><div><u>nic2</u>:</div><div><ul><li>ip: 10.0.0.11</li><li>netmask 255.255.255.0</li></ul></div><div><u>hostname</u>:</div><div><ul><li>deb1<br></li></ul></div><div><br></div><div><u>dns</u></div><div><div><ul><li>127.0.0.1 localhost</li><li>10.0.0.11 deb1</li><li>10.0.0.12 deb2</li></ul></div></div><div><br></div><div>volumegroup drbdpool:</div><div><ul><li>/dev/sdb</li></ul></div><div><br></div><div><div><u><b>Node2:</b></u></div><div><br></div><div><u>nic1</u>: </div><div><ul><li>ip: 192.168.2.104 </li><li>netmask: 255.255.255.0<br></li><li>gateway: 192.168.2.1</li></ul></div><div><u>nic2</u>:</div><div><ul><li>ip: 10.0.0.12</li><li>netmask 255.255.255.0</li></ul></div><div><u>hostname</u>:</div><div><ul><li>deb2<br></li></ul></div><div><br></div><div><u>dns:</u></div><div><ul><li>127.0.0.1 localhost</li><li>10.0.0.11 deb1</li><li>10.0.0.12 deb2</li></ul></div></div><div><div>volumegroup drbdpool:</div><div><ul><li>/dev/sdb</li></ul></div></div><div><br></div><div><br></div><div><br></div><div>It seems DRBD cannot figure out who/what is primary...</div><div>DRBD drives me insane... Sometimes it work and sometimes it doesn't...</div><div><br></div><div>drbdmanage init 10.0.0.11 got stuck 2 times... and at the 3rd try it worked like a charm. Hä!?!?!</div><div><br></div><div><br></div><div>Here's the ouput after trying to add a secondary node:</div><div><br></div><div><br></div><div>------------------- 1st node ------------------------</div><div><br></div><div><div><font face="monospace, monospace">root@deb1:~# drbdmanage add-node deb2 10.0.0.12</font></div><div><font face="monospace, monospace">Operation completed successfully</font></div><div><font face="monospace, monospace">Operation completed successfully</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Executing join command using ssh.</font></div><div><font face="monospace, monospace">IMPORTANT: The output you see comes from deb2</font></div><div><font face="monospace, monospace">IMPORTANT: Your input is executed on deb2</font></div><div><font face="monospace, monospace">You are going to join an existing drbdmanage cluster.</font></div><div><font face="monospace, monospace">CAUTION! Note that:</font></div><div><font face="monospace, monospace"> * Any previous drbdmanage cluster information may be removed</font></div><div><font face="monospace, monospace"> * Any remaining resources managed by a previous drbdmanage installation</font></div><div><font face="monospace, monospace"> that still exist on this system will no longer be managed by drbdmanage</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Confirm:</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> yes/no: yes</font></div><div><font face="monospace, monospace">Operation completed successfully</font></div><div><font face="monospace, monospace">root@deb1:~#</font></div></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><div>root@deb1:~# drbdsetup status</div><div>.drbdctrl role:Primary</div><div> volume:0 disk:UpToDate</div><div> volume:1 disk:UpToDate</div><div><br></div><div>root@deb1:~#</div><div>root@deb1:~#</div><div>root@deb1:~# drbdsetup status</div><div>.drbdctrl role:Primary</div><div> volume:0 disk:UpToDate</div><div> volume:1 disk:UpToDate</div><div> deb2 role:Secondary</div><div> volume:0 replication:SyncSource peer-disk:Inconsistent done:15.78</div><div> volume:1 replication:SyncSource peer-disk:Inconsistent done:15.78</div><div><br></div><div>root@deb1:~# drbdsetup status</div><div>.drbdctrl role:Primary</div><div> volume:0 disk:UpToDate</div><div> volume:1 disk:UpToDate</div><div> deb2 role:Secondary</div><div> volume:0 peer-disk:UpToDate</div><div> volume:1 peer-disk:UpToDate</div><div><br></div></font></div><div><br></div><div><br></div><div>------------------- 2nd node ------------------------<br></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">root@deb2:~# drbdsetup status</font></div><div><font face="monospace, monospace">.drbdctrl role:Secondary</font></div><div><font face="monospace, monospace"> volume:0 disk:Inconsistent</font></div><div><font face="monospace, monospace"> volume:1 disk:Inconsistent</font></div><div><font face="monospace, monospace"> deb1 role:Primary</font></div><div><font face="monospace, monospace"> volume:0 replication:SyncTarget peer-disk:UpToDate done:81.46</font></div><div><font face="monospace, monospace"> volume:1 replication:SyncTarget peer-disk:UpToDate done:81.46</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">root@deb2:~# drbdsetup status</font></div><div><font face="monospace, monospace">.drbdctrl role:Secondary</font></div><div><font face="monospace, monospace"> volume:0 disk:UpToDate</font></div><div><font face="monospace, monospace"> volume:1 disk:UpToDate</font></div><div><font face="monospace, monospace"> deb1 role:Primary</font></div><div><font face="monospace, monospace"> volume:0 peer-disk:UpToDate</font></div><div><font face="monospace, monospace"> volume:1 peer-disk:UpToDate</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">root@deb2:~# drbdsetup status</font></div><div><font face="monospace, monospace">.drbdctrl role:Secondary</font></div><div><font face="monospace, monospace"> volume:0 disk:UpToDate</font></div><div><font face="monospace, monospace"> volume:1 disk:UpToDate</font></div><div><font face="monospace, monospace"> deb1 role:Primary</font></div><div><font face="monospace, monospace"> volume:0 peer-disk:UpToDate</font></div><div><font face="monospace, monospace"> volume:1 peer-disk:UpToDate</font></div></div><div><br></div><div><br></div><div>What is going on there and why does it work and sometimes does not?</div><div><br></div><div><br></div><div>Best Regards,</div><div>Toni (Still a big fan)</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2016-10-31 10:07 GMT+01:00 Roland Kammerer <span dir="ltr"><<a href="mailto:roland.kammerer@linbit.com" target="_blank">roland.kammerer@linbit.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On Sat, Oct 29, 2016 at 08:55:45PM +0200, Toni Bolduan wrote:<br>
> Hi list,<br>
><br>
> Today I've updated to drbdmanage 0.98 on my 2 ubuntu server nodes.<br>
> After setting up the volume group on both nodes I started and the<br>
> initialization on node 1. That worked fine.<br>
><br>
><br>
> Then I tried to add the second node to my cluster with "drbdmanage add-node<br>
> node2 10.0.0.12" and drbdmanage get stuck after confirmation.<br>
<br>
</span>During startup drbdmanage now has to handle more things, so it might<br>
take longer (~15-30 seconds).<br>
<div><div class="gmail-h5"><br>
><br>
> On the second node dmesg shows the following:<br>
><br>
> [...]<br>
> [ 1103.413457] drbd .drbdctrl: Terminating worker thread<br>
> [ 1386.430669] drbd .drbdctrl: Starting worker thread (from drbdsetup<br>
> [2142])<br>
> [ 1386.437482] drbd .drbdctrl node1: Starting sender thread (from drbdsetup<br>
> [2155])<br>
> [ 1386.445330] drbd .drbdctrl/0 drbd0: disk( Diskless -> Attaching )<br>
> [ 1386.445340] drbd .drbdctrl/0 drbd0: Maximum number of peer devices = 31<br>
> [ 1386.445425] drbd .drbdctrl: Method to ensure write ordering: flush<br>
> [ 1386.445427] drbd .drbdctrl/0 drbd0 node1: node_id: 0 idx: 0 bm-uuid: 0x0<br>
> flags: 0x10 max_size: 0 (DUnknown)<br>
> [ 1386.445428] drbd .drbdctrl/0 drbd0: my node_id: 1<br>
> [ 1386.445433] drbd .drbdctrl/0 drbd0 node1: node_id: 0 idx: 0 bm-uuid: 0x0<br>
> flags: 0x10 max_size: 0 (DUnknown)<br>
> [ 1386.445434] drbd .drbdctrl/0 drbd0: my node_id: 1<br>
> [ 1386.445435] drbd .drbdctrl/0 drbd0: drbd_bm_resize called with capacity<br>
> == 8112<br>
> [ 1386.445441] drbd .drbdctrl/0 drbd0: resync bitmap: bits=1014 words=496<br>
> pages=1<br>
> [ 1386.445442] drbd .drbdctrl/0 drbd0: size = 4056 KB (4056 KB)<br>
> [ 1386.446431] drbd .drbdctrl/0 drbd0: recounting of set bits took<br>
> additional 0ms<br>
> [ 1386.446440] drbd .drbdctrl/0 drbd0: disk( Attaching -> Outdated )<br>
> [ 1386.446443] drbd .drbdctrl/0 drbd0: attached to current UUID:<br>
> 120FE59FE04690DE<br>
> [ 1411.289042] drbd .drbdctrl: State change failed: Need access to UpToDate<br>
> data<br>
> [ 1411.289066] drbd .drbdctrl: Failed: role( Secondary -> Primary )<br>
> [ 1434.136862] drbd .drbdctrl: State change failed: Need access to UpToDate<br>
> data<br>
</div></div>> [...]<br>
<span class="gmail-">> <a href="tel:%5B%202033.117704" value="+12033117704">[ 2033.117704</a>] drbd .drbdctrl: Failed: role( Secondary -> Primary )<br>
><br>
> How can I figure what happened here and why?<br>
><br>
<br>
</span>I guess that that happened while the second node was in the leader<br>
election phase, where it tries to become DRBD Primary on the control<br>
volume (.drbdctrl). That is how leader election basically works. All<br>
nodes race to become Primary until one succeeds, the others then see a<br>
Primary and give up and become satellite nodes. The problem is that<br>
there is no UpToDate data.<br>
<br>
I would run "drbdsetup status" in a second window and check if the<br>
resource (.drbdctrl) makes any progress. Does it sync up to the second<br>
node or does it get stuck after some percentage? Or does it not start<br>
syncing at all? Are they in some strange network state,... The output of<br>
"drbdsetup status" of both nodes would help a lot.<br>
<br>
Regards, rck<br>
<br>
______________________________<wbr>_________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/<wbr>mailman/listinfo/drbd-user</a><br>
</blockquote></div><br></div></div>