<div dir="ltr">Thanks Mats, this was the reason for me to give up on drbd 9 when I was testing about 2-3 months ago.<br><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 30, 2016 at 9:50 PM, Mats Ramnefors <span dir="ltr"><<a href="mailto:mats@ramnefors.com" target="_blank">mats@ramnefors.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Patch below.<div><br></div><div>For me it did not work to apply the patch â€as isâ€. I had to manually edit in the changes. </div><div><br></div><div>Do not understand why, but I am no expert on patching so it may work for you.</div><div><br></div><div>With the changes applied, the RA seems to work OK.</div><div><br></div><div>/Mats<br><div><br></div><div><div>++++++ support-drbd9-ra.patch ++++++</div><div>diff --git a/scripts/drbd.ocf b/scripts/drbd.ocf</div><div>index 632e16e..91990fc 100755</div><div>--- a/scripts/drbd.ocf</div><div>+++ b/scripts/drbd.ocf</div><div>@@ -328,6 +328,23 @@ remove_master_score() {</div><div>    do_cmd ${HA_SBIN_DIR}/crm_master -l reboot -D</div><div> }</div><div> </div><div>+_peer_node_process() {</div><div>+    # _since drbd9 support multiple connections</div><div>+    : ${_peer_node_id:=0}</div><div>+    DRBD_PER_NAME[$_peer_node_id]=$_conn_name</div><div>+    DRBD_PER_ID[$_peer_node_id]=$_peer_node_id</div><div>+    DRBD_PER_CSTATE[$_peer_node_id]=$_cstate</div><div>+    DRBD_PER_ROLE_REMOTE[$_peer_node_id]=${_peer:-Unknown}</div><div>+    DRBD_PER_DSTATE_REMOTE[$_peer_node_id]=${_pdsk:-DUnknown}</div><div>+</div><div>+    : == DEBUG == _peer_node_id             == ${_peer_node_id} ==</div><div>+    : == DEBUG == DRBD_PER_NAME[_peer_node_id]      == ${DRBD_PER_NAME[${_peer_node_id}]} ==</div><div>+    : == DEBUG == DRBD_PER_ID[_peer_node_id]       == ${DRBD_PER_ID[${_peer_node_id}]} ==</div><div>+    : == DEBUG == DRBD_PER_CSTATE[_peer_node_id]     == ${DRBD_PER_CSTATE[${_peer_node_id}]} ==</div><div>+    : == DEBUG == DRBD_PER_ROLE_REMOTE[_peer_node_id]  == ${DRBD_PER_ROLE_REMOTE[${_peer_node_id}]} ==</div><div>+    : == DEBUG == DRBD_PER_DSTATE_REMOTE[_peer_node_id] == ${DRBD_PER_DSTATE_REMOTE[${_peer_node_id}]} ==</div><div>+}</div><div>+</div><div> _sh_status_process() {</div><div>    # _volume not present should not happen,</div><div>    # but may help make this agent work even if it talks to drbd 8.3.</div><div>@@ -335,11 +352,36 @@ _sh_status_process() {</div><div>    # not-yet-created volumes are reported as -1</div><div>    (( _volume >= 0 )) || _volume=$[1 << 16]</div><div>    DRBD_ROLE_LOCAL[$_volume]=${_role:-Unconfigured}</div><div>-    DRBD_ROLE_REMOTE[$_volume]=${_peer:-Unknown}</div><div>-    DRBD_CSTATE[$_volume]=$_cstate</div><div>    DRBD_DSTATE_LOCAL[$_volume]=${_disk:-Unconfigured}</div><div>-    DRBD_DSTATE_REMOTE[$_volume]=${_pdsk:-DUnknown}</div><div>+</div><div>+    if $DRBD_VERSION_9 ; then</div><div>+        #Get from _peer_node_process</div><div>+        DRBD_NAME[$_volume]=${DRBD_PER_NAME[@]}</div><div>+        DRBD_ID[$_volume]=${DRBD_PER_ID[@]}</div><div>+        DRBD_VOLUME[$_volume]=${_volume}</div><div>+        DRBD_CSTATE[$_volume]=${DRBD_PER_CSTATE[@]}</div><div>+        DRBD_ROLE_REMOTE[$_volume]=${DRBD_PER_ROLE_REMOTE[@]}</div><div>+        DRBD_DSTATE_REMOTE[$_volume]=${DRBD_PER_DSTATE_REMOTE[@]}</div><div>+</div><div>+        DRBD_PER_NAME=()</div><div>+        DRBD_PER_ID=()</div><div>+        DRBD_PER_CSTATE=()</div><div>+        DRBD_PER_ROLE_REMOTE=()</div><div>+        DRBD_PER_DSTATE_REMOTE=()</div><div>+</div><div>+        : == DEBUG == _volume       == ${_volume} ==</div><div>+        : == DEBUG == DRBD_ROLE_LOCAL   == ${DRBD_ROLE_LOCAL[${_volume}]} ==</div><div>+        : == DEBUG == DRBD_DSTATE_LOCAL  == ${DRBD_DSTATE_LOCAL[${_volume}]} ==</div><div>+        : == DEBUG == DRBD_CSTATE     == ${DRBD_CSTATE[${_volume}]} ==</div><div>+        : == DEBUG == DRBD_ROLE_REMOTE  == ${DRBD_ROLE_REMOTE[${_volume}]} ==</div><div>+        : == DEBUG == DRBD_DSTATE_REMOTE == ${DRBD_DSTATE_REMOTE[${_volume}]} ==</div><div>+    else</div><div>+        DRBD_CSTATE[$_volume]=$_cstate</div><div>+        DRBD_ROLE_REMOTE[$_volume]=${_peer:-Unknown}</div><div>+        DRBD_DSTATE_REMOTE[$_volume]=${_pdsk:-DUnknown}</div><div>+    fi</div><div>}</div><div>+</div><div> drbd_set_status_variables() {</div><div>    # drbdsetup sh-status prints these values to stdout,</div><div>    # and then prints _sh_status_process.</div><div>@@ -352,6 +394,15 @@ drbd_set_status_variables() {</div><div>    local _resynced_percent</div><div>    local out</div><div> </div><div>+    if $DRBD_VERSION_9 ; then</div><div>+        local _peer_node_id _conn_name</div><div>+        DRBD_PER_NAME=()</div><div>+        DRBD_PER_ID=()</div><div>+        DRBD_PER_CSTATE=()</div><div>+        DRBD_PER_ROLE_REMOTE=()</div><div>+        DRBD_PER_DSTATE_REMOTE=()</div><div>+    fi</div><div>+</div><div>    DRBD_ROLE_LOCAL=()</div><div>    DRBD_ROLE_REMOTE=()</div><div>    DRBD_CSTATE=()</div><div>@@ -369,16 +420,20 @@ drbd_set_status_variables() {</div><div>    # if there was no output at all, or a weird output</div><div>    # make sure the status arrays won't be empty.</div><div>    [[ ${#DRBD_ROLE_LOCAL[@]}   != 0 ]] || DRBD_ROLE_LOCAL=(Unconfigured)</div><div>-    [[ ${#DRBD_ROLE_REMOTE[@]}  != 0 ]] || DRBD_ROLE_REMOTE=(Unknown)</div><div>-    [[ ${#DRBD_CSTATE[@]}     != 0 ]] || DRBD_CSTATE=(Unconfigured)</div><div>    [[ ${#DRBD_DSTATE_LOCAL[@]}  != 0 ]] || DRBD_DSTATE_LOCAL=(Unconfigured)</div><div>+    [[ ${#DRBD_CSTATE[@]}     != 0 ]] || DRBD_CSTATE=(Unconfigured)</div><div>+    [[ ${#DRBD_ROLE_REMOTE[@]}  != 0 ]] || DRBD_ROLE_REMOTE=(Unknown)</div><div>    [[ ${#DRBD_DSTATE_REMOTE[@]} != 0 ]] || DRBD_DSTATE_REMOTE=(DUnknown)</div><div> </div><div>-</div><div>+    if $DRBD_VERSION_9 ; then</div><div>+        : == DEBUG == DRBD_NAME   == ${DRBD_NAME[@]} ==</div><div>+        : == DEBUG == DRBD_ID   == ${DRBD_ID[@]} ==</div><div>+        : == DEBUG == DRBD_VOLUME   == ${DRBD_VOLUME[@]} ==</div><div>+    fi</div><div>    : == DEBUG == DRBD_ROLE_LOCAL   == ${DRBD_ROLE_LOCAL[@]} ==</div><div>-    : == DEBUG == DRBD_ROLE_REMOTE  == ${DRBD_ROLE_REMOTE[@]} ==</div><div>-    : == DEBUG == DRBD_CSTATE     == ${DRBD_CSTATE[@]} ==</div><div>    : == DEBUG == DRBD_DSTATE_LOCAL  == ${DRBD_DSTATE_LOCAL[@]} ==</div><div>+    : == DEBUG == DRBD_CSTATE     == ${DRBD_CSTATE[@]} ==</div><div>+    : == DEBUG == DRBD_ROLE_REMOTE  == ${DRBD_ROLE_REMOTE[@]} ==</div><div>    : == DEBUG == DRBD_DSTATE_REMOTE == ${DRBD_DSTATE_REMOTE[@]} ==</div><div> }</div><div> </div><div>@@ -414,6 +469,9 @@ maybe_outdate_self()</div><div>    ocf_is_true $OCF_RESKEY_stop_outdates_secondary || return 1</div><div> </div><div>    local host stop_uname</div><div>+    if $DRBD_VERSION_9 ; then</div><div>+        local master temp_nmber outdate_self</div><div>+    fi</div><div>    # We ignore $OCF_RESKEY_CRM_meta_notify_promote_uname here</div><div>    # because: if demote and promote for a _stacked_ resource</div><div>    # (or a "floating" one, where DRBD sits on top of some SAN)</div><div>@@ -437,6 +495,29 @@ maybe_outdate_self()</div><div>        return 1</div><div>    done</div><div> </div><div>+    if $DRBD_VERSION_9 ; then</div><div>+        temp_name=($DRBD_NAME[@])</div><div>+        temp_dstate=($DRBD_DSTATE_REMOTE[@])</div><div>+        temp_number=${#temp_name[@]}</div><div>+        outdate_self=false</div><div>+</div><div>+        for master in $OCF_RESKEY_CRM_meta_notify_master_uname; do</div><div>+            for i in `seq 0 $((temp_number-1))`; do</div><div>+                if [[ ${temp_name[$i]} == "$master" ]] &&</div><div>+                 [[ ${temp_dstate[$i]} == "DUnknown" ]]; then</div><div>+                    outdate_self=true</div><div>+                    break</div><div>+                fi</div><div>+            done</div><div>+            temp_number=${#temp_name[@]}</div><div>+        done</div><div>+</div><div>+        if ! $outdate_self; then</div><div>+            #The disconnecting node is not in Primary</div><div>+            return 1</div><div>+        fi</div><div>+   fi</div><div>+</div><div>    # e.g. post/promote of some other peer.</div><div>    # Should not happen, fencing constraints should take care of that.</div><div>    # But in case it does, scream out loud.</div><div>@@ -993,6 +1074,7 @@ drbd_validate_all () {</div><div>    DRBDADM="drbdadm"</div><div>    DRBDSETUP="drbdsetup"</div><div>    DRBD_HAS_MULTI_VOLUME=false</div><div>+    DRBD_VERSION_9=false</div><div> </div><div>    # these will _exit_ if they don't find the binaries</div><div>    check_binary $DRBDADM</div><div>@@ -1015,18 +1097,23 @@ drbd_validate_all () {</div><div>            modinfo -F version drbd |</div><div>            sed -ne 's/^\([0-9]\+\)\.\([0-9]\+\)\.\([0-9]\+\).*$/\1 \2 \3/p'))</div><div>    fi</div><div>-    if (( $DRBD_KERNEL_VERSION_CODE >= 0x080400 )); then</div><div>+    if (( $DRBD_KERNEL_VERSION_CODE >= 0x090000 )); then</div><div>+        DRBD_HAS_MULTI_VOLUME=true</div><div>+        DRBD_VERSION_9=true</div><div>+        ocf_log warn "RA for DRBD version 9 is in experiment, do not using multiple primaries in DRBD9.0"</div><div>+    elif (( $DRBD_KERNEL_VERSION_CODE >= 0x080400 )); then</div><div>        DRBD_HAS_MULTI_VOLUME=true</div><div>-    elif (( $DRBD_KERNEL_VERSION_CODE >= 0x090000 )) ; then</div><div>-        ocf_log err "This resource agent does (still) only support DRBD version 8.x"</div><div>-        exit $OCF_ERR_INSTALLED</div><div>    fi</div><div>    check_crm_feature_set</div><div> </div><div>    # Check clone and M/S options.</div><div>-    meta_expect clone-max -le 2</div><div>+    # Drbd9 support more than two nodes</div><div>+    if ! $DRBD_VERSION_9 ; then</div><div>+        meta_expect clone-max -le 2</div><div>+    fi</div><div>    meta_expect clone-node-max = 1</div><div>    meta_expect master-node-max = 1</div><div>+    # With current DRBD-9.0 version more than two primaries at the same time is not support.</div><div>    meta_expect master-max -le 2</div><div> </div><div>    # Rather than returning $OCF_ERR_CONFIGURED, we sometimes return</div><div>@@ -1080,7 +1167,8 @@ drbd_validate_all () {</div><div>    # DRBD_DEVICES will be a shell array!</div><div>    # FIXME we should double check that we explicitly restrict the set of</div><div>    # valid characters in device names...</div><div>-    if DRBD_DEVICES=($($DRBDADM --stacked sh-dev $DRBD_RESOURCE 2>/dev/null)); then</div><div>+    # In DRBD9, no matter stacked or not "$DRBDADM --stacked sh-dev $DRBD_RESOURCE" will return true</div><div>+    if ! $DRBD_VERSION_9 && DRBD_DEVICES=($($DRBDADM --stacked sh-dev $DRBD_RESOURCE 2>/dev/null)); then</div><div>        # apparently a "stacked" resource. Remember for future DRBDADM calls.</div><div>        DRBDADM="$DRBDADM -S"</div><div>    elif DRBD_DEVICES=($($DRBDADM sh-dev $DRBD_RESOURCE 2>/dev/null)); then</div><div>diff --git a/user/v9/drbdsetup.c b/user/v9/drbdsetup.c</div><div>index 053b9d3..fba72e1 100644</div><div>--- a/user/v9/drbdsetup.c</div><div>+++ b/user/v9/drbdsetup.c</div><div>@@ -251,6 +251,7 @@ static int del_resource_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int show_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int status_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int role_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div>+static int sh_status_9compat_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int cstate_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int dstate_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int check_resize_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div>@@ -478,6 +479,9 @@ struct drbd_cmd commands[] = {</div><div>    {"role", CTX_RESOURCE, 0, NO_PAYLOAD, role_cmd,</div><div>     .lockless = true,</div><div>     .summary = "Show the current role of a resource." },</div><div>+    {"sh-status", CTX_RESOURCE | CTX_ALL, 0, 0, sh_status_9compat_cmd,</div><div>+     .lockless = true,</div><div>+     .summary = "Show all status of resource." },</div><div>    {"cstate", CTX_PEER_NODE, 0, NO_PAYLOAD, cstate_cmd,</div><div>     .lockless = true,</div><div>     .summary = "Show the current state of a connection." },</div><div>@@ -2576,6 +2580,87 @@ static int role_cmd(struct drbd_cmd *cm, int argc, char **argv)</div><div>    return 0;</div><div> }</div><div> </div><div>+</div><div>+static int sh_status_9compat_cmd(struct drbd_cmd *cm, int argc, char **argv)</div><div>+{</div><div>+</div><div>+    struct resources_list *resources_list, *resource;</div><div>+    char *old_objname = objname;</div><div>+</div><div>+    resources_list = sort_resources(list_resources());</div><div>+</div><div>+    for (resource = resources_list; resource; resource = resource->next) {</div><div>+        struct devices_list *devices, *device;</div><div>+        struct connections_list *connections, *connection;</div><div>+        struct peer_devices_list *peer_devices = NULL, *peer_device;</div><div>+        struct nlattr *nla;</div><div>+</div><div>+        if (strcmp(old_objname, "all") && strcmp(old_objname, resource->name))</div><div>+            continue;</div><div>+</div><div>+        objname = resource->name;</div><div>+        printI("_res_name=%s\n", objname);</div><div>+</div><div>+        nla = nla_find_nested(resource->res_opts, __nla_type(T_node_id));</div><div>+        if (nla)</div><div>+            printI("_node_id=%d\n\n", *(uint32_t *)nla_data(nla));</div><div>+        else</div><div>+            printI("_node_id=\n\n");</div><div>+</div><div>+        devices = list_devices(resource->name);</div><div>+        connections = sort_connections(list_connections(resource->name));</div><div>+        if (devices && connections)</div><div>+            peer_devices = list_peer_devices(resource->name);</div><div>+</div><div>+        link_peer_devices_to_devices(peer_devices, devices);</div><div>+</div><div>+        for (device = devices; device; device = device->next) {</div><div>+            ++indent;</div><div>+            printI("_minor=%d\n", device->minor);</div><div>+            printI("_volume=%d\n", device->ctx.ctx_volume);</div><div>+            //refer to v84</div><div>+            //printI("_known=%s\n", xxx);</div><div>+            printI("_role=%s\n", drbd_role_str(resource->info.res_role));</div><div>+            printI("_disk=%s\n\n", drbd_disk_str(device->info.dev_disk_state));</div><div>+</div><div>+            for (connection = connections; connection; connection = connection->next) {</div><div>+                ++indent;</div><div>+                for (peer_device = peer_devices; peer_device; peer_device = peer_device->next) {</div><div>+                    if (connection->ctx.ctx_peer_node_id != peer_device->ctx.ctx_peer_node_id</div><div>+                        || device->ctx.ctx_volume != peer_device->ctx.ctx_volume)</div><div>+                        continue;</div><div>+                    printI("_conn_name=%s\n", connection->ctx.ctx_conn_name);</div><div>+                    printI("_peer_node_id=%d\n", connection->ctx.ctx_peer_node_id);</div><div>+                    printI("_cstate=%s\n", drbd_conn_str(connection->info.conn_connection_state));</div><div>+                    if (connection->info.conn_connection_state == C_CONNECTED) {</div><div>+                        printI("_peer=%s\n", drbd_role_str(connection->info.conn_role));</div><div>+                        printI("_pdsk=%s\n\n", drbd_disk_str(peer_device->info.peer_disk_state));</div><div>+                    } else {</div><div>+                        printI("_peer=\n");</div><div>+                        printI("_pdsk=\n");</div><div>+                    }</div><div>+                    wrap_printf(0, "_peer_node_process\n\n");</div><div>+                }</div><div>+                //Dummy</div><div>+                //printI("_flags_susp==%s\n", xxx);</div><div>+                //...</div><div>+                --indent;</div><div>+            }</div><div>+</div><div>+            wrap_printf(0, "_sh_status_process\n\n");</div><div>+            --indent;</div><div>+        }</div><div>+</div><div>+        free_connections(connections);</div><div>+        free_devices(devices);</div><div>+        free_peer_devices(peer_devices);</div><div>+    }</div><div>+</div><div>+    free(resources_list);</div><div>+    objname = old_objname;</div><div>+    return 0;</div><div>+}</div><div>+</div><div> static int cstate_cmd(struct drbd_cmd *cm, int argc, char **argv)</div><div> {</div><div>    struct connections_list *connections, *connection;</div><div><div class="h5"><div><br></div><div><br></div><div><br></div><div><br></div><div><blockquote type="cite"><div>30 maj 2016 kl. 13:36 skrev Igor Cicimov <<a href="mailto:icicimov@gmail.com" target="_blank">icicimov@gmail.com</a>>:</div><br><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 17, 2016 at 8:21 AM, Mats Ramnefors <span dir="ltr"><<a href="mailto:mats@ramnefors.com" target="_blank">mats@ramnefors.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am testing a DRBD 9 and 8.4 in simple 2 node active - passive clusters with NFS.<br>
<br>
Copying files form a third server to the NFS share using dd, I typically see an average of 20% CPU load (with v9) on the primary during transfer of larger files, testing with 0,5 and 2 GB.<br>
<br>
At the very end of the transfer DRBD process briefly peaks at 70 - 100% CPU.<br>
<br>
This causes occasional problems with Corosync believing the node is down. Increasing the token time for Corosync to 2000 ms fixes the symptom but I am wondering about the root cause and any possible fixes?<br>
<br>
This is the DRBD configuration.<br>
<br>
resource san_data {<br>
 protocol C;<br>
 meta-disk internal;<br>
 device /dev/drbd1;<br>
 disk  /dev/nfs/share;<br>
 net {<br>
  verify-alg sha1;<br>
  cram-hmac-alg sha1;<br>
  shared-secret â€****************";<br>
  after-sb-0pri discard-zero-changes;<br>
  after-sb-1pri discard-secondary;<br>
  after-sb-2pri disconnect;<br>
 }<br>
 on san1 {<br>
  address <a href="http://192.168.1.86:7789/" rel="noreferrer" target="_blank">192.168.1.86:7789</a>;<br>
 }<br>
 on san2 {<br>
  address <a href="http://192.168.1.87:7789/" rel="noreferrer" target="_blank">192.168.1.87:7789</a>;<br>
 }<br>
}<br>
<br>
The nodes are two VM on different ESXi hosts (Dell T620). Hosts are very lightly loaded. Network is 1 Gb at the moment through a Catalyst switch. Network appears not saturated.<br>
<br>
BTW when can we expect a DRBD resource agent for v9? It took me a while to figure out why DRBD9 was not working with Pacemaker and then finding a patch to the agent :)<br>
<br>
Cheers Mats<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a></blockquote><div><br></div><div>Hi Mats,<br><br></div><div>Can you please share the patch if you don't mind?<br><br></div><div>Thanks,<br></div><div>Igor <br></div></div><br></div></div>
</div></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div></div></div>