<div dir="ltr">You are using a over 3 year old DRBD version, there have been numerous bug fixes to DRBD since.<div>So first upgrade to the latest DRBD version and check if you can still reproduce your problem.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 23, 2020 at 7:20 AM 박기혁 <<a href="mailto:korea.oops@gmail.com">korea.oops@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div id="gmail-m_-600838170073727939gmail-show_issue"><div id="gmail-m_-600838170073727939gmail-discussion_bucket"><div><div><div><div><div><div id="gmail-m_-600838170073727939gmail-issue-722904788"><div><div><span disabled><table><tbody><tr><td><p>Hello, Community</p><p>My system is using Pacemaker + DRBD + MySQL DB.<br> There is something unusual about your system.</p><p>kernel version: 3.10.0-693.el7.x86_64<br> drbd version: drbd90-utils-9.0.0-1.el7.elrepo.x86_64<br> kmod-drbd90-9.0.9-1.el7_4.elrepo.x86_64<br> DB version: mariadb-10.3.22</p><p>Issue Time: October 15, 2020, 18:13:47 to 18:13:55</p><ul><li>When monitoring with IOSTAT, it detects 100% Utiliztion which cannot be IO-handled for the /dev/drbd0 device.<br> command: iostat -td 1 -x</li></ul><p>10/15/2020 06:13:47 PM<br> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util<br> sda 0.00 0.00 0.00 6.00 0.00 24.00 8.00 0.00 0.17 0.00 0.17 0.17 0.10<br> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> dm-0 0.00 0.00 0.00 6.00 0.00 24.00 8.00 0.00 0.17 0.00 0.17 0.17 0.10<br> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> sdc 0.00 0.00 3.00 0.00 1.50 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00<br> drbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00 <<----- **</p><p>10/15/2020 06:13:52 PM<br> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util<br> drbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00</p><p>10/15/2020 06:13:53 PM<br> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util<br> drbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00</p><p>10/15/2020 06:13:54 PM<br> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util<br> sda 0.00 0.00 0.00 6.00 0.00 24.00 8.00 0.00 0.67 0.00 0.67 0.67 0.40<br> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> dm-0 0.00 0.00 0.00 6.00 0.00 24.00 8.00 0.00 0.67 0.00 0.67 0.67 0.40<br> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br> sdc 0.00 0.00 3.00 0.00 1.50 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00<br> drbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00</p><ul><li>When monitoring the DRBD status, it detects that an upper-pending has occurred.<br> exists resource name:drbd01 role:Primary suspended:no write-ordering:flush<br> exists connection name:drbd01 peer-node-id:2 conn-name:node2 connection:Connected role:Secondary congested:no<br> exists device name:drbd01 volume:0 minor:0 disk:UpToDate client:no size:1610559452 read:17730265 written:69192955 al-writes:16100 bm-writes:0 upper-pending:1 lower-pending:0 al-suspended:no blocked:no<br> exists peer-device name:drbd01 peer-node-id:2 conn-name:node2 volume:0 replication:Established peer-disk:UpToDate peer-client:no resync-suspended:no received:8483 sent:69184366 out-of-sync:0 <strong>pending:1</strong> unacked:0<br> exists -</li></ul><ul><li>upper-pending (application pending) : Number of block I/O requests forwarded to DRBD, but not yet answered by DRBD</li></ul><ul><li><p>When you check Mysql Slow Query, the response was received after 9 seconds after the IO Hang was finished after the Query request.<br> User@Host: nodeapp[nodeapp] @ [100.100.100.142]<br> Thread_id: 7879 Schema: MYMQDB QC_hit: No<br> Query_time: 9.492522 Lock_time: 0.000058 Rows_sent: 0 Rows_examined: 1<br> Rows_affected: 1 Bytes_sent: 52<br> use MYMQDB;<br> SET timestamp=1602753235;<br> UPDATE ACTIVEMQ_LOCK SET BROKER_NAME='node2', TIME=1602753250881 WHERE BROKER_NAME='node2' AND ID = 1;</p></li><li><p>drbd configuration<br> disk {<br> on-io-error detach;<br> no-disk-flushes ;<br> no-disk-barrier;<br> c-plan-ahead 0;<br> c-fill-target 24M;<br> c-min-rate 80M;<br> c-max-rate 720M;<br> }<br> net {<br> max-buffers 36k;<br> sndbuf-size 1024k ;<br> rcvbuf-size 2048k;<br> }</p></li></ul><p>In conclusion, the %util level in the DRBD device is 100%, but there is no read write at this time, and the slow time of MySQL is the same as the time of 100% duration.</p><p>Does anyone know a similar case or solution to this phenomenon?</p><p>Hang does not occur if drbd is operated as single.<br></p></td></tr></tbody></table></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Star us on GITHUB: <a href="https://github.com/LINBIT" rel="noreferrer" target="_blank">https://github.com/LINBIT</a><br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="https://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">https://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div>