<div dir="ltr">I think disconnecting drbd before umount and connecting it after umount is a good idea to avoid this.<br><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/6 Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Tue, May 21, 2013 at 11:52:58PM +0800, Mia Lueng wrote:<br>
> I have 16G RAM in this server. Using a low dirty configuration may lead<br>
> to a pool I/O performance?<br>
<br>
</div>Maybe this blog post, and the links referenced therein help?<br>
<a href="https://blogs.linbit.com/p/548/umount-takes-time/" target="_blank">https://blogs.linbit.com/p/548/umount-takes-time/</a><br>
<br>
<a href="http://lists.linbit.com/pipermail/drbd-user/2012-September/019080.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2012-September/019080.html</a><br>
<span><font color="#888888"><br>
Lars<br>
</font></span><div><div><br>
> 2013/5/14 Lars Ellenberg <<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>><br>
><br>
> > On Thu, May 09, 2013 at 10:33:16AM +0800, Mia Lueng wrote:<br>
> > > # sysctl -a|grep dirty<br>
> > > vm.dirty_background_ratio = 10<br>
> > > vm.dirty_background_bytes = 0<br>
> > > vm.dirty_ratio = 20<br>
> > > vm.dirty_bytes = 0<br>
> > > vm.dirty_writeback_centisecs = 500<br>
> > > vm.dirty_expire_centisecs = 3000<br>
> > ><br>
> > > bandwidth is 100M bps<br>
> ><br>
> > You can replicate around 10 to 12 MByte per second.<br>
> > To avoid long "write-out stalls" when flushing caches,<br>
> > you should not allow more than about 20 MByte dirty,<br>
> > and start write out much earlier.<br>
> ><br>
> > vm.dirty_bytes=20100100<br>
> > vm.dirty_background_bytes=500100<br>
> > vm.dirty_writeback_centisecs=97<br>
> ><br>
> > A ratio of 20 % of available RAM may well mean several GB.<br>
> > How much RAM do you have?<br>
> ><br>
> > Depending on what usage patterns and data characteristics<br>
> > you actually have in production, maybe you want to try drbd-proxy.<br>
> > Or check with LINBIT what other options you have.<br>
> ><br>
> > > 2013/5/9 Lars Ellenberg <<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>><br>
> > ><br>
> > > > On Thu, May 09, 2013 at 12:16:56AM +0800, Mia Lueng wrote:<br>
> > > > > in drbd 8.4.3£¬I do the following test:<br>
> > > > ><br>
> > > > > [root@kvm3 drbd.d]# drbdadm dump drbd0<br>
> > > > > # resource drbd0 on kvm3: not ignored, not stacked<br>
> > > > > # defined at /etc/drbd.d/drbd0.res:1<br>
> > > > > resource drbd0 {<br>
> > > > > on kvm3 {<br>
> > > > > device /dev/drbd0 minor 0;<br>
> > > > > disk /dev/vg_kvm3/drbd0;<br>
> > > > > meta-disk internal;<br>
> > > > > address ipv4 <a href="http://192.168.10.6:7700" target="_blank">192.168.10.6:7700</a>;<br>
> > > > > }<br>
> > > > > on kvm4 {<br>
> > > > > device /dev/drbd0 minor 0;<br>
> > > > > disk /dev/vg_kvm4/drbd0;<br>
> > > > > meta-disk internal;<br>
> > > > > address ipv4 <a href="http://192.168.10.7:7700" target="_blank">192.168.10.7:7700</a>;<br>
> > > > > }<br>
> > > > > net {<br>
> > > > > protocol A;<br>
> > > > > csums-alg md5;<br>
> > > > > verify-alg md5;<br>
> > > > > ping-timeout 30;<br>
> > > > > ping-int 30;<br>
> > > > > max-epoch-size 8192;<br>
> > > > > max-buffers 8912;<br>
> > > > > unplug-watermark 131072;<br>
> > > > > }<br>
> > > > > disk {<br>
> > > > > on-io-error pass_on;<br>
> > > > > disk-barrier no;<br>
> > > > > disk-flushes no;<br>
> > > > > resync-rate 100M;<br>
> > > > > c-plan-ahead 20;<br>
> > > > > c-delay-target 100;<br>
> > > > > c-max-rate 400M;<br>
> > > > > c-min-rate 2M;<br>
> > > > > al-extents 601;<br>
> > > > > }<br>
> > > > > }<br>
> > > > ><br>
> > > > > [root@kvm3 oradata]# dd if=t1 of=t2 bs=1M<br>
> > > > > 5585+1 records in<br>
> > > > > 5585+1 records out<br>
> > > > > 5856305152 bytes (5.9 GB) copied, 286.119 s, 20.5 MB/s<br>
> > > ><br>
> > > > That writes to the page cache, and from there to the block device.<br>
> > > ><br>
> > > > No fsync, no sync: there will still be a few GB in the cache (RAM<br>
> > only).<br>
> > > ><br>
> > > > > [root@kvm3 oradata]# cd<br>
> > > > > [root@kvm3 ~]# umount /oradata<br>
> > > > ><br>
> > > > ><br>
> > > > > it takes lots of time(up to 600 seconds) to umount the drbd mount<br>
> > point.<br>
> > > ><br>
> > > > On umount, the filesystem obviously has to flush all dirty pages first.<br>
> > > ><br>
> > > > What is your replication bandwidth?<br>
> > > ><br>
> > > > > echo "1" >/proc/sys/vm/block_dump<br>
> > > > > show when umount ,<br>
> > > > ><br>
> > > > > [root@kvm3 ~]# dmesg|tail -n 100<br>
> > > > ...<br>
> > > > > umount(3958): WRITE block 100925440 on dm-5<br>
> > > > > umount(3958): WRITE block 100925440 on dm-5<br>
> > > > > umount(3958): WRITE block 100925440 on dm-5<br>
> > > > > umount(3958): WRITE block 0 on dm-5<br>
> > > > > umount(3958): dirtied inode 1053911 (mtab.tmp) on dm-0<br>
> > > > > umount(3958): dirtied inode 1053911 (mtab.tmp) on dm-0<br>
> > > > > umount(3958): WRITE block 33845632 on dm-0<br>
> > > > > umount(3958): dirtied inode 1053912 (?) on dm-0<br>
> > > > ><br>
> > > > ><br>
> > > > > Is the reason that I use protocol A?<br>
> > > ><br>
> > > > No.<br>
> > > ><br>
> > > > But that you need to understand caching, and tunables.<br>
> > > ><br>
> > > > Some hints and keywords for a followup search:<br>
> > > ><br>
> > > > Check how much "dirty" data (writes not yet on stable storage)<br>
> > > > is still in RAM:<br>
> > > > grep Dirty /proc/meminfo<br>
> > > ><br>
> > > > Tune how much dirty data is "allowed"<br>
> > > > sysctl<br>
> > > > vm.dirty_background_bytes<br>
> > > > vm.dirty_bytes<br>
> > > > vm.dirty_writeback_centisecs<br>
> > > > vm.dirty_expire_centisecs<br>
> > > ><br>
> > > > also compare:<br>
> > > > time dd if=t1 of=t2 bs=1M; time sync<br>
> > > > time dd if=t1 of=t2 bs=1M conv=fsync<br>
<br>
<br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
<br>
</div></div><div><div>DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
__<br>
please don't Cc me, but send to list -- I'm subscribed<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</div></div></blockquote></div><br></div></div>