Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello Volker,
On 12/15/2011 01:19 PM, Volker wrote:
> Hi all,
>
> we've been using drbd for about six month now, and so far everything is
> working pretty well. Our setup is like this with two identical machines
> (besides the actual HDDs).
>
> Dell 2950 III
> - 16GB Ram, Dual-Quadcore Xeons 2.0GHz
> - Redhat Enterprise Linux 5.7, 2.6.18-238.19.1.el5
> - PERC6/i Raid-Controller
> - 2 Disk OS, Raid 1 (sda)
> - 4 Disk Content, Raid 10 (sdb)
> - 500GB /dev/sdb5 extended partition
> - LVM-Group 'content' on /dev/sdb5
> - 400GB LVM-Volume 'data' created in LVM-Group 'content'
> - DRBD with /dev/drbd0 on /dev/content/data (content being the
> LVM-Group, data being the LVM-Volume)
> - /dev/drbd0 is mounted with noatime,ext3-ordered-journaling and then
> exported with nfs3 and and mounted by 8 machines (rhel5 entirely)
> - replication is done using a dedicated nic with gbit
>
> The DRBD-Version is
> drbd 8.3.8-1
> kmod 8.3.8.1
>
> here is the information from /proc/drbd:
>
> ####
> version: 8.3.8 (api:88/proto:86-94)
> GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by
> mockbuild at builder10.centos.org, 2010-06-04 08:04:09
> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate B r----
> ns:91617716 nr:15706584 dw:107784232 dr:53529112 al:892898 bm:37118
> lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
> ####
>
> This is the latest "official" Version available for CentOS/Redhat from
> the CentOS-extras repo (as far as i know):
try elrepo.org
>
> http://mirror.centos.org/centos/5/extras/x86_64/RPMS/
>
>
> The configuration is identical on both nodes, looking like this:
>
> # /etc/drbd.d/global_common.conf #
> ##################################
> global {
> usage-count no;
> }
>
> common {
> protocol B;
> handlers {}
> startup {}
> disk {
> no-disk-barrier;
> no-disk-flushes;
> no-disk-drain;
try replacing "no-disk-drain" by "no-md-flushes"
Regards,
Andreas
--
Need help with DRBD?
http://www.hastexo.com/now
> }
> net {
> max-buffers 8000;
> max-epoch-size 8000;
> unplug-watermark 1024;
> sndbuf-size 512k;
> }
> syncer {
> rate 25M;
> al-extents 3833;
> }
> }
> ##################################
>
>
>
> # /etc/drbd.d/production.conf #
> ##################################
> resource eshop
> {
> device /dev/drbd0;
> disk /dev/content/data;
> meta-disk internal;
>
> on nfs01.data.domain.de {
> address 10.110.127.129:7789;
> }
>
> on fallback.dta.domain.de {
> address 10.110.127.130:7789;
> }
> }
> ##################################
>
>
> The problem we have with this setup is quite complicated to explain. The
> read/write-performance in daily production use is sufficient to not
> effect the entire platform. The usual system-load viewed using top is
> pretty low, usually between 0.5 and 3.
>
> As soon as i produce some artifical i/o on /dev/drb0 on the master, the
> load pretty much explodes (up to 15) because of blocking i/o. The i/o is
> done with dd and pretty small files of bout 40MB:
>
> dd if=/dev/zero of=./test-data.dd bs=4096 count=10240
>
> Two successive runs like this, make the load go up as far as 10-12
> rendering the whole system useless. In this state, a running dd can not
> be interruped, the nfs-exports are totally inaccessible and the whole
> production-system is at a stand still.
>
> Using blktrace/blkparse one can see, that absolutely no i/o is possible.
>
> 'top' shows one or two cores at 100% wait:
>
> ###
> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni, 0.0%id,100.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu5 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu6 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu7 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> ###
>
> This lasts for about 3-4 minutes with the load slowly degrading.
>
> This behaviour can also be reproduced by using the megacli and querying
> the raid-controller for information. A couple of successive runs result
> in the above described behaviour.
>
> And here comes the catch:
> This only happens, if the drbd-layer is in use.
>
>
> If i produce heavy i/o (100 runs of writing a 400MB) on
> - the same block-device /dev/sdb
> - in the same volume-group 'content'
> - but on a newly created _different_ LVM-Volume
> - without usind the drbd-layer
>
> the system-load marginally rises, but i/o is never blocking.
>
> Things i have tried:
> - switching from protocol C to B
> - disk: no-disk-barrier / no-disk-flushes / no-disk-drain;
> - net: max-buffers / max-epoch-size / unplug-watermark / sndbuf-size
> - syncer: rate, al-extents
> - various caching settings on the raid-controller itself (using megacli)
> - completely disabling drbd-replication to rule out the replication-setup
> - ruled out faulty raid-controller battery which could force the
> controller into writethrough mode instead of writeback
> - checked/updated firmwares of hdds and raid-controllers
> - went through the git-changelogs for drbd 8.3, but i cannot say for
> sure which bug might or might not cause this behaviour. there are quite
> a lot of changes which mention sync/resync/replication and the like
> - newest RHEL5 kernel kernel-2.6.18-274.12.1.el5
>
> Well, if you have gotten this far, thank you for your patience and for
> staying with me :-)
>
> If you have any suggestions on what could cause this behaviour, please
> let me know.
>
> greetings from germany
> volker
>
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 286 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20111215/08fbafa4/attachment.pgp>