<HTML>
<HEAD>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<META content="OPENWEBMAIL" name=GENERATOR>
</HEAD>
<BODY bgColor=#ffffff>
<font size="2">Hello all,
<br />
<br />I have a fresh question that's bugging me for the past week or so. Been unable to determine why my cluster is acting this way so I will throw the question out here to the list for your opinions. I have a 2node cluster running CentOS 5.3 x86_64, with DRBD 8.3.6, Heartbeat 3.0.1-1, and Pacemaker 1.0.6 all on a pair of IBM x3500 servers each running 2 x QuadCore 3.0GHz and 32GB RAM using 8 x 450GB SAS drives running in RAID10 configuration with the onboard IBM AACRAID (ServeRAID 8k) controllers.
<br />
<br />What we are seeing is an extremely high load during rcp data transfer from a remote machine to the cluster. I cannot reproduce this on a test cluster, it only seems to happen on our production machine and only during the day with 200+ users connected to the machine. This machine has a lot of users and activity, but nothing too extreme - we rarely see our 1min load avgerage go over a 1 or a 2.
<br />First time we had someone inhouse transfer about 20GB of data to this cluster from a remote machine using rcp - the system load on the primary cluster node soared to over a 33! I can now reproduce this on demand at any time. After a fresh install and reload of the OS and all our data on both machines, things looked better (still loaded up but not as bad). Now here a week later and we are seeing the very same thing again. Transfer of 10 -30GB of data from elsewhere on the LAN to this cluster causes it to spin out of control on disk I/O (CPU uses is almost non-existant when this happens). I don't see anything posted about possible fragmentation of the DRBD meta-data, so I am trying to forget that the cluster acted more civilized when it was freshly installed. I have about only about 25% of the available space in use on my /dev/drbd0 partition (1.6TB total size).
<br />
<br />I followed the DRBD Users Guide and did a lot of throughput and latency testing months ago and arrived at what I thought would be optimal for our uses, but this is something new. Below is my current drbd.conf file. System almost acts like it is getting bogged down while replicating data to the secondary, but the secondary is showing almost no load of any kind going on.
<br />
<br />I have compiled up some numbers to make the issue a bit more evident to you. Using sar I pulled some numbers on disk activity.
<br />These are the numbers I see when both cluster nodes are up and running normally with (sar -b) during an rcp data transfer:
<br />
<br /> tps rtps wtps bread/s bwrtn/s
<br />13:08:07 163.75 46.22 117.53 454.18 1568.13
<br />13:08:12 2627.89 23.90 2603.98 286.85 59270.52
<br />13:08:17 3052.60 8.40 3044.20 76.80 68673.60
<br />13:08:22 3050.89 2.39 3048.51 19.09 68895.43
<br />13:08:27 3035.91 17.26 3018.65 242.86 67758.73
<br />13:08:32 3029.11 15.45 3013.66 123.56 67873.27
<br />13:08:37 2990.12 0.00 2990.12 0.00 68151.78
<br />13:08:42 3038.69 33.33 3005.36 276.19 68150.79
<br />13:08:47 3009.86 14.79 2995.07 250.89 67747.53
<br />13:08:52 3046.34 17.03 3029.31 242.38 68451.88
<br />13:08:57 2962.77 4.16 2958.61 33.27 68069.70
<br />13:09:02 2996.63 3.56 2993.07 28.51 68000.00
<br />13:09:07 2982.57 0.00 2982.57 0.00 67719.60
<br />13:09:12 3008.13 1.19 3006.94 9.52 68449.21
<br />13:09:17 2998.42 0.00 2998.42 0.00 68071.29
<br />13:09:22 1886.36 4.74 1881.62 56.92 35857.71
<br />13:09:27 432.27 17.73 414.54 277.29 4231.47
<br />13:09:32 134.13 32.34 101.80 297.01 1191.22
<br />
<br />I then shutdown the sceondary server to compare what happens when there is no DRDB replication going on, and the numbers shot up dramatically:
<br />
<br /> tps rtps wtps bread/s bwrtn/s
<br />3:58:14 127.69 1.79 125.90 23.90 1730.68
<br />13:58:19 20386.06 11.75 20374.30 210.36 467987.25
<br />13:58:24 22312.57 98.80 22213.77 1216.77 510267.47
<br />13:58:29 260.44 38.17 222.27 419.88 2874.75
<br />
<br />
<br />From these numbers you can see how my disk activity shot up dramatically when there was no DRBD replication going on in the background.
<br />
<br />First thing I looked into here was enabling jumbo frames on the replication link, but my NIC's don't seem to support this - I cannot set anything above 1500 MTU. In the past when I was doing some legwork on optimization, the deadline scheduler didn't seem to help me out at all. Does anyone have any ideas why I get such an odd bottle neck here? It almost seems like the server is getting behind on DRBD replication which appears to be loading up the system. After the rcp ends, it takes the system several minutes to calm down to a "normal" load again. We decided on protocol A early on to minimize this effect. From what I can tell, the protocol version is the only thing the should affect replication speed. We are using a straight connection between the secondary onboard 10/100/1000 NICs so there is nothing else on that interface other than secondary heartbeat communications between machines. Any insight into this issue would be greatly appreciated!
<br />
<br />
<br />global { usage-count yes; }
<br />common {
<br /> startup {
<br /> # If a cluster starts up in degraded mode, it will echo a message to all
<br /> # users. It'll wait 60 seconds then halt the system.
<br /> wfc-timeout 120; # drbd init script will wait infinitely on resources.
<br /> degr-wfc-timeout 120; # 2 minutes.
<br /> }
<br /> syncer {
<br /> rate 100M; # Sync rate, in megabytes. 10M is good for 100Mb network.
<br /> verify-alg md5; # can also use md5, crc32c, ect
<br /> csums-alg md5; # can also use md5, crc32c, ect
<br /> al-extents 3833; # Must be prime, number of active sets.
<br /> }
<br /> handlers {
<br /> pri-on-incon-degr "/usr/local/bin/support_drbd_deg";
<br /> split-brain "/usr/local/bin/support_drbd_sb";
<br /> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
<br /> fence-peer "/usr/lib64/heartbeat/drbd-peer-outdater -t 5";
<br /> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
<br /> }
<br /> disk {
<br /> on-io-error detach; # What to do when the lower level device errors.
<br /> no-disk-barrier;
<br /> no-disk-flushes;
<br /> no-disk-drain;
<br /> no-md-flushes;
<br /> fencing resource-only;
<br /> }
<br /> net {
<br /> unplug-watermark 8192;
<br /> max-buffers 8192;
<br /> max-epoch-size 8192;
<br /> sndbuf-size 512k;
<br /> rcvbuf-size 0;
<br /> ko-count 4; # Peer is dead if this count is exceeded.
<br /> after-sb-0pri discard-zero-changes;
<br /> after-sb-1pri discard-secondary;
<br /> after-sb-2pri disconnect;
<br /> }
<br />}
<br />resource drbd0 {
<br /> protocol A;
<br /> device /dev/drbd0;
<br /> disk /dev/sda4;
<br /> meta-disk internal;
<br /> on supportHA1 {
<br /> address 10.0.0.1:7789;
<br /> }
<br /> on supportHA2 {
<br /> address 10.0.0.2:7789;
<br /> }
<br />}
<br />
<br />
<br />Kenneth M
DeChick
<br />Linux Systems
Administrator
<br />Community Computer Service,
Inc.
<br />(315)-255-1751
ext154
<br />http://www.medent.com
<br />kend@medent.com
<br />Registered Linux User
#497318
<br />-- -- -- -- -- -- -- -- -- --
--
<br />
"You canna change the laws of physics, Captain; I've got to have
thirty
minutes!
"
<br />
<br />.
<center><img src="https://www.medent.com/openwebmail/images/defmailsig.jpg" /></center>
<br />
</font>
</BODY>
</HTML>
This message has been scanned for viruses and dangerous content by MailScanner, SpamAssassin & ClamAV. <BR>
<BR>
This message and any attachments may contain information that is protected by law as privileged and confidential, and is transmitted for the sole use <BR>
of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, copying or retention of this e-mail <BR>
or the information contained herein is strictly prohibited. If you received this e-mail in error, please immediately notify the sender by e-mail, and permanently <BR>
delete this e-mail. <BR>