[DRBD-user] IOwaits after upgrading

[Olivier Le Cam]

Hi -

I have upgraded my DRBD storage from Debian/Lenny to Debian/Squeeze and 
iowait increased significantly. I have uploaded a graph for the last 
week CPU utilisation (with a long interrupt in the middle, snmp was not 
working as expected after upgrade) here:
http://olecam.online.fr/cpu_last_week.png

The reason of this upgrade is mainly the heavy load I occasionally face 
and I'd like to give flashcache a try. Now I wonder if I should continue 
that way or downgrade back to Lenny due to the aggravated performance!... :)

I have been able to optimize things a bit (iowait were actually even 
worse the first hours after upgrade which you can see on the graph), 
thanks to good readings here and in the drbd manual. Here is the current 
common config:

common {
	protocol C;

	handlers {
		...
		...
	}

	startup {
		degr-wfc-timeout 120;
	}

	disk {
		on-io-error detach;
		no-disk-barrier;
		no-disk-flushes;
		no-md-flushes;
	}

	net {
		max-buffers 10000;
		max-epoch-size 10000;
		sndbuf-size 0;
		unplug-watermark 10000;
	}

	syncer {
		rate 90M;
		al-extents 3389;
	}
}

- scheduler is deadline (tested as well cfq with same results)
- front_merges=0, read_expire=150, write_expire=1500, writes_starved=16
- current DRBD version deployed is 8.3.7

Servers are both identical, 48MB, 3ware RAID controler (9690SA-4I, 
512MB+BBU), direct ethernet bonding 2x1GB. DRBD runs over LVM over 3 x 5 
SAS 10k drives in RAID-5.

The servers always have 500~1000 write IOPS. Stopping DRBD replication 
reduces iowaits by about 50% which seems correct according to the C 
protocol.

Is that some of you have seen similar behavior? Any 
pointers/comments/feedbacks will be greatly appreciated! Thanks in 
anticipation.

regards,
-- 
Olivier