Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Fri, Jan 30, 2009 at 11:34:22AM -0800, John Du wrote:
> Hi,
>
> I upgraded DRBD 8.2.0 to 8.3.0. The upgrade went smoothly. However,
> the upgraded version is very slow and the system load is near 100.
> Before the upgrade the load never exceeded 1.
>
> The DRBD worker thread runs into "un-interruptible state" very often for
> long time.
>
> iostat shows:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.52 0.00 0.57 1.47 0.00 97.44
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sdb 31.02 548.36 668.39 27950963 34068995
> drbd1 87.19 99.38 648.63 5065610 33061496
>
> sdb is the underlying storage, a SAN partition of 1.2 TB.
>
> Note that the read speed for sdb is 548 but for drbd1 99. The write
> speeds for sdb and drbd1 are about the same. Note the numbers of blocks
> read for sdb and drbd1 are very different.
strange.
> top shows:
>
> Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 98.9%id, 0.7%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 8175372k total, 8120248k used, 55124k free, 849224k buffers
> Swap: 2031608k total, 0k used, 2031608k free, 4206700k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 5507 root 15 0 0 0 0 D 0 0.0
> 0:29.36 drbd1_worker
>
> Here is the background information
>
> OS: Linux 2.6.18-8.1.15.el5 #1 SMP Thu Oct 4 04:06:39 EDT 2007 x86_64
> x86_64 x86_64 GNU/Linux
>
> The primary DRBD is not connected to the secondary.
>
> DRBD configuration:
>
> global {
> usage-count no;
> }
>
> common {
>
> net {
> sndbuf-size 512k;
> timeout 60;
> connect-int 10;
> ping-int 10;
> max-buffers 2048;
> max-epoch-size 2048;
> }
>
> resource drbd0 {
> protocol A;
>
> startup {
> wfc-timeout 30;
>
> degr-wfc-timeout 120; # 2 minutes.
> }
>
> on host1 {
> device /dev/drbd1;
> disk /dev/sdb1;
> address 10.100.2.232:7789;
> meta-disk internal;
> }
>
> on host2 {
> device /dev/drbd1;
> disk /dev/sdb1;
> address 10.101.152.36:7789;
> meta-disk internal;
> }
> }
>
>
> My questions are:
>
> 1.. Does DRBD 8.3 re-organize the data on disk after the upgrade
No.
> making the IO on drbd1 slow now and it will return to normal after it
> is done?
>
> 2. Can I rollback to 8.2.0 while investigating the cause of the
> slowness?
yes.
> Does 8.3.0 make any changes that 8.2.0 does not rcognize?
no.
> 3. What else can I do to improve the performance to a level close to
> what it was before the upgrade?
I have no idea what goes on there.
It certainly is unexpected behaviour.
--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
please don't Cc me, but send to list -- I'm subscribed