Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Mon, Feb 09, 2009 at 03:10:22PM -0800, John Du wrote: > Lars Ellenberg wrote: >> On Mon, Feb 09, 2009 at 01:04:37PM -0800, John Du wrote: >> >>>>> I still do not understand why iostat only shows DRBD devices on >>>>> this particular node with 8.2.7 and 8.3.0 but not other nodes >>>>> with the same hardware, same Linux Kernel and same DRBD version. >>>>> >>>> io stats accounting was introduced only in drbd-8.0.12 respective 8.2.6. >>>> if you don't see drbd in iostats, you probably use an older DRBD version. >>>> >>>> >>> I obviously did not make myself clear. We were running 8.3 on six >>> nodes and only this node showed DRBD in iostat and only this node >>> was having the problem I reported. I reverted to 8.2 on this node to >>> make our production going. >>> >> >> so you say >> six nodes. >> same hardware. same linux kernel. same drbd. >> but ONE node behaves different. >> >> pretty non-deterministic behaviour for software. >> >> > Yes. Everything is identical. Only this node works with 8.2 but not > 8.3. I know it is hard to believe. It is hard for me to believe too. > Assume something is different on this node, what difference would make > DRBD 8.3 not work but 8.2 do? is that possible that 8.3 sees the meta > data differently than 8.2? > > According to your message, iostat should show DRBD with 8.3. But it > does not on all of the other five nodes. >> I doubt I can help, as if that is true, >> circumstantial evidence suggests that it has nothing to do with drbd, >> but everything to do with whatever makes the non-behaving node behave >> different. >> >> though my guess is >> that either these nodes are not all that identical as you think they are. >> or you installed the new kernel module, but did not actually reload it. >> >> > The log from the problematic node is as follows: You can see it went > from 8.3.0 to 8.2.7 to 8.2.0. You cannot see the server was slow from > the log though. Trust me, it was very very slow. Also I ran the > different versions of DRBD with the same config file shown in my > original message. > > Feb 6 22:22:17 newimapn kernel: drbd: initialised. Version: 8.3.0 (api:88/proto:86-89) yes. I did not doubt the "slow" server. I think that "slowness" is related to the barriers and flushes, and you can disable those, as explained previously. from disabling those barriers and flushes, I'd expect the same performance as with older drbd versions which did not yet use them. but I seriously doubt that you ever loaded drbd 8.3.0 on the _other_ servers, those you say you did not notice any slowdown, and you did not see any iostats either. because first, if you _did_ load drbd 8.3 there, you would have seen iostats (because we support those since 8.2.6). and I'd have expected the same slowdown (because you say its the same hardware and linux kernel). so to find proof of that goof, you'd have to grep the logs of those other servers for the "kernel: drbd: initialised. Version: 8.3.0". and I dare say you won't find it. did I make myself clear this time? -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ please don't Cc me, but send to list -- I'm subscribed