Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Mon, Jun 11, 2012 at 11:42:35PM +0200, Florian Haas wrote: > On 06/11/12 23:00, Matthias Hensler wrote: > > On Mon, Jun 11, 2012 at 10:31:16PM +0200, Florian Haas wrote: > >> On 06/11/12 22:14, Matthias Hensler wrote: > >>> Indeed, the problem lies within the kernel version used to build the > >>> drbd.ko module. I double checked by using all userland tools from 8.3.13 > >>> elrepo build together with my drbd.ko build on 2.6.32-71 (but run from > >>> 2.6.32-220). > >>> > >>> Just to be clear: all tests were made with kernel 2.6.32-220, and the > >>> userland version does not matter. > >>> > >>> drbd.ko | 8.3.11 | 8.3.13 > >>> ---------------------+--------+------- > >>> build on 2.6.32-71 | good | good > >>> build on 2.6.32-220 | bad | bad > >>> > >>> > >>> So, how to debug this further? I would suspect looking at the symbols of > >>> both modules might give a clue? > >> > >> As a knee-jerk response based on a hunch -- you've been warned :) --, > >> this could be related to the BIO_RW_BARRIER vs. FLUSH/FUA dance that the > >> RHEL 6 kernel has been doing between the initial RHEL 6 release, and > >> more recent updates (when they've been backporting the "let's kill > >> barriers" upstream changes from post-2.6.32). > > > > OK. > > > >> Try configuring your disk section with no-disk-barrier, no-disk-flushes > >> and no-md-flushes (in both configurations) and see if your kernel module > >> change still makes a difference. > > > > Just did that: > > > > Using the drbd.ko build on 2.6.32-71 shows minor increase in > > performance (108,5 MByte/s, so some 5% more or so). > > Yeah, that's what you tend to top out at, throughput wise, over a GbE > network. > > > Using the drbd.ko build on 2.6.32-220.17.1 now finally brings the > > expected performance (same as with the 2.6.32-71 built). > > > >> Of course, in production you should only use those options if you have > >> no volatile caches involved in the I/O path. > > > > Yes, that is clear. I did not plan to disable barriers, as the > > bottleneck in my setup should be clearly the network. > > Slightly misguided conclusions, or incorrect assumptions, or both. :) > What you should be doing is get a good cache with a BBU or > capacitor-backed flash, and then disable barriers and flushes. That's > all in the User's Guide. > > >> Not sure if this is useful, but I sure hope it is. :) > > > > Well, what does that mean: are the modules build on 2.6.32-71 broken in > > a way that they do not use barriers (and therefore dangerous to use), or > > is everything fine with the 2.6.32-71 builds and just building on a > > newer kernel produces broken modules? > > For whether the newly produced modules are "broken", and if so in what > way and how to prevent any such breakage, I'll defer to the devs. As for > the alleged performance regression that was also reported in this thread > I'm curious to find out what this means for upstream kernel users, as > (IIUC) the current 8.3.13 codebase just resynced with upstream for the > 3.5 kernel merge window. > > I'm fairly confident Lars will weigh in on this, so let's see what he says. All said already. But if you insist... The RHEL kernel changed in a very peculiar way between RHEL 6.0 and RHEL 6.1: They kept kABI (in theory), but changed the meaning of certain bitflags in the bio. So what used to cause a "BARRIER" now causes it to fail more or less noisily. And what used to be unused bits now cause FUA and/or FLUSH requests. Which bitflags are used, and their meaning, is hardcoded at compiletime, based on the kernel headers used. If you use a drbd.ko, built against the old headers, on a new kernel, all weak-modules and modversion and whatnot magic would think it is kABI compatible, modprobe will find and load it, but in fact the bio bitflag meanings have changed. Explicit barriers/flushes done internally by DRBD simply would not be possible, regardless of drbd configuration. The performance degradation you experience is simply that your config did not disable these, and if compiled against the right headers, they are used and effective. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com