[DRBD-user] DRBD no longer working after RHEL 7 kernel upgrade

[Lars Ellenberg]

Well, obviously DRBD *does* still work just fine.
Though not for those that only upgrade the kernel
without upgrading the module, or vice versa. See below.


As we get more and more reports of people having problems with their
DRBD after a RHEL upgrade, let me quickly state the facts:

RHEL promises a "stable kABI", at least for symbols that are "whitelisted".

Unfortunately, DRBD uses some symbols that are not whitelisted.

But that is not all: RHEL sometimes breaks the kABI,
while *pretending* it was still compatible.

[technical detail: apparently sometimes they don't notice, or sometimes
their devs think using the (*RH_KABI_*) "magic macros" magically makes
their change kABI compatible.
It does not; it only helps to hide the change from modversions magic,
and annotates a compatible change, *IF* the change was in fact compatible;
if the change was NOT compatible, it still hides that fact,
but now *pretends* it was compatible when it is not.
It does not magically make the change compatible.
Makeing a change kABI compatible is the responsibility of the dev!]

Going from 7.7 to 7.8 kernels,
there is a kABI change that is hidden from modversions magic,
so a module that was built against an older kernel loads
(because the loader checks have been told it was okay),
but then does not properly operate.
Same for "too new" module loaded into "too old" kernel.

One typical DRBD kernel message in this case is:
> kernel: drbd: error sending genl reply

It would have been nice that, if the change is incompatible, and it was
unavoidable anyways, to *not* hide that fact and *pretend* it was
compatible, but leave it for modversions to *see* that it was
incompatible, and refuse the module load in the first place.

Would have resulted in a much cleaner failure mode
than "load, but not fully operate" :-(

Anyways.

Please make sure that your DRBD module was built for the kernel (... dot series)
you are running.  Our kmod-drbd package encode a "kernel version hint"
in their release part, so you can match the "uname -r" of your running kernel
with that kmod-drbd kernel version hint:

That is:
RHEL 7.6, *-957* kernels: kmod-drbd-9.0.22_3.10.0_957-2.x86_64.rpm
RHEL 7.7, *-1049* kernels: kmod-drbd-9.0.22_3.10.0_1049-2.x86_64.rpm
RHEL 7.7, *-1062* kernels: kmod-drbd-9.0.22_3.10.0_1062-2.x86_64.rpm
RHEL 7.8, *-1127* kernels: kmod-drbd-9.0.22_3.10.0_1127-2.x86_64.rpm
(and 9.0.23 or later, once released)

you can check your repository urls:
# yum repolist -v
and list all the kmod-drbd your yum can see:
# yum --showduplicates list kmod-drbd
and pick a specific one, if the one yum choses for you does not match
the kernel you intend to use.

Similarly for DRBD 8.4, obviously:
RHEL 7.6, *-957* kernels: kmod-drbd-8.4.11_3.10.0_957-1.el7.x86_64.rpm
RHEL 7.7, *-1049* kernels: kmod-drbd-8.4.11_3.10.0_1049-1.el7.x86_64.rpm
RHEL 7.8, *-1127* kernels: kmod-drbd-8.4.11_3.10.0_1127-1.el7.x86_64.rpm

    Lars