[Drbd-dev] running DRBD in usermode

[David Butterfield]

On 6/14/19 3:49 AM, Robert Altnoeder wrote:
> Only changing errno to _errno to fix this is probaby not the best idea,
> so if we're going to change this, then I'd rather use a new variable
> name that does not even suggest being related to errno, otherwise
> whoever sees it the next time might be tempted to change it back to errno.

Sure.  I just thought I'd point out the few things I noticed along the way.
How (or if) they should be fixed isn't my aim.  There was remarkably little
trouble getting 50,000 lines of DRBD kernel code running in usermode (using
as a base the project I did a few years ago bringing SCST to usermode).

I compile the DRBD source nearly unmodified, providing a simulated 2.6.32
environment around it so it feels at home (using its kernel_compat stuff).
The only changes to DRBD source code are temporary #ifndef around rb_augment,
netlink multicast, and nested spinlocking, which I haven't written
emulations for yet, plus maybe another dozen places with 1-2 lines of
change, most of which I hope to eventually be able to clean out.

> While I'm at it, I am wondering a bit about the modification of DRBD to
> run in usermode. It's not that I couldn't see a whole lot of reasons why
> you'd want to do this, quite the opposite, but I believe it would make a
> lot more sense if you did that on top of a true microkernel OS such as
> e.g. Minix, seL4, Integrity or QNX. That might even turn the whole thing
> into a very useful project, as such a port of DRBD might be interesting
> for use in e.g. embedded systems running those OSs.

Any of that could be done.  The "platform" I ported to was "POSIX system
calls along with a few libraries like pthreads and libfuse and my multi-
threaded event engine".  (I probably use a couple of Linux-specific system
calls or options, but those should be isolated and easily replaceable.)

The "usermode_compat" ("UMC") code that (partially) emulates the Linux
kernel environment is in a separate layer (separate repository) from the
"Multithreaded Engine" ("MTE") layer that provides basic system services
(memory, timers, threads, socket event polling, debugging, etc), with MTE
services accessed through a "sys_service" ops vector (which was designed
*without* having in mind using it as a base to run Linux kernel code).
(I wrote MTE first, then got the idea to write UMC using MTE services and
use it to port SCST to usermode.)

The UMC "Linux kernel emulation" layer should be able to operate with any
"sys_service provider" that implements the functions behind the sys_service
ops vector.  The MTE is about 7500 lines of code that could probably be
re-implemented as a smaller shim to an alternative mem/time/threading
provider such as a microkernel, without disturbing the Linux kernel
emulation layer too much.

These are a couple of diagrams from the SCST project that show the layering:
https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_usermode_service_map.pdf
https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_usermode_includes.pdf

Regards,
David Butterfield