[Drbd-dev] [PATCH 0/2] exec: refactor how call_usermodehelper works, and update the sense of the core_pipe recursion check (v3)

[Neil Horman]

Ok, version 3, taking olegs next set of notes into account.  Really just made
the subprocess_info struct visible from the header file (not opaque), and
modified the cleanup and init routines such that they accept a subprocess_info
parameter.


Hey all-
	So, about 6 months ago, I made a set of changes to how the
core-dump-to-a-pipe feature in the kernel works.  We had reports of several
races, including some reports of apps bypassing our recursion check so that a
process that was forked as part of a core_pattern setup could infinitely crash
and refork until the system crashed.

	We fixes those by improving our recursion checks.  The new check
basically refuses to fork a process if its core limit is zero, which works well.

	Unfortunately, I've been getting grief from maintainer of user space
programs that are inserted as the forked process of core_pattern.  They contend
that in order for their programs (such as abrt and apport) to work, all the
running processes in a system must have their core limits set to a non-zero
value, to which I say 'yes'.  I did this by design, and think thats the right
way to do things.

	But I've been asked to ease this burden on user space enough times that
I thought I would take a look at it.  The first suggestion was to make the
recursion check fail on a non-zero 'special' number, like one.  That way the
core collector process could set its core size ulimit to 1, and enable the
kernel's recursion detection.  This isn't a bad idea on the surface, but I don't
like it since its opt-in, in that if a program like abrt or apport has a bug and
fails to set such a core limit, we're left with a recursively crashing system
again.

	So I've come up with this.  What I've done is modify the
call_usermodehelper api such that an extra parameter is added, a function
pointer which will be called by the user helper task, after it forks, but before
it exec's the required process.  This will give the caller the opportunity to
get a call back in the processes context, allowing it to do whatever it needs to
to the process in the kernel prior to exec-ing the user space code.  In the case
of do_coredump, this callback is ues to set the core ulimit of the helper
process to 1.  This elimnates the opt-in problem that I had above, as it allows
the ulimit for core sizes to be set to the value of 1, which is what the
recursion check looks for in do_coredump.

	This patch has been tested successfully by some of the Abrt maintainers
in fedora, with good results.  Patch applies to the latest -mm tree as of this
AM.

Neil

Signed-off-by: Neil Horman <nhorman at tuxdriver.com>
Tested-by: Jiri Moskovcak <jmoskovc at redhat.com>
CC: Ingo Molnar <mingo at redhat.com>
CC: drbd-dev at lists.linbit.com
CC: Benjamin Herrenschmidt <benh at kernel.crashing.org>
CC: Thomas Sailer <t.sailer at alumni.ethz.ch>
CC: Adam Belay <abelay at mit.edu>
CC: Greg Kroah-Hartman <gregkh at suse.de>
CC: Michal Januszewski <spock at gentoo.org>
CC: Al Viro <viro at zeniv.linux.org.uk>
CC: Neil Brown <neilb at suse.de>
CC: Mark Fasheh <mfasheh at suse.com>
CC: Paul Menage <menage at google.com>
CC: Stephen Hemminger <shemminger at linux-foundation.org>
CC: Kentaro Takeda <takedakn at nttdata.co.jp>