[Csync2] Csync failing with xinetd, working in standalone mode

[Lars Ellenberg]

On Tue, Jun 21, 2016 at 04:17:24AM +0000, Hamish Currie wrote:
> Hi,
> 
> I'm having a problem with csync2 failing if the server is being run
> via xinetd (csync2 -i), but is working in stand-alone mode (csync2 -ii).
> 
> This is only happening on a pair of RHEL7 servers . Its working fine on our centos7 servers.
> 
> I've provided information from my attempts to debug the problem below,
> Any suggestions you can provide would be appreciated.

Your xinetd seems to fail to even spawn csync2.  See below.
If so, that cannot possibly be fixed in csync2.

> The csync software was compiled from the current latest release.

see http://git.linbit.com/csync2.git
    http://github.com/LINBIT/csync2.git

> /etc/xinetd.d/csync2 contains:
> service csync2
> {
>         disable = no
>         protocol = tcp
>         socket_type = stream
>         wait = no
>         user = root
>         group = root
>         server = /usr/local/sbin/csync2
>         server_args = -i
>         log_type = FILE /tmp/data_csync2.log
>         log_on_failure  += USERID
> }
> 
> When running the server in standalone mode, I run it as root.
> 
> I found this stack overflow question
> http://stackoverflow.com/questions/30235086/error-in-csync2-two-server-data-using-csync2-tools

That would be (x)inetd mixing stderr of csync2 into stdout,
thereby "corrupting" the csync2 protocol.

add "-l" to the inetd invocation.  as in
/etc/inetd.conf:
-csync2          stream  tcp     nowait  root    /usr/sbin/csync2 csync2 -i
+csync2          stream  tcp     nowait  root    /usr/sbin/csync2 csync2 -i -l

xinetd: server_args = -i -l

or add up to two -v (the third -v will break protocol again,
when run from inetd, see below).

currently csync2 defaults to using syslog when run from inetd
when "-v" is present only.

Back in May 2016, I pushed workarounds for defaulting to syslog
when run from inetd, disregarding the debug level.

Csync2 may still use stderr (or even stdout!) directly in some cases.
This optional "debug" logging of csync2 is badly grown
$swearword_of_choice; not sure if I eliminated all of it yet.

> - but adding -l to the server_args did not solve the problem for me.

Too bad :(

> I ran strace on the client side and diffed the 2 scenarios.
> When connecting to the xinetd version of the server I get:
> write(5, "CONFIG \n", 8)                = 8
> read(5, 0x631f60, 512)                  = -1 ECONNRESET (Connection reset by peer)
> write(2, "Config command failed.\n", 23) = 23
> 
> When connecting to the standalone server I get:
> write(5, "CONFIG \n", 8)                = 8
> read(5, "OK (cmd_finished).\n", 512)    = 19
> lstat("/var/data/www/a.txt", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> write(5, "HELLO wfe01\n", 15)        = 15
> read(5, "OK (cmd_finished).\n", 512)    = 19
> 
> I did an strace on the server - when running standalone the config file is read almost immediately, but in xinetd mode the server fails before attempting to read the config file.

> 13:43:30.709033 connect(3, {sa_family=AF_LOCAL, sun_path="/var/lib/sss/pipes/nss"}, 110) = -1 ENOENT (No such file or directory)
> 13:43:30.709113 close(3)                = 0
> 13:43:30.709177 setgroups(1, [0])       = 0
> 13:43:30.709239 setuid(0)               = 0
> 13:43:30.709292 umask(02)               = 022
> 13:43:30.709441 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2197, ...}) = 0
> 13:43:30.709493 write(6, "16/6/21 at 13:43:30: FAIL: csync2 a"..., 56) = 56

Now, what did xinetd complain about there?
That truncated log message might have been the key :-/
This looks like xinetd did not even want to spawn csync2.
Possibly be because some problem with your auth setup,
but that's speculation now.

> 13:43:30.709695 exit_group(0)           = ?
> 13:43:30.709888 +++ exited with 0 +++
> 
> 
> SELinux is set to permissive.
> 
> # sestatus -v
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   permissive
> Mode from config file:          permissive
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Max kernel policy version:      28
> 
> Process contexts:
> Current context:                unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> Init context:                   system_u:system_r:init_t:s0
> /usr/sbin/sshd                  system_u:system_r:sshd_t:s0-s0:c0.c1023
> 
> File contexts:
> Controlling terminal:           unconfined_u:object_r:user_devpts_t:s0
> /etc/passwd                     system_u:object_r:passwd_file_t:s0
> /etc/shadow                     system_u:object_r:shadow_t:s0
> /bin/bash                       system_u:object_r:shell_exec_t:s0
> /bin/login                      system_u:object_r:login_exec_t:s0
> /bin/sh                         system_u:object_r:bin_t:s0 -> system_u:object_r:shell_exec_t:s0
> /sbin/agetty                    system_u:object_r:getty_exec_t:s0
> /sbin/init                      system_u:object_r:bin_t:s0 -> system_u:object_r:init_exec_t:s0
> /usr/sbin/sshd                  system_u:object_r:sshd_exec_t:s0
> /lib/libc.so.6                  system_u:object_r:lib_t:s0 -> system_u:object_r:lib_t:s0
> /lib/ld-linux.so.2              system_u:object_r:lib_t:s0 -> system_u:object_r:ld_so_t:s0


-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT