slaprunner: supervisor handles better log files

software/slaprunner/common.cfg

@@ -44,7 +44,7 @@ mode = 0644
 recipe = slapos.recipe.template
 url = ${:_profile_base_location_}/instance-runner.cfg
 output = ${buildout:directory}/template-runner.cfg.in
-md5sum = d3821fef9182b2f1c096117aaeb118da
+md5sum = 2c1ea4e4e2fdb6ee2f3e893c9da9b4f8
 mode = 0644
 
 [template-runner-import-script]
@@ -146,7 +146,7 @@ mode = 0644
 [template-supervisord]
 recipe = hexagonit.recipe.download
 url = ${:_profile_base_location_}/template/${:filename}
-md5sum = f1f08f789195dc62b66c45e130f4114d
+md5sum = 15500060605c0224d3534af60b07c6be
 location = ${buildout:parts-directory}/${:_buildout_section_name_}
 filename = supervisord.conf.in
 download-only = true

software/slaprunner/instance-runner.cfg

@@ -627,6 +627,7 @@ autostart = false
 directory = $${buildout:directory}
 exitcodes = 0
 logfile = $${directory:log}/supervisord.log
+no_logfile = NONE
 numprocs = 1
 path = $${environ:PATH}
 pidfile = $${directory:run}/supervisord.pid
@@ -644,7 +645,9 @@ slapgrid-sr-startretries = 2
 socket_name = unix://$${:socket_path}
 socket_path = $${directory:tmp}/supervisord.sock
 startsecs = 1
-stdout_logfile = NONE
+# This file logs errors from listeners. Supervisord has its own logfile.
+# Processes should handle their logs by themselves
+stderr_logfile = $${directory:log}/supervisord-errors.log
 
 [supervisord-conf]
 recipe = slapos.recipe.template:jinja2

software/slaprunner/template/supervisord.conf.in

@@ -23,8 +23,8 @@ exitcodes = {{ supervisord['exitcodes'] }}
 startretries = {{ supervisord['slapgrid-sr-startretries'] }}
 startsecs = {{ supervisord['startsecs'] }}
 autorestart = {{ supervisord['autorestart'] }}
-stdout_logfile = {{ supervisord['stdout_logfile'] }}
-stderr_logfile = {{ supervisord['stdout_logfile'] }}
+stdout_logfile = {{ supervisord['no_logfile'] }}
+stderr_logfile = {{ supervisord['no_logfile'] }}
 directory = {{ supervisord['directory'] }}
 environment = PATH="{{- supervisord['path'] -}}"
 
@@ -37,11 +37,13 @@ exitcodes = {{ supervisord['exitcodes'] }}
 startretries = {{ supervisord['slapgrid-cp-startretries'] }}
 startsecs = {{ supervisord['startsecs'] }}
 autorestart = {{ supervisord['autorestart'] }}
-stdout_logfile = {{ supervisord['stdout_logfile'] }}
-stderr_logfile = {{ supervisord['stdout_logfile'] }}
+stdout_logfile = {{ supervisord['no_logfile'] }}
+stderr_logfile = {{ supervisord['no_logfile'] }}
 directory = {{ supervisord['directory'] }}
 environment = PATH="{{- supervisord['path'] -}}"
 
 [eventlistener:listener-slapgrid]
 command = {{ listener_slapgrid }}
 events = PROCESS_STATE_EXITED
+stdout_logfile = {{ supervisord['no_logfile'] }}
+stderr_logfile = {{ supervisord['stderr_logfile'] }}

[Jérome Perrin]

I guess this ("processes are supposed to create their log files, so no need to keep logs in supervisor") is the original reason of this change, but what about this:

• processes manage their own log files in ~/var/log/
• supervisord keep logs of processes stdout/stderr

The concern is that with this setting this to NONE, stdout/stderr is completely discarded. Even if we prefer that our programs logs to proper log files, we may be interested in program outputs on stdout/stderr.

I assume that if slapos proxy process gets an exception, it's silently restarted and we don't have the traceback that should go to stderr.

But thinking more about this, it's probably not important [edit: important in the context of Romain's bug when proxy db "updates" from 14 to 13 Slow (unusable) webrunner editor ] because errors during request processing are probably catched and logged in flask log file.

/cc @romain

---

for reference:

relevant supervisor doc: http://supervisord.org/configuration.html .

our generated section is like this, with NONE for logfiles:

[program:slapproxy]
command = /opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/bin/slapos proxy start --logfile /srv/slapgrid/slappart8/var//log//slapproxy.log --cfg /srv/slapgrid/slappart8/etc//slapos.cfg
process_name = slapproxy
numprocs = 1
autostart = true
startsecs = 1
exitcodes = 0
autorestart = true
stdout_logfile = NONE
stderr_logfile = NONE
directory = /srv/slapgrid/slappart8
environment = PATH="/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/nano/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/vim/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/screen/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/git/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/curl/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/python2.7/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/tig/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/zip/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/mosh/bin:/opt/slapgrid/b0bcd9831b23f334bc85e4a193c748a0/parts/bash/bin:/srv/slapgrid/slappart8/bin/:/usr/bin:/bin/"

[Nicolas Wavrant]

Indeed, I think we should keep logging stderr.

Please don't trust the wisdom of this commit, it was done when I was a trainee...