CLI: --log-color option, disabled by default. Safe for log redirection.

slapos/cli/coloredlogs/LICENSE.txt

+Copyright (c) 2013 Peter Odding
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

slapos/cli/coloredlogs/README.rst

+coloredlogs: Colored stream handler for Python's logging module
+===============================================================
+
+.. image:: https://travis-ci.org/xolox/python-coloredlogs.svg?branch=master
+   :target: https://travis-ci.org/xolox/python-coloredlogs
+
+The ``coloredlogs.ColoredStreamHandler`` class is a simple logging handler that
+inherits from `logging.StreamHandler`_ and uses `ANSI escape sequences`_ to
+render your logging messages in color. It uses only standard colors so it
+should work on any UNIX terminal. It's currently tested on Python 2.6, 2.7 and
+3.4. This module does not support non-UNIX terminals (e.g. the Windows
+console). Here is a screen shot of the demo that is printed when the command
+``python -m coloredlogs.demo`` is executed:
+
+.. image:: https://peterodding.com/code/python/coloredlogs/screenshots/terminal.png
+
+Note that the screenshot above includes the custom logging level ``VERBOSE``
+defined by my verboselogs_ module: if you install both ``coloredlogs`` and
+``verboselogs`` it will Just Work (``verboselogs`` is of course not
+required to use ``coloredlogs``).
+
+The logging handler does not use ANSI escape sequences when output redirection
+applies (for example when the standard error stream is being redirected to a
+file or another program) so if you like the format (see below) you can use it
+for your log files as well.
+
+Format of log messages
+----------------------
+
+As can be seen in the screenshot above, the logging handler includes four
+fields in every logged message by default:
+
+1. A timestamp indicating when the event was logged. This field is visible by
+   default. To hide it you can pass the keyword argument
+   ``show_timestamps=False`` when you create the handler.
+2. The hostname of the system on which the event was logged. This field is
+   visible by default. To hide it you can pass the keyword argument
+   ``show_hostname=False`` when you create the handler.
+3. The name of the logger that logged the event. This field is visible by
+   default. To hide it you can pass the keyword argument ``show_name=False``
+   when you create the handler.
+4. The human friendly name of the log level / severity.
+5. The message that was logged.
+
+Usage
+-----
+
+Here's an example of how you would use the logging handler::
+
+   # Create a logger object.
+   import logging
+   logger = logging.getLogger('your-module')
+
+   # Initialize coloredlogs.
+   import coloredlogs
+   coloredlogs.install(level=logging.DEBUG)
+
+   # Some examples.
+   logger.debug("this is a debugging message")
+   logger.info("this is an informational message")
+   logger.warn("this is a warning message")
+   logger.error("this is an error message")
+   logger.fatal("this is a fatal message")
+   logger.critical("this is a critical message")
+
+You can change the formatting of the output to a limited amount by subclassing
+``ColoredStreamHandler`` and overriding the method(s) of your choice. For
+details take a look at the `source code`_ (it's only +/- 160 lines of code,
+including documentation).
+
+For people who like Vim
+-----------------------
+
+Although the logging handler was originally meant for interactive use, it can
+also be used to generate log files. In this case the ANSI escape sequences are
+not used so the log file will contain plain text and no colors. If you use Vim_
+and ``coloredlogs`` and would like to view your log files in color, you can try
+the two Vim scripts included in the ``coloredlogs`` source distributions and
+git repository:
+
+.. image:: https://peterodding.com/code/python/coloredlogs/screenshots/vim.png
+
+For people who like cron
+------------------------
+
+When ``coloredlogs`` is used in a cron_ job, the output that's e-mailed to you
+by cron won't contain any ANSI escape sequences because ``coloredlogs``
+realizes that it's not attached to an interactive terminal. If you'd like to
+have colors e-mailed to you by cron there's a simple way to set it up::
+
+    MAILTO="your-email-address@here"
+    CONTENT_TYPE="text/html"
+    * * * * * root ansi2html your-command
+
+The ``ansi2html`` program is installed when you install ``coloredlogs``. It
+runs ``your-command`` under the external program ``script`` (you need to have
+this installed to get ``ansi2html`` working). This makes ``your-command`` think
+that it's attached to an interactive terminal which means it will output ANSI
+escape sequences and ``ansi2html`` converts these to HTML. Yes, this is a bit
+convoluted, but it works great :-)
+
+You can use ``ansi2html`` without ``coloredlogs``, but please note that it only
+supports normal text, bold text and text with one of the foreground colors
+black, red, green, yellow, blue, magenta, cyan and white (these are the
+portable ANSI color codes).
+
+Contact
+-------
+
+The latest version of ``coloredlogs`` is available on PyPi_ and GitHub_. For
+bug reports please create an issue on GitHub_. If you have questions,
+suggestions, etc. feel free to send me an e-mail at `peter@peterodding.com`_.
+
+License
+-------
+
+This software is licensed under the `MIT license`_.
+
+© 2013 Peter Odding.
+
+.. External references:
+.. _ANSI escape sequences: http://en.wikipedia.org/wiki/ANSI_escape_code#Colors
+.. _cron: https://en.wikipedia.org/wiki/Cron
+.. _GitHub: https://github.com/xolox/python-coloredlogs
+.. _logging.StreamHandler: http://docs.python.org/2/library/logging.handlers.html#streamhandler
+.. _MIT license: http://en.wikipedia.org/wiki/MIT_License
+.. _peter@peterodding.com: peter@peterodding.com
+.. _PyPi: https://pypi.python.org/pypi/coloredlogs
+.. _source code: https://github.com/xolox/python-coloredlogs/blob/master/coloredlogs/__init__.py
+.. _verboselogs: https://pypi.python.org/pypi/verboselogs
+.. _Vim: http://www.vim.org/

slapos/cli/coloredlogs/__init__.py

+"""
+Colored terminal output for Python's logging module.
+
+Author: Peter Odding <peter@peterodding.com>
+Last Change: May 10, 2014
+URL: https://github.com/xolox/python-coloredlogs
+"""
+
+# Semi-standard module versioning.
+__version__ = '0.5'
+
+# Standard library modules.
+import copy
+import logging
+import os
+import re
+import socket
+import sys
+import time
+
+# Portable color codes from http://en.wikipedia.org/wiki/ANSI_escape_code#Colors.
+ansi_color_codes = dict(black=0, red=1, green=2, yellow=3, blue=4, magenta=5, cyan=6, white=7)
+
+# The logging handler attached to the root logger (initialized by install()).
+root_handler = None
+
+def ansi_text(message, color='black', bold=False):
+    """
+    Wrap text in ANSI escape sequences for the given color and/or style.
+
+    :param message: The text message (a string).
+    :param color: The name of a color (one of the strings black, red, green,
+                  yellow, blue, magenta, cyan or white).
+    :param bold: ``True`` if the text should be bold, ``False`` otherwise.
+    :returns: The text message wrapped in ANSI escape sequences.
+    """
+    return '\x1b[%i;3%im%s\x1b[0m' % (bold and 1 or 0, ansi_color_codes[color], message)
+
+def install(level=logging.INFO, **kw):
+    """
+    Install a :py:class:`ColoredStreamHandler` for the root logger. Calling
+    this function multiple times will never install more than one handler.
+
+    :param level: The logging level to filter on (defaults to ``INFO``).
+    :param kw: Optional keyword arguments for :py:class:`ColoredStreamHandler`.
+    """
+    global root_handler
+    if not root_handler:
+        # Create the root handler.
+        root_handler = ColoredStreamHandler(level=level, **kw)
+        # Install the root handler.
+        root_logger = logging.getLogger()
+        root_logger.setLevel(logging.NOTSET)
+        root_logger.addHandler(root_handler)
+
+# TODO Move these functions into ColoredStreamHandler?
+
+def increase_verbosity():
+    """
+    Increase the verbosity of the root handler by one defined level.
+    Understands custom logging levels like defined by my ``verboselogs``
+    module.
+    """
+    defined_levels = find_defined_levels()
+    current_level = get_level()
+    closest_level = min(defined_levels, key=lambda l: abs(l - current_level))
+    set_level(defined_levels[max(0, defined_levels.index(closest_level) - 1)])
+
+def is_verbose():
+    """
+    Check whether the log level of the root handler is set to a verbose level.
+
+    :returns: ``True`` if the root handler is verbose, ``False`` if not.
+    """
+    return get_level() < logging.INFO
+
+def get_level():
+    """
+    Get the logging level of the root handler.
+
+    :returns: The logging level of the root handler (an integer).
+    """
+    install()
+    return root_handler.level
+
+def set_level(level):
+    """
+    Set the logging level of the root handler.
+
+    :param level: The logging level to filter on (an integer).
+    """
+    install()
+    root_handler.level = level
+
+def find_defined_levels():
+    """
+    Find the defined logging levels.
+    """
+    defined_levels = set()
+    for name in dir(logging):
+        if name.isupper():
+            value = getattr(logging, name)
+            if isinstance(value, int):
+                defined_levels.add(value)
+    return sorted(defined_levels)
+
+class ColoredStreamHandler(logging.StreamHandler):
+
+    """
+    The :py:class:`ColoredStreamHandler` class enables colored terminal output
+    for a logger created with Python's :py:mod:`logging` module. The log
+    handler formats log messages including timestamps, logger names and
+    severity levels. It uses `ANSI escape sequences`_ to highlight timestamps
+    and debug messages in green and error and warning messages in red. The
+    handler does not use ANSI escape sequences when output redirection applies,
+    for example when the standard error stream is being redirected to a file.
+    Here's an example of its use::
+
+        # Create a logger object.
+        import logging
+        logger = logging.getLogger('your-module')
+
+        # Initialize coloredlogs.
+        import coloredlogs
+        coloredlogs.install()
+        coloredlogs.set_level(logging.DEBUG)
+
+        # Some examples.
+        logger.debug("this is a debugging message")
+        logger.info("this is an informational message")
+        logger.warn("this is a warning message")
+        logger.error("this is an error message")
+        logger.fatal("this is a fatal message")
+        logger.critical("this is a critical message")
+
+    .. _ANSI escape sequences: http://en.wikipedia.org/wiki/ANSI_escape_code#Colors
+    """
+
+    def __init__(self, stream=sys.stderr, level=logging.NOTSET, isatty=None,
+                 show_name=True, show_severity=True, show_timestamps=True,
+                 show_hostname=True, use_chroot=True):
+        logging.StreamHandler.__init__(self, stream)
+        self.level = level
+        self.show_timestamps = show_timestamps
+        self.show_hostname = show_hostname
+        self.show_name = show_name
+        self.show_severity = show_severity
+        if isatty is not None:
+            self.isatty = isatty
+        else:
+            # Protect against sys.stderr.isatty() not being defined (e.g. in
+            # the Python Interface to Vim).
+            try:
+                self.isatty = stream.isatty()
+            except Exception:
+                self.isatty = False
+        if show_hostname:
+            chroot_file = '/etc/debian_chroot'
+            if use_chroot and os.path.isfile(chroot_file):
+                with open(chroot_file) as handle:
+                    self.hostname = handle.read().strip()
+            else:
+                self.hostname = re.sub(r'\.local$', '', socket.gethostname())
+        if show_name:
+            self.pid = os.getpid()
+
+    def emit(self, record):
+        """
+        Called by the :py:mod:`logging` module for each log record. Formats the
+        log message and passes it onto :py:func:`logging.StreamHandler.emit()`.
+        """
+        # If the message doesn't need to be rendered we take a shortcut.
+        if record.levelno < self.level:
+            return
+        # Make sure the message is a string.
+        message = record.msg
+        try:
+            if not isinstance(message, basestring):
+                message = unicode(message)
+        except NameError:
+            if not isinstance(message, str):
+                message = str(message)
+        # Colorize the log message text.
+        severity = record.levelname
+        if severity == 'CRITICAL':
+            message = self.wrap_color('red', message, bold=True)
+        elif severity == 'ERROR':
+            message = self.wrap_color('red', message)
+        elif severity == 'WARNING':
+            message = self.wrap_color('yellow', message)
+        elif severity == 'VERBOSE':
+            # The "VERBOSE" logging level is not defined by Python's logging
+            # module; I've extended the logging module to support this level.
+            message = self.wrap_color('blue', message)
+        elif severity == 'DEBUG':
+            message = self.wrap_color('green', message)
+        # Compose the formatted log message as:
+        #   timestamp hostname name severity message
+        # Everything except the message text is optional.
+        parts = []
+        if self.show_timestamps:
+            parts.append(self.wrap_color('green', self.render_timestamp(record.created)))
+        if self.show_hostname:
+            parts.append(self.wrap_color('magenta', self.hostname))
+        if self.show_name:
+            parts.append(self.wrap_color('blue', self.render_name(record.name)))
+        if self.show_severity:
+            parts.append(self.wrap_color('black', severity, bold=True))
+        parts.append(message)
+        message = ' '.join(parts)
+        # Copy the original record so we don't break other handlers.
+        record = copy.copy(record)
+        record.msg = message
+        # Use the built-in stream handler to handle output.
+        logging.StreamHandler.emit(self, record)
+
+    def render_timestamp(self, created):
+        """
+        Format the time stamp of the log record. Receives the time when the
+        LogRecord was created (as returned by :py:func:`time.time()`). By
+        default this returns a string in the format ``YYYY-MM-DD HH:MM:SS``.
+        """
+        return time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(created))
+
+    def render_name(self, name):
+        """
+        Format the name of the logger. Receives the name of the logger used to
+        log the call. By default this returns a string in the format
+        ``NAME[PID]`` (where PID is the process ID reported by
+        :py:func:`os.getpid()`).
+        """
+        return '%s[%s]' % (name, self.pid)
+
+    def wrap_color(self, colorname, message, bold=False):
+        """
+        Wrap text in ANSI escape sequences for the given color [and optionally
+        to enable bold font].
+        """
+        if self.isatty:
+            return ansi_text(message, color=colorname, bold=bold)
+        else:
+            return message
+
+# vim: ts=4 sw=4 et

slapos/cli/coloredlogs/converter.py

+"""
+Program to convert text with ANSI escape sequences to HTML.
+
+Author: Peter Odding <peter@peterodding.com>
+Last Change: May 10, 2014
+URL: https://github.com/xolox/python-coloredlogs
+"""
+
+# Standard library modules.
+import pipes
+import re
+import subprocess
+import sys
+import tempfile
+import webbrowser
+
+# Portable color codes from http://en.wikipedia.org/wiki/ANSI_escape_code#Colors.
+EIGHT_COLOR_PALETTE = ('black',
+                       'red',
+                       'rgb(78, 154, 6)', # green
+                       'rgb(196, 160, 0)', # yellow
+                       'blue',
+                       'rgb(117, 80, 123)', # magenta
+                       'cyan',
+                       'white')
+
+# Regular expression that matches strings we want to convert. Used to separate
+# all special strings and literal output in a single pass (this allows us to
+# properly encode the output without resorting to nasty hacks).
+token_pattern = re.compile('(https?://\\S+|www\\.\\S+|\x1b\\[.*?m)', re.UNICODE)
+
+def main():
+    """
+    Command line interface for the ``ansi2html`` program. Takes a command (and
+    its arguments) and runs the program under ``script`` (emulating an
+    interactive terminal), intercepts the output of the command and converts
+    ANSI escape sequences in the output to HTML.
+    """
+    command = ['script', '-qe']
+    command.extend(['-c', ' '.join(pipes.quote(a) for a in sys.argv[1:])])
+    command.append('/dev/null')
+    program = subprocess.Popen(command, stdout=subprocess.PIPE)
+    stdout, stderr = program.communicate()
+    html_output = convert(stdout)
+    if sys.stdout.isatty():
+        fd, filename = tempfile.mkstemp(suffix='.html')
+        with open(filename, 'w') as handle:
+            handle.write(html_output)
+        webbrowser.open(filename)
+    else:
+        print(html_output)
+
+def convert(text):
+    """
+    Convert text with ANSI escape sequences to HTML.
+
+    :param text: The text with ANSI escape sequences (a string).
+    :returns: The text converted to HTML (a string).
+    """
+    output = []
+    for token in token_pattern.split(text):
+        if token.startswith(('http://', 'https://', 'www.')):
+            url = token
+            if '://' not in token:
+                url = 'http://' + url
+            text = url.partition('://')[2]
+            token = u'<a href="%s" style="color: inherit;">%s</a>' % (html_encode(url), html_encode(text))
+        elif token.startswith('\x1b['):
+            ansi_codes = token[2:-1].split(';')
+            if ansi_codes == ['0']:
+                token = '</span>'
+            else:
+                styles = []
+                for code in ansi_codes:
+                    if code == '1':
+                        styles.append('font-weight: bold;')
+                    elif code.startswith('3') and len(code) == 2:
+                        styles.append('color: %s;' % EIGHT_COLOR_PALETTE[int(code[1])])
+                if styles:
+                    token = '<span style="%s">' % ' '.join(styles)
+                else:
+                    token = ''
+        else:
+            token = html_encode(token)
+            token = encode_whitespace(token)
+        output.append(token)
+    return ''.join(output)
+
+def encode_whitespace(text):
+    """
+    Encode whitespace in text as HTML so that all whitespace (specifically
+    indentation and line breaks) is preserved when the text is rendered in a
+    web browser.
+
+    :param text: The plain text (a string).
+    :returns: The text converted to HTML (a string).
+    """
+    text = text.replace('\r\n', '\n')
+    text = text.replace('\n', '<br>\n')
+    text = text.replace(' ', '&nbsp;')
+    return text
+
+def html_encode(text):
+    """
+    Encode special characters as HTML so that web browsers render the
+    characters literally instead of messing up the rendering :-).
+
+    :param text: The plain text (a string).
+    :returns: The text converted to HTML (a string).
+    """
+    text = text.replace('&', '&amp;')
+    text = text.replace('<', '&lt;')
+    text = text.replace('>', '&gt;')
+    text = text.replace('"', '&quot;')
+    return text
+
+# vim: ts=4 sw=4 et

slapos/cli/coloredlogs/demo.py

+# Demonstration of the coloredlogs package.
+#
+# Author: Peter Odding <peter@peterodding.com>
+# Last Change: May 10, 2014
+# URL: https://github.com/xolox/python-coloredlogs
+
+# Standard library modules.
+import logging
+import time
+
+# Modules included in our package.
+import coloredlogs
+
+# If my verbose logger is installed, we'll use that for the demo.
+try:
+    from verboselogs import VerboseLogger as DemoLogger
+except ImportError:
+    from logging import getLogger as DemoLogger
+
+# Initialize the logger and handler.
+logger = DemoLogger('coloredlogs')
+
+def main():
+
+    # Initialize colored output to the terminal.
+    coloredlogs.install(level=logging.DEBUG)
+
+    # Print some examples with different timestamps.
+    for level in ['debug', 'verbose', 'info', 'warn', 'error', 'critical']:
+        if hasattr(logger, level):
+            getattr(logger, level)("message with level %r", level)
+            time.sleep(1)
+
+    # Show how exceptions are logged.
+    try:
+        class RandomException(Exception):
+            pass
+        raise RandomException("Something went horribly wrong!")
+    except Exception as e:
+        logger.exception(e)
+
+    logger.info("Done, exiting ..")
+
+if __name__ == '__main__':
+    main()

slapos/cli/entry.py

@@ -131,9 +131,6 @@ class SlapOSApp(App):
 
     log = logging.getLogger('slapos')
 
-    CONSOLE_MESSAGE_FORMAT = '%(message)s'
-    LOG_FILE_MESSAGE_FORMAT = '[%(asctime)s] %(levelname)-8s %(message)s'
-
     def __init__(self):
         super(SlapOSApp, self).__init__(
             description='SlapOS client %s' % slapos.version.version,
@@ -177,6 +174,18 @@ class SlapOSApp(App):
             default=None,
             help='Specify a file to log output (default: console only)',
         )
+        parser.add_argument(
+            '--log-color',
+            action='store_true',
+            help='Colored log output in console (stripped if redirected)',
+            default=True,
+        )
+        parser.add_argument(
+            '--log-time',
+            action='store_false',
+            default=True,
+            help='Include timestamp in console log',
+        )
         parser.add_argument(
             '-h', '--help',
             action=SlapOSHelpAction,
@@ -198,6 +207,54 @@ class SlapOSApp(App):
         if self.options.verbose_level > 2:
             self.log.debug('clean_up %s', cmd.__class__.__name__)
 
+    @property
+    def CONSOLE_MESSAGE_FORMAT(self):
+        fmt = []
+        if self.options.log_time and not self.options.log_color:
+            fmt.append('[%(asctime)s]')
+        if not self.options.log_color:
+            fmt.append('%(levelname)s')
+        fmt.append('%(message)s')
+        return ' '.join(fmt)
+
+    @property
+    def LOG_FILE_MESSAGE_FORMAT(self):
+        return '[%(asctime)s] %(levelname)-8s %(message)s'
+
+    def configure_logging(self):
+        """Create logging handlers for any log output.
+        """
+        root_logger = logging.getLogger('')
+        root_logger.setLevel(logging.DEBUG)
+
+        # Set up logging to a file
+        if self.options.log_file:
+            file_handler = logging.FileHandler(
+                filename=self.options.log_file,
+            )
+            formatter = logging.Formatter(self.LOG_FILE_MESSAGE_FORMAT)
+            file_handler.setFormatter(formatter)
+            root_logger.addHandler(file_handler)
+
+        # Always send higher-level messages to the console via stderr
+        if self.options.log_color:
+            import coloredlogs
+            console = coloredlogs.ColoredStreamHandler(show_name=True,     # logger name (slapos) and PID
+                                                       show_severity=True,
+                                                       show_timestamps=self.options.log_time,
+                                                       show_hostname=False)
+        else:
+            console = logging.StreamHandler(self.stderr)
+        console_level = {0: logging.WARNING,
+                         1: logging.INFO,
+                         2: logging.DEBUG,
+                         }.get(self.options.verbose_level, logging.DEBUG)
+        console.setLevel(console_level)
+        formatter = logging.Formatter(self.CONSOLE_MESSAGE_FORMAT)
+        console.setFormatter(formatter)
+        root_logger.addHandler(console)
+        return
+
     def run(self, argv):
         # same as cliff.app.App.run except that it won't re-raise
         # a logged exception, and doesn't use --debug