Monitor
=======

This stack has for purpose to know if all promises went/are ok.

It provides a web interface, to see which promises failed. It also provide a rss
feed to easily know the actual state of your instance, and to know when it
started to went bad.

THIS STACK IS A KIND OF FORK OF THE `stack/monitor`. THIS ONE WAS CREATED AS A
REDESIGNED ONE TO REMOVE UNWANTED FEATURES AND TO GO FURTHER TO THE GOOD DESIGN
DIRECTION. PLEASE, DO NOT USE THE OLD MONITORING INTERFACE OR ONLY FOR BACKWARD
COMPATIBILITY REASON.

Summary:

- Activate monitoring for you software
- Add a monitor promise
- Information about URL access
- Monitor promise configuration example
- Promise requirements
- monitor.haljson example
- monitor.conf example


Activate monitoring for your software
-------------------------------------

You just have to extend the monitor stack from your software.cfg.

You can also create a new buildout which extends your software, and the
monitoring stack:

    [buildout]
    extends =
      monitor_url
      my_software_url

In your instance.cfg, your publish section should be named `[publish]` in order
to extends the one of the monitoring stack.

Then, in the same file you can configure the monitor by adding this section:

    [monitor-instance-parameter]
    monitor-httpd-ipv6 = ...
    monitor-httpd-port = ...
    monitor-title = ...


Add a monitor promise
---------------------

For instance, we want to create a promise for KVM log parsing. Add these
sections in its instance.cfg:

    [directory]
    monitor-promise = ${:etc}/monitor-promise

    [kvm-log-parser-promise]
    recipe = ....
    filename = kvm-log-parser
    rendered = ${directory:monitor-promise}/${:filename}
    mode = 0755

    [buildout]
    parts += kvm-log-parser-promise

We can optionaly add promise title:

    [kvm-log-parser-promise-parameter]
    # fill with -> see "Service config example" below
    title = Kvm log parse

    [kvm-log-parser-promise-cfg]
    recipe = slapos.recipe.template:jinja2
    rendered = ${directory:monitor-promise}/${kvm-log-parser-promise:filename}.cfg
    template = service.cfg
    context = section parameter_dict kvm-log-parser-promise-parameter

    [buildout]
    parts += kvm-log-parser-promise-cfg

... and optionaly a specific frequency:

    [kvm-log-parser-promise-parameter]
    frequency = */5 * * * *

Optionaly, we also want a custom interface:

    [directory]
    kvm-log-parser-promise-interface-dir = ....../interface

    [kvm-log-parser-promise-parameter]
    private-path-list += ${directory:kvm-log-parser-promise-interface-dir}

    [kvm-log-parser-promise-interface]
    recipe = ....
    rendered = ${directory:kvm-log-parser-interface-dir}/index.html

    [buildout]
    parts += kvm-log-parser-promise-interface

service.cfg:

    [service]
    {% for key, value in parameter_dict.items() -%}
    {{ key }} = {{ value.strip().replace("\n", "\n  ") }}
    {% endfor -%}


Information about URL access
----------------------------

Open HTTP GET on static files, open HTTP POST on cgi

    GET  <root_monitor>/                              // classical monitoring interface
    GET  <root_monitor>/monitor.haljson               // monitor conf
    GET  <root_monitor>/public/<service>.status.json  // service status json

Example for KVM log parsing promise

    GET  <kvm_monitor>/monitor.haljson
    GET  <kvm_monitor>/public/kvm-log-parser.status.json
    GET  <kvm_monitor>/public/kvm-log-parser/index.html
    POST <kvm_monitor>/cgi-bin/monitor-run-promise.cgi?service=kvm-log-parse  // rerun the promise


Information about internal file tree
------------------------------------

Tree for monitor runtime:

    etc/monitor.conf                 // generated by slapos
    etc/cron.d/monitor               // generated by slapos
    bin/monitor.py                   // generated by slapos
    srv/monitor/web/index.html       // static
    srv/monitor/web/monitor.css      // static
    srv/monitor/web/monitor.js       // static
    srv/monitor/web/monitor.haljson  // generated by monitor.py
    srv/monitor/public/....          // generated by monitor.py
    srv/monitor/private/....         // generated by monitor.py
    srv/monitor/cgi-bin/....         // generated by monitor.py

Example for KVM log parsing promise

    etc/monitor-promise/kvm-log-parse.cfg                                                                    // generated by slapos (kvm-log-parser-promise)
    etc/monitor-promise/kvm-log-parse                                                                        // generated by slapos (kvm-log-parser-promise)
    var/kvm-log-parser-promise/interface/index.html                                                          // generated by slapos (kvm-log-parser-promise)
    var/log/kvm.log                                                                                          // generated by kvm
    var/log/kvm-log-parse-last-report.csv                                                                    // generated by kvm-log-parse
    srv/monitor/public/kvm-log-parse.status.json                                                             // generated by kvm-log-parse (indirectly by the monitor promise executor)
    srv/monitor/public/kvm-log-parse/kvm.log -> var/log/kvm.log                                              // generated by monitor.py
    srv/monitor/public/kvm-log-parse/kvm-log-parse-last-report.csv -> var/log/kvm-log-parse-last-report.csv  // genareted by monitor.py
    srv/monitor/private/kvm-log-parse/interface -> var/kvm-log-parser-promise/interface                      // generated by monitor.py
    srv/monitor/cgi-bin/kvm-log-parse/action.cgi -> var/kvm-log-parser-promise/action.cgi                    // generated by monitor.py


Monitor promise config example
------------------------------

Example for KVM log parsing promise

    # etc/monitor-promise/kvm-log-parse.cfg
    [service]
    title =             Kvm log parse
    frequency =         <Cron Syntax>
    cgi-path-list =                                   # automatically symlink to srv/monitor/cgi-bin/$service/
      $instance/var/kvm-log-parser-promise/action.cgi
    public-path-list =                                # automatically symlink to srv/monitor/public/$service/
      $instance/var/log/kvm.log
    private-path-list =                               # automatically symlink to srv/monitor/private/$service/
      $instance/var/log
      $instance/var/kvm-log-parser-promise/interface

On cron, the command will be something like:

    ${service:frequency} ${monitor:promise-executor-path} '${monitor:service-pid-folder}/${service:name}.pid' '${service:status-path}' '${promise_path}'

and "monitor:promise-executor-path" is a script that would run a promise if not
already on going (see `run-promise.py`).

TODO cron accepts 999 characters maximum for a command, so we should reduce the
size of the cron command

TODO put `run-promise.py` in the software


Promise requirements
--------------------

A promise should check something (like web page is well cached, there's not too
much slow queries, ...):

- MUST output the status.json in stdout
- SHOULD output on stdout
- MUST return 0 if status is good else != 0
- the status.json MUST contain "message" (string) which explains why the status is OK or bad


monitor.haljson example
-----------------------

    {
      "_links": {
        "related_monitor": [
          { "href": "<url>/static" },
          { "href": "http://my.other.monitor" }
        ]
      },
      "_embedded": {
        "service": [
          {
            "_links": {
              "status": { "href": "<url>/kvm-log-parse/status.json" },
              "interface": { "href": "<url>/kvm-log-parse/index.html" }
            },
            "title": "KVM log parse",
            "id": "kvm-log-parse"
          },
          {
            "_links": {
              "status": { "href": "<url>/<service>/status.json" },
              "interface": { "href": "<url>/<service>/index.html" }
            },
            "title": "Service name",
            "id": "<service>"
          }
        ]
      },
      "title": "KVM Monitoring interface"
    }


monitor.conf example
--------------------

    [monitor]
    title =              KVM Monitoring interface
    monitor-hal-json =   $instance/srv/monitor/web/monitor.haljson
    public-folder =      $instance/srv/monitor/public
    private-folder =     $instance/srv/monitor/private
    web-folder =         $instance/srv/monitor/web
    cgi-folder =         $instance/srv/monitor/cgi-bin
    service-pid-folder = $instance/var/monitor/service-pid
    public-path-list =
      $instance/var/log
    private-path-list =
      $instance/srv/backup/log_rotate
    monitor-url-list =
      https://[...]/
      https://[...]/
