Commit 83d87101 authored by Alain Takoudjou's avatar Alain Takoudjou

monitor: update README.md

parent cb3c506b
Monitor
=======
Monitor stack
=============
This stack has for purpose to know if all promises went/are ok.
* This stack has for purpose to know if all promises, services, custom monitoring scripts went/are ok.
* It also provides a web interface, to see which promises, instance and hosting subscription status. It also provide a rss feed to easily know the actual state of your instance, and to know when it started to went bad. You can also add your own monitoring scripts.
It provides a web interface, to see which promises failed. It also provide a rss
feed to easily know the actual state of your instance, and to know when it
started to went bad.
THIS STACK IS A KIND OF FORK OF THE `stack/monitor`. THIS ONE WAS CREATED AS A
REDESIGNED ONE TO REMOVE UNWANTED FEATURES AND TO GO FURTHER TO THE GOOD DESIGN
DIRECTION. PLEASE, DO NOT USE THE OLD MONITORING INTERFACE OR ONLY FOR BACKWARD
COMPATIBILITY REASON.
Implementation :
----------------
Summary:
- Activate monitoring for you software
- Add a monitor promise
- Information about URL access
- Monitor promise configuration example
- Promise requirements
- monitor.haljson example
- monitor.conf example
Activate monitoring for your software
-------------------------------------
You just have to extend the monitor stack from your software.cfg.
You can also create a new buildout which extends your software, and the
monitoring stack:
1/ In your software.cfg, extends the stack:
[buildout]
extends =
monitor_url
my_software_url
In your instance.cfg, your publish section should be named `[publish]` in order
to extends the one of the monitoring stack.
../../stack/monitor/buildout.cfg
...
Then, in the same file you can configure the monitor by adding this section:
2/ In your instance.cfg file or instance template, override monitor configuration section to define your custom parameters.
[monitor-instance-parameter]
monitor-httpd-ipv6 = ...
monitor-title = ${slap-configuration:instance-title}
monitor-httpd-ipv6 = ${slap-configuration:ipv6-random}
monitor-httpd-port = ...
monitor-title = ...
monitor-base-url = ${monitor-frontend-promise:url}
root-instance-title = ${slap-configuration:root-instance-title}
opml-url-list =
cors-domains = monitor.app.officejs.com
collector-db = ...
password = ${monitor-htpasswd:passwd}
username = ${monitor-htpasswd:username}
instance-configuration = ...
configuration-file-path = ...
You don't need to define all parameters, you can only set what is required to be changed. ie:
[monitor-instance-parameter]
monitor-httpd-port = 8333
- monitor-title: is the title of the current software instance.
- root-instance-title: it the title of the hosting subscription.
- monitor-httpd-ipv6: is the ipv6 of the computer partition.
- monitor-httpd-port: the port to bind monitor httpd server on.
- monitor-base-url: this url that will be used/showed in monitor interface. This url is present in some monitor generated output files. There can be two value, the default: ${monitor-frontend-promise:url} which access monitor httpd server through the frontend and ${monitor-httpd-conf-parameter:url} which is the url with ipv6 (https://[IPv6]:port/).
- opml-url-list: list of OPML URL of monitor sub-instances, if this is the root instance with at least one child.
- cors-domains: the domain used by the monitor web interface. The default is: monitor.app.officejs.com.
- username: monitor username, this should be the same in all sub-instances. Default is: admin.
- password: monitor password, this should be the same in all sub-instances. Default is generated (${monitor-htpasswd:username}).
- instance-configuration: instance custom information or configuration to show in monitor web interface. There is many possibility:
raw CONFIG_KEY VALUE => non editable configuration, ie: raw monitor-password resqdsdsd34
file CONFIG_KEY PATH_TO_RESULT_FILE => editable configuration.
httpdcors CONFIG_KEY PATH_TO_HTTP_CORS_CFG_FILE PATH_HTTPD_GRACEFUL_WRAPPER => show/edit cors domain in monitor
- configuration-file-path: path of knowledge0 cfg file where instance configuration will be written.
Example of custom monitor-instance-parameter: https://lab.nexedi.com/nexedi/slapos/blob/master/software/slaprunner/instance-runner.cfg#L726
Add a monitor promise
---------------------
For instance, we want to create a promise for KVM log parsing. Add these
sections in its instance.cfg:
By default, monitor stack will include slapos promise directory etc/promise to promise folder. All files in that directory will be considered as a promise.
[directory]
monitor-promise = ${:etc}/monitor-promise
[monitor-conf-parameters]
promise-folder-list =
${directory:promises}
${directory:monitor-promise}
Monitor will run each promise every minutes and save the result in a json file. Here is an exemple of promise result:
[kvm-log-parser-promise]
recipe = ....
filename = kvm-log-parser
rendered = ${directory:monitor-promise}/${:filename}
mode = 0755
{"status": "ERROR", "change-time": 1466415901.53, "hosting_subscription": "XXXX", "title": "vnc_promise", "start-date": "2016-06-21 10:47:01", "instance": "XXXX-title", "_links": {"monitor": {"href": "MONITOR_PRIVATE_URL"}}, "message": "PROMISE_OUPT_MESSAGE", "type": "status"}
[buildout]
parts += kvm-log-parser-promise
We can optionaly add promise title:
Add log directory to monitor
----------------------------
[kvm-log-parser-promise-parameter]
# fill with -> see "Service config example" below
title = Kvm log parse
Log or others files can be added in monitor public or private directory:
[kvm-log-parser-promise-cfg]
recipe = slapos.recipe.template:jinja2
rendered = ${directory:monitor-promise}/${kvm-log-parser-promise:filename}.cfg
template = service.cfg
context = section parameter_dict kvm-log-parser-promise-parameter
[monitor-conf-parameters]
public-path-list =
...
private-path-list =
${directory:log}
[buildout]
parts += kvm-log-parser-promise-cfg
files in public directory are accessible at MONITOR_BASE_URL/public, and for private directory: MONITOR_BASE_URL/private.
... and optionaly a specific frequency:
[kvm-log-parser-promise-parameter]
frequency = */5 * * * *
Add custom scripts to monitor
-----------------------------
Optionaly, we also want a custom interface:
Custom script will be automatically run by the monitor and result will be reported in monitor private folder. To add your custom script in ${monitor-directory:reports} folder. Here is an example:
[directory]
kvm-log-parser-promise-interface-dir = ....../interface
[monitor-check-webrunner-internal-instance]
recipe = slapos.recipe.template:jinja2
template = ${monitor-check-webrunner-internal-instance:location}/${monitor-check-webrunner-internal-instance:filename}
rendered = $${monitor-directory:reports}/$${:filename}
filename = monitor-check-webrunner-internal-instance
mode = 0744
[kvm-log-parser-promise-parameter]
private-path-list += ${directory:kvm-log-parser-promise-interface-dir}
The script will be executed every minutes by default. To change, put periodicity in script name:
- monitor-check-webrunner-internal-instance_every_1_minute
- monitor-check-webrunner-internal-instance_every_5_minute
- monitor-check-webrunner-internal-instance_every_1_hour
- monitor-check-webrunner-internal-instance_every_3_hour
- ...
[kvm-log-parser-promise-interface]
recipe = ....
rendered = ${directory:kvm-log-parser-interface-dir}/index.html
the script name should end with _every_XX_hour or _every_XX_minute. With this, we can set filename as:
[buildout]
parts += kvm-log-parser-promise-interface
filename = monitor-check-webrunner-internal-instance_every_2_minute
service.cfg:
You can get custom script results files at MONITOR_BASE_URL/private/FILE_NAME.
[service]
{% for key, value in parameter_dict.items() -%}
{{ key }} = {{ value.strip().replace("\n", "\n ") }}
{% endfor -%}
Access to Monitor
-----------------
In monitor instance.cfg file, the section [publish] contain information about monitor access.
Usefull information are monitor-base-url, monitor-url, monitor-user and monitor-password.
Information about URL access
----------------------------
- ${publish:monitor-base-url} is the url of monitor httpd server.
- ${publish:monitor-base-url}/public/feed is the RSS url of this monitor instance.
- ${publish:monitor-base-url}/public/feeds is the OPML URL of this monitor instance. To setup monitor instance in your monitoring interface, use OPML URL of the root instance. It should contain URL to others monitor instances.
- ${publish:monitor-base-url}/private is the monitor private directory. Username and password are reqired to connect.
Open HTTP GET on static files, open HTTP POST on cgi
To publish configuration URL in your instance.cfg, you can do like this:
GET <root_monitor>/ // classical monitoring interface
GET <root_monitor>/monitor.haljson // monitor conf
GET <root_monitor>/public/<service>.status.json // service status json
[publish-connection-information]
...
monitor-setup-url = https://monitor.app.officejs.com/#page=settings_configurator&url=${publish:monitor-url}&username=${publish:monitor-user}&password=${publish:monitor-password}
Example for KVM log parsing promise
GET <kvm_monitor>/monitor.haljson
GET <kvm_monitor>/public/kvm-log-parser.status.json
GET <kvm_monitor>/public/kvm-log-parser/index.html
POST <kvm_monitor>/cgi-bin/monitor-run-promise.cgi?service=kvm-log-parse // rerun the promise
Information about internal file tree
------------------------------------
Tree for monitor runtime:
etc/monitor.conf // generated by slapos
etc/cron.d/monitor // generated by slapos
bin/monitor.py // generated by slapos
srv/monitor/web/index.html // static
srv/monitor/web/monitor.css // static
srv/monitor/web/monitor.js // static
srv/monitor/web/monitor.haljson // generated by monitor.py
srv/monitor/public/.... // generated by monitor.py
srv/monitor/private/.... // generated by monitor.py
srv/monitor/cgi-bin/.... // generated by monitor.py
Example for KVM log parsing promise
etc/monitor-promise/kvm-log-parse.cfg // generated by slapos (kvm-log-parser-promise)
etc/monitor-promise/kvm-log-parse // generated by slapos (kvm-log-parser-promise)
var/kvm-log-parser-promise/interface/index.html // generated by slapos (kvm-log-parser-promise)
var/log/kvm.log // generated by kvm
var/log/kvm-log-parse-last-report.csv // generated by kvm-log-parse
srv/monitor/public/kvm-log-parse.status.json // generated by kvm-log-parse (indirectly by the monitor promise executor)
srv/monitor/public/kvm-log-parse/kvm.log -> var/log/kvm.log // generated by monitor.py
srv/monitor/public/kvm-log-parse/kvm-log-parse-last-report.csv -> var/log/kvm-log-parse-last-report.csv // genareted by monitor.py
srv/monitor/private/kvm-log-parse/interface -> var/kvm-log-parser-promise/interface // generated by monitor.py
srv/monitor/cgi-bin/kvm-log-parse/action.cgi -> var/kvm-log-parser-promise/action.cgi // generated by monitor.py
Monitor promise config example
------------------------------
Example for KVM log parsing promise
# etc/monitor-promise/kvm-log-parse.cfg
[service]
title = Kvm log parse
frequency = <Cron Syntax>
cgi-path-list = # automatically symlink to srv/monitor/cgi-bin/$service/
$instance/var/kvm-log-parser-promise/action.cgi
public-path-list = # automatically symlink to srv/monitor/public/$service/
$instance/var/log/kvm.log
private-path-list = # automatically symlink to srv/monitor/private/$service/
$instance/var/log
$instance/var/kvm-log-parser-promise/interface
On cron, the command will be something like:
${service:frequency} ${monitor:promise-executor-path} '${monitor:service-pid-folder}/${service:name}.pid' '${service:status-path}' '${promise_path}'
and "monitor:promise-executor-path" is a script that would run a promise if not
already on going (see `run-promise.py`).
TODO cron accepts 999 characters maximum for a command, so we should reduce the
size of the cron command
TODO put `run-promise.py` in the software
Promise requirements
--------------------
A promise should check something (like web page is well cached, there's not too
much slow queries, ...):
- MUST output the status.json in stdout
- SHOULD output on stdout
- MUST return 0 if status is good else != 0
- the status.json MUST contain "message" (string) which explains why the status is OK or bad
monitor.haljson example
-----------------------
{
"_links": {
"related_monitor": [
{ "href": "<url>/static" },
{ "href": "http://my.other.monitor" }
]
},
"_embedded": {
"service": [
{
"_links": {
"status": { "href": "<url>/kvm-log-parse/status.json" },
"interface": { "href": "<url>/kvm-log-parse/index.html" }
},
"title": "KVM log parse",
"id": "kvm-log-parse"
},
{
"_links": {
"status": { "href": "<url>/<service>/status.json" },
"interface": { "href": "<url>/<service>/index.html" }
},
"title": "Service name",
"id": "<service>"
}
]
},
"title": "KVM Monitoring interface"
}
monitor.conf example
--------------------
[monitor]
title = KVM Monitoring interface
monitor-hal-json = $instance/srv/monitor/web/monitor.haljson
public-folder = $instance/srv/monitor/public
private-folder = $instance/srv/monitor/private
web-folder = $instance/srv/monitor/web
cgi-folder = $instance/srv/monitor/cgi-bin
service-pid-folder = $instance/var/monitor/service-pid
public-path-list =
$instance/var/log
private-path-list =
$instance/srv/backup/log_rotate
monitor-url-list =
https://[...]/
https://[...]/
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment