Commit 2f25b1ba authored by Tristan Cavelier's avatar Tristan Cavelier

monitor2: Add stack/monitor2

More information available on the README file.
parent 01d5ebd9
Monitor
=======
This stack has for purpose to know if all promises went/are ok.
It provides a web interface, to see which promises failed. It also provide a rss
feed to easily know the actual state of your instance, and to know when it
started to went bad.
THIS STACK IS A KIND OF FORK OF THE `stack/monitor`. THIS ONE WAS CREATED AS A
REDESIGNED ONE TO REMOVE UNWANTED FEATURES AND TO GO FURTHER TO THE GOOD DESIGN
DIRECTION. PLEASE, DO NOT USE THE OLD MONITORING INTERFACE OR ONLY FOR BACKWARD
COMPATIBILITY REASON.
Summary:
- Activate monitoring for you software
- Add a monitor promise
- Information about URL access
- Monitor promise configuration example
- Promise requirements
- monitor.haljson example
- monitor.conf example
Activate monitoring for your software
-------------------------------------
You just have to extend the monitor stack from your software.cfg.
You can also create a new buildout which extends your software, and the
monitoring stack:
[buildout]
extends =
monitor_url
my_software_url
In your instance.cfg, your publish section should be named `[publish]` in order
to extends the one of the monitoring stack.
Then, in the same file you can configure the monitor by adding this section:
[monitor-instance-parameter]
monitor-httpd-ipv6 = ...
monitor-httpd-port = ...
monitor-title = ...
Add a monitor promise
---------------------
For instance, we want to create a promise for KVM log parsing. Add these
sections in its instance.cfg:
[directory]
monitor-promise = ${:etc}/monitor-promise
[kvm-log-parser-promise]
recipe = ....
filename = kvm-log-parser
rendered = ${directory:monitor-promise}/${:filename}
mode = 0755
[buildout]
parts += kvm-log-parser-promise
We can optionaly add promise title:
[kvm-log-parser-promise-parameter]
# fill with -> see "Service config example" below
title = Kvm log parse
[kvm-log-parser-promise-cfg]
recipe = slapos.recipe.template:jinja2
rendered = ${directory:monitor-promise}/${kvm-log-parser-promise:filename}.cfg
template = service.cfg
context = section parameter_dict kvm-log-parser-promise-parameter
[buildout]
parts += kvm-log-parser-promise-cfg
... and optionaly a specific frequency:
[kvm-log-parser-promise-parameter]
frequency = */5 * * * *
Optionaly, we also want a custom interface:
[directory]
kvm-log-parser-promise-interface-dir = ....../interface
[kvm-log-parser-promise-parameter]
private-path-list += ${directory:kvm-log-parser-promise-interface-dir}
[kvm-log-parser-promise-interface]
recipe = ....
rendered = ${directory:kvm-log-parser-interface-dir}/index.html
[buildout]
parts += kvm-log-parser-promise-interface
service.cfg:
[service]
{% for key, value in parameter_dict.items() -%}
{{ key }} = {{ dumps(value) }}
{% endfor -%}
Information about URL access
----------------------------
Open HTTP GET on static files, open HTTP POST on cgi
GET <root_monitor>/ // classical monitoring interface
GET <root_monitor>/monitor.haljson // monitor conf
GET <root_monitor>/public/<service>.status.json // service status json
Example for KVM log parsing promise
GET <kvm_monitor>/monitor.haljson
GET <kvm_monitor>/public/kvm-log-parser.status.json
GET <kvm_monitor>/public/kvm-log-parser/index.html
POST <kvm_monitor>/cgi-bin/monitor-run-promise.cgi?service=kvm-log-parse // rerun the promise
Information about internal file tree
------------------------------------
Tree for monitor runtime:
etc/monitor.conf // generated by slapos
etc/cron.d/monitor // generated by slapos
bin/monitor.py // generated by slapos
srv/monitor/web/index.html // static
srv/monitor/web/monitor.css // static
srv/monitor/web/monitor.js // static
srv/monitor/web/monitor.haljson // generated by monitor.py
srv/monitor/public/.... // generated by monitor.py
srv/monitor/private/.... // generated by monitor.py
Example for KVM log parsing promise
etc/monitor-promise/kvm-log-parse.cfg // generated by slapos (kvm-log-parser-promise)
etc/monitor-promise/kvm-log-parse // generated by slapos (kvm-log-parser-promise)
var/kvm-log-parser-promise/interface/index.html // generated by slapos (kvm-log-parser-promise)
var/log/kvm.log // generated by kvm
var/log/kvm-log-parse-last-report.csv // generated by kvm-log-parse
srv/monitor/public/kvm-log-parse.status.json // generated by kvm-log-parse (indirectly by the monitor promise executor)
srv/monitor/public/kvm-log-parse/kvm.log -> var/log/kvm.log // generated by monitor.py
srv/monitor/public/kvm-log-parse/kvm-log-parse-last-report.csv -> var/log/kvm-log-parse-last-report.csv // genareted by monitor.py
srv/monitor/private/kvm-log-parse/interface -> var/kvm-log-parser-promise/interface // generated by monitor.py
Monitor promise config example
------------------------------
Example for KVM log parsing promise
# etc/monitor-promise/kvm-log-parse.cfg
[service]
title = Kvm log parse
frequency = <Cron Syntax>
public-path-list = $instance/var/log/kvm.log # automatically symlink to srv/monitor/public/$service/
private-path-list = $instance/var/log # automatically symlink to srv/monitor/private/$service/
On cron, the command will be something like:
${service:frequency} ${monitor:promise-executor-path} '${monitor:service-pid-folder}/${service:name}.pid' '${service:status-path}' '${promise_path}'
and "monitor:promise-executor-path" is a script that would run a promise if not
already on going (see `run-promise.py`).
TODO cron accepts 999 characters maximum for a command, so we should reduce the
size of the cron command
TODO put `run-promise.py` in the software
Promise requirements
--------------------
A promise should check something (like web page is well cached, there's not too
much slow queries, ...):
- MUST output the status.json in stdout
- SHOULD output on stdout
- MUST return 0 if status is good else != 0
- the status.json MUST contain "message" (string) which explains why the status is OK or bad
monitor.haljson example
-----------------------
{
"_links": {
"related_monitor": [
{ "href": "<url>/static" },
{ "href": "http://my.other.monitor" }
]
},
"_embedded": {
"service": [
{
"_links": {
"status": { "href": "<url>/kvm-log-parse/status.json" },
"interface": { "href": "<url>/kvm-log-parse/index.html" }
},
"title": "KVM log parse",
"id": "kvm-log-parse"
},
{
"_links": {
"status": { "href": "<url>/<service>/status.json" },
"interface": { "href": "<url>/<service>/index.html" }
},
"title": "Service name",
"id": "<service>"
}
]
},
"title": "KVM Monitoring interface"
}
monitor.conf example
--------------------
[monitor]
title = KVM Monitoring interface
monitor-hal-json = $instance/srv/monitor/web/monitor.haljson
public-folder = $instance/srv/monitor/public
private-folder = $instance/srv/monitor/private
web-folder = $instance/srv/monitor/web
cgi-folder = $instance/srv/monitor/cgi-bin
service-pid-folder = $instance/var/monitor/service-pid
public-path-list =
$instance/var/log
private-path-list =
$instance/srv/backup/log_rotate
monitor-url-list =
https://[...]/
https://[...]/
[buildout]
# XXX THIS STACK IS A KIND OF FORK OF `stack/monitor`. THIS ONE WAS
# CREATED AS A REDESIGNED ONE TO REMOVE UNWANTED FEATURES AND
# TO GO FURTHER TO THE GOOD DESIGN DIRECTION. SEE THE README FOR
# MORE INFORMATION.
extends =
../../component/apache/buildout.cfg
../../component/curl/buildout.cfg
../../component/dash/buildout.cfg
../../component/dcron/buildout.cfg
../../component/openssl/buildout.cfg
parts +=
slapos-cookbook
dcron
monitor-eggs
extra-eggs
monitor-conf
monitor-bin
monitor-web-index-html
monitor-web-monitor-css
monitor-web-monitor-js
monitor-web-monitor-logout-cgi
monitor-web-monitor-logout-page
monitor-template
rss-bin
[monitor-download-base]
recipe = hexagonit.recipe.download
download-only = true
url = ${:_profile_base_location_}/${:filename}
mode = 0644
[monitor-eggs]
recipe = zc.recipe.egg
eggs =
collective.recipe.template
cns.recipe.symlink
[extra-eggs]
recipe = zc.recipe.egg
interpreter = pythonwitheggs
eggs =
PyRSS2Gen
Jinja2
[make-rss-script]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/make-rss.sh.in
md5sum = 98c8f6fd81e405b0ad10db07c3776321
output = ${buildout:directory}/template-make-rss.sh.in
mode = 0644
[monitor-conf]
<= monitor-download-base
filename = monitor.conf.in
md5sum = 2db5c08c7e8658981b4b1e3f27fd5967
[monitor-bin]
<= monitor-download-base
filename = monitor.py.in
md5sum = 2484cb185c391890a05db26c2163af8e
[monitor-web-default-promise-interface]
<= monitor-download-base
filename = default-promise-interface.html
md5sum = 29c899529f4539c9dd1432907b37b7b1
[monitor-web-index-html]
<= monitor-download-base
filename = index.html
md5sum = 262db07691c145301252a49b6b51d11d
[monitor-web-monitor-css]
<= monitor-download-base
filename = monitor.css
md5sum = a18ab932e5e2e656995f47c7d4a7853a
[monitor-web-monitor-js]
<= monitor-download-base
filename = monitor.js.in
md5sum = 8bc4b8368a752f90da2571866768e81f
[monitor-web-monitor-logout-cgi]
recipe = slapos.recipe.template:jinja2
filename = monitor-logout.py.cgi
md5sum = 5b3c0aa559722a3bae5a692ea9a0a441
mode = 0755
template = ${:_profile_base_location_}/${:filename}
rendered = ${buildout:directory}/monitor-logout.cgi
context = key python_executable buildout:executable
[monitor-web-monitor-logout-page]
<= monitor-download-base
filename = monitor-logout.html
md5sum = b210c6842df541305d299081bc1bf81e
[monitor-web-monitor-promise-runner-cgi]
<= monitor-download-base
filename = monitor-run-promise.py.cgi
md5sum = 15625e5bf6c1b57b9199250951ffc16e
[monitor-template]
recipe = slapos.recipe.template:jinja2
filename = template-monitor.cfg
template = ${:_profile_base_location_}/instance-monitor.cfg.jinja2.in
rendered = ${buildout:directory}/template-monitor.cfg
md5sum = fde8b1c9ff04c64a3b7bb0ae11ffe0c5
context =
key apache_location apache:location
key gzip_location gzip:location
raw monitor_bin ${monitor-bin:location}/${monitor-bin:filename}
raw monitor_conf_template ${monitor-conf:location}/${monitor-conf:filename}
raw monitor_web_default_promise_interface ${monitor-web-default-promise-interface:location}/${monitor-web-default-promise-interface:filename}
raw monitor_web_index_html ${monitor-web-index-html:location}/${monitor-web-index-html:filename}
raw monitor_web_monitor_css ${monitor-web-monitor-css:location}/${monitor-web-monitor-css:filename}
key monitor_web_monitor_logout_cgi monitor-web-monitor-logout-cgi:rendered
raw monitor_web_monitor_logout_page ${monitor-web-monitor-logout-page:location}/${monitor-web-monitor-logout-page:filename}
raw monitor_web_monitor_promise_runner_cgi ${monitor-web-monitor-promise-runner-cgi:location}/${monitor-web-monitor-promise-runner-cgi:filename}
raw monitor_web_monitor_js ${monitor-web-monitor-js:location}/${monitor-web-monitor-js:filename}
raw curl_executable_location ${curl:location}/bin/curl
raw dash_executable_location ${dash:location}/bin/dash
raw dcron_executable_location ${dcron:location}/sbin/crond
raw logrotate_executable_location ${logrotate:location}/usr/sbin/logrotate
raw monitor_httpd_promise_conf ${monitor-httpd-promise-conf:location}/${monitor-httpd-promise-conf:filename}
raw monitor_httpd_template ${monitor-httpd-conf:location}/${monitor-httpd-conf:filename}
raw monitor_service_run ${monitor-service-template-run:location}/${monitor-service-template-run:filename}
raw openssl_executable_location ${openssl:location}/bin/openssl
raw python_executable ${buildout:executable}
raw promise_executor_py ${run-promise-py:location}/${run-promise-py:filename}
raw template_wrapper ${template-wrapper:output}
[monitor-httpd-conf]
<= monitor-download-base
md5sum = 625d3d948c0af7b4848d7fad92bfb844
filename = monitor-httpd.conf.in
[monitor-httpd-promise-conf]
<= monitor-download-base
filename = monitor-httpd-promise.conf.in
md5sum = 5913d2a0096b50537f394a49b762b3e5
[monitor-service-template-run]
<= monitor-download-base
md5sum = d5f29fa859a45696e1ff1bb174ab1111
filename = monitor-service-run.in
[run-promise-py]
<= monitor-download-base
filename = run-promise.py
md5sum = 6db26ce13becf8a190e34c14cb8b6f9f
[monitor-httpd-template]
<= monitor-download-base
md5sum = 93e1dda50cb71bfe29966b2946c02dd1
filename = cgi-httpd.conf.in
[index]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/webfile-directory/${:filename}
download-only = true
md5sum = e759977b21c70213daa4c2701f2c2078
destination = ${buildout:parts-directory}/monitor-index
filename = index.cgi.in
mode = 0644
[index-template]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/webfile-directory/${:filename}
download-only = true
destination = ${buildout:parts-directory}/monitor-template-index
md5sum = 7400c8cfa16a15a0d41f512b8bbb1581
filename = index.html.jinja2
mode = 0644
[status-cgi]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/webfile-directory/${:filename}
download-only = true
md5sum = e43d79bec8824265e22df7960744113a
destination = ${buildout:parts-directory}/monitor-template-status-cgi
filename = status.cgi.in
mode = 0644
[status-history-cgi]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/webfile-directory/${:filename}
download-only = true
#md5sum = 4fb26753ee669b8ac90ffe33dbd12e8f
destination = ${buildout:parts-directory}/monitor-template-status-history-cgi
filename = status-history.cgi.in
mode = 0644
[settings-cgi]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/webfile-directory/${:filename}
download-only = true
md5sum = b4cef123a3273e848e8fe496e22b20a8
destination = ${buildout:parts-directory}/monitor-template-settings-cgi
filename = settings.cgi.in
mode = 0644
[monitor-password-cgi]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/webfile-directory/${:filename}
download-only = true
md5sum = c7ba7ecb09d0d1d24e7cb73a212cc33f
destination = ${buildout:parts-directory}/monitor-template-monitor-password-cgi
filename = monitor-password.cgi.in
mode = 0644
[rss-bin]
recipe = hexagonit.recipe.download
url = ${:_profile_base_location_}/${:filename}
download-only = true
md5sum = 6c84a826778cb059754623f39b33651b
destination = ${buildout:parts-directory}/monitor-template-rss-bin
filename = status2rss.py
mode = 0644
[dcron-service]
recipe = slapos.recipe.template
url = ${template-dcron-service:output}
output = $${directory:services}/crond
mode = 0700
logfile = $${directory:log}/crond.log
[template-wrapper]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/wrapper.in
output = ${buildout:directory}/template-wrapper.cfg
mode = 0644
md5sum = 8cde04bfd0c0e9bd56744b988275cfd8
PidFile "{{ httpd_configuration.get('pid-file') }}"
StartServers 1
ServerLimit 1
ThreadLimit 4
ThreadsPerChild 4
ServerName example.com
ServerAdmin someone@email
<IfDefine !MonitorPort>
Listen [{{ httpd_configuration.get('listening-ip') }}]:{{ monitor_parameters.get('port') }}
Define MonitorPort
</IfDefine>
DocumentRoot "{{ directory.get('www') }}"
ErrorLog "{{ httpd_configuration.get('error-log') }}"
LoadModule unixd_module modules/mod_unixd.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule mime_module modules/mod_mime.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule dir_module modules/mod_dir.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule alias_module modules/mod_alias.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule rewrite_module modules/mod_rewrite.so
# SSL Configuration
<IfDefine !SSLConfigured>
Define SSLConfigured
SSLCertificateFile {{ httpd_configuration.get('certificate') }}
SSLCertificateKeyFile {{ httpd_configuration.get('key') }}
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLRandomSeed startup /dev/urandom 256
SSLRandomSeed connect builtin
SSLProtocol -ALL +SSLv3 +TLSv1
SSLHonorCipherOrder On
SSLCipherSuite RC4-SHA:HIGH:!ADH
</IfDefine>
SSLEngine On
ScriptSock {{ httpd_configuration.get('cgid-pid-file') }}
<Directory {{ directory.get('www') }}>
SSLVerifyDepth 1
SSLRequireSSL
SSLOptions +StrictRequire
# XXX: security????
Options +ExecCGI
AddHandler cgi-script .cgi
DirectoryIndex {{ monitor_parameters.get('index-filename') }}
</Directory>
Alias /private/ {{ directory.get('private-directory') }}/
<Directory {{ directory.get('private-directory') }}>
Order Deny,Allow
Deny from env=AUTHREQUIRED
<Files ".??*">
Order Allow,Deny
Deny from all
</Files>
AuthType Basic
AuthName "Private access"
AuthUserFile "{{ monitor_parameters.get('htaccess-file') }}"
Require valid-user
Options Indexes FollowSymLinks
Satisfy all
</Directory>
<Location /rewrite>
AuthType Basic
AuthName "Private access"
AuthUserFile "{{ monitor_parameters.get('htaccess-file') }}"
Require valid-user
</Location>
ProxyVia On
RewriteEngine On
{% for key, value in monitor_rewrite_rule.iteritems() %}
RewriteRule ^/rewrite/{{ key }}($|/.*) {{ value }}/$1 [P,L]
{% endfor %}
<!DOCTYPE html>
<html>
<head>
<title>Promise status</title>
<script>
function getServiceName() {
var match = /(?:&|\?)service_name=([^&]*)/.exec(location.search);
if (match) {
return match[1];
}
throw new Error("no service name found");
}
var service_name = getServiceName(),
monitor_json_url = "/monitor.haljson",
status_json_url = "/public/" + service_name + ".status.json",
rerun_cgi_url = "/cgi-bin/monitor-run-promise.cgi?service=" + service_name;
function newDeferred() {
var d = {
"promise": undefined,
"resolve": undefined,
"reject": undefined
};
d.promise = new Promise(function (resolve, reject) {
d.resolve = resolve;
d.reject = reject;
});
return d;
}
function xhr(param) {
/*global XMLHttpRequest */
var d = newDeferred(), xhr = new XMLHttpRequest(), k, i, l, a;
d.promise.cancel = function () { xhr.abort(); };
xhr.open((param.method || "GET").toUpperCase(), param.url, true);
xhr.responseType = param.responseType || "";
if (param.withCredentials !== undefined) {
xhr.withCredentials = param.withCredentials;
}
if (param.headers) {
a = Object.keys(param.headers);
l = a.length;
for (i = 0; i < l; i += 1) {
k = a[i];
xhr.setRequestHeader(k, param.headers[k]);
}
}
xhr.addEventListener("load", function (e) {
var r, t = e.target, callback;
if (param.noStatusCheck) {
d.resolve(t);
} else if (t.status < 400) {
d.resolve(t);
} else {
d.reject(new Error("HTTP: " + (t.status ? t.status + " " : "") + (t.statusText || "Unknown")));
}
}, false);
xhr.addEventListener("error", function (e) {
return d.reject(new Error("HTTP: Error"));
}, false);
xhr.addEventListener("abort", function (e) {
return d.reject(new Error("HTTP: Aborted"));
}, false);
xhr.send(param.data);
return d.promise;
}
function unexpectedError(reason) {
console.error(reason);
alert(reason);
}
function PromiseStatusInterface(config) {
var it = this,
statusP = document.createElement("p"),
descriptionH2 = document.createElement("h2"),
descriptionP = document.createElement("p"),
errorH2 = document.createElement("h2"),
errorPre = document.createElement("pre"),
header = document.createElement("header"),
h1 = document.createElement("h1"),
h2 = document.createElement("h2"),
a = document.createElement("a"),
button = document.createElement("button");
this.element = config.rootElement || document.createElement("div");
this.statusP = statusP;
this.descriptionP = descriptionP;
this.errorH2 = errorH2;
this.errorPre = errorPre;
this.element.appendChild(header);
header.appendChild(a);
a.setAttribute("tabindex", "-1");
a.setAttribute("href", "/");
a.appendChild(button);
button.textContent = "Home";
a = document.createElement("a");
button = document.createElement("button");
header.appendChild(a);
a.setAttribute("tabindex", "-1");
a.setAttribute("href", "");
a.appendChild(button);
button.textContent = "Refresh";
button = document.createElement("button");
header.appendChild(button);
button.textContent = "Run promise now";
button.onclick = function () {
this.runPromiseNow();
}.bind(this);
this.runPromiseNowButton = button;
this.element.appendChild(h1);
h1.textContent = "Promise status";
this.element.appendChild(statusP);
this.element.appendChild(descriptionH2);
descriptionH2.textContent = "Description";
this.element.appendChild(descriptionP);
this.element.appendChild(errorH2);
errorH2.textContent = "Error output";
errorH2.style.display = "none";
this.element.appendChild(errorPre);
errorPre.style.display = "none";
this.loadStatusUi();
this.loadDescriptionUi();
this.loadErrorUi();
}
PromiseStatusInterface.prototype.loadStatusJson = function () {
if (this.status_json_promise) { return; }
this.status_json_promise = Promise.resolve().then(function () {
return xhr({url: status_json_url, withCredentials: true, responseType: "json"});
}).then(function (xhr) {
return xhr.response;
});
this.status_json_promise.catch(function () { return; }).then(function () {
setTimeout(function () {
delete this.status_json_promise;
}.bind(this), 1000);
}.bind(this));
return this.status_json_promise;
};
PromiseStatusInterface.prototype.loadStatusUi = function () {
this.loadStatusJson();
this.statusP.textContent = "Loading status...";
return this.status_json_promise.then(function (status_json) {
if (status_json.status === "OK") {
this.statusP.textContent = "Status: OK.";
} else {
this.statusP.textContent = "Status: BAD (" + status_json.status + ").";
}
if (status_json.message) {
this.statusP.appendChild(document.createTextNode(" " + status_json.message));
}
}.bind(this), function (reason) {
var message = reason && (reason.target && (reason.target.statusText || "Unknown") || reason.message);
this.statusP.textContent = "Status Json Error: " + (message || "Unknown error");
}.bind(this)).catch(unexpectedError);
};
PromiseStatusInterface.prototype.loadDescriptionUi = function () {
this.loadStatusJson();
this.descriptionP.textContent = "Loading description...";
return this.status_json_promise.then(function (status_json) {
if (status_json.description) {
this.descriptionP.textContent = status_json.description;
} else {
this.descriptionP.textContent = "No description";
}
}.bind(this), function (reason) {
var message = reason && (reason.target && (reason.target.statusText || "Unknown") || reason.message);
this.descriptionP.textContent = "Status Json Error: " + (message || "Unknown error");