pax_global_header 0000666 0000000 0000000 00000000064 13012616531 0014510 g ustar 00root root 0000000 0000000 52 comment=523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b
slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/ 0000775 0000000 0000000 00000000000 13012616531 0023446 5 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/ 0000775 0000000 0000000 00000000000 13012616531 0024553 5 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/ 0000775 0000000 0000000 00000000000 13012616531 0026551 5 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/README.rst 0000664 0000000 0000000 00000021723 13012616531 0030245 0 ustar 00root root 0000000 0000000
Base resilient stack
====================
This stack is meant to be extended by SR profiles, or other stacks, that need to provide
automated backup/restore, election of backup candidates, and instance failover.
As reference implementations, both stack/lamp and stack/lapp define resilient behavior for
MySQL and Postgres respectively.
This involves three different software_types:
* pull-backup
* {mysoftware}_export
* {mysoftware}_import
where 'mysoftware' is the component that needs resiliency (can be postgres, mysql, erp5, and so on).
pull-backup
-----------
This software type is defined in
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/stack/resilient/instance-pull-backup.cfg.in
and there should be no reason to modify or extend it.
An instance of type 'pull-backup' will receive data from an 'export' instance and immediately populate an 'import' instance.
The backup data is automatically used to build an historical, incremental archive in srv/backup/pbs.
export
------
example:
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/stack/lapp/postgres/instance-postgres-export.cfg.in
This is the *active* instance - the one providing live data to the application.
A backup is run via the bin/exporter script: it will
1) run bin/{mysoftware}-backup
and 2) notify the pull-backup instance that data is ready.
The pull-backup, upon receiving the notification, will make a copy of the data and transmit it to the 'import' instances.
You should provide the bin/{mysoftware}-exporter script, see for instance
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/slapos/recipe/postgres/__init__.py#L207
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/slapos/recipe/mydumper.py#L71
By default, as defined in
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/stack/resilient/pbsready-export.cfg.in#L27
the bin/exporter script is run every 60 minutes.
import
------
example:
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/stack/lapp/postgres/instance-postgres-import.cfg.in
This is the *fallback* instance - the one that can be activated and thus become active.
Any number of import instances can be used. Deciding which one should take over can be done manually
or through a monitoring + election script.
You should provide the bin/{mysoftware}-importer script, see for instance
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/slapos/recipe/postgres/__init__.py#L233
https://lab.nexedi.com/nexedi/slapos/blob/HEAD/slapos/recipe/mydumper.py#L71
In practice
-----------
Add resilience to your software
Let's say you already have a file instance-mysoftware.cfg.in that instantiates your
software. In which there is a part [mysoftware] where there is the main recipe
that instantiates the program.
You need to create two new files, instance-mysoftware-import.cfg.in and
instance-mysoftware-export.cfg.in, following this layout:
IMPORT:
[buildout]
extends = ${instance-mysoftware:output}
${pbsready-import:output}
parts +=
mysoftware
import-on-notification
[importer]
recipe = YourImportRecipe
wrapper = $${rootdirectory:bin}/$${slap-parameter:namebase}-importer
backup-directory = $${directory:backup}
...
EXPORT:
[buildout]
extends = ${instance-mysoftware:output}
${pbsready-export:output}
parts +=
mysoftware
cron-entry-backup
[exporter]
recipe = YourExportRecipe
wrapper = $${rootdirectory:bin}/$${slap-parameter:namebase}-exporter
backup-directory = $${directory:backup}
...
In the [exporter] / [importer] part, you are free to do whatever you want, but
you need to dump / import your data from $${directory:backup} and specify a
wrapper. I suggest you only add options and specify your export/import recipe.
Checking that it works
----------------------
To check that your software instance is resilient you can proceed this way:
Once all instances are successfully deployed, go to your export instance, connect as the instance user and run:
$ ~/bin/exporter
It is the script responsible for triggering the resiliency stack on your instance. After doing a backup of your data, it will notify the pull-backup instances of a new backup, triggering the transfer of this data to the import instances.
Once this script is run successfully, go to your import instance, connect as its instance user and check ~/srv/backup/"your sofwtare"/, the location of the data you wanted to receive. The last part of the resiliency is up to your import script.
DEBUGGING:
Here is a partial list of things you can check to understand what is causing the problem:
- Check that your import script does not fail and successfully places your data in ~/srv/backup/"your software" (as the import instance user) by runnig:
$ ~/bin/"your software"-exporter
- Check the export instance script is run successfully as this instance user by running:
$ ~/bin/exporter
- Check the pull-instance system did its job by going to one of your pull-backup instance, connect as its user and check the log : ~/var/log/equeue.log
-----------------------------------------------------------------------------------------
Finally, instance-mysoftware-import.cfg.in and
instance-mysoftware-export.cfg.in need to be downloaded and accessible by
switch_softwaretype, and you need to extend stack/resilient/buildout.cfg and
stack/resilient/switchsoftware.cfg to download the whole resiliency bundle.
Here is how it's done in the mariadb case for the lamp stack:
** buildout.cfg **
extends =
../resilient/buildout.cfg
[instance-mariadb-import]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/mariadb/instance-mariadb-import.cfg.in
output = ${buildout:directory}/instance-mariadb-import.cfg
md5sum = ...
mode = 0644
[instance-mariadb-export]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/mariadb/instance-mariadb-export.cfg.in
output = ${buildout:directory}/instance-mariadb-export.cfg
md5sum = ...
mode = 0644
** instance.cfg.in **
extends =
../resilient/switchsoftware.cfg
[switch-softwaretype]
...
mariadb = ${instance-mariadb:output}
mariadb-import = ${instance-mariadb-import:output}
mariadb-export = ${instance-mariadb-export:output}
...
Then, in the .cfg file where you want to instantiate your software, you can do, instead of requesting your software
* template-resilient.cfg.in *
[buildout]
...
parts +=
{{ parts.replicate("Name","3") }}
...
[...]
...
[ArgLeader]
...
[ArgBackup]
...
{{ replicated.replicate("Name", "3",
"mysoftware-export", "mysoftware-import",
"ArgLeader","ArgBackup", slapparameter_dict=slapparameter_dict) }}
and it'll expend into the sections require to request Name0, Name1 and Name2,
backuped and resilient. The leader will expend the section [ArgLeader], backups
will expend [ArgBackup]. slapparameter_dict is the dict containing the parameters given to the instance. If you don't need to specify any options, you can
omit the last three arguments in replicate().
Since you will compile your template with jinja2, there should be no $${},
because it is not yet possible to use jinja2 -> buildout template.
To compile with jinja2, see jinja2's recipe.
Deploying your resilient software
---------------------------------
You can provide sla parameters to each request you make (a lot: for export, import and pbs).
example:
Here is a small example of parameters you can provide to control the deployment (case of a runner):
COMP-GRP1
COMP-PBS1
COMP-GROUP2
COMP-RUN2
NET-2
COMP-RUN0
Consequence on sla parameters by request:
* runner0: computer_guid = COMP-RUN0 (provided directly)
* runner1: computer_guid = COMP-GRP1 (provided by group 1)
* runner2: computer_guid = COMP-RUN2 (provided by group 2 but overided directly)
network_guid = NET-2 (provided by group 2)
* PBS 1: computer_guid = COMP-PBS1 (provided by group 1 but overided directly)
* PBS 2: computer_guid = COMP-GRP2 (provided by group 2)
network_guid = NET-2 (provided by group 2)
Parameters are analysed this way:
* If it starts with "-sla-" it is not transmitted to requested instance and is used to do the request as sla.
* -sla-foo-bar=example (foo being a magic key) will be use for each request considering "foo" as a key to use and the sla parameter is "bar". So for each group using the "foo" key, sla parameter "bar" is used with value "example"
About magic keys:
We can find 2 kinds of magic keys:
* id : example, in "-sla-2-foo" 2 is the magic key and the parameter will be used for each request with id 2 (in case of kvm: kvm2 and PBS 2)
* nameid : example, in "-sla-kvm2-foo", foo will be used for kvm2 request. Name for pbs is "pbs" -> "-sla-pbs2-foo".
IMPORTANT NOTE: in case the same foo parameter is asked for the group, the nameid key prevail
slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/TODO.rst 0000664 0000000 0000000 00000002700 13012616531 0030047 0 ustar 00root root 0000000 0000000 * Report, from pbs and from clone, when a backup failed
* Make sure, when a takeover is done, that "importer" script finishes to run while importer instance is changed into exporter.
* Test that, after a successful backup/takeover, another backup is possible and will be successful.
* PBSs and mirrors should monitor/replace themselves
* Report errors from backup
* If a PBS master is down and then back again, it might want to participate in the ongoing election, then.. what happens?
* If the network is partitioned (the two backups don't see each other, but each can see the slapos master) there will be two concurrent elections taking place, with two winners and two renames.
* How to ensure "synchronization" between two main instances? example: Wordpress: mysql is down, then replaced, then inconsistency between apache and the new mysql
* How to deal with big data? I.e how to have working backup/restore system of 1TB data with slow connection?
* How to be sure that elected importer contains a/ the latest data and b/ has finished to pull. We should prevent importer not having a/ and b/ to become the main.
* Should we crypt backed up data?
* If a PBS is lost, a new PBS should be created from another one, in order ot keep history
* If an election takes place and asks for a rename, but no slapgrid is running (or takes too much) then another election will take place
and ask to rename the previously selected winner, thus breaking the resiliency system.
slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/buildout.cfg 0000664 0000000 0000000 00000007477 13012616531 0031100 0 ustar 00root root 0000000 0000000 [buildout]
extends =
../../component/apache/buildout.cfg
../../component/bash/buildout.cfg
../../component/dropbear/buildout.cfg
../../component/openssh/buildout.cfg
../../component/gzip/buildout.cfg
../../component/rdiff-backup/buildout.cfg
../../component/rsync/buildout.cfg
../monitor/buildout.cfg
parts =
pbs-recipe-egg
pbsready
pbsready-import
pbsready-export
template-replicated
template-parts
instance-frozen
# needed tools for resiliency
gzip
rdiff-backup
dash
[pbs-recipe-egg]
recipe = zc.recipe.egg
eggs =
collective.recipe.template
collective.recipe.environment
#----------------
#--
#-- Profiles needed to setup automated backup and recovery.
#--
[pbsready]
# Common parts for pbsready-import and pbsready-export.
# Provides rdiff-backup, notification queue, ssh authentication,
# dropbear server, and the takeover script.
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/pbsready.cfg.in
output = ${buildout:directory}/pbsready.cfg
md5sum = 6e3bd92750407c8d6eaffde2fc94d000
mode = 0644
[pbsready-import]
# An import instance has an importer script, which is called
# by the parent PBS instance when the dump content is propagated.
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/pbsready-import.cfg.in
output = ${buildout:directory}/pbsready-import.cfg
md5sum = 8b4d5288f18b5404a8d87279e5dd1fde
mode = 0644
[pbsready-export]
# An export instance has an exporter script, and communicates
# to parent PBS instances to deliver the exported dump.
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/pbsready-export.cfg.in
output = ${buildout:directory}/pbsready-export.cfg
md5sum = 195ab54f73883d150e45cd3339b91
mode = 0644
[template-pull-backup]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance-pull-backup.cfg.in
output = ${buildout:directory}/instance-pull-backup.cfg
md5sum = 71c24e37426910af05ac484f5e206388
mode = 0644
[template-replicated]
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/template-replicated.cfg.in
md5sum = 7a6234465ae845cb262d4f94c158764e
mode = 0644
destination = ${buildout:directory}/template-replicated.cfg.in
[template-parts]
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/template-parts.cfg.in
md5sum = 071b1034ee8f5cc14f79b16fdeba2813
mode = 0644
destination = ${buildout:directory}/template-parts.cfg.in
[instance-frozen]
# When an instance is detected as broken, its software type is changed to "frozen".
# On the next run of slapgrid-cp, the buildout profile is replaced by instance-frozen.cfg,
# which will run without removing any content because it raises an error.
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance-frozen.cfg.in
md5sum = d21472f0e58f928fb827f2cbf22c4d4a
output = ${buildout:directory}/instance-frozen.cfg
[resilient-web-takeover-cgi-script-download]
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/resilient-web-takeover-cgi-script.py.in
md5sum = c46c8e3e4ce4376c98ad2fc0e2ff0fe4
mode = 0644
destination = ${buildout:directory}/resilient-web-takeover-cgi-script.py.in
# Provide an empty wrapper
[template-wrapper]
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/templates/wrapper.in
output = ${buildout:directory}/template-wrapper.cfg
mode = 0644
md5sum = 8cde04bfd0c0e9bd56744b988275cfd8
##################
# Monitor element
#
[template-monitor-check-resilient-feed]
recipe = hexagonit.recipe.download
ignore-existing = true
url = ${:_profile_base_location_}/templates/monitor-check-resilient-feed.in
download-only = true
md5sum = 19ee9055de961acf402e2dfe5b9581d2
filename = monitor-check-resilient-feed.in
mode = 0644
[rdiff-backup-build]
# use our own version
find-links = http://www.nexedi.org/static/packages/source/rdiff-backup-1.3.4nxd2.tar.gz
[versions]
rdiff-backup = 1.3.4nxd2
instance-frozen.cfg.in 0000664 0000000 0000000 00000000214 13012616531 0032662 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient [buildout]
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
offline = true
parts =
instance-pull-backup.cfg.in 0000664 0000000 0000000 00000025652 13012616531 0033613 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient [buildout]
parts =
publish-connection-information
pbs
logrotate
logrotate-entry-notifier
cron
cron-entry-logrotate
pbs-sshkeys-authority
sshkeys-openssh
backup-checksum-integrity-promise
resilient-genstatrss-wrapper
pbs-push-history-log
backup-signature-link
cron-pbs-status-feed
pull-push-stalled-promise
## Monitor for pbs
monitor-base
monitor-check-resilient-feed-file
extends = ${monitor2-template:rendered}
eggs-directory = ${buildout:eggs-directory}
develop-eggs-directory = ${buildout:develop-eggs-directory}
offline = true
#----------------
#--
#-- Creation of all needed directories.
[rootdirectory]
recipe = slapos.cookbook:mkdirectory
etc = $${buildout:directory}/etc
srv = $${buildout:directory}/srv
bin = $${buildout:directory}/bin
tmp = $${buildout:directory}/tmp
var = $${buildout:directory}/var
[basedirectory]
recipe = slapos.cookbook:mkdirectory
log = $${rootdirectory:var}/log
services = $${rootdirectory:etc}/service
run = $${rootdirectory:var}/run
backup = $${rootdirectory:srv}/backup
promises = $${rootdirectory:etc}/promise
ssh-home = $${rootdirectory:etc}/ssh
notifier = $${rootdirectory:etc}/notifier
[directory]
recipe = slapos.cookbook:mkdirectory
logrotate-entries = $${rootdirectory:etc}/logrotate.d
logrotate-backup = $${basedirectory:backup}/logrotate
cronstamps = $${rootdirectory:etc}/cronstamps
cron-entries = $${rootdirectory:etc}/cron.d
crontabs = $${rootdirectory:etc}/crontabs
cronoutput = $${basedirectory:log}/cron-ouput
pbs-backup = $${basedirectory:backup}/pbs
sshkeys = $${rootdirectory:srv}/sshkeys
pbs-wrappers = $${rootdirectory:bin}/pbs
dot-ssh = $${basedirectory:ssh-home}/.ssh
notifier-feeds = $${basedirectory:notifier}/feeds
notifier-callbacks = $${basedirectory:notifier}/callbacks
notifier-status-items = $${basedirectory:notifier}/status-items
monitor-resilient = $${monitor-directory:private}/resilient
#----------------
#--
#-- Set up the equeue and notifier.
[equeue]
recipe = slapos.cookbook:equeue
socket = $${basedirectory:run}/equeue.sock
lockfile = $${basedirectory:run}/equeue.lock
log = $${basedirectory:log}/equeue.log
database = $${rootdirectory:srv}/equeue.db
wrapper = $${basedirectory:services}/equeue
equeue-binary = ${buildout:bin-directory}/equeue
[notifier-port]
recipe = slapos.cookbook:free_port
minimum = 8088
maximum = 8097
ip = $${notifier:host}
# notifier.notify adds the [exporter, notifier] to the execution queue
# notifier.notify.callback sets up a callback
[notifier]
recipe = slapos.recipe.template:jinja2
template = ${template-wrapper:output}
rendered = $${:wrapper}
wrapper = $${basedirectory:services}/notifier
mode = 0700
command = ${buildout:bin-directory}/pubsubserver --callbacks $${directory:notifier-callbacks} --feeds $${directory:notifier-feeds} --equeue-socket $${equeue:socket} --logfile $${basedirectory:log}/notifier.log $${:host} $${:port}
host = $${slap-network-information:global-ipv6}
port = $${notifier-port:port}
context =
key content notifier:command
[logrotate-entry-equeue]
<= logrotate
recipe = slapos.cookbook:logrotate.d
name = equeue
log = $${equeue:log}
frequency = daily
rotate-num = 30
#----------------
#--
#-- The pull-backup-server contains every backup (incremental)
#-- to prevent a corrupt dump from destroying everything.
[pbs]
<= notifier
recipe = slapos.cookbook:pbs
client = true
feeds = $${directory:notifier-feeds}
callbacks = $${directory:notifier-callbacks}
equeue-socket = $${equeue:socket}
notifier-binary = ${buildout:bin-directory}/pubsubnotifier
rdiffbackup-binary = ${buildout:bin-directory}/rdiff-backup
sshclient-binary = $${openssh-client:wrapper-path}
known-hosts = $${directory:dot-ssh}/known_hosts
promises-directory = $${basedirectory:promises}
directory = $${directory:pbs-backup}
cron-entries = $${cron:cron-entries}
wrappers-directory = $${directory:pbs-wrappers}
run-directory = $${basedirectory:run}
# XXX: this should be named "notifier-host"
notifier-url = http://[$${notifier:host}]:$${notifier:port}
slave-instance-list = $${slap-parameter:slave_instance_list}
ignore-known-hosts-file = $${slap-parameter:ignore-known-hosts-file}
# To get a verbose feed about PBS state
instance-root-name = $${instance-info-parameters:root-name}
log-url = $${publish:monitor-base-url}/private/notifier/
status-item-directory = $${directory:notifier-status-items}
[pbs-resilient-status-feed]
recipe = slapos.cookbook:wrapper
command-line = ${buildout:directory}/bin/generatefeed --output $${:feed-path} --status-item-path $${pbs:status-item-directory} --title "Status feed for $${instance-info-parameters:root-name}-PBS" --link $${pbs:log-url}
feed-path = $${directory:monitor-resilient}/pbs-status-rss
wrapper-path = $${rootdirectory:bin}/resilient-genstatusrss.py
[cron-pbs-status-feed]
<= cron
recipe = slapos.cookbook:cron.d
name = resilient-pbs-status-feed
frequency = 5 * * * *
command = $${pbs-resilient-status-feed:wrapper-path}
#----------------
#--
#-- Deploy cron.
[cron]
recipe = slapos.cookbook:cron
dcrond-binary = ${dcron:location}/sbin/crond
cron-entries = $${directory:cron-entries}
crontabs = $${directory:crontabs}
cronstamps = $${directory:cronstamps}
catcher = $${cron-simplelogger:wrapper}
binary = $${basedirectory:services}/crond
[cron-simplelogger]
recipe = slapos.cookbook:simplelogger
wrapper = $${rootdirectory:bin}/cron_simplelogger
log = $${basedirectory:log}/crond.log
#----------------
#--
#-- Deploy logrotate.
[cron-entry-logrotate]
<= cron
recipe = slapos.cookbook:cron.d
name = logrotate
frequency = 0 0 * * *
command = $${logrotate:wrapper}
[logrotate]
recipe = slapos.cookbook:logrotate
# Binaries
logrotate-binary = ${logrotate:location}/usr/sbin/logrotate
gzip-binary = ${gzip:location}/bin/gzip
gunzip-binary = ${gzip:location}/bin/gunzip
# Directories
wrapper = $${rootdirectory:bin}/logrotate
conf = $${rootdirectory:etc}/logrotate.conf
logrotate-entries = $${directory:logrotate-entries}
backup = $${directory:logrotate-backup}
state-file = $${rootdirectory:srv}/logrotate.status
[logrotate-entry-cron]
<= logrotate
recipe = slapos.cookbook:logrotate.d
name = cron
log = $${cron-simplelogger:log}
frequency = daily
rotate-num = 30
[logrotate-entry-notifier]
recipe = collective.recipe.template
mode = 600
input = inline:
$${directory:notifier-feeds}/* {
rotate 5
weekly
nocompress
missingok
olddir $${directory:logrotate-backup}
}
output = $${logrotate:logrotate-entries}/notifier
#----------------
#--
#-- sshkeys
[sshkeys-directory]
recipe = slapos.cookbook:mkdirectory
requests = $${directory:sshkeys}/openssl-requests
keys = $${directory:sshkeys}/openssl-keys
[pbs-sshkeys-authority]
recipe = slapos.cookbook:sshkeys_authority
request-directory = $${sshkeys-directory:requests}
keys-directory = $${sshkeys-directory:keys}
wrapper = $${basedirectory:services}/pbs_sshkeys_authority
keygen-binary = ${openssh:location}/bin/ssh-keygen
[sshkeys-openssh]
<= pbs-sshkeys-authority
recipe = slapos.cookbook:sshkeys_authority.request
name = pbs-client
type = rsa
executable = $${openssh-client:wrapper-path}
public-key = $${openssh-client:identity-file}.pub
private-key = $${openssh-client:identity-file}
wrapper = $${rootdirectory:bin}/do_backup
#----------------
#--
#-- OpenSSH.
[openssh-client]
recipe = slapos.cookbook:wrapper
home = $${basedirectory:ssh-home}
identity-file = $${:home}/id_rsa
command-line = ${openssh:location}/bin/ssh -T -o "UserKnownHostsFile $${pbs:known-hosts}" -i $${:identity-file}
wrapper-path = $${rootdirectory:bin}/ssh
parameters-extra = true
#----------------
#--
#-- Slave instance list (empty default).
[htpasswd]
recipe = slapos.cookbook:generate.password
storage-path = $${directory:etc}/.monitor_user
bytes = 8
username = admin
[slap-parameter]
slave_instance_list = []
ignore-known-hosts-file = false
monitor-cors-domains =
monitor-httpd-port = 8070
monitor-title = PBS Instance
monitor-password = $${htpasswd:passwd}
monitor-username = $${htpasswd:username}
#----------------
#--
#-- Resiliency promises.
[backup-checksum-integrity-promise]
recipe = slapos.recipe.template:jinja2
template = inline:
#!${dash:location}/bin/dash
# Raise an error if signatures are different
# Error cannot be deduced if files do not exist
cd $${directory:pbs-backup}
if [ ! -f "proof.signature" ]; then exit 0; fi
backup_signature=$(find . -maxdepth 2 -name backup.signature)
if [ -z "$backup_signature" ]; then
exit 0;
else
diff -q "proof.signature" "$backup_signature";
if [ "$?" -eq 0 ]; then
exit 0;
else
echo "Signature file is not the same before and after transfer"
exit 1
fi
fi
rendered = $${basedirectory:promises}/backup-checksum-integrity
mode = 700
[resilient-genstatrss-wrapper]
recipe = slapos.cookbook:wrapper
# XXX - hard-coded Urls
command-line = ${buildout:directory}/bin/rdiffbackup.genstatrss --output '$${monitor-directory:public}/resilient-feed' --rdiff_backup_data_folder '$${pbs:rdiff-backup-data-folder}' --feed_url '$${monitor-conf-parameters:base-url}/public/resilient-feed'
wrapper-path = $${directory:bin}/resilient-genstatrss.py
[pbs-push-history-log]
recipe = cns.recipe.symlink
symlink = $${pbs:rdiff-backup-data-folder}/restore.log = $${basedirectory:log}/pbs-push-history-log
[backup-signature-link]
recipe = cns.recipe.symlink
symlink = $${directory:pbs-backup}/proof.signature = $${directory:monitor-resilient}/backup.signature
[pull-push-stalled-promise]
recipe = slapos.cookbook:wrapper
# time-buffer is 18h : cron for backup is run once a day - 6h of random sleep
command-line = ${buildout:bin-directory}/check-feed-as-promise --feed-path $${pbs-resilient-status-feed:feed-path} --title --ok-pattern 'OK' --time-buffer 64800
wrapper-path = $${basedirectory:promises}/stalled-pull-push
#----------------
#--
#-- Publish instance parameters.
[publish-connection-information]
recipe = slapos.cookbook:publish
ssh-key = $${sshkeys-openssh:public-key-value}
notification-url = http://[$${notifier:host}]:$${notifier:port}/notify
feeds-url = http://[$${notifier:host}]:$${notifier:port}/get/
monitor-base-url = $${publish:monitor-base-url}
monitor-url = $${publish:monitor-url}
monitor-user = $${publish:monitor-user}
monitor-password = $${publish:monitor-password}
#----------------
#--
#-- Monitor
[monitor-instance-parameter]
monitor-httpd-port = $${slap-parameter:monitor-httpd-port}
monitor-title = $${slap-parameter:monitor-title}
cors-domains = $${slap-parameter:monitor-cors-domains}
username = $${slap-parameter:monitor-username}
password = $${slap-parameter:monitor-password}
[monitor-conf-parameters]
private-path-list +=
$${directory:logrotate-backup}
$${basedirectory:log}
[monitor-check-resilient-feed-file]
recipe = slapos.recipe.template:jinja2
template = ${template-monitor-check-resilient-feed:location}/${template-monitor-check-resilient-feed:filename}
rendered = $${monitor-directory:reports}/check-create-resilient-feed-files
mode = 700
context =
key input_feed_directory directory:notifier-feeds
key monitor_feed_directory monitor-directory:public
key base_url publish-connection-information:feeds-url
raw python_executable ${buildout:executable}
parameter-schema.json 0000664 0000000 0000000 00000002257 13012616531 0032611 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient {
"$schema": "http://json-schema.org/draft-04/schema",
"title": "Resiliency Parameters",
"description": "List of possible parameters used in the resilient stack",
"type": "object",
"properties": {
"-sla-0-computer_guid": {
"title": "Target computer for main instance",
"description": "Target computer GUID for main instance.",
"type": "string"
},
"-sla-1-computer_guid": {
"title": "Target computer for first clone",
"description": "Target computer for first clone and PBS.",
"type": "string"
},
"-sla-2-computer_guid": {
"title": "Target computer for second clone",
"description": "Target computer for second clone and PBS.",
"type": "string"
},
"resiliency-backup-periodicity": {
"title": "Periodicity of backup",
"description": "Periodicity of backup, in cron format. Default is every hour.",
"type": "string"
},
"remove-backup-older-than": {
"title": "Remove backups older than...",
"description": "Remove all the backups in PBS that are older than specified value. It should be rdiff-backup-compatible.",
"type": "string",
"default": "2W"
}
}
}
pbsready-export.cfg.in 0000664 0000000 0000000 00000004733 13012616531 0032717 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient [buildout]
extends = ${pbsready:output}
# Explicitely define extended parts from pbsready
# then add local parts
parts =
logrotate
logrotate-entry-cron
logrotate-entry-equeue
cron
cron-entry-logrotate
resilient-sshkeys-authority
sshd-raw-server
sshd-graceful
sshkeys-sshd
sshd-promise
resilient-sshkeys-sshd-promise
sshd-pbs-authorized-key
notifier
notifier-exporter-promise
cron-entry-backup
[resilient-directory]
recipe = slapos.cookbook:mkdirectory
home = $${buildout:directory}
var = $${:home}/var
pid = $${:var}/pid
# Define port of ssh server. It has to be different from import so that it
# supports export/import using same IP (slaprunner, slapos-in-partition,
# ipv4...)
[sshd-port]
recipe = slapos.cookbook:free_port
minimum = 22200
maximum = 22209
ip = $${slap-network-information:global-ipv6}
[notifier-port]
recipe = slapos.cookbook:free_port
minimum = 65526
maximum = 65535
ip = $${notifier:host}
[resilient-publish-connection-parameter]
notification-id = http://[$${notifier:host}]:$${notifier:port}/get/$${notifier-exporter:name}
[notifier-exporter]
# notifier.notify launches an (exporter) executable, and when finished,
# notifies the the pull-backup-servers.
<= notifier
recipe = slapos.cookbook:notifier.notify
name = exporter
title = Dumping $${slap-parameter:namebase}
executable = $${exporter:wrapper}
wrapper = $${rootdirectory:bin}/exporter
notify = $${slap-parameter:notify}
pidfile = $${resilient-directory:pid}/$${:name}.pid
max-run = 3
[logrotate-entry-notifier]
output = $${rootdirectory:etc}/logrotate_notifier.conf
[notifier-exporter-promise]
recipe = slapos.recipe.template:jinja2
mode = 700
template = inline:
#!${bash:location}/bin/bash
EXPORTER_FEED="$${notifier-exporter:log-file}"
FAILURE_PATTERN="FAILURE"
if [ -s "$EXPORTER_FEED" ]; then
tail -n 1 $EXPORTER_FEED | grep -vq FAILURE_PATTERN
fi
rendered = $${basedirectory:promises}/exporter-status
[cron-entry-backup]
# Schedule the periodic database dump.
# Through notifications, this triggers (one or more) incremental backups on PBS instances.
<= cron
recipe = slapos.cookbook:cron.d
name = backup
frequency = $${slap-parameter:resiliency-backup-periodicity}
once-a-day = true
command = ${logrotate:location}/sbin/logrotate -s $${basedirectory:run}/logrotate.status $${logrotate-entry-notifier:output}; $${notifier-exporter:wrapper} --transaction-id `date +%s`
[slap-parameter]
# In cron.d format (i.e things like */15 * * * * are accepted).
resiliency-backup-periodicity =
pbsready-import.cfg.in 0000664 0000000 0000000 00000014126 13012616531 0032705 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient [buildout]
extends = ${pbsready:output}
# Explicitely define extended parts from pbsready
# then add local parts
parts =
logrotate
logrotate-entry-cron
logrotate-entry-equeue
logrotate-entry-notifier
cron
cron-entry-logrotate
resilient-sshkeys-authority
sshd-raw-server
sshd-graceful
sshkeys-sshd
sshd-promise
resilient-sshkeys-sshd-promise
sshd-pbs-authorized-key
notifier
resiliency-takeover-script
resilient-web-takeover-cgi-script
resilient-web-takeover-httpd-wrapper
resilient-web-takeover-httpd-promise
check-backup-integrity-on-notification
import-on-notification
backup-checksum-integrity-promise
resilient-publish-connection-parameter
backup-signature-link
[resilient-publish-connection-parameter]
notification-url = http://[$${notifier:host}]:$${notifier:port}/notify
takeover-url = http://[$${resilient-web-takeover-httpd-configuration-file:listening-ip}]:$${resilient-web-takeover-httpd-configuration-file:listening-port}/
takeover-password = $${resilient-web-takeover-password:passwd}
# Define port of ssh server. It has to be different from import so that it
# supports export/import using same IP (slaprunner, slapos-in-partition,
# ipv4...)
[sshd-port]
recipe = slapos.cookbook:free_port
minimum = 22210
maximum = 22219
ip = $${slap-network-information:global-ipv6}
# Define port of notifier (same reason)
[notifier-port]
recipe = slapos.cookbook:free_port
minimum = 65516
maximum = 65525
ip = $${notifier:host}
[import-on-notification]
# notifier.callback runs a script when a notification (sent by a parent PBS)
# is received
<= notifier
recipe = slapos.cookbook:notifier.callback
on-notification-id = $${slap-parameter:on-notification}
callback = $${importer:wrapper}
[post-notification-run]
recipe = collective.recipe.template
diff-file = $${basedirectory:backup}/backup.diff
proof-signature-file = $${basedirectory:backup}/proof.signature
input = inline:
#!/${bash:location}/bin/bash
cd $${directory:backup}
find -type f ! -name backup.signature ! -wholename "./rdiff-backup-data/*" -print0 | xargs -P4 -0 sha256sum | LC_ALL=C sort -k 66 > $${:proof-signature-file}
diff -ruw backup.signature $${:proof-signature-file} > $${:diff-file}
output = $${rootdirectory:bin}/post-notification-run
mode = 0700
[check-backup-integrity-on-notification]
<= notifier
recipe = slapos.cookbook:notifier.callback
on-notification-id = $${slap-parameter:on-notification}
callback = $${post-notification-run:output}
[backup-checksum-integrity-promise]
recipe = slapos.recipe.template:jinja2
template = inline:
#!/${bash:location}/bin/bash
backup_diff_file=$${post-notification-run:diff-file}
if [ -f "$backup_diff_file" ]; then
if [ $(wc -l "$backup_diff_file" | cut -d \ -f1) -eq 0 ]; then
exit 0;
else
exit 1;
fi
else
# If file doesn't exist, promise shouldnt raise false positive
exit 0;
fi
rendered = $${basedirectory:promises}/backup-checksum-integrity
mode = 700
###########
# Generate the takeover script
###########
[resiliency-takeover-script]
recipe = slapos.cookbook:addresiliency
wrapper-takeover = $${rootdirectory:bin}/takeover
takeover-triggered-file-path = $${rootdirectory:srv}/takeover_triggered
# Add path of file created by takeover script when takeover is triggered
# Takeover script will create this file
# equeue process will watch for file existence.
[equeue]
takeover-triggered-file-path = $${resiliency-takeover-script:takeover-triggered-file-path}
###########
# Deploy a webserver allowing to do takeover from a web browser.
###########
[resilient-web-takeover-password]
recipe = slapos.cookbook:generate.password
storage-path = $${directory:srv}/passwd
bytes = 8
[resilient-web-takeover-cgi-script]
recipe = collective.recipe.template
input = ${resilient-web-takeover-cgi-script-download:destination}
output = $${directory:cgi-bin}/web-takeover.cgi
password = $${resilient-web-takeover-password:passwd}
mode = 700
proof-signature-url = $${publish:monitor-base-url}/private/resilient/backup.signature
# XXX could it be something lighter?
# XXX Add SSL
[resilient-web-takeover-httpd-configuration-file]
recipe = collective.recipe.template
input = inline:
PidFile "$${:pid-file}"
Listen [$${:listening-ip}]:$${:listening-port}
ServerAdmin someone@email
DocumentRoot "$${:document-root}"
ErrorLog "$${:error-log}"
LoadModule unixd_module modules/mod_unixd.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule mime_module modules/mod_mime.so
LoadModule cgid_module modules/mod_cgid.so
LoadModule dir_module modules/mod_dir.so
ScriptSock $${:cgid-pid-file}
# XXX: security????
Options +ExecCGI
AddHandler cgi-script .cgi
DirectoryIndex web-takeover.cgi
output = $${directory:etc}/resilient-web-takeover-httpd.conf
# md5sum =
listening-ip = $${slap-network-information:global-ipv6}
# XXX: randomize-me
listening-port = 9263
htdocs = $${directory:cgi-bin}
pid-file = $${directory:run}/resilient-web-takeover-httpd.pid
cgid-pid-file = $${directory:run}/resilient-web-takeover-httpd-cgid.pid
document-root = $${directory:cgi-bin}
error-log = $${directory:log}/resilient-web-takeover-httpd-error-log
[resilient-web-takeover-httpd-wrapper]
recipe = slapos.cookbook:wrapper
apache-executable = ${apache:location}/bin/httpd
command-line = $${:apache-executable} -f $${resilient-web-takeover-httpd-configuration-file:output} -DFOREGROUND
wrapper-path = $${basedirectory:services}/resilient-web-takeover-httpd
[resilient-web-takeover-httpd-promise]
recipe = slapos.cookbook:check_url_available
path = $${basedirectory:promises}/resilient-web-takeover-httpd
url = http://[$${resilient-web-takeover-httpd-configuration-file:listening-ip}]:$${resilient-web-takeover-httpd-configuration-file:listening-port}/
dash_path = ${dash:location}/bin/dash
curl_path = ${curl:location}/bin/curl
###########
# Symlinks
###########
[backup-signature-link]
recipe = cns.recipe.symlink
symlink = $${post-notification-run:proof-signature-file} = $${directory:monitor-resilient}/backup.signature
slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/pbsready.cfg.in 0000664 0000000 0000000 00000020020 13012616531 0031442 0 ustar 00root root 0000000 0000000 [buildout]
parts =
logrotate
logrotate-entry-cron
logrotate-entry-equeue
logrotate-entry-notifier
cron
cron-entry-logrotate
resilient-sshkeys-authority
sshd-graceful
sshkeys-sshd
sshd-promise
resilient-sshkeys-sshd-promise
sshd-pbs-authorized-key
notifier
#----------------
#--
#-- Creation of all needed directories.
[rootdirectory]
recipe = slapos.cookbook:mkdirectory
etc = $${buildout:directory}/etc
var = $${buildout:directory}/var
srv = $${buildout:directory}/srv
bin = $${buildout:directory}/bin
[basedirectory]
recipe = slapos.cookbook:mkdirectory
log = $${rootdirectory:var}/log
services = $${rootdirectory:etc}/service
run = $${rootdirectory:var}/run
scripts = $${rootdirectory:etc}/run
backup = $${rootdirectory:srv}/backup
promises = $${rootdirectory:etc}/promise
services = $${rootdirectory:etc}/service
cache = $${rootdirectory:var}/cache
notifier = $${rootdirectory:etc}/notifier
[directory]
recipe = slapos.cookbook:mkdirectory
backup = $${basedirectory:backup}/$${slap-parameter:namebase}
ssh = $${rootdirectory:etc}/ssh/
sshkeys = $${rootdirectory:srv}/sshkeys
notifier-feeds = $${basedirectory:notifier}/feeds
notifier-callbacks = $${basedirectory:notifier}/callbacks
cron-entries = $${rootdirectory:etc}/cron.d
crontabs = $${rootdirectory:etc}/crontabs
cronstamps = $${rootdirectory:etc}/cronstamps
logrotate-entries = $${rootdirectory:etc}/logrotate.d
logrotate-backup = $${basedirectory:backup}/logrotate
cgi-bin = $${rootdirectory:srv}/cgi-bin
monitor-resilient = $${monitor-directory:private}/resilient
#----------------
#--
#-- Deploy cron.
[cron]
recipe = slapos.cookbook:cron
dcrond-binary = ${dcron:location}/sbin/crond
cron-entries = $${directory:cron-entries}
crontabs = $${directory:crontabs}
cronstamps = $${directory:cronstamps}
catcher = $${cron-simplelogger:wrapper}
binary = $${basedirectory:services}/crond
[cron-simplelogger]
recipe = slapos.cookbook:simplelogger
wrapper = $${rootdirectory:bin}/cron_simplelogger
log = $${basedirectory:log}/crond.log
#----------------
#--
#-- Deploy logrotate.
[cron-entry-logrotate]
<= cron
recipe = slapos.cookbook:cron.d
name = logrotate
frequency = 0 0 * * *
command = $${logrotate:wrapper}
[logrotate]
recipe = slapos.cookbook:logrotate
# Binaries
logrotate-binary = ${logrotate:location}/usr/sbin/logrotate
gzip-binary = ${gzip:location}/bin/gzip
gunzip-binary = ${gzip:location}/bin/gunzip
# Directories
wrapper = $${rootdirectory:bin}/logrotate
conf = $${rootdirectory:etc}/logrotate.conf
logrotate-entries = $${directory:logrotate-entries}
backup = $${directory:logrotate-backup}
state-file = $${rootdirectory:srv}/logrotate.status
[logrotate-entry-mariadb]
<= logrotate
recipe = slapos.cookbook:logrotate.d
name = mariadb
log = $${mariadb:error-log}
frequency = daily
rotate-num = 30
post = $${mariadb:logrotate-post}
sharedscripts = true
notifempty = true
create = true
[logrotate-entry-cron]
<= logrotate
recipe =slapos.cookbook:logrotate.d
name = crond
log = $${cron-simplelogger:log}
frequency = daily
rotate-num = 30
notifempty = true
create = true
[logrotate-entry-equeue]
<= logrotate
recipe = slapos.cookbook:logrotate.d
name = equeue
log = $${equeue:log} $${sshd-server:log}
frequency = daily
rotate-num = 30
[logrotate-entry-notifier]
recipe = collective.recipe.template
mode = 600
input = inline:
$${notifier:feeds}/* {
rotate 5
weekly
nocompress
missingok
olddir $${directory:logrotate-backup}
}
output = $${logrotate:logrotate-entries}/notifier
#----------------
#--
#-- Sets up an rdiff-backup server (with a openssh server for ssh)
[rdiff-backup-server]
recipe = slapos.cookbook:pbs
client = false
path = $${directory:backup}
wrapper = $${rootdirectory:bin}/rdiffbackup-server
rdiffbackup-binary = ${buildout:bin-directory}/rdiff-backup
#----------------
#--
#-- Set up the equeue and notifier.
[equeue]
recipe = slapos.cookbook:equeue
socket = $${basedirectory:run}/equeue.sock
lockfile = $${basedirectory:run}/equeue.lock
log = $${basedirectory:log}/equeue.log
database = $${rootdirectory:srv}/equeue.db
wrapper = $${basedirectory:services}/equeue
equeue-binary = ${buildout:bin-directory}/equeue
# notifier.notify adds the [exporter, notifier] to the execution queue
# notifier.notify.callback sets up a callback
[notifier]
recipe = slapos.recipe.template:jinja2
template = ${template-wrapper:output}
rendered = $${:wrapper}
wrapper = $${basedirectory:services}/notifier
mode = 0700
feeds = $${directory:notifier-feeds}
callbacks = $${directory:notifier-callbacks}
command = ${buildout:bin-directory}/pubsubserver --callbacks $${directory:notifier-callbacks} --feeds $${directory:notifier-feeds} --equeue-socket $${equeue:socket} --logfile $${basedirectory:log}/notifier.log $${:host} $${:port}
notifier-binary = ${buildout:bin-directory}/pubsubnotifier
host = $${slap-network-information:global-ipv6}
port = $${notifier-port:port}
context =
key content notifier:command
#----------------
#--
#-- OpenSSH.
[resilient-sshd-config]
# XXX: Add timeout support
recipe = slapos.recipe.template:jinja2
rendered = $${directory:etc}/resilient-sshd.conf
path_pid = $${directory:run}/resilient-sshd.pid
template = inline:
PidFile $${:path_pid}
Port $${sshd-port:port}
ListenAddress $${slap-network-information:global-ipv6}
Protocol 2
UsePrivilegeSeparation no
HostKey $${directory:ssh}/server_key.rsa
AuthorizedKeysFile $${directory:ssh}/.ssh/authorized_keys
PasswordAuthentication no
PubkeyAuthentication yes
ForceCommand $${rdiff-backup-server:wrapper}
[sshd-raw-server]
recipe = slapos.cookbook:wrapper
host = $${slap-network-information:global-ipv6}
rsa-keyfile = $${directory:ssh}/server_key.rsa
home = $${directory:ssh}
command-line = ${openssh:location}/sbin/sshd -D -e -f $${resilient-sshd-config:rendered}
wrapper-path = $${rootdirectory:bin}/raw_sshd
[sshd-pbs-authorized-key]
<= sshd-raw-server
recipe = slapos.cookbook:dropbear.add_authorized_key
key = $${slap-parameter:authorized-key}
[sshd-server]
recipe = collective.recipe.template
log = $${basedirectory:log}/sshd.log
input = inline:#!/bin/sh
exec $${sshd-raw-server:wrapper-path} >> $${:log} 2>&1
output = $${rootdirectory:bin}/raw_sshd_log
mode = 700
[sshd-graceful]
recipe = slapos.cookbook:wrapper
command-line = $${directory:bin}/killpidfromfile $${resilient-sshd-config:path_pid} SIGHUP
wrapper-path = $${basedirectory:scripts}/sshd-graceful
[sshd-promise]
recipe = slapos.cookbook:check_port_listening
path = $${basedirectory:promises}/sshd
hostname = $${slap-network-information:global-ipv6}
port = $${sshd-port:port}
#----------------
#--
#-- sshkeys
[sshkeys-directory]
recipe = slapos.cookbook:mkdirectory
requests = $${directory:sshkeys}/resilient-requests
keys = $${directory:sshkeys}/resilient-keys
[resilient-sshkeys-authority]
recipe = slapos.cookbook:sshkeys_authority
request-directory = $${sshkeys-directory:requests}
keys-directory = $${sshkeys-directory:keys}
wrapper = $${basedirectory:services}/resilient_sshkeys_authority
keygen-binary = ${openssh:location}/bin/ssh-keygen
[sshkeys-sshd]
<= resilient-sshkeys-authority
recipe = slapos.cookbook:sshkeys_authority.request
name = sshd
type = rsa
executable = $${sshd-server:output}
public-key = $${sshd-raw-server:rsa-keyfile}.pub
private-key = $${sshd-raw-server:rsa-keyfile}
wrapper = $${basedirectory:services}/sshd
[resilient-sshkeys-sshd-promise]
# Check that public key file exists and is not empty
recipe = collective.recipe.template
input = inline:#!${bash:location}/bin/bash
PUBLIC_KEY_CONTENT="$${sshkeys-sshd:public-key-value}"
if [[ ! -n "$PUBLIC_KEY_CONTENT" || "$PUBLIC_KEY_CONTENT" == *None* ]]; then
exit 1
fi
output = $${basedirectory:promises}/public-key-existence
mode = 700
#----------------
#--
#-- Connection informations to re-use.
[user-info]
recipe = slapos.cookbook:userinfo
# XXX-Cedric: when "aggregation" system is done in libslap, directly publish.
[resilient-publish-connection-parameter]
recipe = slapos.cookbook:publish
ssh-public-key = $${sshkeys-sshd:public-key-value}
ssh-url = ssh://$${user-info:pw-name}@[$${sshd-raw-server:host}]:$${sshd-port:port}/$${rdiff-backup-server:path}
ip = $${slap-network-information:global-ipv6}
resilient-web-takeover-cgi-script.py.in 0000664 0000000 0000000 00000006726 13012616531 0036115 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient #!${buildout:executable}
equeue_database = '${equeue:database}'
equeue_lockfile = '${equeue:lockfile}'
takeover_script = '${resiliency-takeover-script:wrapper-takeover}'
import cgi
import cgitb
import datetime
import gdbm
import os
import shutil
import subprocess
import sys
import tempfile
if os.path.exists('resilient_software_release_information.py'):
from resilient_software_release_information import main as resilient_main
else:
resilient_main = lambda: {}
cgitb.enable()
def getLatestBackupDate():
"""
Get the date of the latest successful backup.
"""
# Create a copy of the db (locked by equeue process)
temporary_directory = tempfile.mkdtemp()
equeue_database_copy = os.path.join(temporary_directory, 'equeue.db')
shutil.copyfile(equeue_database, equeue_database_copy)
db = gdbm.open(equeue_database_copy)
# Usually, there is only one callback (so only one key
# in the db), but if there are several:
# Take the "oldest" one (oldest value).
if not db.keys():
result = False
else:
last_backup = db[db.keys()[0]]
for callback in db.keys():
timestamp = float(db[callback])
if timestamp < last_backup:
last_backup = timestamp
result = datetime.datetime.fromtimestamp(last_backup)
db.close()
shutil.rmtree(temporary_directory)
return result
def isBackupInProgress():
"""
Check if backup is in progress (importer script is running)
by checking if equeue lockfile exists.
"""
# XXX: check if file is valid
return os.path.exists(equeue_lockfile)
def getInformationFromSoftwareRelease():
result = resilient_main()
if isinstance(result, dict):
return result
else:
return {'Custom Information': 'Error, received information is malformed'}
def getSoftwareReleaseInformationFormatted():
result_string = ""
for key, value in getInformationFromSoftwareRelease().items():
result_string += "
%s: %s
" % (key, value)
return result_string
latest_backup_date = getLatestBackupDate()
if latest_backup_date == False:
latest_backup_message = "No backup downloaded yet, takeover should not happen now."
else:
latest_backup_message = latest_backup_date.strftime('%Y-%m-%d %H:%M:%S')
print "Content-Type: text/html"
print
form = cgi.FieldStorage()
if "password" not in form:
print """
This is takeover web interface.
Calling takeover will stop and freeze the current main instance, and make this clone instance the new main instance, replacing the old one.
Warning: submit the form only if you understand what you are doing.
Note: the password asked here can be found within the parameters of your SlapOS instance page.
Last valid backup: %s
Importer script(s) of backup in progress: %s
Backup Signature: ${resilient-web-takeover-cgi-script:proof-signature-url}
%s
""" % (latest_backup_message, isBackupInProgress(), getSoftwareReleaseInformationFormatted())
sys.exit(0)
if form['password'].value != '${:password}':
print "Error
"
print "Password is invalid."
sys.exit(1)
# XXX hardcoded location
result = subprocess.check_output([takeover_script], stderr=subprocess.STDOUT)
print 'Success.'
print '%s
' % result
template-parts.cfg.in 0000664 0000000 0000000 00000001155 13012616531 0032524 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient ## Parts Needed for the resiliency
{% macro replicate(namebase, nbbackup) %}
request-{{namebase}}
resilient-request-{{namebase}}-public-key-promise
{% for id in range(1,nbbackup|int) %}
request-{{namebase}}-pseudo-replicating-{{id}}
resilient-request-{{namebase}}-pseudo-replicating-{{id}}-public-key-promise
{% endfor %}
{% for id in range(1,nbbackup|int) %}
request-pbs-{{namebase}}-{{id}}
resilient-request-pbs-{{namebase}}-{{id}}-public-key-promise
request-pull-backup-server-{{namebase}}-{{id}}
request-pull-backup-server-{{namebase}}-backup-{{id}}
{% endfor %}
{% endmacro %}
template-replicated.cfg.in 0000664 0000000 0000000 00000026260 13012616531 0033513 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient {% macro replicate(namebase, nbbackup, typeexport, typeimport, heriteLeader='', heriteBackup='', slapparameter_dict={}, monitor_parameter_dict={}) %}
{% set sla_parameter_dict = {} -%}
{% set monitor_dict = monitor_parameter_dict.get('parameter', {}) -%}
{% set monitor_return = ' '.join(monitor_parameter_dict.get('return', [])) -%}
{% set monitor_url_list = [] -%}
# prepare sla-parameters
{% if slapparameter_dict is defined -%}
{% for key in slapparameter_dict.keys() -%}
{% if key.startswith('-sla-') -%}
{% do sla_parameter_dict.__setitem__(key, slapparameter_dict.pop(key)) -%}
{% endif -%}
{% endfor -%}
{% endif -%}
[resilient-directory]
recipe = slapos.cookbook:mkdirectory
home = ${buildout:directory}
etc = ${:home}/etc
promise = ${:etc}/promise
## Tells the Backupable recipe that we want a backup
[resilient]
recipe = slapos.cookbook:request
config-namebase = {{namebase}}
software-url = ${slap-connection:software-release-url}
[request-{{namebase}}]
<= resilient
slap-connection
{{heriteLeader}}
software-type = {{typeexport}}
name = {{namebase}}0
return = ssh-public-key ssh-url notification-id ip {{ monitor_return }}
config-number = 0
config-authorized-key = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}{% endfor %}
config-notify = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-notification-url}{% endfor %}
config-name = {{namebase}}0
# Bubble up all the instance parameters to the requested export instance.
{% if slapparameter_dict is defined %}
{% for parameter_name, parameter_value in slapparameter_dict.items() %}config-{{parameter_name}} = {{parameter_value}}
{% endfor %}
{% endif %}
{% for key, value in monitor_dict.iteritems() -%}
config-{{ key }} = {{ value }}
{% endfor -%}
{% if sla_parameter_dict == {} -%}
sla-mode = unique_by_network
{% else %}
{% set sla_key_main = "-sla-%s%s-" % (namebase, 0) -%}
{% set sla_key_secondary = "-sla-%s-" % (0) -%}
{% set sla_key_main_length = sla_key_main | length -%}
{% set sla_key_secondary_length = sla_key_secondary | length -%}
{% set sla_dict = {} -%}
{% for key in sla_parameter_dict.keys() -%}
{% if key.startswith(sla_key_main) -%}
{% do sla_dict.__setitem__(key[sla_key_main_length:], sla_parameter_dict.get(key)) -%}
{% elif key.startswith(sla_key_secondary) and not sla_dict.has_key(key[sla_key_secondary_length:]) -%}
{% do sla_dict.__setitem__(key[sla_key_secondary_length:], sla_parameter_dict.get(key)) -%}
{% endif -%}
{% endfor -%}
{% for key, value in sla_dict.iteritems() -%}
sla-{{ key }} = {{ value }}
{% endfor -%}
{% endif -%}
{% if 'monitor-base-url' in monitor_return and not monitor_parameter_dict.get('set-monitor-url', False) -%}
{% do monitor_url_list.append('${request-' ~ namebase ~ ':connection-monitor-base-url}') -%}
{% endif -%}
{% for id in range(1,nbbackup|int) %}
[request-{{namebase}}-pseudo-replicating-{{id}}]
<= slap-connection
resilient
{{heriteBackup}}
recipe = slapos.cookbook:request
name = {{namebase}}{{id}}
software-url = ${slap-connection:software-release-url}
software-type = {{typeimport}}
return = ssh-public-key ssh-url notification-url ip takeover-url takeover-password {{ monitor_return }}
pbs-notification-id = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-{{id}}-push
config-number = {{id}}
config-name = {{namebase}}{{id}}
config-authorized-key = ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}
config-on-notification = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${:pbs-notification-id}
{% for key, value in monitor_dict.iteritems() -%}
config-{{ key }} = {{ value }}
{% endfor -%}
{% if sla_parameter_dict == {} -%}
sla-mode = unique_by_network
{% else %}
{% set sla_key_main = "-sla-%s%s-" % (namebase, id) -%}
{% set sla_key_secondary = "-sla-%s-" % (id) -%}
{% set sla_key_main_length = sla_key_main | length -%}
{% set sla_key_secondary_length = sla_key_secondary | length -%}
{% set sla_dict = {} -%}
{% for key in sla_parameter_dict.keys() -%}
{% if key.startswith(sla_key_main) -%}
{% do sla_dict.__setitem__(key[sla_key_main_length:], sla_parameter_dict.get(key)) -%}
{% elif key.startswith(sla_key_secondary) and not sla_dict.has_key(key[sla_key_secondary_length:]) -%}
{% do sla_dict.__setitem__(key[sla_key_secondary_length:], sla_parameter_dict.get(key)) -%}
{% endif -%}
{% endfor -%}
{% for key, value in sla_dict.iteritems() -%}
sla-{{ key }} = {{ value }}
{% endfor -%}
{% endif %}
[publish-connection-information]
feed-url-{{namebase}}-{{id}}-push = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${request-{{namebase}}-pseudo-replicating-{{id}}:pbs-notification-id}
takeover-{{namebase}}-{{id}}-url = ${request-{{namebase}}-pseudo-replicating-{{id}}:connection-takeover-url}
takeover-{{namebase}}-{{id}}-password = ${request-{{namebase}}-pseudo-replicating-{{id}}:connection-takeover-password}
{% if 'monitor-base-url' in monitor_return -%}
{% do monitor_url_list.append('${request-' ~ namebase ~ '-pseudo-replicating-' ~ id ~ ':connection-monitor-base-url}') -%}
{% endif -%}
{% endfor -%}
[resilient-request-{{namebase}}-public-key-promise]
# Check that public-key-value parameter exists and is not empty
# XXX: maybe we should consider empty values to be non-nexistent.
recipe = collective.recipe.template
# XXX: don't use system executable
input = inline:#!/bin/sh
PUBLIC_KEY_CONTENT="${request-{{namebase}}:connection-ssh-public-key})"
if [[ ! -n "$PUBLIC_KEY_CONTENT" || "$PUBLIC_KEY_CONTENT" == *None* ]]; then
exit 1
fi
output = ${resilient-directory:promise}/resilient-request-{{namebase}}-public-key
mode = 700
{% for id in range(1,nbbackup|int) %}
[resilient-request-{{namebase}}-pseudo-replicating-{{id}}-public-key-promise]
# Check that public-key-value parameter exists and is not empty
# XXX: maybe we should consider empty values to be non-nexistent.
recipe = collective.recipe.template
# XXX: don't use system executable
input = inline:#!/bin/sh
PUBLIC_KEY_CONTENT="${request-{{namebase}}-pseudo-replicating-{{id}}:connection-ssh-public-key})"
if [[ ! -n "$PUBLIC_KEY_CONTENT" || "$PUBLIC_KEY_CONTENT" == *None* ]]; then
exit 1
fi
output = ${resilient-directory:promise}/resilient-request-{{namebase}}-pseudo-replicating-{{id}}-public-key
mode = 700
{% endfor %}
## The PBS and their push / pull slaves
## Adding a PBS provides resiliency
## Adding a backup server provides availability
## Having 3 backups pulling from the same PBS provides
##only availability, not resiliency
[request-pbs-common]
<= slap-connection
recipe = slapos.cookbook:request
software-url = ${slap-connection:software-release-url}
software-type = pull-backup
{% for id in range(1,nbbackup|int) %}
[request-pbs-{{namebase}}-{{id}}]
<= request-pbs-common
name = PBS ({{namebase}} / {{id}})
config-ignore-known-hosts-file = ${slap-parameter:ignore-known-hosts-file}
config-monitor-title = PBS ${slap-connection:computer-id}-{{namebase}}-{{id}}
{% for key, value in monitor_dict.iteritems() -%}
config-{{ key }} = {{ value }}
{% endfor -%}
return = ssh-key notification-url feeds-url {{ monitor_return }}
slave = false
{% if sla_parameter_dict == {} -%}
sla-mode = unique_by_network
{% else %}
{% set sla_key_main = "-sla-%s%s-" % ("pbs", id) -%}
{% set sla_key_secondary = "-sla-%s-" % (id) -%}
{% set sla_key_main_length = sla_key_main | length -%}
{% set sla_key_secondary_length = sla_key_secondary | length -%}
{% set sla_dict = {} -%}
{% for key in sla_parameter_dict.keys() -%}
{% if key.startswith(sla_key_main) -%}
{% do sla_dict.__setitem__(key[sla_key_main_length:], sla_parameter_dict.get(key)) -%}
{% elif key.startswith(sla_key_secondary) and not sla_dict.has_key(key[sla_key_secondary_length:]) -%}
{% do sla_dict.__setitem__(key[sla_key_secondary_length:], sla_parameter_dict.get(key)) -%}
{% endif -%}
{% endfor -%}
{% for key, value in sla_dict.iteritems() -%}
sla-{{ key }} = {{ value }}
{% endfor %}
{% endif %}
[resilient-request-pbs-{{namebase}}-{{id}}-public-key-promise]
# Check that public-key-value parameter exists and is not empty
# XXX: maybe we should consider empty values to be non-nexistent.
recipe = collective.recipe.template
# XXX: don't use system executable
input = inline:#!/bin/sh
PUBLIC_KEY_CONTENT="${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}:connection-ssh-key})"
if [[ ! -n "$PUBLIC_KEY_CONTENT" || "$PUBLIC_KEY_CONTENT" == *None* ]]; then
exit 1
fi
output = ${resilient-directory:promise}/resilient-request-{{namebase}}-pseudo-replicating-{{id}}-public-key
mode = 700
[request-pull-backup-server-{{namebase}}-{{id}}]
<= request-pbs-common
name = PBS {{id}} pulling from ${request-{{namebase}}:name}
config-url = ${request-{{namebase}}:connection-ssh-url}
config-type = pull
config-server-key = ${request-{{namebase}}:connection-ssh-public-key}
config-on-notification = ${request-{{namebase}}:connection-notification-id}
config-notify = ${request-pbs-{{namebase}}-{{id}}:connection-notification-url}
config-notification-id = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-{{id}}-pull
config-name = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-{{id}}
config-title = Pulling from {{namebase}}
config-remove-backup-older-than = {{ slapparameter_dict.get('remove-backup-older-than', '2W') }}
slave = true
sla-instance_guid = ${request-pbs-{{namebase}}-{{id}}:instance_guid}
[publish-connection-information]
feed-url-{{namebase}}-{{id}}-pull = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${request-pull-backup-server-{{namebase}}-{{id}}:config-notification-id}
{% if 'monitor-base-url' in monitor_return -%}
{% do monitor_url_list.append('${request-pbs-' ~ namebase ~ '-' ~ id ~ ':connection-monitor-base-url}') -%}
{% endif -%}
{% if monitor_parameter_dict.get('set-monitor-url', False) -%}
[request-{{namebase}}]
config-monitor-url-list = {{ monitor_url_list | join(' ') }}
{% else -%}
[monitor-conf-parameters]
monitor-url-list = {{ monitor_url_list | join(' ') }}
{% endif -%}
[request-pull-backup-server-{{namebase}}-backup-{{id}}]
<= request-pbs-common
name = PBS pushing on ${request-{{namebase}}-pseudo-replicating-{{id}}:name}
config-url = ${request-{{namebase}}-pseudo-replicating-{{id}}:connection-ssh-url}
config-type = push
config-server-key = ${request-{{namebase}}-pseudo-replicating-{{id}}:connection-ssh-public-key}
config-on-notification = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${request-pull-backup-server-{{namebase}}-{{id}}:config-notification-id}
config-notify = ${request-{{namebase}}-pseudo-replicating-{{id}}:connection-notification-url}
config-notification-id = ${request-{{namebase}}-pseudo-replicating-{{id}}:pbs-notification-id}
config-name = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-{{id}}
config-title = Pushing to {{namebase}} backup {{id}}
slave = true
sla-instance_guid = ${request-pbs-{{namebase}}-{{id}}:instance_guid}
{% endfor %}
[slap-parameter]
# Default parameters for distributed deployment
# I.e state "backup1 of maria should go there, ..."
{% for id in range(1,nbbackup|int) %}
{{namebase}}{{id}}-computer-guid =
pbs-{{namebase}}{{id}}-computer-guid =
{% endfor %}
ignore-known-hosts-file = false
{% endmacro %}
slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/templates/ 0000775 0000000 0000000 00000000000 13012616531 0030547 5 ustar 00root root 0000000 0000000 monitor-check-resilient-feed.in 0000664 0000000 0000000 00000001202 13012616531 0036452 0 ustar 00root root 0000000 0000000 slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/templates #!{{ python_executable }}
import os
import urllib2
import sys
input_feed_directory = '{{ input_feed_directory }}'
monitor_feed_directory = '{{ monitor_feed_directory }}'
base_url = "{{ base_url }}"
feed_file_list = os.listdir(input_feed_directory)
rss_ok = True
for feed_file_name in feed_file_list:
print "Getting %s" % feed_file_name
url = base_url + feed_file_name
try:
feed = urllib2.urlopen(url)
body = feed.read()
open(os.path.join(monitor_feed_directory, feed_file_name + '.rss'), 'w').write(body)
print "FEED is ok"
except urllib2.HTTPError as e:
sys.exit("%s is unvailable: %s" % (feed_file_name, e))
slapos-523ca7b9e10d9dfdef1fb038186b6a3caa1fc01b-stack-resilient/stack/resilient/templates/wrapper.in0000664 0000000 0000000 00000000043 13012616531 0032554 0 ustar 00root root 0000000 0000000 #!${dash-output:dash}
{{ content }}