Commit 75f83ced authored by Marco Mariani's avatar Marco Mariani

bin/takeover script

parent f8124124
master 1.0 alain apache-frontend-httpd2.4 apache-unified-frontend arnau asterisk authorized_keys cedric.ln-erp5-cluster cloudooo-clusterise-split cygwin-0 cygwin-res-resstack davstorage-ajaxplorer dream drop_mt_index_in_BTreeFolder2 drupal erp5 erp5-binary-cache-test erp5-cluster erp5-cluster-monitoring erp5-cluster-scalability erp5-cluster-z2.13 erp5-cluster-z2.13-scalability erp5-cluster-z2.13-scalability-N erp5-component erp5-component-libressl erp5-component-subversion-1.7 erp5-component-subversion-1.8 erp5-component-z2.12 erp5-mysql-insecure-password erp5-postfix erp5-rkm erp5-scikit erp5-shared-test erp5-shared-test2 erp5-z2.12 erp5_kernel_jupyter erp5testnode-scalability erp5testnode-scalability-rebase-erp5testnode erp5testnode_newfirefox etherpad-lite frontend-monitor-slave-configuration func_firefox func_firefox60 gcc-4.7 gitlab-test gitlab-test-8.6.1 glpi grid-computing hexaglobe-watermarking hexaglobe-watermarking-replicated improv-xargs improved-hello-world inkscape-cmake ipnb_st ipython ipython_notebook ivan jupyter_wendelin kvm-iaas lab-resilent lamp-edge-jinja manual-kvm monitor-stack neo-rocks new-frontend nginx_with_auth officejs_testrunner onlyoffice-core openblas-make-ext-options openstack palo palo_etl_wip postfix postgres rafael_agent re6st-registry reafs/heads/ipython request-recipe scalability-testnode-yusei scikit-dev scikit-notmerge shebang_limitation shellinabox_password_promise_delay slapos-master-cluster slapprunner-shellinabox-wip slaprunner systematic systematic-nginx testnode_software_link ttrm wendelin_notebook zerovm zerovm-test zimbra zimbra-kvm 1.0.49 1.0.48 1.0.47 1.0.46.1 1.0.46 1.0.45 1.0.44 1.0.43 1.0.42 1.0.41 1.0.40.1 1.0.40 1.0.39 1.0.38.1 1.0.38 1.0.37 1.0.36 1.0.35 1.0.34 1.0.33 1.0.32 1.0.31 1.0.30 1.0.29 1.0.28 1.0.27 1.0.26 1.0.25 1.0.24 1.0.23 1.0.22 1.0.21 1.0.20 1.0.19 1.0.18 1.0.17 1.0.16 1.0.15 1.0.14 1.0.13 1.0.12 1.0.11 1.0.10 1.0.9 1.0.8.1 1.0.8 1.0.7 1.0.6 1.0.5 1.0.4 1.0.3 1.0.2 1.0.1 vifib-20150331 vifib-20150203 vifib-20150130 vifib-20150123 vifib-20150116 vifib-20141222 slaprunner-resilient-v1 slaprunner-resilient-v1-RC1 slapos.core-1.0.0rc6-1 slapos.cookbook-0.103 slapos.cookbook-0.102 slapos.cookbook-0.101 slapos.cookbook-0.100 slapos.cookbook-0.99 slapos.cookbook-0.98 slapos.cookbook-0.97 slapos.cookbook-0.96 slapos.cookbook-0.95 slapos.cookbook-0.94 slapos.cookbook-0.93 slapos.cookbook-0.92 slapos.cookbook-0.91 slapos.cookbook-0.90 slapos.cookbook-0.89 slapos.cookbook-0.88 slapos.cookbook-0.87 slapos.cookbook-0.86 slapos.cookbook-0.85 slapos.cookbook-0.84.2 slapos.cookbook-0.84.1 slapos.cookbook-0.84 slapos.cookbook-0.83.1 slapos.cookbook-0.83 slapos.cookbook-0.82 slapos.cookbook-0.81 slapos.cookbook-0.80 slapos.cookbook-0.79 slapos.cookbook-0.78.5 slapos.cookbook-0.78.4 slapos.cookbook-0.78.3 slapos.cookbook-0.78.2 slapos.cookbook-0.78.1 slapos.cookbook-0.78.0 slapos.cookbook-0.77.1 slapos.cookbook-0.77.0 slapos.cookbook-0.76.0 slapos.cookbook-0.75.0 slapos.cookbook-0.74.0 slapos.cookbook-0.73.1 slapos.cookbook-0.72.0 slapos.cookbook-0.71.4 slapos.cookbook-0.71.3 slapos.cookbook-0.71.2 slapos.cookbook-0.71.1 slapos.cookbook-0.71 slapos-legacy-flask-stack slapos-159 slapos-0.271 slapos-0.270 slapos-0.269 slapos-0.268 slapos-0.267 slapos-0.266 slapos-0.265 slapos-0.264 slapos-0.263 slapos-0.262 slapos-0.261 slapos-0.260 slapos-0.259.2 slapos-0.259.1 slapos-0.259 slapos-0.258 slapos-0.257 slapos-0.256 slapos-0.255 slapos-0.254 slapos-0.253.3 slapos-0.253.2 slapos-0.253.1 slapos-0.253 slapos-0.252.1 slapos-0.252 slapos-0.251 slapos-0.250.2 slapos-0.250.1 slapos-0.250 slapos-0.249.1 slapos-0.249 slapos-0.248.1 slapos-0.248 slapos-0.247 slapos-0.246 slapos-0.245 slapos-0.244 slapos-0.243 slapos-0.242 slapos-0.241 slapos-0.240.0 slapos-0.240 slapos-0.239 slapos-0.238 slapos-0.237 slapos-0.236 slapos-0.235 slapos-0.234 slapos-0.233 slapos-0.232 slapos-0.231 slapos-0.230 slapos-0.229 slapos-0.228 slapos-0.227 slapos-0.226 slapos-0.225 slapos-0.224 slapos-0.223.1 slapos-0.223 slapos-0.222 slapos-0.221 slapos-0.220 slapos-0.219 slapos-0.218.2 slapos-0.218.1 slapos-0.218 slapos-0.217 slapos-0.217-B1 slapos-0.216 slapos-0.215 slapos-0.214 slapos-0.213 slapos-0.212 slapos-0.211 slapos-0.210 slapos-0.209 slapos-0.208 slapos-0.207 slapos-0.206 slapos-0.205 slapos-0.204 slapos-0.203 slapos-0.202 slapos-0.201 slapos-0.200 slapos-0.199 slapos-0.199-hotfix-20140409 slapos-0.198 slapos-0.197 slapos-0.196 slapos-0.195 slapos-0.194-RC3 slapos-0.194-RC2 slapos-0.194-RC1 slapos-0.193 slapos-0.192 slapos-0.191 slapos-0.190 slapos-0.189 slapos-0.188 slapos-0.187 slapos-0.186 slapos-0.185 slapos-0.184 slapos-0.183 slapos-0.182 slapos-0.181 slapos-0.180 slapos-0.179 slapos-0.178 slapos-0.177.5 slapos-0.177.4 slapos-0.177.3 slapos-0.177.2 slapos-0.177.1 slapos-0.177 slapos-0.176.4 slapos-0.176.3 slapos-0.176.2 slapos-0.176.1 slapos-0.176 slapos-0.175 slapos-0.174 slapos-0.173 slapos-0.172 slapos-0.171 slapos-0.170 slapos-0.169 slapos-0.168.9 slapos-0.168.8 slapos-0.168.7 slapos-0.168.6 slapos-0.168.5 slapos-0.168.4 slapos-0.168.3 slapos-0.168.2 slapos-0.168.1 slapos-0.168 slapos-0.165 slapos-0.164 slapos-0.163 slapos-0.162 slapos-0.161 slapos-0.160 slapos-0.159 slapos-0.158 slapos-0.157 slapos-0.156 slapos-0.155 slapos-0.154 slapos-0.153 slapos-0.152 slapos-0.151 slapos-0.150 slapos-0.149 slapos-0.148 slapos-0.147 slapos-0.146 slapos-0.145 slapos-0.144 slapos-0.143 slapos-0.142 slapos-0.141 mr80_review_2 frontend_parameter_doc_review1 erp5-cluster-0.1.1 cygwin-resstack-3 cygwin-resstack-2 cygwin-resstack-1 cygwin-cachetest-4 cygwin-cachetest-3 cygwin-cachetest-2 cygwin-cachetest-1 apache_frontend_prod_test
No related merge requests found
...@@ -31,7 +31,10 @@ import os ...@@ -31,7 +31,10 @@ import os
class Recipe(GenericSlapRecipe): class Recipe(GenericSlapRecipe):
""" This class provides the installation of the resilience """ This class provides the installation of the resilience
script on the partition. scripts on the partition.
bin/takeover will perform a rename (must be run manually).
bin/bully will monitor, run elections and perform renames when needed.
""" """
def _install(self): def _install(self):
...@@ -55,12 +58,12 @@ class Recipe(GenericSlapRecipe): ...@@ -55,12 +58,12 @@ class Recipe(GenericSlapRecipe):
slap_connection = self.buildout['slap-connection'] slap_connection = self.buildout['slap-connection']
if self.optionIsTrue('enable-bully-service', default=False): if self.optionIsTrue('enable-bully-service', default=False):
wrapper_dir = self.options['services'] bully_dir = self.options['services']
else: else:
wrapper_dir = self.options['bin'] bully_dir = self.options['bin']
wrapper = self.createPythonScript( bully_wrapper = self.createPythonScript(
name=os.path.join(wrapper_dir, self.parameter_dict['wrapper']), name=os.path.join(bully_dir, self.options['wrapper-bully']),
absolute_function='slapos.recipe.addresiliency.bully.run', absolute_function='slapos.recipe.addresiliency.bully.run',
arguments={ arguments={
'confpath': confpath, 'confpath': confpath,
...@@ -73,7 +76,22 @@ class Recipe(GenericSlapRecipe): ...@@ -73,7 +76,22 @@ class Recipe(GenericSlapRecipe):
'namebase': self.parameter_dict['namebase'], 'namebase': self.parameter_dict['namebase'],
}) })
path_list.append(wrapper) path_list.append(bully_wrapper)
takeover_wrapper = self.createPythonScript(
name=os.path.join(self.options['bin'], self.options['wrapper-takeover']),
absolute_function='slapos.recipe.addresiliency.takeover.run',
arguments={
'server_url': slap_connection['server-url'],
'key_file': slap_connection.get('key-file'),
'cert_file': slap_connection.get('cert-file'),
'computer_id': slap_connection['computer-id'],
'partition_id': slap_connection['partition-id'],
'software': slap_connection['software-release-url'],
'namebase': self.parameter_dict['namebase'],
})
path_list.append(takeover_wrapper)
return path_list return path_list
......
...@@ -6,7 +6,7 @@ import socket ...@@ -6,7 +6,7 @@ import socket
import thread import thread
import time import time
from slapos import slap as slapmodule import slapos.recipe.addresiliency.renamer
import slapos import slapos
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
...@@ -29,86 +29,6 @@ STATE_REORGANIZATION = 'reorganization' ...@@ -29,86 +29,6 @@ STATE_REORGANIZATION = 'reorganization'
class Renamer(object):
def __init__(self, server_url, key_file, cert_file, computer_guid,
partition_id, software_release, namebase):
self.server_url = server_url
self.key_file = key_file
self.cert_file = cert_file
self.computer_guid = computer_guid
self.partition_id = partition_id
self.software_release = software_release
self.namebase = namebase
def _failover(self):
"""\
This method does
- retrieve the broken computer partition
- change its reference to 'broken-...' and its software type to 'frozen'
- retrieve the winner computer partition (attached to this process)
- change its reference and software type to replace the broken one
Then, after running slapgrid-cp a few times, the winner takes over and
a new cp is created to replace it as an importer.
"""
# TODO: replace hardcoded strings with values from the API
slap = slapmodule.slap()
slap.initializeConnection(self.server_url, self.key_file, self.cert_file)
# partition that will take over.
cp_winner = slap.registerComputerPartition(computer_guid=self.computer_guid,
partition_id=self.partition_id)
# XXX although we can already rename cp_winner, to change its software type we need to
# get hold of the root cp as well
root_partition_id = 'slappart0' # XXX hardcoded. what's the API for this?
cp_root = slap.registerComputerPartition(computer_guid=self.computer_guid,
partition_id=root_partition_id)
cp_exporter_ref = self.namebase + '0' # this is ok. the boss is always number zero.
# partition to be deactivated
cp_broken = cp_root.request(software_release=self.software_release,
software_type='frozen',
state='stopped',
partition_reference=cp_exporter_ref)
broken_new_ref = 'broken-{}'.format(time.strftime("%d-%b_%H:%M:%S", time.gmtime()))
# XXX can we retrieve and log the old reference name?
log.debug("Renaming {}: {}".format(cp_broken.getId(), broken_new_ref))
cp_broken.rename(new_name=broken_new_ref)
cp_broken.stopped()
log.debug("Renaming {}: {}".format(cp_broken.getId(), cp_exporter_ref))
# update software type and name for the partition that will take over
cp_winner_ref = self.namebase + '2' # XXX hardcoded. what's the API for this?
cp_root.request(software_release=self.software_release,
software_type=self.namebase+'-export',
partition_reference=cp_winner_ref).rename(new_name=cp_exporter_ref)
def failover(self):
try:
self._failover()
log.info('Renaming done')
except slapos.slap.slap.ServerError:
log.info('Internal server error')
## Leader is always number 0 ## Leader is always number 0
class ResilientInstance(object): class ResilientInstance(object):
...@@ -291,13 +211,13 @@ class Wrapper(object): ...@@ -291,13 +211,13 @@ class Wrapper(object):
def run(args): def run(args):
confpath = args.pop('confpath') confpath = args.pop('confpath')
renamer = Renamer(server_url = args.pop('server_url'), renamer = slapos.recipe.addresiliency.renamer.Renamer(server_url = args.pop('server_url'),
key_file = args.pop('key_file'), key_file = args.pop('key_file'),
cert_file = args.pop('cert_file'), cert_file = args.pop('cert_file'),
computer_guid = args.pop('computer_id'), computer_guid = args.pop('computer_id'),
partition_id = args.pop('partition_id'), partition_id = args.pop('partition_id'),
software_release = args.pop('software'), software_release = args.pop('software'),
namebase = args.pop('namebase')) namebase = args.pop('namebase'))
if args: if args:
raise ValueError('Unknown arguments: %s' % ', '.join(args)) raise ValueError('Unknown arguments: %s' % ', '.join(args))
......
# -*- coding: utf-8 -*-
import logging
import time
from slapos import slap as slapmodule
import slapos
log = logging.getLogger(__name__)
logging.basicConfig(level=logging.DEBUG)
class Renamer(object):
def __init__(self, server_url, key_file, cert_file, computer_guid,
partition_id, software_release, namebase):
self.server_url = server_url
self.key_file = key_file
self.cert_file = cert_file
self.computer_guid = computer_guid
self.partition_id = partition_id
self.software_release = software_release
self.namebase = namebase
def _failover(self):
"""\
This method does
- retrieve the broken computer partition
- change its reference to 'broken-...' and its software type to 'frozen'
- retrieve the winner computer partition (attached to this process)
- change its reference and software type to replace the broken one
Then, after running slapgrid-cp a few times, the winner takes over and
a new cp is created to replace it as an importer.
"""
# TODO: replace hardcoded strings with values from the API
slap = slapmodule.slap()
slap.initializeConnection(self.server_url, self.key_file, self.cert_file)
# partition that will take over.
cp_winner = slap.registerComputerPartition(computer_guid=self.computer_guid,
partition_id=self.partition_id)
# XXX although we can already rename cp_winner, to change its software type we need to
# get hold of the root cp as well
root_partition_id = 'slappart0' # XXX hardcoded. what's the API for this?
cp_root = slap.registerComputerPartition(computer_guid=self.computer_guid,
partition_id=root_partition_id)
cp_exporter_ref = self.namebase + '0' # this is ok. the boss is always number zero.
# partition to be deactivated
cp_broken = cp_root.request(software_release=self.software_release,
software_type='frozen',
state='stopped',
partition_reference=cp_exporter_ref)
broken_new_ref = 'broken-{}'.format(time.strftime("%d-%b_%H:%M:%S", time.gmtime()))
# XXX can we retrieve and log the old reference name?
log.debug("Renaming {}: {}".format(cp_broken.getId(), broken_new_ref))
cp_broken.rename(new_name=broken_new_ref)
cp_broken.stopped()
log.debug("Renaming {}: {}".format(cp_broken.getId(), cp_exporter_ref))
# update software type and name for the partition that will take over
cp_winner_ref = self.namebase + '2' # XXX hardcoded. what's the API for this?
cp_root.request(software_release=self.software_release,
software_type=self.namebase+'-export',
partition_reference=cp_winner_ref).rename(new_name=cp_exporter_ref)
def failover(self):
try:
self._failover()
log.info('Renaming done')
except slapos.slap.slap.ServerError:
log.info('Internal server error')
# -*- coding: utf-8 -*-
import slapos.recipe.addresiliency.renamer
def run(args):
renamer = slapos.recipe.addresiliency.renamer.Renamer(server_url = args.pop('server_url'),
key_file = args.pop('key_file'),
cert_file = args.pop('cert_file'),
computer_guid = args.pop('computer_id'),
partition_id = args.pop('partition_id'),
software_release = args.pop('software'),
namebase = args.pop('namebase'))
renamer.failover()
...@@ -22,7 +22,7 @@ parts = ...@@ -22,7 +22,7 @@ parts =
recipe = slapos.recipe.template recipe = slapos.recipe.template
url = ${:_profile_base_location_}/pbsready.cfg.in url = ${:_profile_base_location_}/pbsready.cfg.in
output = ${buildout:directory}/pbsready.cfg output = ${buildout:directory}/pbsready.cfg
md5sum = b6102416d000cae81dd2b06268946ea9 md5sum = af02afc439530a6cef9f5a21a25f3363
mode = 0644 mode = 0644
[pbsready-import] [pbsready-import]
...@@ -53,7 +53,7 @@ mode = 0644 ...@@ -53,7 +53,7 @@ mode = 0644
[template-replicated] [template-replicated]
recipe = slapos.recipe.download recipe = slapos.recipe.download
url = ${:_profile_base_location_}/template-replicated.cfg.in url = ${:_profile_base_location_}/template-replicated.cfg.in
md5sum = 1017d919dbf41904f04f5c17dcb574fa md5sum = 63b5649f3cf1c9a77315382793d9593f
mode = 0644 mode = 0644
destination = ${buildout:directory}/template-replicated.cfg.in destination = ${buildout:directory}/template-replicated.cfg.in
......
...@@ -45,6 +45,8 @@ notifier-callbacks = $${basedirectory:notifier}/callbacks ...@@ -45,6 +45,8 @@ notifier-callbacks = $${basedirectory:notifier}/callbacks
# If false, they can be run with bin/bullly for all the PBSReady instances. # If false, they can be run with bin/bullly for all the PBSReady instances.
enable-bully-service = False enable-bully-service = False
recipe = slapos.cookbook:addresiliency recipe = slapos.cookbook:addresiliency
wrapper-bully = bully
wrapper-takeover = takeover
services = $${basedirectory:services} services = $${basedirectory:services}
bin = $${rootdirectory:bin} bin = $${rootdirectory:bin}
etc = $${rootdirectory:etc} etc = $${rootdirectory:etc}
......
...@@ -2,8 +2,6 @@ ...@@ -2,8 +2,6 @@
## Tells the Backupable recipe that we want a backup ## Tells the Backupable recipe that we want a backup
[resilient] [resilient]
config-script = bully.py
config-wrapper = bully
config-namebase = {{namebase}} config-namebase = {{namebase}}
## Every request is double to provide the 3 IPs. ## Every request is double to provide the 3 IPs.
...@@ -15,7 +13,7 @@ software-type = {{typeexport}} ...@@ -15,7 +13,7 @@ software-type = {{typeexport}}
name = {{namebase}}0 name = {{namebase}}0
return = url ssh-public-key ssh-url notification-id ip return = url ssh-public-key ssh-url notification-id ip
config = number script wrapper authorized-key notify ip-list namebase config = number authorized-key notify ip-list namebase
config-number = 0 config-number = 0
config-authorized-key = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}{% endfor %} config-authorized-key = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}{% endfor %}
config-notify = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-notification-url}{% endfor %} config-notify = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-notification-url}{% endfor %}
...@@ -36,7 +34,7 @@ return = url ssh-public-key ssh-url notification-url ip ...@@ -36,7 +34,7 @@ return = url ssh-public-key ssh-url notification-url ip
pbs-notification-id = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-push pbs-notification-id = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-push
config = number script wrapper authorized-key on-notification ip-list namebase config = number authorized-key on-notification ip-list namebase
config-number = {{id}} config-number = {{id}}
config-authorized-key = ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key} config-authorized-key = ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}
config-on-notification = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${:pbs-notification-id} config-on-notification = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${:pbs-notification-id}
...@@ -60,7 +58,7 @@ software-url = ${slap-connection:software-release-url} ...@@ -60,7 +58,7 @@ software-url = ${slap-connection:software-release-url}
software-type = {{typeexport}} software-type = {{typeexport}}
return = url ssh-public-key ssh-url notification-id ip return = url ssh-public-key ssh-url notification-id ip
config = number script wrapper authorized-key notify ip-list namebase config = number authorized-key notify ip-list namebase
config-number = 0 config-number = 0
config-authorized-key = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}{% endfor %} config-authorized-key = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}{% endfor %}
config-notify = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-notification-url}{% endfor %} config-notify = {% for id in range(1,nbbackup|int) %} ${request-pbs-{{namebase}}-{{id}}:connection-notification-url}{% endfor %}
...@@ -81,7 +79,7 @@ return = url ssh-public-key ssh-url notification-url ...@@ -81,7 +79,7 @@ return = url ssh-public-key ssh-url notification-url
pbs-notification-id = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-push pbs-notification-id = ${slap-connection:computer-id}-${slap-connection:partition-id}-{{namebase}}-push
config = number script wrapper authorized-key on-notification ip-list namebase config = number authorized-key on-notification ip-list namebase
config-number = {{id}} config-number = {{id}}
config-authorized-key = ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key} config-authorized-key = ${request-pbs-{{namebase}}-{{id}}:connection-ssh-key}
config-on-notification = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${:pbs-notification-id} config-on-notification = ${request-pbs-{{namebase}}-{{id}}:connection-feeds-url}${:pbs-notification-id}
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment