pax_global_header 0000666 0000000 0000000 00000000064 11602033661 0014510 g ustar 00root root 0000000 0000000 52 comment=182cdf43b3e66a610f38c6e26a87b7b69e058cc5
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/ 0000775 0000000 0000000 00000000000 11602033661 0022253 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/ 0000775 0000000 0000000 00000000000 11602033661 0023554 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/README.console.txt 0000664 0000000 0000000 00000000020 11602033661 0026703 0 ustar 00root root 0000000 0000000 console
=======
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/README.format.txt 0000664 0000000 0000000 00000001052 11602033661 0026537 0 ustar 00root root 0000000 0000000 format
======
slapformat is an application to prepare SlapOS ready node (machine).
It "formats" the machine by:
- creating users and groups
- creating bridge interface
- creating needed tap interfaces
- creating needed directories with proper ownership and permissions
In the end special report is generated and information are posted to
configured SlapOS server.
This program shall be only run by root.
Requirements
------------
Linux with IPv6, bridging and tap interface support.
Binaries:
* brctl
* groupadd
* ip
* tunctl
* useradd
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/README.grid.txt 0000664 0000000 0000000 00000005270 11602033661 0026202 0 ustar 00root root 0000000 0000000 grid
====
slapgrid is a client of SLAPos. SLAPos provides support for deploying a SaaS
system in a minute.
Slapgrid allows you to easily deploy instances of softwares based on buildout
profiles.
For more informations about SLAP and SLAPos, please see the SLAP documentation.
Requirements
------------
A working SLAP server with informations about your computer, in order to
retrieve them.
As Vifib servers use IPv6 only, we strongly recommend an IPv6 enabled UNIX
box.
For the same reasons, Python >= 2.6 with development headers is also strongly
recommended (IPv6 support is not complete in previous releases).
For now, gcc and glibc development headers are required to build most software
releases.
Concepts
--------
Here are the fundamental concepts of slapgrid :
A Software Release (SR) is just a software.
A Computer Partition (CP) is an instance of a Software Release.
Imagine you want to install with slapgrid some software and run it. You will
have to install the software as a Software Release, and then instantiate it,
i.e configuring it for your needs, as a Computer Partition.
How it works
------------
When run, slapgrid will authenticate to the SLAP library with a computer_id and
fetch the list of Software Releases to install or remove and Computer
Partitions to start or stop.
Then, it will process each Software Release, and each Computer Partition.
It will also periodically send to SLAP the usage report of each Computer
Partition.
Installation
------------
With easy_install::
$ easy_install slapgrid
slapgrid needs several directories to be created and configured before being
able to run : a software releases directory, and an instances directory with
configured computer partition directory(ies).
You should create for each Computer Partition directory created a specific user
and associate it with its Computer Partition directory. Each Computer Partition
directory should belongs to this specific user, with permissions of 0750.
Usage
-----
slapgrid needs several informations in order to run. You can specify them by
adding arguments to the slapgrid command line, or by putting then in a
configuration file.
Beware : you need a valid computer resource on server side.
Examples
--------
simple example :
Just run slapgrid:
$ slapgrid --instance-root /path/to/instance/root --software-root
/path/to/software_root --master-url https://some.server/some.resource
--computer-id my.computer.id
configuration file example::
[slapgrid]
instance_root = /path/to/instance/root
software_root = /path/to/software/root
master_url = https://slapos.server/slap_service
computer_id = my.computer.id
then run slapgrid::
$ slapgrid --configuration-file = path/to/configuration/file
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/README.proxy.txt 0000664 0000000 0000000 00000000014 11602033661 0026425 0 ustar 00root root 0000000 0000000 proxy
=====
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/README.slap.txt 0000664 0000000 0000000 00000002714 11602033661 0026214 0 ustar 00root root 0000000 0000000 slap
====
Simple Language for Accounting and Provisioning python library.
Developer note - python version
-------------------------------
This library is used on client (slapgrid) and server side.
Server is using python2.4 and client is using python2.6
Having this in mind, code of this library *have* to work
on python2.4
How it works
------------
The SLAP main server which is in charge of service coordination receives from participating servers the number of computer paritions which are available, the type of resource which a party is ready provide, and request from parties for resources which are needed.
Each participating server is identified by a unique ID and runs a slap-server daemon. This dameon collects from the main server the installation tasks and does the installation of resources, then notifies the main server of completion whenever a resource is configured, installed and available.
The data structure on the main server is the following:
A - Action: an action which can happen to provide a resource or account its usage
CP - Computer Partition: provides a URL to Access a Cloud Resource
RI - Resource Item: describes a resource
CI - Contract Item: describes the contract to attach the DL to (This is unclear still)
R - Resource: describes a type of cloud resource (ex. MySQL Table) is published on slapgrid.org
DL - Delivery Line: Describes an action happening on a resource item on a computer partition
D - Delivery: groups multiple Delivery Lines
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/__init__.py 0000664 0000000 0000000 00000000364 11602033661 0025670 0 ustar 00root root 0000000 0000000 # See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/console.py 0000664 0000000 0000000 00000002550 11602033661 0025572 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
def run():
__import__("code").interact(banner="", local=globals())
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/format/ 0000775 0000000 0000000 00000000000 11602033661 0025044 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/format/__init__.py 0000664 0000000 0000000 00000105520 11602033661 0027160 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2010, 2011 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from optparse import OptionParser, Option
from xml_marshaller import xml_marshaller
from pwd import getpwnam
import ConfigParser
import grp
import logging
import netaddr
import netifaces
import os
import random
import slapos.slap as slap
import socket
import subprocess
import sys
import time
class OS(object):
_os = os
def __init__(self, config):
self._dry_run = config.dry_run
self._verbose = config.verbose
self._logger = config.logger
add = self._addWrapper
add('chown')
add('chmod')
add('makedirs')
add('mkdir')
def _addWrapper(self, name):
def wrapper(*args, **kw):
if self._verbose:
arg_list = [repr(x) for x in args] + [
'%s=%r' % (x, y) for x, y in kw.iteritems()]
self._logger.debug('%s(%s)' % (
name,
', '.join(arg_list)
))
if not self._dry_run:
getattr(self._os, name)(*args, **kw)
setattr(self, name, wrapper)
def __getattr__(self, name):
return getattr(self._os, name)
class SlapError(Exception):
"""
Slap error
"""
def __init__(self, message):
self.msg = message
class UsageError(SlapError):
pass
class ExecError(SlapError):
pass
def callAndRead(argument_list, raise_on_error=True):
popen = subprocess.Popen(argument_list, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
result = popen.communicate()[0]
if raise_on_error and popen.returncode != 0:
raise ValueError('Issue during invoking %r, result was:\n%s' % (argument_list, result))
return popen.returncode, result
def isGlobalScopeAddress(a):
"""Returns True if a is global scope IP v4/6 address"""
ip = netaddr.IPAddress(a)
return not ip.is_link_local() and not ip.is_loopback() and \
not ip.is_reserved() and ip.is_unicast()
def netmaskToPrefixIPv4(netmask):
"""Convert string represented netmask to its integer prefix"""
return netaddr.strategy.ipv4.netmask_to_prefix[
netaddr.strategy.ipv4.str_to_int(netmask)]
def netmaskToPrefixIPv6(netmask):
"""Convert string represented netmask to its integer prefix"""
return netaddr.strategy.ipv6.netmask_to_prefix[
netaddr.strategy.ipv6.str_to_int(netmask)]
def _getDict(instance):
"""
Serialize an object instance into dictionaries. List and dict will remains
the same, basic type too. But encapsulated object will be returned as dict.
Set, collections and other aren't handle for now.
Args:
instance: an object of any type.
Returns:
A dictionary if the given object wasn't a list, a list otherwise.
"""
if isinstance(instance, list):
return [_getDict(item) for item in instance]
elif isinstance(instance, dict):
result = {}
for key in instance.keys():
result[key] = _getDict(instance[key])
return result
else:
try:
result = {}
for key in instance.__dict__.keys():
result[key] = _getDict(instance.__dict__[key])
return result
except AttributeError:
return instance
class Error(Exception):
"Base class for exceptions in this module."
def __str__(self):
return self.message
class NoAddressOnBridge(Error):
"""
Exception raised if there's not address on the bridge to construct IPv6
address with.
Attributes:
brige: String, the name of the bridge.
"""
def __init__(self, bridge):
self.message = 'No IPv6 found on bridge %s to construct IPv6 with.' % bridge
class AddressGenerationError(Error):
"""
Exception raised if the generation of an IPv6 based on the prefix obtained
from the bridge failed.
Attributes:
addr: String, the invalid address the exception is raised for.
"""
def __init__(self, addr):
self.message = 'Generated IPv6 %s seems not to be a valid IP.' % addr
class Computer:
"Object representing the computer"
def __init__(self, reference, bridge=None, addr = None, netmask = None):
"""
Attributes:
reference: String, the reference of the computer.
bridge: String, if it has one, the name of the computer's bridge.
"""
self.reference = str(reference)
self.bridge = bridge
self.partition_list = []
self.address = addr
self.netmask = netmask
def __getinitargs__(self):
return (self.reference, self.bridge)
def getAddress(self):
"""
Return a list of the bridge address not attributed to any partition, (which
are therefore free for the computer itself).
Returns:
False if the bridge isn't available, else the list of the free addresses.
"""
if self.bridge is None:
return dict(addr=self.address, netmask=self.netmask)
computer_partition_address_list = []
for partition in self.partition_list:
for address in partition.address_list:
if netaddr.valid_ipv6(address['addr']):
computer_partition_address_list.append(address['addr'])
# Going through addresses of the computer's bridge interface
for address_dict in self.bridge.getGlobalScopeAddressList():
# Comparing with computer's partition addresses
if address_dict['addr'] not in computer_partition_address_list:
return address_dict
return None
def send(self, config):
"""
Send a marshalled dictionary of the computer object serialized via_getDict.
"""
slap_instance = slap.slap()
connection_dict = {}
if config.key_file and config.cert_file:
connection_dict.update(
key_file=config.key_file,
cert_file=config.cert_file)
slap_instance.initializeConnection(config.master_url,
**connection_dict)
slap_computer = slap_instance.registerComputer(self.reference)
if config.dry_run:
return
return slap_computer.updateConfiguration(
xml_marshaller.dumps(_getDict(self)))
def dump(self, path_to_xml):
"""
Dump the computer object to an xml file via xml_marshaller.
Args:
path_to_xml: String, path to the file to load.
users: List of User, list of user needed to be add to the dump
(even if they are not related to any tap interface).
"""
computer_dict = _getDict(self)
output_file = open(path_to_xml,'w')
output_file.write(xml_marshaller.dumps(computer_dict))
output_file.close()
@classmethod
def load(cls, path_to_xml, reference):
"""
Create a computer object from a valid xml file.
Arg:
path_to_xml: String, a path to a valid file containing
a valid configuration.
Return:
A Computer object if the path where pointing on a valid
file, False otherwise.
"""
dumped_dict = xml_marshaller.loads(open(path_to_xml).read())
# Reconstructing the computer object from the xml
computer = Computer(
reference = reference,
addr = dumped_dict['address'],
netmask = dumped_dict['netmask'],
)
for partition_dict in dumped_dict['partition_list']:
if partition_dict['user']:
user = User(partition_dict['user']['name'])
else:
user = User('root')
if partition_dict['tap']:
tap = Tap(partition_dict['tap']['name'])
else:
tap = Tap(partition_dict['reference'])
address_list = partition_dict['address_list']
partition = Partition(
reference = partition_dict['reference'],
path = partition_dict['path'],
user = user,
address_list = address_list,
tap = tap,
)
computer.partition_list.append(partition)
return computer
def construct(self, alter_user=True, alter_network=True):
"""
Construct the computer object as it is.
"""
if alter_network and self.address is not None:
self.bridge.addAddr(self.address, self.netmask)
for path in self.instance_root, self.software_root:
if not os.path.exists(path):
os.makedirs(path, 0755)
else:
os.chmod(path, 0755)
# own self.software_root by slapsoft
slapsoft = User('slapsoft')
slapsoft.path = self.software_root
if alter_user:
slapsoft.create()
slapsoft_pw = getpwnam(slapsoft.name)
os.chown(self.software_root, slapsoft_pw.pw_uid, slapsoft_pw.pw_gid)
os.chmod(self.software_root, 0755)
for partition in self.partition_list:
# Reconstructing User's
partition.path = os.path.join(self.instance_root, partition.reference)
partition.user.setPath(partition.path)
partition.user.additional_group_list = [slapsoft.name]
if alter_user:
partition.user.create()
# Reconstructing Tap
if partition.user and partition.user.isAvailable():
owner = partition.user
else:
owner = User('root')
if alter_network:
partition.tap.createWithOwner(owner)
self.bridge.addTap(partition.tap)
# Reconstructing partition's directory
partition.createPath(alter_user)
# Reconstructing partition's address
# There should be two addresses on each Computer Partition:
# * global IPv6
# * local IPv4, took from slapformat:ipv4_local_network
if len(partition.address_list) == 0:
# regenerate
partition.address_list.append(self.bridge.addIPv4LocalAddress())
partition.address_list.append(self.bridge.addAddr())
elif alter_network:
# regenerate list of addresses
old_partition_address_list = partition.address_list
partition.address_list = []
if len(old_partition_address_list) != 2:
raise ValueError('There should be exactly 2 stored addresses')
if not any([netaddr.valid_ipv6(q['addr']) for q in old_partition_address_list]):
raise ValueError('Not valid ipv6 addresses loaded')
if not any([netaddr.valid_ipv4(q['addr']) for q in old_partition_address_list]):
raise ValueError('Not valid ipv6 addresses loaded')
for address in old_partition_address_list:
if netaddr.valid_ipv6(address['addr']):
partition.address_list.append(self.bridge.addAddr(address['addr'],
address['netmask']))
elif netaddr.valid_ipv4(address['addr']):
partition.address_list.append(self.bridge.addIPv4LocalAddress(address['addr']))
else:
raise ValueError('Address %r is incorrect' % address['addr'])
class Partition:
"Represent a computer partition"
def __init__(self, reference, path, user, address_list, tap):
"""
Attributes:
reference: String, the name of the partition.
path: String, the path to the partition folder.
user: User, the user linked to this partition.
address_list: List of associated IP addresses.
tap: Tap, the tap interface linked to this partition.
"""
self.reference = str(reference)
self.path = str(path)
self.user = user
self.address_list = address_list or []
self.tap = tap
def __getinitargs__(self):
return (self.reference, self.path, self.user, self.address_list, self.tap)
def createPath(self, alter_user=True):
"""
Create the directory of the partition, assign to the partition user and give
it the 750 permission. In case if path exists just modifies it.
"""
self.path = os.path.abspath(self.path)
owner = self.user if self.user else User('root')
if not os.path.exists(self.path):
os.mkdir(self.path, 0750)
if alter_user:
owner_pw = getpwnam(owner.name)
os.chown(self.path, owner_pw.pw_uid, owner_pw.pw_gid)
os.chmod(self.path, 0750)
class User:
"User: represent and manipulate a user on the system."
def __init__(self, user_name, additional_group_list=None):
"""
Attributes:
user_name: string, the name of the user, who will have is home in
"""
self.name = str(user_name)
self.additional_group_list = additional_group_list
def __getinitargs__(self):
return (self.name,)
def setPath(self, path):
self.path = path
def create(self):
"""
Create a user on the system who will be named after the self.name with its
own group and directory.
Returns:
True: if the user creation went right
"""
# XXX: This method shall be no-op in case if all is correctly setup
# This method shall check if all is correctly done
# This method shall not reset groups, just add them
try:
grp.getgrnam(self.name)
except KeyError:
callAndRead(['groupadd', self.name])
user_parameter_list = ['-d', self.path, '-g', self.name, '-s', '/bin/false']
if self.additional_group_list is not None:
user_parameter_list.extend(['-G', ','.join(self.additional_group_list)])
user_parameter_list.append(self.name)
try:
getpwnam(self.name)
except KeyError:
callAndRead(['useradd'] + user_parameter_list)
else:
callAndRead(['usermod'] + user_parameter_list)
return True
def isAvailable(self):
"""
Determine the availability of a user on the system
Return:
True: if available
False: otherwise
"""
try:
getpwnam(self.name)
return True
except KeyError:
return False
class Tap:
"Tap represent a tap interface on the system"
def __init__(self, tap_name):
"""
Attributes:
tap_name: String, the name of the tap interface.
user: User, the owner of the tap interface.
"""
self.name = str(tap_name)
def __getinitargs__(self):
return (self.name,)
def createWithOwner(self,owner):
"""
Create a tap interface on the system.
Return:
True: Everything went right.
"""
# some systems does not have -p switch for tunctl
#callAndRead(['tunctl', '-p', '-t', self.name, '-u', owner.name])
check_file = '/sys/devices/virtual/net/%s/owner' % self.name
owner_id = None
if os.path.exists(check_file):
try:
owner_id = int(open(check_file).read().strip())
except Exception:
pass
if (owner_id is None) or (owner_id != getpwnam(owner.name).pw_uid):
callAndRead(['tunctl', '-t', self.name, '-u', owner.name])
callAndRead(['ip', 'link', 'set', self.name, 'up'])
return True
class Bridge:
"Bridge represent a bridge on the system"
def __init__(self, name, ipv4_local_network):
"""
Attributes:
name: String, the name of the bridge
"""
self.name = str(name)
self.ipv4_local_network = ipv4_local_network
def __getinitargs__(self):
return (self.name,)
def getIPv4LocalAddressList(self):
"""Returns currently configured local IPv4 addresses which are in ipv4_local_network"""
if not socket.AF_INET in netifaces.ifaddresses(self.name):
return []
return [dict(addr=q['addr'], netmask=q['netmask']) for q in
netifaces.ifaddresses(self.name)[socket.AF_INET] if netaddr.IPAddress(
q['addr'], 4) in netaddr.glob_to_iprange(
netaddr.cidr_to_glob(self.ipv4_local_network))]
def getGlobalScopeAddressList(self):
"""Returns currently configured global scope IPv6 addresses"""
address_list = [q for q in netifaces.ifaddresses(self.name)[socket.AF_INET6]
if isGlobalScopeAddress(q['addr'].split('%')[0])]
# XXX: Missing implementation of Unique Local IPv6 Unicast Addresses as
# defined in http://www.rfc-editor.org/rfc/rfc4193.txt
# XXX: XXX: XXX: IT IS DISALLOWED TO IMPLEMENT link-local addresses as
# Linux and BSD are possibly wrongly implementing it -- it is "too local"
# it is impossible to listen or access it on same node
# XXX: IT IS DISALLOWED to implement ad hoc solution like inventing node
# local addresses or anything which does not exists in RFC!
return address_list
def getInterfaceList(self):
"""Returns list of interfaces already present on bridge"""
interface_list = []
returncode, result = callAndRead(['brctl', 'show'])
in_bridge = False
for line in result.split('\n'):
if len(line.split()) > 1:
if self.name in line:
interface_list.append(line.split()[-1])
in_bridge = True
continue
if in_bridge:
break
elif in_bridge:
if line.strip():
interface_list.append(line.strip())
return interface_list
def addTap(self, tap):
"""
Add the tap interface tap to the bridge.
Args:
tap: Tap, the tap interface.
"""
if tap.name not in self.getInterfaceList():
callAndRead(['brctl', 'addif', self.name, tap.name])
def _addSystemAddress(self, address, netmask, ipv6=True):
"""Adds system address to bridge
Returns True if address was added successfully.
Returns False if there was issue.
"""
if ipv6:
address_string = '%s/%s' % (address, netmaskToPrefixIPv6(netmask))
af = socket.AF_INET6
else:
af = socket.AF_INET
address_string = '%s/%s' % (address, netmaskToPrefixIPv4(netmask))
# check if address is already took by any other interface
for interface in netifaces.interfaces():
if interface != self.name:
address_dict = netifaces.ifaddresses(interface)
if af in address_dict:
if address in [q['addr'].split('%')[0] for q in address_dict[af]]:
return False
if not af in netifaces.ifaddresses(self.name) or not address in [q['addr'].split('%')[0] for q in netifaces.ifaddresses(self.name)[af]]:
# add an address
callAndRead(['ip', 'addr', 'add', address_string, 'dev', self.name])
# wait few moments
time.sleep(2)
# check existence on interface
returncode, result = callAndRead(['ip', 'addr', 'list', self.name])
for l in result.split('\n'):
if address in l:
if 'tentative' in l:
# duplicate, remove
callAndRead(['ip', 'addr', 'del', address_string, 'dev', self.name])
return False
# found and clean
return True
# even when added not found, this is bad...
return False
def _generateRandomIPv4Address(self, netmask):
# no addresses found, generate new one
# Try 10 times to add address, raise in case if not possible
try_num = 10
while try_num > 0:
addr = random.choice([q for q in netaddr.glob_to_iprange(
netaddr.cidr_to_glob(self.ipv4_local_network))]).format()
if dict(addr=addr, netmask=netmask) not in self.getIPv4LocalAddressList():
# Checking the validity of the IPv6 address
if self._addSystemAddress(addr, netmask, False):
return dict(addr=addr, netmask=netmask)
try_num -= 1
raise AddressGenerationError(addr)
def addIPv4LocalAddress(self, addr=None):
"""Adds local IPv4 address in ipv4_local_network"""
netmask = '255.255.255.255'
local_address_list = self.getIPv4LocalAddressList()
if addr is None:
return self._generateRandomIPv4Address(netmask)
elif dict(addr=addr, netmask=netmask) not in local_address_list:
if self._addSystemAddress(addr, netmask, False):
return dict(addr=addr, netmask=netmask)
else:
return self._generateRandomIPv4Address(netmask)
else:
# confirmed to be configured
return dict(addr=addr, netmask=netmask)
def addAddr(self, addr = None, netmask = None):
"""
Adds IP address to bridge.
If addr is specified and exists already on bridge does nothing.
If addr is specified and does not exists on bridge, tries to add given address.
In case if it is not possible (ex. because network changed) calculates new address.
Args:
addr: Wished address to be added to bridge.
netmask: Wished netmask to be used.
Returns:
Tuple of (address, netmask).
Raises:
AddressGenerationError: Couldn't construct valid address with existing
one's on the bridge.
NoAddressOnBridge: There's no address on the bridge to construct
an address with.
"""
# Getting one address of the bridge as base of the next addresses
bridge_addr_list = self.getGlobalScopeAddressList()
# No address found
if len(bridge_addr_list) == 0:
raise NoAddressOnBridge(self.name)
address_dict = bridge_addr_list[0]
if addr is not None:
if dict(addr=addr, netmask=netmask) in bridge_addr_list:
# confirmed to be configured
return dict(addr=addr, netmask=netmask)
if netmask == address_dict['netmask']:
# same netmask, so there is a chance to add good one
bridge_network = netaddr.ip.IPNetwork('%s/%s' % (address_dict['addr'],
netmaskToPrefixIPv6(address_dict['netmask'])))
requested_network = netaddr.ip.IPNetwork('%s/%s' % (addr, netmaskToPrefixIPv6(netmask)))
if bridge_network.network == requested_network.network:
# same network, try to add
if self._addSystemAddress(addr, netmask):
# succeed, return it
return dict(addr=addr, netmask=netmask)
# Try 10 times to add address, raise in case if not possible
try_num = 10
netmask = address_dict['netmask']
while try_num > 0:
addr = ':'.join(address_dict['addr'].split(':')[:-1] + ['%x' % random.randint(1, 65000)])
socket.inet_pton(socket.AF_INET6, addr)
if dict(addr=addr, netmask=netmask) not in self.getGlobalScopeAddressList():
# Checking the validity of the IPv6 address
if self._addSystemAddress(addr, netmask):
return dict(addr=addr, netmask=netmask)
try_num -= 1
raise AddressGenerationError(addr)
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-x", "--computer_xml",
help="Path to file with computer's XML. If does not exists, "
"will be created",
default=None,
type=str),
Option("-l", "--log_file",
help="The path to the log file used by the script.",
type=str),
Option("-i", "--input_definition_file",
help="Path to file to read definition of computer instead of "
"declaration. Using definition file allows to disable "
"'discovery' of machine services and allows to define computer "
"configuration in fully controlled manner.",
type=str),
Option("-o", "--output_definition_file",
help="Path to file to write definition of computer from "
"declaration.",
type=str),
Option("-n", "--dry_run",
help="Don't actually do anything.",
default=False,
action="store_true"),
Option("-v", "--verbose",
default=False,
action="store_true",
help="Verbose output."),
Option("-c", "--console",
default=False,
action="store_true",
help="Console output."),
Option('--alter_user', choices=['True', 'False'],
help="Shall slapformat alter user database [default: True]"),
Option('--alter_network', choices=['True', 'False'],
help="Shall slapformat alter network configuration [default: True]"),
])
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
return options, args[0]
def run(config):
try:
# Define the computer
if config.input_definition_file:
filepath = os.path.abspath(config.input_definition_file)
config.logger.info('Using definition file %r' % filepath)
computer_definition = ConfigParser.RawConfigParser()
computer_definition.read(filepath)
bridge = None
address = None
netmask = None
if computer_definition.has_option('computer', 'address'):
address, netmask = computer_definition.get('computer', 'address').split('/')
if config.alter_network and config.bridge_name is not None \
and config.ipv4_local_network is not None:
bridge = Bridge(config.bridge_name, config.ipv4_local_network)
computer = Computer(
reference=config.computer_id,
bridge=bridge,
addr=address,
netmask=netmask,
)
partition_list = []
for partition_number in range(int(config.partition_amount)):
section = 'partition_%s' % partition_number
user = User(computer_definition.get(section, 'user'))
address_list = []
for a in computer_definition.get(section, 'address').split():
address, netmask = a.split('/')
address_list.append(dict(addr=address, netmask=netmask))
tap = Tap(computer_definition.get(section, 'network_interface'))
partition_list.append(Partition(reference=computer_definition.get(section, 'pathname'),
path=os.path.join(config.instance_root, computer_definition.get(section, 'pathname')),
user=user,
address_list=address_list,
tap=tap,
))
computer.partition_list = partition_list
else:
# no definition file, figure out computer
if os.path.exists(config.computer_xml):
config.logger.info('Loading previous computer data from %r' % config.computer_xml)
computer = Computer.load(config.computer_xml, reference=config.computer_id)
# Connect to the bridge interface defined by the configuration
computer.bridge = Bridge(config.bridge_name, config.ipv4_local_network)
else:
# If no pre-existent configuration found, creating a new computer object
config.logger.warning('Creating new data computer with id %r' % config.computer_id)
computer = Computer(
reference=config.computer_id,
bridge=Bridge(config.bridge_name, config.ipv4_local_network),
addr=None,
netmask=None,
)
partition_amount = int(config.partition_amount)
existing_partition_amount = len(computer.partition_list)
if existing_partition_amount > partition_amount:
raise ValueError('Requested amount of computer partitions (%s) is lower '
'then already configured (%s), cannot continue' % (partition_amount,
len(computer.partition_list)))
config.logger.info('Adding %s new partitions' %
(partition_amount-existing_partition_amount))
for nb_iter in range(existing_partition_amount, partition_amount):
# add new ones
user = User("%s%s" % (config.user_base_name, nb_iter))
tap = Tap("%s%s" % (config.tap_base_name, nb_iter))
path = os.path.join(config.instance_root, "%s%s" % (
config.partition_base_name, nb_iter))
computer.partition_list.append(
Partition(
reference="%s%s" % (config.partition_base_name, nb_iter),
path=path,
user=user,
address_list=None,
tap=tap,
))
computer.instance_root = config.instance_root
computer.software_root = config.software_root
config.logger.info('Updating computer')
address = computer.getAddress()
computer.address = address['addr']
computer.netmask = address['netmask']
if config.output_definition_file:
computer_definition = ConfigParser.RawConfigParser()
computer_definition.add_section('computer')
if computer.address is not None and computer.netmask is not None:
computer_definition.set('computer', 'address', '/'.join([computer.address, computer.netmask]))
partition_number = 0
for partition in computer.partition_list:
section = 'partition_%s' % partition_number
computer_definition.add_section(section)
address_list = []
for address in partition.address_list:
address_list.append('/'.join([address['addr'], address['netmask']]))
computer_definition.set(section, 'address', ' '.join(address_list))
computer_definition.set(section, 'user', partition.user.name)
computer_definition.set(section, 'user', partition.user.name)
computer_definition.set(section, 'network_interface', partition.tap.name)
computer_definition.set(section, 'pathname', partition.reference)
partition_number += 1
filepath = os.path.abspath(config.output_definition_file)
computer_definition.write(open(filepath, 'w'))
config.logger.info('Stored computer definition in %r' % filepath)
computer.construct(alter_user=config.alter_user,
alter_network=config.alter_network)
# Dumping and sending to the erp5 the current configuration
if not config.dry_run:
computer.dump(config.computer_xml)
config.logger.info('Posting information to %r' % config.master_url)
computer.send(config)
except:
config.logger.exception('Uncaught exception:')
raise
class Config:
def checkRequiredBinary(self, binary_list):
missing_binary_list = []
for b in binary_list:
try:
callAndRead([b])
except ValueError:
pass
except OSError:
missing_binary_list.append(b)
if missing_binary_list:
raise UsageError('Some required binaries are missing or not functional: %s'%
','.join(missing_binary_list))
def setConfig(self, option_dict, configuration_file_path):
"""
Set options given by parameters.
"""
self.key_file = None
self.cert_file = None
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration
for section in ("slapformat", "slapos"):
configuration_dict = dict(configuration_parser.items(section))
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
# setup some nones
for parameter in ['bridge_name', 'partition_base_name', 'user_base_name',
'tap_base_name', 'ipv4_local_network']:
if getattr(self, parameter, None) is None:
setattr(self, parameter, None)
# Set defaults lately
if self.alter_network is None:
self.alter_network = 'True'
if self.alter_user is None:
self.alter_user = 'True'
# set up logging
self.logger = logging.getLogger("slapformat")
self.logger.setLevel(logging.INFO)
if self.console:
self.logger.addHandler(logging.StreamHandler())
# Convert strings to booleans
root_needed = False
for o in ['alter_network', 'alter_user']:
if getattr(self, o).lower() == 'true':
root_needed = True
setattr(self, o, True)
elif getattr(self, o).lower() == 'false':
setattr(self, o, False)
else:
message = 'Option %r needs to be "True" or "False", wrong value: %r' % (
o, getattr(self, o))
self.logger.error(message)
raise UsageError(message)
if not self.dry_run:
if self.alter_user:
self.checkRequiredBinary(['groupadd', 'useradd', 'usermod'])
if self.alter_network:
self.checkRequiredBinary(['ip', 'tunctl'])
# Required, even for dry run
if self.alter_network:
self.checkRequiredBinary(['brctl'])
if self.dry_run:
root_needed = False
# check root
if root_needed and os.getuid() != 0:
message = "Root rights are needed"
self.logger.error(message)
raise UsageError(message)
if self.log_file:
if not os.path.isdir(os.path.dirname(self.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(self.log_file), self.log_file))
else:
file_handler = logging.FileHandler(self.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
self.logger.addHandler(file_handler)
self.logger.info('Configured logging to file %r' % self.log_file)
# Check mandatory options
for parameter in ('computer_id', 'instance_root', 'master_url',
'software_root', 'computer_xml'):
if not getattr(self, parameter, None):
raise UsageError("Parameter '%s' is not defined." % parameter)
self.logger.info("Started.")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
self.logger.debug("Verbose mode enabled.")
if self.dry_run:
self.logger.info("Dry-run mode enabled.")
# Calculate path once
self.computer_xml = os.path.abspath(self.computer_xml)
def main():
"Run default configuration."
global os
global callAndRead
global getpwnam
real_callAndRead = callAndRead
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
try:
# Parse arguments
options, configuration_file_path = Parser(usage=usage).check_args()
config = Config()
config.setConfig(options, configuration_file_path)
os = OS(config)
if config.dry_run:
def dry_callAndRead(argument_list, raise_on_error=True):
if argument_list == ['brctl', 'show']:
return real_callAndRead(argument_list, raise_on_error)
else:
return 0, ''
callAndRead = dry_callAndRead
real_addSystemAddress = Bridge._addSystemAddress
def fake_addSystemAddress(*args, **kw):
real_addSystemAddress(*args, **kw)
# Fake success
return True
Bridge._addSystemAddress = fake_addSystemAddress
def fake_getpwnam(user):
class result:
pw_uid = 12345
pw_gid = 54321
return result
getpwnam = fake_getpwnam
else:
dry_callAndRead = real_callAndRead
if config.verbose:
def logging_callAndRead(argument_list, raise_on_error=True):
config.logger.debug(' '.join(argument_list))
return dry_callAndRead(argument_list, raise_on_error)
callAndRead = logging_callAndRead
run(config)
except UsageError, err:
print >>sys.stderr, err.msg
print >>sys.stderr, "For help use --help"
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/format/slapos.xsd 0000664 0000000 0000000 00000001671 11602033661 0027072 0 ustar 00root root 0000000 0000000
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/ 0000775 0000000 0000000 00000000000 11602033661 0024501 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/SlapObject.py 0000664 0000000 0000000 00000042770 11602033661 0027113 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import os
import shutil
import subprocess
import pkg_resources
import shutil
import stat
import tempfile
from supervisor import xmlrpc
import xmlrpclib
import pwd
from utils import launchBuildout, getCleanEnvironment,\
dropPrivileges, bootstrapBuildout, updateFile,\
getSoftwareUrlHash, SlapPopen
from svcbackend import getSupervisorRPC
from exception import BuildoutFailedError, WrongPermissionError, \
PathDoesNotExistError
REQUIRED_COMPUTER_PARTITION_PERMISSION = '0750'
class Software(object):
"""This class is responsible of installing a software release"""
def __init__(self, url, software_root, console, buildout):
"""Initialisation of class parameters
"""
self.url = url
self.software_root = software_root
self.software_path = os.path.join(self.software_root,
getSoftwareUrlHash(self.url))
self.buildout = buildout
self.logger = logging.getLogger('BuildoutManager')
self.console = console
def install(self):
""" Fetches buildout configuration from the server, run buildout with
it. If it fails, we notify the server.
"""
self.logger.info("Installing software release %s..." % self.url)
if not os.path.isdir(self.software_path):
os.mkdir(self.software_path)
if os.getuid() == 0:
# In case when running as root copy ownership, to simplify logic
root_stat_info = os.stat(self.software_root)
for path in [self.software_path]:
path_stat_info = os.stat(path)
if root_stat_info.st_uid != path_stat_info.st_uid or\
root_stat_info.st_gid != path_stat_info.st_gid:
os.chown(path, root_stat_info.st_uid,
root_stat_info.st_gid)
extends_cache = tempfile.mkdtemp()
try:
buildout_parameter_list = [
'buildout:extends-cache=%s' % extends_cache,
'buildout:directory=%s' % self.software_path,
'-c', self.url]
bootstrapBuildout(self.software_path, self.buildout,
additional_buildout_parametr_list=buildout_parameter_list,
console=self.console)
launchBuildout(self.software_path,
os.path.join(self.software_path, 'bin', 'buildout'),
additional_buildout_parametr_list=buildout_parameter_list,
console=self.console)
finally:
shutil.rmtree(extends_cache)
def remove(self):
"""Removes the part that was installed.
"""
try:
shutil.rmtree(self.software_path)
except IOError as error:
error_string = "I/O error while removing software (%s): %s" % (self.url,
error)
raise IOError(error_string)
class Partition(object):
"""This class is responsible of the installation of a instance
"""
def __init__(self,
software_path,
instance_path,
supervisord_partition_configuration_path,
supervisord_socket,
computer_partition,
computer_id,
partition_id,
server_url,
software_release_url,
buildout,
certificate_repository_path=None,
console=False
):
"""Initialisation of class parameters"""
self.buildout = buildout
self.software_path = software_path
self.instance_path = instance_path
self.run_path = os.path.join(self.instance_path, 'etc', 'run')
self.supervisord_partition_configuration_path = \
supervisord_partition_configuration_path
self.supervisord_socket = supervisord_socket
self.computer_partition = computer_partition
self.logger = logging.getLogger('Partition')
self.computer_id = computer_id
self.partition_id = partition_id
self.server_url = server_url
self.software_release_url = software_release_url
self.console = console
self.key_file = ''
self.cert_file = ''
if certificate_repository_path is not None:
self.key_file = os.path.join(certificate_repository_path,
self.partition_id + '.key')
self.cert_file = os.path.join(certificate_repository_path,
self.partition_id + '.crt')
self._updateCertificate()
def _updateCertificate(self):
if not os.path.exists(self.key_file) or \
not os.path.exists(self.cert_file):
self.logger.info('Certificate and key not found, downloading to %r and '
'%r' % (self.cert_file, self.key_file))
partition_certificate = self.computer_partition.getCertificate()
open(self.key_file, 'w').write(partition_certificate['key'])
open(self.cert_file, 'w').write(partition_certificate['certificate'])
for f in [self.key_file, self.cert_file]:
os.chmod(f, 0400)
os.chown(f, *self.getUserGroupId())
def getUserGroupId(self):
"""Returns tuple of (uid, gid) of partition"""
stat_info = os.stat(self.instance_path)
uid = stat_info.st_uid
gid = stat_info.st_gid
return (uid, gid)
def install(self):
""" Creates configuration file from template in software_path, then
installs the software partition with the help of buildout
"""
# XXX: Shall be no op in case if revision had not changed
# It requires implementation of revision on server
self.logger.info("Installing Computer Partition %s..." \
% self.computer_partition.getId())
# Checks existence and permissions of Partition directory
# Note : Partitions have to be created and configured before running slapgrid
if not os.path.isdir(self.instance_path):
raise PathDoesNotExistError('Please create partition directory %s'
% self.instance_path)
permission = oct(stat.S_IMODE(os.stat(self.instance_path).st_mode))
if permission != REQUIRED_COMPUTER_PARTITION_PERMISSION:
raise WrongPermissionError('Wrong permissions in %s : actual ' \
'permissions are : %s, wanted ' \
'are %s' %
(self.instance_path, permission,
REQUIRED_COMPUTER_PARTITION_PERMISSION))
# Generates buildout part from template
# TODO how to fetch the good template? Naming conventions?
template_location = os.path.join(self.software_path, 'template.cfg')
config_location = os.path.join(self.instance_path, 'buildout.cfg')
self.logger.debug("Coping %r to %r" % (template_location, config_location))
shutil.copy(template_location, config_location)
# fill generated buildout with additional information
buildout_text = open(config_location).read()
buildout_text += '\n\n' + pkg_resources.resource_string(__name__,
'templates/buildout-tail.cfg.in') % dict(
computer_id=self.computer_id,
partition_id=self.partition_id,
server_url=self.server_url,
software_release_url=self.software_release_url,
key_file=self.key_file,
cert_file=self.cert_file
)
open(config_location, 'w').write(buildout_text)
os.chmod(config_location, 0640)
# Try to find the best possible buildout:
# *) if software_root/bin/bootstrap exists use this one to bootstrap
# locally
# *) as last resort fallback to buildout binary from software_path
bootstrap_candidate_dir = os.path.abspath(os.path.join(self.software_path,
'bin'))
if os.path.isdir(bootstrap_candidate_dir):
bootstrap_candidate_list = [q for q in os.listdir(bootstrap_candidate_dir)
if q.startswith('bootstrap')]
else:
bootstrap_candidate_list = []
uid, gid = self.getUserGroupId()
os.chown(config_location, -1, int(gid))
if len(bootstrap_candidate_list) == 0:
buildout_binary = os.path.join(self.software_path, 'bin', 'buildout')
self.logger.warning("Falling back to default buildout %r" %
buildout_binary)
else:
if len(bootstrap_candidate_list) != 1:
raise ValueError('More then one bootstrap candidate found.')
# Reads uid/gid of path, launches buildout with thoses privileges
bootstrap_file = os.path.abspath(os.path.join(bootstrap_candidate_dir,
bootstrap_candidate_list[0]))
file = open(bootstrap_file, 'r')
line = file.readline()
file.close()
invocation_list = []
if line.startswith('#!'):
invocation_list = line[2:].split()
invocation_list.append(bootstrap_file)
self.logger.debug('Invoking %r in %r' % (' '.join(invocation_list),
self.instance_path))
kw = dict()
if not self.console:
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid), cwd=self.instance_path,
env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw)
result_std = process_handler.communicate()[0]
if self.console:
result_std = 'Please consult messages above.'
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to bootstrap buildout in %r:\n%s\n' % (
self.instance_path, result_std)
raise BuildoutFailedError(message)
buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout')
if not os.path.exists(buildout_binary):
# use own buildout generation
bootstrapBuildout(self.instance_path, self.buildout,
['buildout:bin-directory=%s'% os.path.join(self.instance_path,
'sbin')], console=self.console)
buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout')
# Launches buildout
launchBuildout(self.instance_path,
buildout_binary, console=self.console)
# Generates supervisord configuration file from template
self.logger.info("Generating supervisord config file from template...")
# check if CP/etc/run exists and it is a directory
# iterate over each file in CP/etc/run
# if at least one is not 0750 raise -- partition has something funny
runner_list = []
if os.path.exists(self.run_path):
if os.path.isdir(self.run_path):
runner_list = os.listdir(self.run_path)
if len(runner_list) == 0:
self.logger.warning('No runners found for partition %r' %
self.partition_id)
if os.path.exists(self.supervisord_partition_configuration_path):
os.unlink(self.supervisord_partition_configuration_path)
else:
partition_id = self.computer_partition.getId()
program_partition_template = pkg_resources.resource_stream(__name__,
'templates/program_partition_supervisord.conf.in').read()
group_partition_template = pkg_resources.resource_stream(__name__,
'templates/group_partition_supervisord.conf.in').read()
partition_supervisor_configuration = group_partition_template % dict(
instance_id=partition_id,
program_list=','.join(['_'.join([partition_id, runner])
for runner in runner_list]))
for runner in runner_list:
partition_supervisor_configuration += '\n' + \
program_partition_template % dict(
program_id='_'.join([partition_id, runner]),
program_directory=self.instance_path,
program_command=os.path.join(self.run_path, runner),
program_name=runner,
instance_path=self.instance_path,
user_id=uid,
group_id=gid,
# As supervisord has no environment to inherit setup minimalistic one
HOME=pwd.getpwuid(uid).pw_dir,
USER=pwd.getpwuid(uid).pw_name,
)
updateFile(self.supervisord_partition_configuration_path,
partition_supervisor_configuration)
self.updateSupervisor()
def start(self):
"""Asks supervisord to start the instance. If this instance is not
installed, we install it.
"""
supervisor = self.getSupervisorRPC()
partition_id = self.computer_partition.getId()
supervisor.startProcessGroup(partition_id, False)
self.logger.info("Requested start of %s..." % self.computer_partition.getId())
def stop(self):
"""Asks supervisord to stop the instance."""
supervisor = self.getSupervisorRPC()
partition_id = self.computer_partition.getId()
try:
supervisor.stopProcessGroup(partition_id, False)
except xmlrpclib.Fault, e:
if e.faultString.startswith('BAD_NAME:'):
self.logger.info('Partition %s not known in supervisord, ignoring' % partition_id)
else:
self.logger.info("Requested stop of %s..." % self.computer_partition.getId())
def destroy(self):
"""Destroys the partition and makes it available for subsequent use."
"""
self.logger.info("Destroying Computer Partition %s..." \
% self.computer_partition.getId())
# Gets actual buildout binary
buildout_binary = os.path.join(self.instance_path, 'sbin', 'buildout')
if not os.path.exists(buildout_binary):
buildout_binary = os.path.join(self.software_path, 'bin', 'buildout')
# Launches "destroy" binary if exists
destroy_executable_location = os.path.join(self.instance_path, 'sbin',
'destroy')
if os.path.exists(destroy_executable_location):
# XXX: we should factorize this code
uid, gid = None, None
stat_info = os.stat(self.instance_path)
uid = stat_info.st_uid
gid = stat_info.st_gid
self.logger.debug('Invoking %r' % destroy_executable_location)
kw = dict()
if not self.console:
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen([destroy_executable_location],
preexec_fn=lambda: dropPrivileges(uid, gid), cwd=self.instance_path,
env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw)
result_std = process_handler.communicate()[0]
if self.console:
result_std = 'Please consult messages above'
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to destroy Computer Partition in %r:\n%s\n' % (
self.instance_path, result_std)
raise subprocess.CalledProcessError(message)
# Manually cleans what remains
try:
for f in [self.key_file, self.cert_file]:
if f:
if os.path.exists(f):
os.unlink(f)
for root, dirs, file_list in os.walk(self.instance_path):
for directory in dirs:
shutil.rmtree(os.path.join(self.instance_path, directory))
for file in file_list:
os.remove(os.path.join(self.instance_path, file))
if os.path.exists(self.supervisord_partition_configuration_path):
os.remove(self.supervisord_partition_configuration_path)
self.updateSupervisor()
except IOError as error:
error_string = "I/O error while freeing partition (%s): %s" \
% (self.instance_path, error)
raise IOError(error_string)
def fetchInformations(self):
"""Fetch usage informations with buildout, returns it.
"""
raise NotImplementedError
def getSupervisorRPC(self):
return getSupervisorRPC(self.supervisord_socket)
def updateSupervisor(self):
"""Forces supervisord to reload its configuration"""
# Note: This method shall wait for results from supervisord
# In future it will be not needed, as update command
# is going to be implemented on server side.
self.logger.debug('Updating supervisord')
supervisor = self.getSupervisorRPC()
# took from supervisord.supervisorctl.do_update
result = supervisor.reloadConfig()
added, changed, removed = result[0]
for gname in removed:
results = supervisor.stopProcessGroup(gname)
fails = [res for res in results
if res['status'] == xmlrpc.Faults.FAILED]
if fails:
self.logger.warning('Problem while stopping process %r, will try later' % gname)
else:
self.logger.info('Stopped %r' % gname)
supervisor.removeProcessGroup(gname)
self.logger.info('Removed %r' % gname)
for gname in changed:
results = supervisor.stopProcessGroup(gname)
self.logger.info('Stopped %r' % gname)
supervisor.removeProcessGroup(gname)
supervisor.addProcessGroup(gname)
self.logger.info('Updated %r' % gname)
for gname in added:
supervisor.addProcessGroup(gname)
self.logger.info('Updated %r' % gname)
self.logger.debug('Supervisord updated')
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/__init__.py 0000664 0000000 0000000 00000002442 11602033661 0026614 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/exception.py 0000664 0000000 0000000 00000002713 11602033661 0027054 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
"""Exposed exceptions"""
class PathDoesNotExistError(Exception):
pass
class WrongPermissionError(Exception):
pass
class BuildoutFailedError(Exception):
pass
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/slapgrid.py 0000664 0000000 0000000 00000064206 11602033661 0026670 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import os
import sys
import pkg_resources
import warnings
if sys.version_info < (2, 6):
warnings.warn('Used python version (%s) is old and have problems with'
' IPv6 connections' % sys.version.split('\n')[0])
import socket
import subprocess
import traceback
#from time import strftime
from SlapObject import Software, Partition, WrongPermissionError, \
PathDoesNotExistError
import argparse
import ConfigParser
from utils import updateFile
from utils import createPrivateDirectory
from utils import setRunning
from utils import setFinished
from utils import getSoftwareUrlHash
from slapos import slap
from slapos.slap import NotFoundError
from utils import dropPrivileges
from utils import SlapPopen
from svcbackend import launchSupervisord
import tempfile
import StringIO
from lxml import etree
MANDATORY_PARAMETER_LIST = [
'computer_id',
'instance_root',
'master_url',
'software_root',
]
def parseArgumentTupleAndReturnSlapgridObject(*argument_tuple):
"""Parses arguments either from command line, from method parameters or from
config file. Then returns a new instance of slapgrid.Slapgrid with those
parameters. Also returns the options dict and unused variable list, and
configures logger.
"""
parser = argparse.ArgumentParser()
parser.add_argument("--instance-root",
help="The instance root directory location.")
parser.add_argument("--software-root",
help="The software_root directory location.")
parser.add_argument("--master-url",
help="The master server URL. Mandatory.")
parser.add_argument("--computer-id",
help="The computer id defined in the server.")
parser.add_argument("--supervisord-socket",
help="The socket supervisor will use.")
parser.add_argument("--supervisord-configuration-path",
help="The location where supervisord configuration " \
"will be stored.")
parser.add_argument("--usage-report-periodicity",
type=int, default="24",
help="The periodicity of usage report sends, in hours.")
parser.add_argument("--buildout", help="Location of buildout binary.",
default=None)
parser.add_argument("--pidfile",
help="The location where pidfile will be created.")
parser.add_argument("--logfile",
help="The location where slapgrid logfile will be " \
"created.")
parser.add_argument("--key_file", help="SSL Authorisation key file.")
parser.add_argument("--cert_file",
help="SSL Authorisation certificate file.")
parser.add_argument("--master_ca_file", help="Root certificate of SlapOS "
"master key.")
parser.add_argument("--certificate_repository_path",
help="Path to directory where downloaded certificates would be stored.")
parser.add_argument("-c", "--console", action="store_true", default=False,
help="Enables console output and live output from subcommands.")
parser.add_argument("-v", "--verbose", action="store_true", default=False,
help="Be verbose.")
parser.add_argument("configuration_file", nargs=1, type=argparse.FileType(),
help="SlapOS configuration file.")
# Parses arguments
if argument_tuple == ():
# No arguments given to entry point : we parse sys.argv.
argument_option_instance = parser.parse_args()
else:
argument_option_instance = \
parser.parse_args(list(argument_tuple))
# Parses arguments from config file, if needed, then merge previous arguments
option_dict = {}
configuration_file = argument_option_instance.configuration_file[0]
# Loads config (if config specified)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.readfp(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
for argument_key, argument_value in vars(argument_option_instance
).iteritems():
if argument_value is not None:
option_dict.update({argument_key: argument_value})
# Configures logger.
#XXX: We need to configure it as soon as possible, so I do it here.
logger_format = '%(asctime)s %(name)-18s: %(levelname)-8s %(message)s'
if option_dict['verbose']:
level = logging.DEBUG
else:
level = logging.INFO
if option_dict.get('logfile'):
logging.basicConfig(filename=option_dict['logfile'],
format=logger_format, level=level)
if option_dict['console']:
logging.basicConfig(level=level)
missing_mandatory_parameter_list = []
for mandatory_parameter in MANDATORY_PARAMETER_LIST:
if not mandatory_parameter in option_dict:
missing_mandatory_parameter_list.append(mandatory_parameter)
repository_required = False
if 'key_file' in option_dict:
repository_required = True
if not 'cert_file' in option_dict:
missing_mandatory_parameter_list.append('cert_file')
if 'cert_file' in option_dict:
repository_required = True
if not 'key_file' in option_dict:
missing_mandatory_parameter_list.append('key_file')
if repository_required:
if 'certificate_repository_path' not in option_dict:
missing_mandatory_parameter_list.append('certificate_repository_path')
if len(missing_mandatory_parameter_list) > 0:
parser.error('Missing mandatory parameters:\n%s' % '\n'.join(
missing_mandatory_parameter_list))
key_file = option_dict.get('key_file')
cert_file = option_dict.get('cert_file')
master_ca_file = option_dict.get('master_ca_file')
for f in [key_file, cert_file, master_ca_file]:
if f is not None:
if not os.path.exists(f):
parser.error('File %r does not exists.' % f)
certificate_repository_path = option_dict.get('certificate_repository_path')
if certificate_repository_path is not None:
if not os.path.isdir(certificate_repository_path):
parser.error('Directory %r does not exists' %
certificate_repository_path)
# Supervisord configuration location
if not option_dict.get('supervisord_configuration_path'):
option_dict['supervisord_configuration_path'] = \
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf')
# Supervisord socket
if not option_dict.get('supervisord_socket'):
option_dict['supervisord_socket'] = \
os.path.join(option_dict['instance_root'], 'supervisord.socket')
# Returning new Slapgrid instance and options
return ([Slapgrid(software_root=option_dict['software_root'],
instance_root=option_dict['instance_root'],
master_url=option_dict['master_url'],
computer_id=option_dict['computer_id'],
supervisord_socket=option_dict['supervisord_socket'],
supervisord_configuration_path=option_dict[
'supervisord_configuration_path'],
usage_report_periodicity=option_dict['usage_report_periodicity'],
key_file=key_file,
cert_file=cert_file,
master_ca_file=master_ca_file,
certificate_repository_path=certificate_repository_path,
console=option_dict['console'],
buildout=option_dict.get('buildout')),
option_dict])
def realRun(argument_tuple, method_list):
clean_run = True
slapgrid_object, option_dict = \
parseArgumentTupleAndReturnSlapgridObject(*argument_tuple)
pidfile = option_dict.get('pidfile')
if pidfile:
setRunning(pidfile)
try:
for method in method_list:
if not getattr(slapgrid_object, method)():
clean_run = False
finally:
if pidfile:
setFinished(pidfile)
if clean_run:
sys.exit(0)
else:
sys.exit(1)
def run(*argument_tuple):
"""Hooks for generic entry point to proces Software Releases (sr),
Computer Partitions (cp) and Usage Reports (ur)
Will run one by one each task (sr, cp, ur). If specified,
will run in the user wanted order.
"""
realRun(argument_tuple, ['processSoftwareReleaseList',
'processComputerPartitionList', 'agregateAndSendUsage'])
def runSoftwareRelease(*argument_tuple):
"""Hook for entry point to process Software Releases only
"""
realRun(argument_tuple, ['processSoftwareReleaseList'])
def runComputerPartition(*argument_tuple):
"""Hook for entry point to process Computer Partitions only
"""
realRun(argument_tuple, ['processComputerPartitionList'])
def runUsageReport(*argument_tuple):
"""Hook for entry point to process Usage Reports only
"""
realRun(argument_tuple, ['agregateAndSendUsage'])
class Slapgrid(object):
""" Main class for SlapGrid. Fetches and processes informations from master
server and pushes usage information to master server.
"""
def __init__(self,
software_root,
instance_root,
master_url,
computer_id,
supervisord_socket,
supervisord_configuration_path,
usage_report_periodicity,
buildout,
key_file=None,
cert_file=None,
master_ca_file=None,
certificate_repository_path=None,
console=False):
"""Makes easy initialisation of class parameters"""
# Parses arguments
self.software_root = os.path.abspath(software_root)
self.instance_root = os.path.abspath(instance_root)
self.master_url = master_url
self.computer_id = computer_id
self.supervisord_socket = supervisord_socket
self.supervisord_configuration_path = supervisord_configuration_path
self.usage_report_periodicity = usage_report_periodicity
self.key_file = key_file
self.cert_file = cert_file
self.master_ca_file = master_ca_file
self.certificate_repository_path = certificate_repository_path
# Configures logger
self.logger = logging.getLogger('Slapgrid')
# Creates objects from slap module
self.slap = slap.slap()
self.slap.initializeConnection(self.master_url, key_file=self.key_file,
cert_file=self.cert_file, master_ca_file=self.master_ca_file)
self.computer = self.slap.registerComputer(self.computer_id)
# Defines all needed paths
self.instance_etc_directory = os.path.join(self.instance_root, 'etc')
self.supervisord_configuration_directory = \
os.path.join(self.instance_etc_directory, 'supervisord.conf.d')
self.console = console
self.buildout = buildout
def checkEnvironmentAndCreateStructure(self):
"""Checks for software_root and instance_root existence, then creates
needed files and directories.
"""
# Checks for software_root and instance_root existence
if not os.path.isdir(self.software_root):
error = "%s does not exist." % self.software_root
raise OSError(error)
if not os.path.isdir(self.instance_root):
error = "%s does not exist." % self.instance_root
raise OSError(error)
# Creates everything needed
try:
# Creates instance_root structure
createPrivateDirectory(self.instance_etc_directory)
createPrivateDirectory(os.path.join(self.instance_root, 'var'))
createPrivateDirectory(os.path.join(self.instance_root, 'var', 'log'))
createPrivateDirectory(os.path.join(self.instance_root, 'var', 'run'))
createPrivateDirectory(self.supervisord_configuration_directory)
# Creates supervisord configuration
updateFile(self.supervisord_configuration_path,
pkg_resources.resource_stream(__name__,
'templates/supervisord.conf.in').read() % dict(
supervisord_configuration_directory=self.supervisord_configuration_directory,
supervisord_socket=os.path.abspath(self.supervisord_socket),
supervisord_loglevel='info',
supervisord_logfile=os.path.abspath(os.path.join(
self.instance_root, 'var', 'log', 'supervisord.log')),
supervisord_logfile_maxbytes='50MB',
supervisord_nodaemon='false',
supervisord_pidfile=os.path.abspath(os.path.join(
self.instance_root, 'var', 'run', 'supervisord.pid')),
supervisord_logfile_backups='10',
))
except (WrongPermissionError, PathDoesNotExistError) as error:
raise error
def getComputerPartitionList(self):
try:
computer_partition_list = self.computer.getComputerPartitionList()
except socket.error as error:
self.logger.fatal(error)
sys.exit(1)
return computer_partition_list
def processSoftwareReleaseList(self):
"""Will process each Software Release.
"""
self.checkEnvironmentAndCreateStructure()
logger = logging.getLogger('SoftwareReleases')
logger.info("Processing software releases...")
clean_run = True
for software_release in self.computer.getSoftwareReleaseList():
try:
software_release.building()
software_release_uri = software_release.getURI()
Software(url=software_release_uri, software_root=self.software_root,
console=self.console, buildout=self.buildout).install()
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
software_release.error(exception)
raise
except Exception:
exception = traceback.format_exc()
logger.error(exception)
software_release.error(exception)
clean_run = False
else:
software_release.available()
logger.info("Finished software releases...")
return clean_run
def _launchSupervisord(self):
launchSupervisord(self.supervisord_socket,
self.supervisord_configuration_path)
def processComputerPartitionList(self):
"""Will start supervisord and process each Computer Partition.
"""
logger = logging.getLogger('ComputerPartitionProcessing')
logger.info("Processing computer partitions...")
# Prepares environment
self.checkEnvironmentAndCreateStructure()
self._launchSupervisord()
# Process Computer Partitions
clean_run = True
for computer_partition in self.getComputerPartitionList():
computer_partition_id = computer_partition.getId()
try:
software_url = computer_partition.getSoftwareRelease().getURI()
except NotFoundError:
software_url = None
software_path = os.path.join(self.software_root,
getSoftwareUrlHash(software_url))
local_partition = Partition(
software_path=software_path,
instance_path=os.path.join(self.instance_root,
computer_partition.getId()),
supervisord_partition_configuration_path=os.path.join(
self.supervisord_configuration_directory, '%s.conf' %
computer_partition_id),
supervisord_socket=self.supervisord_socket,
computer_partition=computer_partition,
computer_id=self.computer_id,
partition_id=computer_partition_id,
server_url=self.master_url,
software_release_url=software_url,
certificate_repository_path=self.certificate_repository_path,
console=self.console, buildout=self.buildout)
# There are no conditions to try to instanciate partition
try:
computer_partition_state = computer_partition.getState()
if computer_partition_state == "started":
local_partition.install()
computer_partition.available()
local_partition.start()
computer_partition.started()
elif computer_partition_state == "stopped":
local_partition.install()
computer_partition.available()
local_partition.stop()
computer_partition.stopped()
elif computer_partition_state == "destroyed":
# Stop, but safely
try:
local_partition.stop()
try:
computer_partition.stopped()
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
raise
except Exception:
pass
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
raise
except Exception:
clean_run = False
exception = traceback.format_exc()
logger.error(exception)
computer_partition.error(exception)
else:
error_string = "Computer Partition %r has unsupported state: %s" % \
(computer_partition_id, computer_partition_state)
computer_partition.error(error_string)
raise NotImplementedError(error_string)
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
raise
except Exception:
clean_run = False
exception = traceback.format_exc()
logger.error(exception)
computer_partition.error(exception)
logger.info("Finished computer partitions...")
return clean_run
def validateXML(self, to_be_validated):
"""Will validate the xml file"""
#We get the xsd model
fed = open('choose_the_path', 'r')
model = StringIO.StringIO(fed.read())
xmlschema_doc = etree.parse(model)
xmlschema = etree.XMLSchema(xmlschema_doc)
string_to_validate = StringIO.StringIO(to_be_validated)
document = etree.parse(string_to_validate)
if xmlschema.validate(document):
return True
return False
def agregateAndSendUsage(self):
"""Will agregate usage from each Computer Partition.
"""
slap_computer_usage = self.slap.registerComputer(self.computer_id)
computer_partition_usage_list = []
logger = logging.getLogger('UsageReporting')
logger.info("Aggregating and sending usage reports...")
clean_run = True
#We loop on the different computer partitions
for computer_partition in slap_computer_usage.getComputerPartitionList():
computer_partition_id = computer_partition.getId()
#We want execute all the script in the report folder
instance_path = os.path.join(self.instance_root,
computer_partition.getId())
report_path = os.path.join(instance_path, 'etc', 'report')
if os.path.isdir(report_path):
script_list_to_run = os.listdir(report_path)
else:
script_list_to_run = []
#We now generate the pseudorandom name for the xml file
# and we add it in the invocation_list
f = tempfile.NamedTemporaryFile()
name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name))
path_to_slapreport = os.path.join(instance_path, 'var', 'xml_report',
name_xml)
failed_script_list = []
for script in script_list_to_run:
invocation_list = []
invocation_list.append(os.path.join(instance_path, 'etc', 'report',
script))
#We add the xml_file name in the invocation_list
#f = tempfile.NamedTemporaryFile()
#name_xml = '%s.%s' % ('slapreport', os.path.basename(f.name))
#path_to_slapreport = os.path.join(instance_path, 'var', name_xml)
invocation_list.append(path_to_slapreport)
#Dropping privileges
uid, gid = None, None
stat_info = os.stat(instance_path)
#stat sys call to get statistics informations
uid = stat_info.st_uid
gid = stat_info.st_gid
kw = dict()
if not self.console:
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid),
cwd=os.path.join(instance_path, 'etc', 'report'),
env=None, **kw)
result = process_handler.communicate()[0]
if self.console:
result = 'Please consult messages above'
if process_handler.returncode is None:
process_handler.kill()
if process_handler.returncode != 0:
clean_run = False
failed_script_list.append("Script %r failed with %s." % (script, result))
logger.warning("Failed to run %r, the result was. \n%s" %
(invocation_list, result))
if len(failed_script_list):
computer_partition.error('\n'.join(failed_script_list))
#Now we loop through the different computer partitions to ggetId()et reports
report_usage_issue_cp_list = []
for computer_partition in slap_computer_usage.getComputerPartitionList():
computer_partition_id = computer_partition.getId()
instance_path = os.path.join(self.instance_root, computer_partition_id)
dir_reports = os.path.join(instance_path, 'var', 'xml_report')
#The directory xml_report contain a number of files equal
#to the number of software instance running inside the same partition
if os.path.isdir(dir_reports):
filename_list = os.listdir(dir_reports)
else:
filename_list = []
#logger.debug('name List %s' % filename_list)
usage = ''
for filename in filename_list:
file_path = os.path.join(dir_reports, filename)
if os.path.exists(file_path):
usage_file = open(file_path, 'r')
usage += usage_file.read()
usage_file.close()
computer_partition_usage = self.slap.registerComputerPartition(
self.computer_id, computer_partition_id)
computer_partition_usage.setUsage(usage)
computer_partition_usage_list.append(computer_partition_usage)
else:
logger.debug("Usage report %r not found, ignored" % file_path)
#-XXX-Here we'll have a "validate_xml" function
#if not self.validateXML(usage):
## logger.debug("WARNING: The file xml %s is not valid " %
# os.path.join(dir_reports, filename))
if filename_list:
logger.debug("Sending usage informations and terminating for "
"partition %s..." % os.path.basename(instance_path))
#logger.debug('file %s' % usage)
else:
logger.debug("No usage file created for partition %s..." \
% os.path.basename(instance_path))
#last_push_date = self.computer.getLastUsagePush()
#periodicity_timedelta = datetime.timedelta(
# self.usage_report_periodicity)
#if periodicity_timedelta + last_push_date < datetime.datetime.today():
# Pushes informations, if any
try:
slap_computer_usage.reportUsage(computer_partition_usage_list)
except Exception:
computer_partition_id = computer_partition.getId()
exception = traceback.format_exc()
issue = "Cannot report usage for %r: %s" % (computer_partition_id,
exception)
logger.info(issue)
computer_partition.error(issue)
report_usage_issue_cp_list.append(computer_partition_id)
#After sending the aggregated file we remove everything in the folder
for filename in filename_list:
os.remove(os.path.join(dir_reports, filename))
for computer_partition in slap_computer_usage.getComputerPartitionList():
computer_partition_id = computer_partition.getId()
try:
software_url = computer_partition.getSoftwareRelease().getURI()
except NotFoundError:
software_url = None
software_path = os.path.join(self.software_root,
getSoftwareUrlHash(software_url))
local_partition = Partition(
software_path=software_path,
instance_path=os.path.join(self.instance_root,
computer_partition.getId()),
supervisord_partition_configuration_path=os.path.join(
self.supervisord_configuration_directory, '%s.conf' %
computer_partition_id),
supervisord_socket=self.supervisord_socket,
computer_partition=computer_partition,
computer_id=self.computer_id,
partition_id=computer_partition_id,
server_url=self.master_url,
software_release_url=software_url,
certificate_repository_path=self.certificate_repository_path,
console=self.console
)
if computer_partition.getState() == "destroyed":
try:
local_partition.stop()
try:
computer_partition.stopped()
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
raise
except Exception:
pass
except (SystemExit, KeyboardInterrupt):
exception = traceback.format_exc()
computer_partition.error(exception)
raise
except Exception:
clean_run = False
exception = traceback.format_exc()
computer_partition.error(exception)
logger.error(exception)
if computer_partition.getId() in report_usage_issue_cp_list:
logger.info('Ignoring destruction of %r, as not report usage was '
'sent' % computer_partition.getId())
continue
local_partition.destroy()
try:
computer_partition.destroyed()
except slap.NotFoundError:
logger.debug('Ignored slap error while trying to inform about '
'destroying not fully configured Computer Partition %r' %
computer_partition.getId())
logger.info("Finished usage reports...")
return clean_run
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/svcbackend.py 0000664 0000000 0000000 00000014453 11602033661 0027165 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from supervisor import xmlrpc
import time
from utils import SlapPopen
import logging
import os
import sys
import xmlrpclib
from optparse import OptionParser
import ConfigParser
import socket as socketlib
import subprocess
def getSupervisorRPC(socket):
supervisor_transport = xmlrpc.SupervisorTransport('', '',
'unix://' + socket)
server_proxy = xmlrpclib.ServerProxy('http://127.0.0.1',
supervisor_transport)
return getattr(server_proxy, 'supervisor')
def launchSupervisord(socket, configuration_file):
logger = logging.getLogger('SVCBackend')
supervisor = getSupervisorRPC(socket)
if os.path.exists(socket):
trynum = 1
while trynum < 6:
try:
status = supervisor.getState()
except xmlrpclib.Fault, e:
if e.faultCode == 6 and e.faultString == 'SHUTDOWN_STATE':
logger.info('Supervisor in shutdown procedure, will check again later.')
trynum += 1
time.sleep(2 * trynum)
except Exception:
# In case if there is problem with connection, assume that supervisord
# is not running and try to run it
break
else:
if status['statename'] == 'RUNNING' and status['statecode'] == 1:
logger.info('Supervisord already running.')
return
elif status['statename'] == 'SHUTDOWN_STATE' and status['statecode'] == 6:
logger.info('Supervisor in shutdown procedure, will check again later.')
trynum += 1
time.sleep(2 * trynum)
else:
log_message = 'Unknown supervisord state %r. Will try to start.' % status
logger.warning(log_message)
break
logger.info("Launching supervisord with clean environment.")
# Extract python binary to prevent shebang size limit
invocation_list = ["supervisord", '-c']
invocation_list.append("import sys ; sys.path=" + str(sys.path) + " ; import "
"supervisor.supervisord ; sys.argv[1:1]=['-c','" +
configuration_file +
"'] ; supervisor.supervisord.main()")
supervisord_popen = SlapPopen(invocation_list,
env={},
executable=sys.executable, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
result = supervisord_popen.communicate()[0]
if supervisord_popen.returncode == 0:
log_message = 'Supervisord command invoked with: %s' % result
logger.info(log_message)
try:
default_timeout = socketlib.getdefaulttimeout()
current_timeout = 1
trynum = 1
while trynum < 6:
try:
socketlib.setdefaulttimeout(current_timeout)
status = supervisor.getState()
if status['statename'] == 'RUNNING' and status['statecode'] == 1:
return
logger.warning('Wrong status name %(statename)r and code '
'%(statecode)r, trying again' % status)
trynum += 1
except Exception:
current_timeout = 5 * trynum
trynum += 1
pass
else:
logger.info('Supervisord started correctly in try %s.' % trynum)
return
logger.warning('Issue while checking supervisord.')
finally:
socketlib.setdefaulttimeout(default_timeout)
else:
log_message = 'Supervisord unknown problem: %s' % result
logger.warning(log_message)
def getOptionDict(*argument_tuple):
usage = """
Typical usage:
* %prog CONFIGURATION_FILE [arguments passed to supervisor]
""".strip()
parser = OptionParser(usage=usage)
# Parses arguments
if argument_tuple == ():
# No arguments given to entry point : we parse sys.argv.
(argument_option_instance, argument_list) = parser.parse_args()
else:
(argument_option_instance, argument_list) = \
parser.parse_args(list(argument_tuple))
if len(argument_list) == 0:
parser.error("Configuration file is obligatory. Consult documentation by "
"calling with -h.")
configuration_file = argument_list[0]
if not os.path.exists(configuration_file):
parser.error("Could not read configuration file : %s" \
% configuration_file)
slapgrid_configuration = ConfigParser.SafeConfigParser()
slapgrid_configuration.read(configuration_file)
# Merges the two dictionnaries
option_dict = dict(slapgrid_configuration.items("slapos"))
# Supervisord configuration location
if not option_dict.get('supervisord_configuration_path'):
option_dict['supervisord_configuration_path'] = \
os.path.join(option_dict['instance_root'], 'etc', 'supervisord.conf')
# Supervisord socket
if not option_dict.get('supervisord_socket'):
option_dict['supervisord_socket'] = \
os.path.join(option_dict['instance_root'], 'supervisord.socket')
return option_dict, argument_list[1:]
def supervisorctl(*argument_tuple):
option_dict, args = getOptionDict(*argument_tuple)
import supervisor.supervisorctl
supervisor.supervisorctl.main(args=['-c',
option_dict['supervisord_configuration_path']] + args)
def supervisord(*argument_tuple):
option_dict, dummy = getOptionDict(*argument_tuple)
dummy = dummy
launchSupervisord(option_dict['supervisord_socket'],
option_dict['supervisord_configuration_path'])
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/templates/ 0000775 0000000 0000000 00000000000 11602033661 0026477 5 ustar 00root root 0000000 0000000 buildout-tail.cfg.in 0000664 0000000 0000000 00000000772 11602033661 0032272 0 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/templates # This is beginning of zc.builodout profile's tail added by slapgrid
[buildout]
# put buildout generated binaries in specific directory
bin-directory = ${buildout:directory}/sbin
# protect software and run parts offline
offline = true
[slap_connection]
computer_id = %(computer_id)s
partition_id = %(partition_id)s
server_url = %(server_url)s
software_release_url = %(software_release_url)s
key_file = %(key_file)s
cert_file = %(cert_file)s
# This is end of zc.builodout profile's tail added by slapgrid
group_partition_supervisord.conf.in 0000664 0000000 0000000 00000000062 11602033661 0035564 0 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/templates [group:%(instance_id)s]
programs=%(program_list)s
program_partition_supervisord.conf.in 0000664 0000000 0000000 00000001073 11602033661 0036102 0 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/templates [program:%(program_id)s]
directory=%(program_directory)s
command=%(program_command)s
process_name=%(program_name)s
autostart=false
autorestart=false
startsecs=0
startretries=0
exitcodes=0
stopsignal=TERM
stopwaitsecs=60
user=%(user_id)s
group=%(group_id)s
serverurl=AUTO
redirect_stderr=true
stdout_logfile=%(instance_path)s/.%(program_id)s.log
stdout_logfile_maxbytes=100KB
stdout_logfile_backups=1
stderr_logfile=%(instance_path)s/.%(program_id)s.log
stderr_logfile_maxbytes=100KB
stderr_logfile_backups=1
environment=USER="%(USER)s",LOGNAME="%(USER)s",HOME="%(HOME)s"
supervisord.conf.in 0000664 0000000 0000000 00000001044 11602033661 0032260 0 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/templates [rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[include]
files = %(supervisord_configuration_directory)s/*.conf
[supervisorctl]
serverurl = unix://%(supervisord_socket)s
[supervisord]
loglevel = %(supervisord_loglevel)s
logfile = %(supervisord_logfile)s
logfile_maxbytes = %(supervisord_logfile_maxbytes)s
nodaemon = %(supervisord_nodaemon)s
pidfile = %(supervisord_pidfile)s
logfile-backups = %(supervisord_logfile_backups)s
[unix_http_server]
file=%(supervisord_socket)s
chmod=0700
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/utils.py 0000664 0000000 0000000 00000027127 11602033661 0026224 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import logging
import hashlib
import os
import pkg_resources
import stat
import subprocess
import sys
import pwd
import grp
from exception import BuildoutFailedError, WrongPermissionError
from hashlib import md5
# Such umask by default will create paths with full permission
# for user, non writable by group and not accessible by others
SAFE_UMASK = 027
PYTHON_ENVIRONMENT_REMOVE_LIST = [
'PYTHONHOME',
'PYTHONPATH',
'PYTHONSTARTUP',
'PYTHONY2K',
'PYTHONOPTIMIZE',
'PYTHONDEBUG',
'PYTHONDONTWRITEBYTECODE',
'PYTHONINSPECT',
'PYTHONNOUSERSITE',
'PYTHONNOUSERSITE',
'PYTHONUNBUFFERED',
'PYTHONVERBOSE',
]
SYSTEM_ENVIRONMENT_REMOVE_LIST = [
'ENV',
'LOGNAME',
'TEMP',
'TMP',
'TMPDIR',
'USER',
]
LOCALE_ENVIRONEMNT_REMOVE_LIST = [
'LANG',
'LANGUAGE',
'LC_ADDRESS',
'LC_COLLATE',
'LC_CTYPE',
'LC_IDENTIFICATION',
'LC_MEASUREMENT',
'LC_MESSAGES',
'LC_MONETARY',
'LC_NAME',
'LC_NUMERIC',
'LC_PAPER',
'LC_SOURCED',
'LC_TELEPHONE',
'LC_TIME',
]
class AlreadyRunning(Exception):
pass
class SlapPopen(subprocess.Popen):
"""Almost normal subprocess with gridish features"""
def __init__(self, *args, **kwargs):
kwargs.update(stdin=subprocess.PIPE)
subprocess.Popen.__init__(self, *args, **kwargs)
self.stdin.flush()
self.stdin.close()
self.stdin = None
def getSoftwareUrlHash(url):
return md5(url).hexdigest()
def getCleanEnvironment(home_path='/tmp'):
logger = logging.getLogger('CleanEnvironment')
changed_env = {}
removed_env = []
env = os.environ.copy()
# Clean python related environment variables
for k in PYTHON_ENVIRONMENT_REMOVE_LIST + SYSTEM_ENVIRONMENT_REMOVE_LIST \
+ LOCALE_ENVIRONEMNT_REMOVE_LIST:
old = env.pop(k, None)
if old is not None:
removed_env.append(k)
changed_env['HOME'] = env['HOME'] = home_path
for k in sorted(changed_env.iterkeys()):
logger.debug('Overriden %s = %r' % (k,changed_env[k]))
logger.debug('Removed from environement: %s' % ', '.join(sorted(removed_env)))
return env
def setRunning(pid_file):
"""Creates a pidfile. If a pidfile already exists, we exit"""
logger = logging.getLogger('Slapgrid')
if os.path.exists(pid_file):
# Pid file is present
logger.warning('pid file already exists : %s' % (pid_file))
try:
pid = int(open(pid_file, 'r').readline())
except ValueError:
pid = None
# XXX This could use psutil library.
if pid is not None and os.path.exists("/proc/%s" % pid):
#XXX: can we trust sys.argv?
process_name = os.path.basename(sys.argv[0])
if process_name in open('/proc/%s/cmdline' % pid, 'r').readline():
# In case process is present, ignore.
raise AlreadyRunning('A slapgrid process is running with pid %s' % pid)
logger.info('Pid file %r was stale one, overwritten' % pid_file)
# Start new process
write_pid(pid_file)
def setFinished(pid_file):
try:
os.remove(pid_file)
except OSError:
pass
def write_pid(pid_file):
logger = logging.getLogger('Slapgrid')
pid = os.getpid()
try:
f = open(pid_file, 'w')
f.write('%s' % pid)
f.close()
except (IOError, OSError):
logger.critical('slapgrid could not write pidfile %s' % pid_file)
raise
def dropPrivileges(uid, gid):
"""Drop privileges to uid, gid if current uid is 0
Do tests to check if dropping was successful and that no system call is able
to re-raise dropped privileges
Does nothing in case if uid and gid are not 0
"""
logger = logging.getLogger('dropPrivileges')
current_uid, current_gid = os.getuid(), os.getgid()
if uid == 0 or gid == 0:
raise OSError('Dropping privileges to uid = %r or ' \
'gid = %r is too dangerous' % (uid, gid))
if not(current_uid == 0 and current_gid == 0):
logger.debug('Running as uid = %r, gid = %r, dropping not needed and not '
'possible' % (current_uid, current_gid))
return
# drop privileges
user_name = pwd.getpwuid(uid)[0]
group_list = [x.gr_gid for x in grp.getgrall() if user_name in x.gr_mem]
group_list.append(gid)
os.initgroups(pwd.getpwuid(uid)[0], gid)
os.setgid(gid)
os.setuid(uid)
# assert that privileges are dropped
message_pre = 'After dropping to uid = %r and gid = %r ' \
'and group_list = %s' % (
uid, gid, group_list)
new_uid, new_gid, new_group_list = os.getuid(), os.getgid(), os.getgroups()
if not (new_uid == uid and new_gid == gid and new_group_list == group_list):
raise OSError('%s new_uid = %r and new_gid = %r and ' \
'new_group_list = %r which is fatal.'
% (message_pre,
new_uid,
new_gid,
new_group_list))
# assert that it is not possible to go back to running one
try:
try:
os.setuid(current_uid)
except OSError:
try:
os.setgid(current_gid)
except OSError:
try:
os.setgroups([current_gid])
except OSError:
raise
except OSError:
pass
else:
raise ValueError('%s it was possible to go back to uid = %r and gid = '
'%r which is fatal.' % message_pre, current_uid, current_gid)
logger.info('Succesfully dropped privileges to uid=%r gid=%r' % (uid, gid))
def bootstrapBuildout(path, buildout=None,
additional_buildout_parametr_list=None, console=False):
if additional_buildout_parametr_list is None:
additional_buildout_parametr_list = []
logger = logging.getLogger('BuildoutManager')
# Reads uid/gid of path, launches buildout with thoses privileges
stat_info = os.stat(path)
uid = stat_info.st_uid
gid = stat_info.st_gid
invocation_list = [sys.executable, '-S']
if buildout is not None:
invocation_list.append(buildout)
else:
logger.warning('Using old style bootstrap of included bootstrap file. '
'Consider setting buildout binary location.')
invocation_list.append(pkg_resources.resource_filename(__name__,
'zc.buildout-bootstap.py'))
invocation_list.extend(additional_buildout_parametr_list)
if buildout is not None:
invocation_list.append('bootstrap')
try:
umask = os.umask(SAFE_UMASK)
logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
path))
kw = dict()
if not console:
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid),
cwd=path, env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir),
**kw
)
result = process_handler.communicate()[0]
if console:
result = 'Please consult messages above'
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to run buildout profile in directory %r:\n%s\n' % (
path, result)
raise BuildoutFailedError(message)
else:
logger.debug('Successful run:\n%s' % result)
except OSError as error:
raise BuildoutFailedError(error)
finally:
old_umask = os.umask(umask)
logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
def launchBuildout(path, buildout_binary,
additional_buildout_parametr_list=None, console=False):
""" Launches buildout."""
logger = logging.getLogger('BuildoutManager')
if additional_buildout_parametr_list is None:
additional_buildout_parametr_list = []
# Reads uid/gid of path, launches buildout with thoses privileges
stat_info = os.stat(path)
uid = stat_info.st_uid
gid = stat_info.st_gid
# Extract python binary to prevent shebang size limit
file = open(buildout_binary, 'r')
line = file.readline()
file.close()
invocation_list = []
if line.startswith('#!'):
line = line[2:]
# Prepares parameters for buildout
invocation_list = line.split() + [buildout_binary]
# Run buildout without reading user defaults
invocation_list.append('-U')
invocation_list.extend(additional_buildout_parametr_list)
try:
umask = os.umask(SAFE_UMASK)
logger.debug('Set umask from %03o to %03o' % (umask, SAFE_UMASK))
logger.debug('Invoking: %r in directory %r' % (' '.join(invocation_list),
path))
kw = dict()
if not console:
kw.update(stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process_handler = SlapPopen(invocation_list,
preexec_fn=lambda: dropPrivileges(uid, gid), cwd=path,
env=getCleanEnvironment(pwd.getpwuid(uid).pw_dir), **kw)
result = process_handler.communicate()[0]
if console:
result = 'Please consult messages above'
if process_handler.returncode is None or process_handler.returncode != 0:
message = 'Failed to run buildout profile in directory %r:\n%s\n' % (
path, result)
raise BuildoutFailedError(message)
else:
logger.debug('Successful run:\n%s' % result)
except OSError as error:
raise BuildoutFailedError(error)
finally:
old_umask = os.umask(umask)
logger.debug('Restore umask from %03o to %03o' % (old_umask, umask))
def updateFile(file_path, content, mode='0600'):
"""Creates an executable with "content" as content."""
altered = False
if not (os.path.isfile(file_path)) or \
not(hashlib.md5(open(file_path).read()).digest() ==\
hashlib.md5(content).digest()):
altered = True
file_file = open(file_path, 'w')
file_file.write(content)
file_file.flush()
file_file.close()
os.chmod(file_path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
if oct(stat.S_IMODE(os.stat(file_path).st_mode)) != mode:
os.chmod(file_path, int(mode, 8))
altered = True
return altered
def updateExecutable(executable_path, content):
"""Creates an executable with "content" as content."""
return updateFile(executable_path, content, '0700')
def createPrivateDirectory(path):
"""Creates directory belonging to root with umask 077"""
if not os.path.isdir(path):
os.mkdir(path)
os.chmod(path, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC)
permission = oct(stat.S_IMODE(os.stat(path).st_mode))
if permission not in ('0700'):
raise WrongPermissionError('Wrong permissions in %s ' \
': is %s, should be 0700'
% path, permission)
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/grid/zc.buildout-bootstap.py 0000664 0000000 0000000 00000023522 11602033661 0031152 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2006 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap a buildout-based project
Simply run this script in a directory containing a buildout.cfg.
The script accepts buildout command-line options, so you can
use the -c option to specify an alternate configuration file.
"""
import os, shutil, sys, tempfile, textwrap, urllib, urllib2, subprocess
from optparse import OptionParser
if sys.platform == 'win32':
def quote(c):
if ' ' in c:
return '"%s"' % c # work around spawn lamosity on windows
else:
return c
else:
quote = str
# See zc.buildout.easy_install._has_broken_dash_S for motivation and comments.
stdout, stderr = subprocess.Popen(
[sys.executable, '-Sc',
'try:\n'
' import ConfigParser\n'
'except ImportError:\n'
' print 1\n'
'else:\n'
' print 0\n'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
has_broken_dash_S = bool(int(stdout.strip()))
# In order to be more robust in the face of system Pythons, we want to
# run without site-packages loaded. This is somewhat tricky, in
# particular because Python 2.6's distutils imports site, so starting
# with the -S flag is not sufficient. However, we'll start with that:
if not has_broken_dash_S and 'site' in sys.modules:
# We will restart with python -S.
args = sys.argv[:]
args[0:0] = [sys.executable, '-S']
args = map(quote, args)
os.execv(sys.executable, args)
# Now we are running with -S. We'll get the clean sys.path, import site
# because distutils will do it later, and then reset the path and clean
# out any namespace packages from site-packages that might have been
# loaded by .pth files.
clean_path = sys.path[:]
import site
sys.path[:] = clean_path
for k, v in sys.modules.items():
if k in ('setuptools', 'pkg_resources') or (
hasattr(v, '__path__') and
len(v.__path__)==1 and
not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
# This is a namespace package. Remove it.
sys.modules.pop(k)
is_jython = sys.platform.startswith('java')
setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
distribute_source = 'http://python-distribute.org/distribute_setup.py'
# parsing arguments
def normalize_to_url(option, opt_str, value, parser):
if value:
if '://' not in value: # It doesn't smell like a URL.
value = 'file://%s' % (
urllib.pathname2url(
os.path.abspath(os.path.expanduser(value))),)
if opt_str == '--download-base' and not value.endswith('/'):
# Download base needs a trailing slash to make the world happy.
value += '/'
else:
value = None
name = opt_str[2:].replace('-', '_')
setattr(parser.values, name, value)
usage = '''\
[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
Bootstraps a buildout-based project.
Simply run this script in a directory containing a buildout.cfg, using the
Python that you want bin/buildout to use.
Note that by using --setup-source and --download-base to point to
local resources, you can keep this script from going over the network.
'''
parser = OptionParser(usage=usage)
parser.add_option("-v", "--version", dest="version",
help="use a specific zc.buildout version")
parser.add_option("-d", "--distribute",
action="store_true", dest="use_distribute", default=False,
help="Use Distribute rather than Setuptools.")
parser.add_option("--setup-source", action="callback", dest="setup_source",
callback=normalize_to_url, nargs=1, type="string",
help=("Specify a URL or file location for the setup file. "
"If you use Setuptools, this will default to " +
setuptools_source + "; if you use Distribute, this "
"will default to " + distribute_source +"."))
parser.add_option("--download-base", action="callback", dest="download_base",
callback=normalize_to_url, nargs=1, type="string",
help=("Specify a URL or directory for downloading "
"zc.buildout and either Setuptools or Distribute. "
"Defaults to PyPI."))
parser.add_option("--eggs",
help=("Specify a directory for storing eggs. Defaults to "
"a temporary directory that is deleted when the "
"bootstrap script completes."))
parser.add_option("-t", "--accept-buildout-test-releases",
dest='accept_buildout_test_releases',
action="store_true", default=False,
help=("Normally, if you do not specify a --version, the "
"bootstrap script and buildout gets the newest "
"*final* versions of zc.buildout and its recipes and "
"extensions for you. If you use this flag, "
"bootstrap and buildout will get the newest releases "
"even if they are alphas or betas."))
parser.add_option("-c", None, action="store", dest="config_file",
help=("Specify the path to the buildout configuration "
"file to be used."))
options, args = parser.parse_args()
# if -c was provided, we push it back into args for buildout's main function
if options.config_file is not None:
args += ['-c', options.config_file]
if options.eggs:
eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
else:
eggs_dir = tempfile.mkdtemp()
if options.setup_source is None:
if options.use_distribute:
options.setup_source = distribute_source
else:
options.setup_source = setuptools_source
if options.accept_buildout_test_releases:
args.append('buildout:accept-buildout-test-releases=true')
args.append('bootstrap')
try:
import pkg_resources
import setuptools # A flag. Sometimes pkg_resources is installed alone.
if not hasattr(pkg_resources, '_distribute'):
raise ImportError
except ImportError:
ez_code = urllib2.urlopen(
options.setup_source).read().replace('\r\n', '\n')
ez = {}
exec ez_code in ez
setup_args = dict(to_dir=eggs_dir, download_delay=0)
if options.download_base:
setup_args['download_base'] = options.download_base
if options.use_distribute:
setup_args['no_fake'] = True
ez['use_setuptools'](**setup_args)
if 'pkg_resources' in sys.modules:
reload(sys.modules['pkg_resources'])
import pkg_resources
# This does not (always?) update the default working set. We will
# do it.
for path in sys.path:
if path not in pkg_resources.working_set.entries:
pkg_resources.working_set.add_entry(path)
cmd = [quote(sys.executable),
'-c',
quote('from setuptools.command.easy_install import main; main()'),
'-mqNxd',
quote(eggs_dir)]
if not has_broken_dash_S:
cmd.insert(1, '-S')
find_links = options.download_base
if not find_links:
find_links = os.environ.get('bootstrap-testing-find-links')
if find_links:
cmd.extend(['-f', quote(find_links)])
if options.use_distribute:
setup_requirement = 'distribute'
else:
setup_requirement = 'setuptools'
ws = pkg_resources.working_set
setup_requirement_path = ws.find(
pkg_resources.Requirement.parse(setup_requirement)).location
env = dict(
os.environ,
PYTHONPATH=setup_requirement_path)
requirement = 'zc.buildout'
version = options.version
if version is None and not options.accept_buildout_test_releases:
# Figure out the most recent final version of zc.buildout.
import setuptools.package_index
_final_parts = '*final-', '*final'
def _final_version(parsed_version):
for part in parsed_version:
if (part[:1] == '*') and (part not in _final_parts):
return False
return True
index = setuptools.package_index.PackageIndex(
search_path=[setup_requirement_path])
if find_links:
index.add_find_links((find_links,))
req = pkg_resources.Requirement.parse(requirement)
if index.obtain(req) is not None:
best = []
bestv = None
for dist in index[req.project_name]:
distv = dist.parsed_version
if _final_version(distv):
if bestv is None or distv > bestv:
best = [dist]
bestv = distv
elif distv == bestv:
best.append(dist)
if best:
best.sort()
version = best[-1].version
if version:
requirement = '=='.join((requirement, version))
cmd.append(requirement)
if is_jython:
import subprocess
exitcode = subprocess.Popen(cmd, env=env).wait()
else: # Windows prefers this, apparently; otherwise we would prefer subprocess
exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
if exitcode != 0:
sys.stdout.flush()
sys.stderr.flush()
print ("An error occurred when trying to install zc.buildout. "
"Look above this message for any errors that "
"were output by easy_install.")
sys.exit(exitcode)
ws.add_entry(eggs_dir)
ws.require(requirement)
import zc.buildout.buildout
zc.buildout.buildout.main(args)
if not options.eggs: # clean up temporary egg directory
shutil.rmtree(eggs_dir)
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/proxy/ 0000775 0000000 0000000 00000000000 11602033661 0024735 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/proxy/__init__.py 0000664 0000000 0000000 00000006467 11602033661 0027063 0 ustar 00root root 0000000 0000000 import os
import sys
from optparse import OptionParser, Option
import logging
import logging.handlers
import ConfigParser
class Parser(OptionParser):
"""
Parse all arguments.
"""
def __init__(self, usage=None, version=None):
"""
Initialize all options possibles.
"""
OptionParser.__init__(self, usage=usage, version=version,
option_list=[
Option("-l", "--log_file",
help="The path to the log file used by the script.",
type=str),
Option("-v", "--verbose",
default=False,
action="store_true",
help="Verbose output."),
Option("-c", "--console",
default=False,
action="store_true",
help="Console output."),
Option("-u", "--database-uri",
type=str,
help="URI for sqlite database"),
])
def check_args(self):
"""
Check arguments
"""
(options, args) = self.parse_args()
if len(args) != 1:
self.error("Incorrect number of arguments")
return options, args[0]
class Config:
def setConfig(self, option_dict, configuration_file_path):
"""
Set options given by parameters.
"""
# Set options parameters
for option, value in option_dict.__dict__.items():
setattr(self, option, value)
# Load configuration file
configuration_parser = ConfigParser.SafeConfigParser()
configuration_parser.read(configuration_file_path)
# Merges the arguments and configuration
for section in ("slapproxy", "slapos"):
configuration_dict = dict(configuration_parser.items(section))
for key in configuration_dict:
if not getattr(self, key, None):
setattr(self, key, configuration_dict[key])
# set up logging
self.logger = logging.getLogger("slapproxy")
self.logger.setLevel(logging.INFO)
if self.console:
self.logger.addHandler(logging.StreamHandler())
if not self.database_uri:
raise ValueError('database-uri is required.')
if self.log_file:
if not os.path.isdir(os.path.dirname(self.log_file)):
# fallback to console only if directory for logs does not exists and
# continue to run
raise ValueError('Please create directory %r to store %r log file' % (
os.path.dirname(self.log_file), self.log_file))
else:
file_handler = logging.FileHandler(self.log_file)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
self.logger.addHandler(file_handler)
self.logger.info('Configured logging to file %r' % self.log_file)
self.logger.info("Started.")
if self.verbose:
self.logger.setLevel(logging.DEBUG)
self.logger.debug("Verbose mode enabled.")
def run(config):
from views import app
app.config['computer_id'] = config.computer_id
app.config['DATABASE_URI'] = config.database_uri
app.run(host=config.host, port=int(config.port), debug=True)
def main():
"Run default configuration."
usage = "usage: %s [options] CONFIGURATION_FILE" % sys.argv[0]
try:
# Parse arguments
config = Config()
config.setConfig(*Parser(usage=usage).check_args())
run(config)
return_code = 0
except SystemExit, err:
# Catch exception raise by optparse
return_code = err
sys.exit(return_code)
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/proxy/schema.sql 0000664 0000000 0000000 00000001270 11602033661 0026716 0 ustar 00root root 0000000 0000000 --version:7
CREATE TABLE IF NOT EXISTS software%(version)s (url VARCHAR(255) UNIQUE);
CREATE TABLE IF NOT EXISTS computer%(version)s (
address VARCHAR(255),
netmask VARCHAR(255),
CONSTRAINT uniq PRIMARY KEY (address, netmask));
CREATE TABLE IF NOT EXISTS partition%(version)s (
reference VARCHAR(255) UNIQUE,
slap_state VARCHAR(255) DEFAULT 'free',
software_release VARCHAR(255),
xml TEXT,
connection_xml TEXT,
software_type VARCHAR(255),
partition_reference VARCHAR(255),
requested_by VARCHAR(255)
);
CREATE TABLE IF NOT EXISTS partition_network%(version)s (
partition_reference VARCHAR(255),
reference VARCHAR(255),
address VARCHAR(255),
netmask VARCHAR(255)
);
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/proxy/views.py 0000664 0000000 0000000 00000027072 11602033661 0026454 0 ustar 00root root 0000000 0000000 from flask import g, Flask, request, abort
import xml_marshaller
from lxml import etree
from slapos.slap.slap import Computer, ComputerPartition, SoftwareRelease, SoftwareInstance
import sqlite3
app = Flask(__name__)
DB_VERSION = app.open_resource('schema.sql').readline().strip().split(':')[1]
class UnauthorizedError(Exception):
pass
def xml2dict(xml):
result_dict = {}
if xml is not None and xml != '':
tree = etree.fromstring(xml.encode('utf-8'))
for element in tree.iter(tag=etree.Element):
if element.tag == 'parameter':
key = element.get('id')
value = result_dict.get(key, None)
if value is not None:
value = value + ' ' + element.text
else:
value = element.text
result_dict[key] = value
return result_dict
def dict2xml(dictionnary):
instance = etree.Element('instance')
for parameter_id, parameter_value in dictionnary.iteritems():
# cast everything to string
parameter_value = str(parameter_value)
etree.SubElement(instance, "parameter",
attrib={'id':parameter_id}).text = parameter_value
return etree.tostring(instance, pretty_print=True,
xml_declaration=True, encoding='utf-8')
def partitiondict2partition(partition):
slap_partition = ComputerPartition(app.config['computer_id'],
partition['reference'])
slap_partition._requested_state = 'started'
if partition['software_release']:
slap_partition._need_modification = 1
else:
slap_partition._need_modification = 0
slap_partition._parameter_dict = xml2dict(partition['xml'])
address_list = []
for address in execute_db('partition_network', 'SELECT * FROM %s WHERE partition_reference=?', [partition['reference']]):
address_list.append((address['reference'], address['address']))
slap_partition._parameter_dict['ip_list'] = address_list
slap_partition._parameter_dict['slap_software_type'] = partition['software_type']
slap_partition._connection_dict = xml2dict(partition['connection_xml'])
slap_partition._software_release_document = SoftwareRelease(
software_release=partition['software_release'],
computer_guid=app.config['computer_id'])
return slap_partition
def execute_db(table, query, args=(), one=False):
try:
cur = g.db.execute(query % (table + DB_VERSION,), args)
except:
app.logger.error('There was some issue during processing query %r on table %r with args %r' % (query, table, args))
raise
rv = [dict((cur.description[idx][0], value)
for idx, value in enumerate(row)) for row in cur.fetchall()]
return (rv[0] if rv else None) if one else rv
def connect_db():
return sqlite3.connect(app.config['DATABASE_URI'])
@app.before_request
def before_request():
g.db = connect_db()
schema = app.open_resource('schema.sql')
schema = schema.read() % dict(version = DB_VERSION)
g.db.cursor().executescript(schema)
g.db.commit()
@app.after_request
def after_request(response):
g.db.commit()
g.db.close()
return response
@app.route('/getComputerInformation', methods=['GET'])
def getComputerInformation():
computer_id = request.args['computer_id']
if app.config['computer_id'] == computer_id:
slap_computer = Computer(computer_id)
slap_computer._software_release_list = []
for sr in execute_db('software', 'select * from %s'):
slap_computer._software_release_list.append(SoftwareRelease(
software_release=sr['url'], computer_guid=computer_id))
slap_computer._computer_partition_list = []
for partition in execute_db('partition', 'SELECT * FROM %s'):
slap_computer._computer_partition_list.append(partitiondict2partition(
partition))
return xml_marshaller.xml_marshaller.dumps(slap_computer)
else:
raise UnauthorizedError, "Only accept request for: %s" % \
app.config['computer_id']
@app.route('/setComputerPartitionConnectionXml', methods=['POST'])
def setComputerPartitionConnectionXml():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
connection_xml = request.form['connection_xml']
connection_dict = xml_marshaller.xml_marshaller.loads(
connection_xml.encode())
connection_xml = dict2xml(connection_dict)
query = 'UPDATE %s SET connection_xml=? WHERE reference=?'
argument_list = [connection_xml, computer_partition_id.encode()]
execute_db('partition', query, argument_list)
return 'done'
@app.route('/buildingSoftwareRelease', methods=['POST'])
def buildingSoftwareRelease():
return 'Ignored'
@app.route('/availableSoftwareRelease', methods=['POST'])
def availableSoftwareRelease():
computer_id = request.form['computer_id']
url = request.form['url']
return 'Ignored'
@app.route('/softwareReleaseError', methods=['POST'])
def softwareReleaseError():
return 'Ignored'
@app.route('/buildingComputerPartition', methods=['POST'])
def buildingComputerPartition():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
return 'Ignored'
@app.route('/availableComputerPartition', methods=['POST'])
def availableComputerPartition():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
return 'Ignored'
@app.route('/softwareInstanceError', methods=['POST'])
def softwareInstanceError():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
error_log = request.form['error_log']
return 'Ignored'
@app.route('/startedComputerPartition', methods=['POST'])
def startedComputerPartition():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
return 'Ignored'
@app.route('/stoppedComputerPartition', methods=['POST'])
def stoppedComputerPartition():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
return 'Ignored'
@app.route('/destroyedComputerPartition', methods=['POST'])
def destroyedComputerPartition():
computer_id = request.form['computer_id']
computer_partition_id = request.form['computer_partition_id']
return 'Ignored'
@app.route('/requestComputerPartition', methods=['POST'])
def requestComputerPartition():
software_release = request.form['software_release'].encode()
# some supported parameters
software_type = request.form.get('software_type', 'RootSoftwareInstance'
).encode()
partition_reference = request.form.get('partition_reference', '').encode()
partition_id = request.form.get('computer_partition_id', '').encode()
partition_parameter_kw = request.form.get('partition_parameter_xml', None)
if partition_parameter_kw:
partition_parameter_kw = xml_marshaller.xml_marshaller.loads(
partition_parameter_kw.encode())
else:
partition_parameter_kw = {}
instance_xml = dict2xml(partition_parameter_kw)
args = []
a = args.append
q = 'SELECT * FROM %s WHERE software_release=?'
a(software_release)
if software_type:
q += ' AND software_type=?'
a(software_type)
if partition_reference:
q += ' AND partition_reference=?'
a(partition_reference)
if partition_id:
q += ' AND requested_by=?'
a(partition_id)
partition = execute_db('partition', q, args, one=True)
if partition is None:
partition = execute_db('partition',
'SELECT * FROM %s WHERE slap_state="free"', (), one=True)
if partition is None:
app.logger.warning('No more free computer partition')
abort(408)
args = []
a = args.append
q = 'UPDATE %s SET software_release=?, slap_state="busy"'
a(software_release)
if software_type:
q += ' ,software_type=?'
a(software_type)
if partition_reference:
q += ' ,partition_reference=?'
a(partition_reference)
if partition_id:
q += ' ,requested_by=?'
a(partition_id)
if instance_xml:
q+= ' ,xml=?'
a(instance_xml)
q += ' WHERE reference=?'
a(partition['reference'].encode())
execute_db('partition', q, args)
args = []
partition = execute_db('partition', 'SELECT * FROM %s WHERE reference=?',
[partition['reference'].encode()], one=True)
address_list = []
for address in execute_db('partition_network', 'SELECT * FROM %s WHERE partition_reference=?', [partition['reference']]):
address_list.append((address['reference'], address['address']))
return xml_marshaller.xml_marshaller.dumps(SoftwareInstance(**dict(
xml=partition['xml'],
connection_xml=partition['connection_xml'],
slap_computer_id=app.config['computer_id'],
slap_computer_partition_id=partition['reference'],
slap_software_release_url=partition['software_release'],
slap_server_url='slap_server_url',
slap_software_type=partition['software_type'],
slave_id_list=[],
ip_list=address_list
)))
abort(408)
computer_id = request.form.get('computer_id')
computer_partition_id = request.form.get('computer_partition_id')
software_type = request.form.get('software_type')
partition_reference = request.form.get('partition_reference')
shared_xml = request.form.get('shared_xml')
partition_parameter_xml = request.form.get('partition_parameter_xml')
filter_xml = request.form.get('filter_xml')
raise NotImplementedError
@app.route('/useComputer', methods=['POST'])
def useComputer():
computer_id = request.form['computer_id']
use_string = request.form['use_string']
return 'Ignored'
@app.route('/loadComputerConfigurationFromXML', methods=['POST'])
def loadComputerConfigurationFromXML():
xml = request.form['xml']
computer_dict = xml_marshaller.xml_marshaller.loads(str(xml))
if app.config['computer_id'] == computer_dict['reference']:
args = []
a = args.append
execute_db('computer', 'INSERT OR REPLACE INTO %s values(:address, :netmask)',
computer_dict)
for partition in computer_dict['partition_list']:
execute_db('partition', 'INSERT OR IGNORE INTO %s (reference) values(:reference)', partition)
execute_db('partition_network', 'DELETE FROM %s WHERE partition_reference = ?', [partition['reference']])
for address in partition['address_list']:
address['reference'] = partition['tap']['name']
address['partition_reference'] = partition['reference']
execute_db('partition_network', 'INSERT OR REPLACE INTO %s (reference, partition_reference, address, netmask) values(:reference, :partition_reference, :addr, :netmask)', address)
return 'done'
else:
raise UnauthorizedError, "Only accept request for: %s" % \
app.config['computer_id']
@app.route('/registerComputerPartition', methods=['GET'])
def registerComputerPartition():
computer_reference = request.args['computer_reference']
computer_partition_reference = request.args['computer_partition_reference']
if app.config['computer_id'] == computer_reference:
partition = execute_db('partition', 'SELECT * FROM %s WHERE reference=?',
[computer_partition_reference.encode()], one=True)
if partition is None:
raise UnauthorizedError
return xml_marshaller.xml_marshaller.dumps(
partitiondict2partition(partition))
else:
raise UnauthorizedError, "Only accept request for: %s" % \
app.config['computer_id']
@app.route('/supplySupply', methods=['POST'])
def supplySupply():
url = request.form['url']
computer_id = request.form['computer_id']
if app.config['computer_id'] == computer_id:
execute_db('software', 'INSERT OR REPLACE INTO %s VALUES(?)', [url])
else:
raise UnauthorizedError, "Only accept request for: %s" % \
app.config['computer_id']
return '%r added' % url
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/ 0000775 0000000 0000000 00000000000 11602033661 0024513 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/__init__.py 0000664 0000000 0000000 00000003007 11602033661 0026624 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import sys
if sys.version_info < (2, 6):
import warnings
warnings.warn('Used python version (%s) is old and have problems with'
' IPv6 connections' % '.'.join([str(q) for q in sys.version_info[:3]]))
from slap import *
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/doc/ 0000775 0000000 0000000 00000000000 11602033661 0025260 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/doc/software_instance.xsd 0000664 0000000 0000000 00000001233 11602033661 0031515 0 ustar 00root root 0000000 0000000
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/interface/ 0000775 0000000 0000000 00000000000 11602033661 0026453 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/interface/__init__.py 0000664 0000000 0000000 00000002442 11602033661 0030566 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/interface/slap.py 0000664 0000000 0000000 00000021276 11602033661 0027774 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from zope.interface import Interface
"""
Note: all strings accepted/returned by the slap library are encoded in UTF-8.
"""
class IException(Interface):
"""
Classes which implement IException are used to report errors.
"""
class INotFoundError(IException):
"""
Classes which implement INotFoundError are used to report missing
informations on the slap server.
"""
class IUnauthorized(IException):
"""
Classes which implement IUnauthorized are used to report missing
permissions on the slap server.
"""
class IBuildoutController(Interface):
"""
Classes which implement IBuildoutController can report the buildout run
status to the slapgrid server.
"""
def available():
"""
Notify (to the slapgrid server) that the software instance is
available.
"""
def building():
"""
Notify (to the slapgrid server) that the buildout is not
available and under creation.
"""
def error(error_log):
"""
Notify (to the slapgrid server) that the buildout is not available
and reports an error.
error_log -- a text describing the error
It can be a traceback for example.
"""
class ISoftwareRelease(IBuildoutController):
"""
Software release interface specification
"""
def getURI():
"""
Returns a string representing the uri of the software release.
"""
class IComputerPartition(IBuildoutController):
"""
Computer Partition interface specification
Classes which implement IComputerPartition can propagate the computer
partition state to the SLAPGRID server and request new computer partition
creation.
"""
def request(software_release, software_type, partition_reference,
shared=False, partition_parameter_kw=None, filter_kw=None):
"""
Request software release instanciation to slapgrid server.
Returns a new computer partition document, where this sofware release will
be installed.
software_release -- uri of the software release
which has to be instanciated
software_type -- type of component provided by software_release
partition_reference -- local reference of the instance used by the recipe
to identify the instances.
shared -- boolean to use a shared service
partition_parameter_kw -- dictionary of parameter used to fill the
parameter dict of newly created partition.
filter_kw -- dictionary of filtering parameter to select the requested
computer partition.
computer_reference - computer of the requested partition
partition_type - virtio, slave, full, limited
port - port provided by the requested partition
Example:
request('http://example.com/toto/titi', 'mysql_1')
"""
def stopped():
"""
Notify (to the slapgrid server) that the software instance is
available and stopped.
"""
def started():
"""
Notify (to the slapgrid server) that the software instance is
available and started.
"""
def getId():
"""
Returns a string representing the identifier of the computer partition
inside the slapgrid server.
"""
def getState():
"""
Returns a string representing the expected state of the computer partition.
The result can be: started, stopped, destroyed
"""
def getSoftwareRelease():
"""
Returns the software release associate to the computer partition.
Raise an INotFoundError if no software release is associated.
"""
def getInstanceParameterDict():
"""
Returns a dictionary of instance parameters.
The contained values can be used to fill the software instanciation
profile.
"""
def setUsage(usage_log):
"""
Associate a usage log to the computer partition.
This method does not report the usage to the slapgrid server. See
IComputer.report.
usage_log -- a text describing the computer partition usage.
It can be an XML for example.
"""
class IComputer(Interface):
"""
Computer interface specification
Classes which implement IComputer can fetch informations from the slapgrid
server to know which Software Releases and Software Instances have to be
installed.
"""
def getSoftwareReleaseList():
"""
Returns the list of software release which has to be supplied by the
computer.
Raise an INotFoundError if computer_guid doesn't exist.
"""
def getComputerPartitionList():
"""
Returns the list of configured computer partitions associated to this
computer.
Raise an INotFoundError if computer_guid doesn't exist.
"""
def reportUsage(computer_partition_list):
"""
Report the computer usage to the slapgrid server.
IComputerPartition.setUsage has to be called on each computer partition to
define each usage.
computer_partition_list -- a list of computer partition for which the usage
needs to be reported.
"""
class IOpenOrder(Interface):
"""
Open Order interface specification
Classes which implement Open Order describe which kind of software instances
is requested by a given client.
"""
def request(software_release):
"""
Request the instanciation of a given software release to the slapgrid
server.
Returns a new computer partition document, where this software release will
be installed.
software_release -- uri of the software release
which has to be instanciated
"""
class ISupply(Interface):
"""
Supply interface specification
Classes which implement Supply describe which kind of software releases
a given client is ready to supply.
"""
def supply(software_release, computer_guid=None):
"""
Tell that given client is ready to supply given sofware release
software_release -- uri of the software release
which has to be instanciated
computer_guid -- the identifier of the computer inside the slapgrid
server.
"""
class slap(Interface):
"""
Initialise slap connection to the slapgrid server
Slapgrid server URL is defined during the slap library installation,
as recipes should not use another server.
"""
def initializeConnection(slapgrid_uri, authentification_key=None):
"""
Initialize the connection parameters to the slapgrid servers.
slapgrid_uri -- uri the slapgrid server connector
authentification_key -- string the authentificate the agent.
Example: https://slapos.server/slap_interface
"""
def registerComputer(computer_guid):
"""
Instanciate a computer in the slap library.
computer_guid -- the identifier of the computer inside the slapgrid server.
"""
def registerComputerPartition(computer_guid, partition_id):
"""
Instanciate a computer partition in the slap library.
computer_guid -- the identifier of the computer inside the slapgrid server.
partition_id -- the identifier of the computer partition inside the
slapgrid server.
Raise an INotFoundError if computer_guid doesn't exist.
"""
def registerSoftwareRelease(software_release):
"""
Instanciate a software release in the slap library.
software_release -- uri of the software release definition
"""
def registerOpenOrder():
"""
Instanciate an open order in the slap library.
"""
def registerSupply():
"""
Instanciate a supply in the slap library.
"""
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/slap.py 0000664 0000000 0000000 00000050330 11602033661 0026025 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
__all__ = ["slap", "ComputerPartition", "Computer", "SoftwareRelease",
"Supply", "OpenOrder", "NotFoundError", "Unauthorized",
"ResourceNotReady"]
from interface import slap as interface
from xml_marshaller import xml_marshaller
import httplib
import socket
import ssl
import urllib
import urlparse
import zope.interface
"""
Simple, easy to (un)marshall classes for slap client/server communication
"""
# httplib.HTTPSConnection with key verification
class HTTPSConnectionCA(httplib.HTTPSConnection):
"""Patched version of HTTPSConnection which verifies server certificate"""
def __init__(self, *args, **kwargs):
self.ca_file = kwargs.pop('ca_file')
if self.ca_file is None:
raise ValueError('ca_file is required argument.')
httplib.HTTPSConnection.__init__(self, *args, **kwargs)
def connect(self):
"Connect to a host on a given (SSL) port and verify its certificate."
sock = socket.create_connection((self.host, self.port),
self.timeout, self.source_address)
if self._tunnel_host:
self.sock = sock
self._tunnel()
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file,
ca_certs=self.ca_file, cert_reqs=ssl.CERT_REQUIRED)
class SlapDocument:
pass
class SoftwareRelease(SlapDocument):
"""
Contains Software Release information
"""
zope.interface.implements(interface.ISoftwareRelease)
def __init__(self, software_release=None, computer_guid=None, **kw):
"""
Makes easy initialisation of class parameters
XXX **kw args only kept for compatibility
"""
self._software_instance_list = []
if software_release is not None:
software_release = software_release.encode('UTF-8')
self._software_release = software_release
self._computer_guid = computer_guid
def __getinitargs__(self):
return (self._software_release, self._computer_guid, )
def error(self, error_log):
# Does not follow interface
self._connection_helper.POST('/softwareReleaseError', {
'url': self._software_release,
'computer_id' : self._computer_guid,
'error_log': error_log})
def getURI(self):
return self._software_release
def available(self):
self._connection_helper.POST('/availableSoftwareRelease', {
'url': self._software_release,
'computer_id': self._computer_guid})
def building(self):
self._connection_helper.POST('/buildingSoftwareRelease', {
'url': self._software_release,
'computer_id': self._computer_guid})
# XXX What is this SoftwareInstance class?
class SoftwareInstance(SlapDocument):
"""
Contains Software Instance information
"""
def __init__(self, **kwargs):
"""
Makes easy initialisation of class parameters
"""
for k, v in kwargs.iteritems():
setattr(self, k, v)
"""Exposed exceptions"""
# XXX Why do we need to expose exceptions?
class ResourceNotReady(Exception):
pass
class ServerError(Exception):
pass
class NotFoundError(Exception):
zope.interface.implements(interface.INotFoundError)
class Unauthorized(Exception):
zope.interface.implements(interface.IUnauthorized)
class Supply(SlapDocument):
zope.interface.implements(interface.ISupply)
def supply(self, software_release, computer_guid=None):
self._connection_helper.POST('/supplySupply', {
'url': software_release,
'computer_id': computer_guid})
class OpenOrder(SlapDocument):
zope.interface.implements(interface.IOpenOrder)
def request(self, software_release, partition_reference,
partition_parameter_kw=None, software_type=None, filter_kw=None,
state=None):
if partition_parameter_kw is None:
partition_parameter_kw = {}
if filter_kw is None:
filter_kw = {}
request_dict = {
'software_release': software_release,
'partition_reference': partition_reference,
'partition_parameter_xml': xml_marshaller.dumps(partition_parameter_kw),
'filter_xml': xml_marshaller.dumps(filter_kw),
'state': xml_marshaller.dumps(state),
}
if software_type is not None:
request_dict['software_type'] = software_type
self._connection_helper.POST('/requestComputerPartition', request_dict)
xml = self._connection_helper.response.read()
software_instance = xml_marshaller.loads(xml)
computer_partition = ComputerPartition(
software_instance.slap_computer_id.encode('UTF-8'),
software_instance.slap_computer_partition_id.encode('UTF-8'))
return computer_partition
def _syncComputerInformation(func):
"""
Synchronize computer object with server information
"""
def decorated(self, *args, **kw):
computer = self._connection_helper.getComputerInformation(self._computer_id)
for key, value in computer.__dict__.items():
if isinstance(value, unicode):
# convert unicode to utf-8
setattr(self, key, value.encode('utf-8'))
else:
setattr(self, key, value)
return func(self, *args, **kw)
return decorated
class Computer(SlapDocument):
zope.interface.implements(interface.IComputer)
def __init__(self, computer_id):
self._computer_id = computer_id
def __getinitargs__(self):
return (self._computer_id, )
@_syncComputerInformation
def getSoftwareReleaseList(self):
"""
Returns the list of software release which has to be supplied by the
computer.
Raise an INotFoundError if computer_guid doesn't exist.
"""
return self._software_release_list
@_syncComputerInformation
def getComputerPartitionList(self):
return [x for x in self._computer_partition_list if x._need_modification]
def reportUsage(self, computer_partition_list):
if computer_partition_list == []:
return
computer = Computer(self._computer_id)
computer.computer_partition_usage_list = computer_partition_list
marshalled_slap_usage = xml_marshaller.dumps(computer)
self._connection_helper.POST('/useComputer', {
'computer_id': self._computer_id,
'use_string': marshalled_slap_usage})
def updateConfiguration(self, xml):
self._connection_helper.POST(
'/loadComputerConfigurationFromXML', { 'xml' : xml })
return self._connection_helper.response.read()
def _syncComputerPartitionInformation(func):
"""
Synchronize computer partition object with server information
"""
def decorated(self, *args, **kw):
computer = self._connection_helper.getComputerInformation(self._computer_id)
found_computer_partition = None
for computer_partition in computer._computer_partition_list:
if computer_partition.getId() == self.getId():
found_computer_partition = computer_partition
break
if found_computer_partition is None:
raise NotFoundError("No software release information for partition %s" %
self.getId())
else:
for key, value in found_computer_partition.__dict__.items():
if isinstance(value, unicode):
# convert unicode to utf-8
setattr(self, key, value.encode('utf-8'))
if isinstance(value, dict):
new_dict = {}
for ink, inv in value.iteritems():
if isinstance(inv, (list, tuple)):
new_inv = []
for elt in inv:
if isinstance(elt, (list, tuple)):
new_inv.append([x.encode('utf-8') for x in elt])
else:
new_inv.append(elt.encode('utf-8'))
new_dict[ink.encode('utf-8')] = new_inv
elif inv is None:
new_dict[ink.encode('utf-8')] = None
else:
new_dict[ink.encode('utf-8')] = inv.encode('utf-8')
setattr(self, key, new_dict)
else:
setattr(self, key, value)
return func(self, *args, **kw)
return decorated
class ComputerPartition(SlapDocument):
zope.interface.implements(interface.IComputerPartition)
def __init__(self, computer_id, partition_id):
self._computer_id = computer_id
self._partition_id = partition_id
def __getinitargs__(self):
return (self._computer_id, self._partition_id, )
# XXX: As request is decorated with _syncComputerPartitionInformation it
# will raise ResourceNotReady really early -- just after requesting,
# and not when try to access to real partition is required.
# To have later raising (like in case of calling methods), the way how
# Computer Partition data are fetch from server shall be delayed
@_syncComputerPartitionInformation
def request(self, software_release, software_type, partition_reference,
shared=False, partition_parameter_kw=None, filter_kw=None,
state=None):
if partition_parameter_kw is None:
partition_parameter_kw = {}
elif not isinstance(partition_parameter_kw, dict):
raise ValueError("Unexpected type of partition_parameter_kw '%s'" % \
partition_parameter_kw)
if filter_kw is None:
filter_kw = {}
elif not isinstance(filter_kw, dict):
raise ValueError("Unexpected type of filter_kw '%s'" % \
filter_kw)
request_dict = { 'computer_id': self._computer_id,
'computer_partition_id': self._partition_id,
'software_release': software_release,
'software_type': software_type,
'partition_reference': partition_reference,
'shared_xml': xml_marshaller.dumps(shared),
'partition_parameter_xml': xml_marshaller.dumps(
partition_parameter_kw),
'filter_xml': xml_marshaller.dumps(filter_kw),
'state': xml_marshaller.dumps(state),
}
self._connection_helper.POST('/requestComputerPartition', request_dict)
xml = self._connection_helper.response.read()
software_instance = xml_marshaller.loads(xml)
computer_partition = ComputerPartition(
software_instance.slap_computer_id.encode('UTF-8'),
software_instance.slap_computer_partition_id.encode('UTF-8'))
return computer_partition
def building(self):
self._connection_helper.POST('/buildingComputerPartition', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id})
def available(self):
self._connection_helper.POST('/availableComputerPartition', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id})
def destroyed(self):
self._connection_helper.POST('/destroyedComputerPartition', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id,
})
def started(self):
self._connection_helper.POST('/startedComputerPartition', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id,
})
def stopped(self):
self._connection_helper.POST('/stoppedComputerPartition', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id,
})
def error(self, error_log):
self._connection_helper.POST('/softwareInstanceError', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id,
'error_log': error_log})
def getId(self):
return self._partition_id
@_syncComputerPartitionInformation
def getState(self):
return self._requested_state
@_syncComputerPartitionInformation
def getInstanceParameterDict(self):
return getattr(self, '_parameter_dict', None) or {}
@_syncComputerPartitionInformation
def getSoftwareRelease(self):
"""
Returns the software release associate to the computer partition.
"""
if self._software_release_document is None:
raise NotFoundError("No software release information for partition %s" %
self.getId())
else:
return self._software_release_document
def setConnectionDict(self, connection_dict):
self._connection_helper.POST('/setComputerPartitionConnectionXml', {
'computer_id': self._computer_id,
'computer_partition_id': self._partition_id,
'connection_xml': xml_marshaller.dumps(connection_dict)})
@_syncComputerPartitionInformation
def getConnectionParameter(self, key):
connection_dict = getattr(self, '_connection_dict', None) or {}
if key in connection_dict:
return connection_dict[key]
else:
raise NotFoundError("%s not found" % key)
def setUsage(self, usage_log):
# XXX: this implementation has not been reviewed
self.usage = usage_log
def getCertificate(self):
self._connection_helper.GET(
'/getComputerPartitionCertificate?computer_id=%s&'
'computer_partition_id=%s' % (self._computer_id, self._partition_id))
return xml_marshaller.loads(self._connection_helper.response.read())
# def lazyMethod(func):
# """
# Return a function which stores a computed value in an instance
# at the first call.
# """
# key = '_cache_' + str(id(func))
# def decorated(self, *args, **kw):
# try:
# return getattr(self, key)
# except AttributeError:
# result = func(self, *args, **kw)
# setattr(self, key, result)
# return result
# return decorated
class ConnectionHelper:
error_message_connect_fail = "Couldn't connect to the server. Please double \
check given master-url argument, and make sure that IPv6 is enabled on \
your machine and that the server is available. The original error was:"
def __init__(self, connection_wrapper, host, path, key_file=None,
cert_file=None, master_ca_file=None, timeout=None):
self.connection_wrapper = connection_wrapper
self.host = host
self.path = path
self.key_file = key_file
self.cert_file = cert_file
self.master_ca_file = master_ca_file
self.timeout = timeout
def getComputerInformation(self, computer_id):
self.GET('/getComputerInformation?computer_id=%s' % computer_id)
return xml_marshaller.loads(self.response.read())
def connect(self):
connection_dict = dict(
host=self.host)
if self.key_file and self.cert_file:
connection_dict.update(
key_file=self.key_file,
cert_file=self.cert_file)
if self.master_ca_file is not None:
connection_dict.update(ca_file=self.master_ca_file)
self.connection = self.connection_wrapper(**connection_dict)
def GET(self, path):
try:
default_timeout = socket.getdefaulttimeout()
socket.setdefaulttimeout(self.timeout)
try:
self.connect()
self.connection.request('GET', self.path + path)
self.response = self.connection.getresponse()
except socket.error, e:
raise socket.error(self.error_message_connect_fail + str(e))
# check self.response.status and raise exception early
if self.response.status == httplib.REQUEST_TIMEOUT:
# resource is not ready
raise ResourceNotReady(path)
elif self.response.status == httplib.NOT_FOUND:
raise NotFoundError(path)
elif self.response.status == httplib.FORBIDDEN:
raise Unauthorized(path)
elif self.response.status != httplib.OK:
message = 'Server responded with wrong code %s with %s' % \
(self.response.status, path)
raise ServerError(message)
finally:
socket.setdefaulttimeout(default_timeout)
def POST(self, path, parameter_dict,
content_type="application/x-www-form-urlencoded"):
try:
default_timeout = socket.getdefaulttimeout()
socket.setdefaulttimeout(self.timeout)
try:
self.connect()
header_dict = {'Content-type': content_type}
self.connection.request("POST", self.path + path,
urllib.urlencode(parameter_dict), header_dict)
except socket.error, e:
raise socket.error(self.error_message_connect_fail + str(e))
self.response = self.connection.getresponse()
# check self.response.status and raise exception early
if self.response.status == httplib.REQUEST_TIMEOUT:
# resource is not ready
raise ResourceNotReady("%s - %s" % (path, parameter_dict))
elif self.response.status == httplib.NOT_FOUND:
raise NotFoundError("%s - %s" % (path, parameter_dict))
elif self.response.status == httplib.FORBIDDEN:
raise Unauthorized("%s - %s" % (path, parameter_dict))
elif self.response.status != httplib.OK:
message = 'Server responded with wrong code %s with %s' % \
(self.response.status, path)
raise ServerError(message)
finally:
socket.setdefaulttimeout(default_timeout)
class slap:
zope.interface.implements(interface.slap)
def initializeConnection(self, slapgrid_uri, key_file=None, cert_file=None,
master_ca_file=None, timeout=60):
self._initialiseConnectionHelper(slapgrid_uri, key_file, cert_file,
master_ca_file, timeout)
def _initialiseConnectionHelper(self, slapgrid_uri, key_file, cert_file,
master_ca_file, timeout):
SlapDocument._slapgrid_uri = slapgrid_uri
scheme, netloc, path, query, fragment = urlparse.urlsplit(
SlapDocument._slapgrid_uri)
if not(query == '' and fragment == ''):
raise AttributeError('Passed URL %r issue: not parseable'%
SlapDocument._slapgrid_uri)
if scheme not in ('http', 'https'):
raise AttributeError('Passed URL %r issue: there is no support for %r p'
'rotocol' % (SlapDocument._slapgrid_uri, scheme))
if scheme == 'http':
connection_wrapper = httplib.HTTPConnection
else:
if master_ca_file is not None:
connection_wrapper = HTTPSConnectionCA
else:
connection_wrapper = httplib.HTTPSConnection
slap._connection_helper = \
SlapDocument._connection_helper = ConnectionHelper(connection_wrapper,
netloc, path, key_file, cert_file, master_ca_file, timeout)
def _register(self, klass, *registration_argument_list):
if len(registration_argument_list) == 1 and type(
registration_argument_list[0]) == type([]):
# in case if list is explicitly passed and there is only one
# argument in registration convert it to list
registration_argument_list = registration_argument_list[0]
document = klass(*registration_argument_list)
return document
def registerSoftwareRelease(self, software_release):
"""
Registers connected representation of software release and
returns SoftwareRelease class object
"""
return SoftwareRelease(software_release=software_release)
def registerComputer(self, computer_guid):
"""
Registers connected representation of computer and
returns Computer class object
"""
self.computer_guid = computer_guid
return self._register(Computer, computer_guid)
def registerComputerPartition(self, computer_guid, partition_id):
"""
Registers connected representation of computer partition and
returns Computer Partition class object
"""
self._connection_helper.GET('/registerComputerPartition?' \
'computer_reference=%s&computer_partition_reference=%s' % (
computer_guid, partition_id))
xml = self._connection_helper.response.read()
return xml_marshaller.loads(xml)
def registerOpenOrder(self):
return OpenOrder()
def registerSupply(self):
return Supply()
slap_recipe_mysqldatabase.py.txt 0000664 0000000 0000000 00000006206 11602033661 0033030 0 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from slapos.slap import slap, ComputerPartition, ResourceNotReady
import sys
import MySQLDB
class Recipe:
def __init__(self, buildout, options):
self.buildout, self.options = buildout, options
def install(self):
# register self on slap
# online computer partition, slap gave connector
computer_partition = slap(self.options['vifib_server_url']).register(
ComputerPartition,
coordinate_kw=dict(
computer_id=self.options['computer_id'], # c09fba86-b690-493d-b2b5-03a0df580b28
partition_id=self.options['partition_id'], # 007
resource_url=self.options['resource_url'], # http://slapgrid.org/resource/mysqldatabase-5.4.5
)
)
# request mysql server instance in version 5.4.5
mysql_server_partition = computer_partition.request(
self.options['parent_resource_url'], # http://slapgrid.org/resource/mysqlserver-5.4.5
self.options['parent_resource_id'], # main_server
)
# invoke installation
try:
connect = MySQLDB(mysql_server_partition.getIP(),
mysql_server_partition.getPort(), mysql_server_partition.getUser(),
mysql_server_partition.getPassword()).connect()
except ResourceNotReady:
# accept asynchronous mode
self.logger.info('Not yet available, postponing...')
return []
if not connect.isDatabaseCreated(self.options['database_name']):
computer_partition.building()
try:
connect.query('CREATE DATABASE IF NOT EXISTS %s' %
self.options['database_name'])
except:
# issue during installation
message = 'Issue during creation %s:%s' % sys.traceback_info_as_list()
computer_partition.error(message)
self.logger.error(message)
if connect.isDatabaseCreated(self.options['database_name']):
computer_partition.available()
return []
update = install
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/tests/ 0000775 0000000 0000000 00000000000 11602033661 0025655 5 ustar 00root root 0000000 0000000 slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/tests/__init__.py 0000664 0000000 0000000 00000002442 11602033661 0027770 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/tests/testinterface.py 0000664 0000000 0000000 00000007757 11602033661 0031107 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import unittest
from zope.interface.verify import verifyClass
import zope.interface
import types
from slapos import slap
def getOnlyImplementationAssertionMethod(klass, method_list):
"""Returns method which verifies if a klass only implements its interfaces"""
def testMethod(self):
implemented_method_list = [x for x in dir(klass) \
if ((not x.startswith('_')) and callable(getattr(klass, x)))]
for interface_method in method_list:
if interface_method in implemented_method_list:
implemented_method_list.remove(interface_method)
if implemented_method_list:
raise AssertionError("Unexpected methods %s" % implemented_method_list)
return testMethod
def getImplementationAssertionMethod(klass, interface):
"""Returns method which verifies if interface is properly implemented by klass"""
def testMethod(self):
verifyClass(interface, klass)
return testMethod
def getDeclarationAssertionMethod(klass):
"""Returns method which verifies if klass is declaring interface"""
def testMethod(self):
self.assertNotEqual(0, len(list(zope.interface.implementedBy(klass))))
return testMethod
def generateTestMethodListOnClass(klass, module):
"""Generate test method on klass"""
for class_id in dir(module):
implementing_class = getattr(module, class_id)
if type(implementing_class) not in (types.ClassType, types.TypeType):
continue
# add methods to assert that publicly available classes are defining
# interfaces
method_name = 'test_%s_declares_interface' % (class_id,)
setattr(klass, method_name, getDeclarationAssertionMethod(
implementing_class))
implemented_method_list = []
for interface in list(zope.interface.implementedBy(implementing_class)):
# for each interface which class declares add a method which verify
# implementation
method_name = 'test_%s_implements_%s' % (class_id,
interface.__identifier__)
setattr(klass, method_name, getImplementationAssertionMethod(
implementing_class, interface))
for interface_klass in interface.__iro__:
implemented_method_list.extend(interface_klass.names())
# for each interface which class declares, check that no other method are
# available
method_name = 'test_%s_only_implements' % class_id
setattr(klass, method_name, getOnlyImplementationAssertionMethod(
implementing_class,
implemented_method_list))
class TestInterface(unittest.TestCase):
"""Tests all publicly available classes of slap
Classes are checked *if* they implement interface and if the implementation
is correct.
"""
# add methods to test class
generateTestMethodListOnClass(TestInterface, slap)
if __name__ == '__main__':
unittest.main()
slapos.core-182cdf43b3e66a610f38c6e26a87b7b69e058cc5-slapos/slapos/slap/tests/testslap.py 0000664 0000000 0000000 00000032241 11602033661 0030070 0 ustar 00root root 0000000 0000000 ##############################################################################
#
# Copyright (c) 2010 Vifib SARL and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import unittest
import slap
import os
from slap.slap import SlapDocument
SERVER_URL = os.environ.get('TEST_SLAP_SERVER_URL',
'https://user:pass@server/path/path')
class UndefinedYetException(Exception):
"""To catch exceptions which are not yet defined"""
class SlapMixin(unittest.TestCase):
"""
Usefull methods for slap tests
"""
server_url = SERVER_URL
def _getTestComputerId(self):
"""
Returns the computer id used by the test
"""
return self.id()
return os.environ.get('TEST_SLAP_COMPUTER_ID', self.id())
class TestSlap(SlapMixin):
"""
Test slap against slap server
"""
def test_slap_initialisation(self):
"""
Asserts that slap initialisation works properly in case of
passing correct url
"""
slap_instance = slap.slap()
slap_instance.initializeConnection(self.server_url)
self.assertTrue(SlapDocument._connection_helper.host in self.server_url)
self.assertTrue(SlapDocument._connection_helper.path in self.server_url)
def test_slap_initialisation_wrong_url(self):
"""
Asserts that slap initialisation raises exception when passed url
is not correct
"""
server_url = 'https://user:pass@server/path/path?parameter=notAcceptable'
slap_instance = slap.slap()
self.assertRaises(AttributeError,
slap_instance.initializeConnection, server_url)
def test_registerComputer_with_new_guid(self):
"""
Asserts that calling slap.registerComputer with new guid returns
Computer object
"""
computer_guid = self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
computer = self.slap.registerComputer(computer_guid)
self.assertTrue(isinstance(computer, slap.Computer))
def test_registerComputer_with_existing_guid(self):
"""
Asserts that calling slap.registerComputer with already used guid
returns Computer object
"""
computer_guid = self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
computer = self.slap.registerComputer(computer_guid)
self.assertTrue(isinstance(computer, slap.Computer))
computer2 = self.slap.registerComputer(computer_guid)
self.assertTrue(isinstance(computer2, slap.Computer))
# XXX: There is naming conflict in slap library.
# SoftwareRelease is currently used as suboject of Slap transmission object
def test_registerSoftwareRelease_with_new_uri(self):
"""
Asserts that calling slap.registerSoftwareRelease with new guid
returns SoftwareRelease object
"""
software_release_uri = 'http://server/' + self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
software_release = self.slap.registerSoftwareRelease(software_release_uri)
self.assertTrue(isinstance(software_release, slap.SoftwareRelease))
def test_registerSoftwareRelease_with_existing_uri(self):
"""
Asserts that calling slap.registerSoftwareRelease with already
used guid returns SoftwareRelease object
"""
software_release_uri = 'http://server/' + self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
software_release = self.slap.registerSoftwareRelease(software_release_uri)
self.assertTrue(isinstance(software_release, slap.SoftwareRelease))
software_release2 = self.slap.registerSoftwareRelease(software_release_uri)
self.assertTrue(isinstance(software_release2, slap.SoftwareRelease))
def test_registerComputerPartition_new_partition_id_known_computer_guid(self):
"""
Asserts that calling slap.registerComputerPartition on known computer
returns ComputerPartition object
"""
self.computer_guid = self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
self.partition_id = 'PARTITION_01'
self.slap.registerComputer(self.computer_guid)
partition = self.slap.registerComputerPartition(self.computer_guid,
self.partition_id)
self.assertTrue(isinstance(partition, slap.ComputerPartition))
def test_registerComputerPartition_existing_partition_id_known_computer_guid(self):
"""
Asserts that calling slap.registerComputerPartition on known computer
returns ComputerPartition object
"""
self.test_registerComputerPartition_new_partition_id_known_computer_guid()
partition = self.slap.registerComputerPartition(self.computer_guid,
self.partition_id)
self.assertTrue(isinstance(partition, slap.ComputerPartition))
def test_registerComputerPartition_unknown_computer_guid(self):
"""
Asserts that calling slap.registerComputerPartition on unknown
computer raises (not defined yet) exception
"""
computer_guid = self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
partition_id = 'PARTITION_01'
self.assertRaises(UndefinedYetException,
self.slap.registerComputerPartition, computer_guid, partition_id)
class TestComputer(SlapMixin):
"""
Tests slap.Computer class functionality
"""
def test_computer_getComputerPartitionList_no_partition(self):
"""
Asserts that calling Computer.getComputerPartitionList without Computer
Partitions returns empty list
"""
self.computer_guid = self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
self.computer = self.slap.registerComputer(self.computer_guid)
self.assertEqual(self.computer.getComputerPartitionList(), [])
def test_computer_getComputerPartitionList_only_partition(self):
"""
Asserts that calling Computer.getComputerPartitionList with only
Computer Partitions returns empty list
"""
self.computer_guid = self._getTestComputerId()
partition_id = 'PARTITION_01'
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
self.computer = self.slap.registerComputer(self.computer_guid)
self.partition = self.slap.registerComputerPartition(self.computer_guid,
partition_id)
self.assertEqual(self.computer.getComputerPartitionList(), [])
def test_computer_reportUsage_non_valid_xml_raises(self):
"""
Asserts that calling Computer.reportUsage with non DTD
(not defined yet) XML raises (not defined yet) exception
"""
self.computer_guid = self._getTestComputerId()
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
self.computer = self.slap.registerComputer(self.computer_guid)
non_dtd_xml = """
value
"""
self.assertRaises(UndefinedYetException, self.computer.reportUsage,
non_dtd_xml)
def test_computer_reportUsage_valid_xml_invalid_partition_raises(self):
"""
Asserts that calling Computer.reportUsage with DTD (not defined
yet) XML which refers to invalid partition raises (not defined yet)
exception
"""
self.computer_guid = self._getTestComputerId()
partition_id = 'PARTITION_01'
self.slap = slap.slap()
self.slap.initializeConnection(self.server_url)
self.computer = self.slap.registerComputer(self.computer_guid)
self.partition = self.slap.registerComputerPartition(self.computer_guid,
partition_id)
# XXX: As DTD is not defined currently proper XML is not known
bad_partition_dtd_xml = """