Commit 9105fd56 authored by Gabriel Monnerat's avatar Gabriel Monnerat

Commit initial version

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk/utils@37406 20353a03-c40f-0410-a6d1-a30d3c3de9de
parents
all: install
PYTHON=python2.6
install:
@$(PYTHON) setup.py install
0.6
===
- removed the configurationmanager.py.
0.5
===
- Removed the possibility of infinite loop in getAvailableOO.
- Corrected the problem when the document in file system isn't
found(IllegalArgumentException).
- Implemented in oohandler to when not exist OOo Instance free. If don't exist,
the requisition is locked.
0.4
===
Created class to manipulate the configuration file(ConfigurationManager).
Now, all paths are taken by this object and in configuration file is possible
set the number of OOo instance that you want. The OOFactory object was implemented
to start the max number of OOo instance and get the instance available to use.
0.3
===
Split start.py into openoffice.py and xvfb.py. The classes openoffice and xvfb have
methods to control the process(start,stop and status about the process). Created one class
and one object for it, for easier reload the configuration.
0.2
===
Extends the code to convert a document to pdf. Initially the connection with OO
was implemented in class worker, but should be removed and implemented separately.
0.1
===
Create a prototype using WSGI and XMLRPC. One class calculator was used as test.
Install Dependencies
--------------------
System Dependencies
- Xvfb
Python Dependencies
- zope.interface
- WSGIUtils
- PasteDeploy
- PasteScript
Instalation in Mandriva:
- urpmi xvfb
- urpmi python-setuptools
- easy_install-2.6 zope.interface PasteDeploy PasteScript WSGIUtils
Instalation of Openoffice
Was used for testing the package's official openoffice.Follow these steps to install:
- Download the package from the official site;
- unpack the tar.gz and Install;
The instalation is in /opt
How to use
----------
- Start the Daemon
./cloudooo start
Description of Daemon
---------------------
- XMLRPC + WSGI will be one bridge for easy access OpenOffice.org. This will implement one XMLRPC server into WSGI (Paster).
- PyUno is used to connect to OpenOffice.org stated with open socket. The features will be handled all by pyuno.
- Xvfb is used to run Openoffice.org. This is controlled by Daemon(cloudooo).
- Only a process will have access to OpenOffice.org by time.
- All clients receive the same object(proxy) when connects with XMLRPC Server.
Xvfb and OpenOffice
- configure and start Xvfb;
- Use a single Xvfb;
- the xvfb will be started with the XMLRPC Server;
- When start the Daemon(cloudooo), it configures Xvfb, next opens the openoffice(with pyuno) and start XMLRPC Server;
- control Xvfb;
- start openoffice;
- Pyuno start the openoffice processes and the communication is through sockets;
- Openoffice processes run in brackground and in virtual display;
- control openoffice;
- The socket can't lose the connection, if this occurs should kill the process and submit again the file;
XMLRPC Server - XMLRPC + WSGI
-----------------------------
- Send document to openoffice and return the document converted with metadata;
- XMLRPC receives a file and connects to a openoffice by pyuno;
- The pyuno opens a new openoffice, write, add metadata and returns the document edited or converted to xmlrpc and it return the document to the user;
- When finalize the use of openoffice, should make sure that it was finalized;
- Export to another format;
- Invite document and return metadata only;
- Edit metadata of the document;
- Problems and possible solution
- OpenOffice is stalled;
- finalize the process, start openoffice and submit the document again(without restart the cloudooo);
- Openoffice is crashed;
- finalize the process, verify if all the process was killed, start openoffice and submit the document again(without restart the cloudooo)
- OpenOffice received the document and stalled;
- if openoffice isn't responding, kill the process and start
- The document that was sent is corrupt;
- write in log the error and verify that the process aren't in memory
##############################################################################
#
# Copyright (c) 2002-2010 Nexedi SA and Contributors. All Rights Reserved.
# Gabriel M. Monnerat <gmonnerat@iff.edu.br>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from zope.interface import implements
from cloudooo.interfaces.application import IApplication
from cloudooo.utils import logger, socketStatus
from psutil import pid_exists, Process
class Application(object):
"""Base object to create an object that is possible manipulation a
process"""
implements(IApplication)
name = "application"
def start(self):
"""Start Application"""
logger.debug("Process Started %s, Port %s. Pid %s" % (self.name,
self.getAddress()[-1],
self.pid()))
def stop(self):
"""Stop the process"""
if hasattr(self, 'process') and self.status():
process_pid = self.process.pid
logger.debug("Stop Pid - %s" % process_pid)
try:
self.process.terminate()
finally:
if pid_exists(process_pid) or self.status():
Process(process_pid).kill()
def loadSettings(self, hostname, port, path_run_dir, display_id, **kwargs):
"""Define attributes for application instance
Keyword arguments:
hostname -- Host to start the instance.
port -- Expected a int number.
path_run_dir -- Full path to create the enviroment.
display_id -- Display to open the OpenOffice.
"""
self.hostname = hostname
self.port = port
self.path_run_dir = path_run_dir
self.display_id = display_id
self.timeout = kwargs.get('start_timeout', 20)
def restart(self):
"""Start and Stop the process"""
self.stop()
self.start()
def status(self):
"""Check by socket if the openoffice work."""
return socketStatus(self.hostname, self.port)
def getAddress(self):
"""Return port and hostname of OOo Instance."""
return self.hostname, self.port
def pid(self):
"""Returns the pid"""
if not hasattr(self, 'process') or not self.status():
return None
return self.process.pid
##############################################################################
#
# Copyright (c) 2002-2010 Nexedi SA and Contributors. All Rights Reserved.
# Gabriel M. Monnerat <gmonnerat@iff.edu.br>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from os import environ, remove
from os.path import exists, join
from subprocess import Popen, PIPE
from threading import Lock
from cloudooo.ooolib import setUpUnoEnvironment
from zope.interface import implements
from application import Application
from cloudooo.interfaces.lockable import ILockable
from cloudooo.utils import logger, waitStartDaemon, removeDirectory, waitStopDaemon
class OpenOffice(Application):
"""Object to control one OOo Instance and all features instance."""
implements(ILockable)
name = "openoffice"
def __init__(self):
"""Creates the variable to save the pid, port and hostname of the object.
The lock is a simple lock python that is used when one requisition is
using the openoffice.
"""
self._bin_soffice = 'soffice.bin'
self._lock = Lock()
self._cleanRequest()
def _testOpenOffice(self, host, port):
"""Test if OpenOffice was started correctly"""
logger.debug("Test OpenOffice %s - Pid %s" % (self.getAddress()[-1], self.pid()))
command = [self.python_path
, self.unoconverter_bin
, "--test"
, "--hostname=%s" % host
, "--port=%s" % port
, "--document_url=%s" % self.document_url
, "--unomimemapper_bin=%s" % self.unomimemapper_bin
, "--python_path=%s" % self.python_path
, "--uno_path=%s" % self.uno_path
, "--office_bin_path=%s" % self.office_bin_path]
logger.debug("Testing Openoffice Instance %s" % port)
stdout, stderr = Popen(" ".join(command), shell=True, stdout=PIPE,
stderr=PIPE).communicate()
if not stdout and stderr != "":
logger.debug(stderr)
return False
else:
url = stdout.replace("\n", "")
logger.debug("Instance %s works" % port)
remove(url)
return True
def _cleanRequest(self):
"""Define request attribute as 0"""
self.request = 0
def loadSettings(self, hostname, port, path_run_dir, display_id,
office_bin_path, uno_path, **kw):
"""Method to load the configuratio to control one OpenOffice Instance
Keyword arguments:
office_path -- Full Path of the OOo executable.
e.g office_bin_path='/opt/openoffice.org3/program'
uno_path -- Full path of the Uno Library
"""
Application.loadSettings(self, hostname, port, path_run_dir, display_id)
setUpUnoEnvironment(uno_path, office_bin_path)
self.office_bin_path = office_bin_path
self.uno_path = uno_path
self.process_name = "soffice.bin"
self.document_url = kw.get('document_url', '')
self.unoconverter_bin = kw.get("unoconverter_bin", "unoconverter")
self.python_path = kw.get('python_path', 'python')
self.unomimemapper_bin = kw.get("unomimemapper_bin")
def _start_process(self, command, env):
"""Start OpenOffice.org process"""
self.process = Popen(' '.join(command),
stdout=PIPE,
shell=True,
close_fds=True,
env=env)
waitStartDaemon(self, self.timeout)
if exists(self.document_url):
return self._testOpenOffice(self.hostname, self.port)
return True
def start(self):
"""Start Instance."""
self.path_user_installation = join(self.path_run_dir, \
"cloudooo_instance_%s" % self.port)
if exists(self.path_user_installation):
removeDirectory(self.path_user_installation)
# Create command with all parameters to start the instance
self.command = [join(self.office_bin_path, self._bin_soffice)
, '-invisible'
, '-nologo'
, '-nodefault'
, '-norestore'
, '-nofirststartwizard'
, '"-accept=socket,host=%s,port=%d;urp;StarOffice.ComponentContext"' % \
(self.hostname, self.port)
, '-env:UserInstallation=file://%s' % self.path_user_installation]
# To run the instance OOo is need a environment. So, the "DISPLAY" of Xvfb
# is passed to env and the environment customized is passed to the process
env = environ.copy()
env["DISPLAY"] = ":%s" % self.display_id
process_started = self._start_process(self.command, env)
if not process_started:
self.stop()
waitStopDaemon(self, self.timeout)
self._start_process(self.command, env)
self._cleanRequest()
Application.start(self)
def stop(self):
"""Stop the instance by pid. By the default
the signal is 15."""
Application.stop(self)
if exists(self.path_user_installation):
removeDirectory(self.path_user_installation)
self._cleanRequest()
def isLocked(self):
"""Verify if OOo instance is being used."""
return self._lock.locked()
def acquire(self):
"""Lock Instance to use."""
self.request += 1
self._lock.acquire()
def release(self):
"""Unlock Instance."""
logger.debug("OpenOffice %s, %s unlocked" % self.getAddress())
self._lock.release()
openoffice = OpenOffice()
##############################################################################
#
# Copyright (c) 2002-2010 Nexedi SA and Contributors. All Rights Reserved.
# Gabriel M. Monnerat <gmonnerat@iff.edu.br>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from subprocess import Popen, PIPE
from cloudooo.utils import logger, waitStartDaemon, remove_file
from zope.interface import implements
from application import Application
from cloudooo.interfaces.application import IApplication
from os.path import exists
class Xvfb(Application):
"""Start and control Xvfb. It is used to open/run
instances OpenOffice.
"""
implements(IApplication)
name = "xvfb"
def loadSettings(self, hostname, port, path_run_dir, display_id, **kwargs):
"""Method to load the configuration to run and monitoring the Xvfb.
Keyword Arguments:
virtual_screen -- the display number
"""
Application.loadSettings(self, hostname, port, path_run_dir, display_id)
self.virtual_screen = kwargs.get('virtual_screen', '0')
self.process_name = "Xvfb"
def start(self):
"""Method to start Virtual Frame Buffer."""
self.command = ["Xvfb", "-ac", ":%s" % self.display_id, \
"-screen", self.virtual_screen, "800x600x16", \
"-fbdir", self.path_run_dir]
self.process = Popen(self.command,
stdout=PIPE,
close_fds=True)
waitStartDaemon(self, self.timeout)
Application.start(self)
logger.debug("Xvfb pid - %s" % self.pid())
def stop(self):
"""Stop Xvfb processes and remove lock file in file system"""
Application.stop(self)
lock_filepath = '/tmp/.X%s-lock' % self.display_id
if exists(lock_filepath):
remove_file(lock_filepath)
display_filepath = '/tmp/X11/X%s' % self.display_id
if exists(display_filepath):
remove_file(display_filepath)
xvfb = Xvfb()
This diff is collapsed.
#!/usr/bin/env python
##############################################################################
#
# Copyright (c) 2002-2010 Nexedi SA and Contributors. All Rights Reserved.
# Gabriel M. Monnerat <gmonnerat@iff.edu.br>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
import sys
from os import environ
from getopt import getopt, GetoptError
from cloudooo.ooolib import getServiceManager
from cloudooo.utils import usage
from types import InstanceType
__doc__ = """
usage: unomimemapper.py [options]
Options:
-h, --help this help screen
--hostname=STRING OpenOffice Instance address
--port=STRING OpenOffice Instance port
--office_bin_path=STRING_URL
Folder path were is the binary openoffice
--uno_path=STRING_URL
Folter path were is the uno library
"""
class UnoMimemapper(object):
""" """
def __init__(self, hostname, port):
""" Receives hostname and port from openoffice and create a service manager"""
self.service_manager = getServiceManager(hostname, port)
def _getElementNameByService(self, uno_service, ignore_name_list=[]):
"""Returns an dict with elements."""
name_list = uno_service.getElementNames()
service_dict = {}
for name in iter(name_list):
element_dict = {}
element_list = uno_service.getByName(name)
for obj in iter(element_list):
if obj.Name in ignore_name_list:
continue
elif type(obj.Value) == InstanceType:
continue
element_dict[obj.Name] = obj.Value
service_dict[name] = element_dict
return service_dict
def getFilterDict(self):
"""Return all filters and your properties"""
filter_service = self.service_manager.createInstance("com.sun.star.document.FilterFactory")
filter_dict = self._getElementNameByService(filter_service, ["UINames", "UserData"])
return filter_dict
def getTypeDict(self):
"""Return all types and your properties"""
type_service = self.service_manager.createInstance("com.sun.star.document.TypeDetection")
type_dict = self._getElementNameByService(type_service, ["UINames", "URLPattern"])
return type_dict
def help():
usage(sys.stderr, __doc__)
sys.exit(1)
def main():
try:
opt_list, arg_list = getopt(sys.argv[1:], "h", ["help",
"uno_path=", "office_bin_path=",
"hostname=", "port="])
except GetoptError, msg:
msg = msg.msg + "\nUse --help or -h"
usage(sys.stderr, msg)
sys.exit(2)
if not opt_list:
help()
for opt, arg in opt_list:
if opt in ("-h", "--help"):
help()
if opt == "--uno_path":
environ['uno_path'] = arg
elif opt == "--office_bin_path":
environ['office_bin_path'] = arg
elif opt == '--hostname':
hostname = arg
elif opt == "--port":
port = arg
mimemapper = UnoMimemapper(hostname, port)
filter_dict = mimemapper.getFilterDict()
type_dict = mimemapper.getTypeDict()
print "filter_dict = %s\ntype_dict = %s" % (filter_dict, type_dict)
if "__main__" == __name__:
main()
#!/bin/sh
lsb_functions="/etc/rc.d/init.d/functions"
if test -f $lsb_functions ; then
. $lsb_functions
else
log_success_msg()
{
echo " SUCCESS! $@"
}
log_failure_msg()
{
echo " ERROR! $@"
}
fi
# complete path of the cloudooo folder
FOLDER=$(cd $(dirname $0); pwd -P)/
# get the pid of the cloudooo
CLOUDOOO_PID=$FOLDER/paster.pid
# complete path of this script
SELF=$FOLDER$(basename $0)
case "$1" in
start)
paster serve ./samples/cloudooo.conf --daemon;;
stop)
kill -1 `cat $CLOUDOOO_PID`;
paster serve ./samples/cloudooo.conf --stop-daemon;;
restart)
$SELF stop;
$SELF start;;
*)
echo "Usage: ./cloudooo {start|stop|restart}";
exit 1;;
esac
exit 0
##############################################################################
#
# Copyright (c) 2002-2010 Nexedi SA and Contributors. All Rights Reserved.
# Gabriel M. Monnerat <gmonnerat@iff.edu.br>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from signal import signal, SIGHUP
from application.openoffice import openoffice
from application.xvfb import xvfb
from wsgixmlrpcapplication import WSGIXMLRPCApplication
from utils import convertStringToBool, configureLogger, cleanDirectory
from os import path
import monitor
from mimemapper import mimemapper
import gc
def stopProcesses():
monitor.stop()
xvfb.stop()
openoffice.stop()
def application(global_config, **local_config):
"""Method to load all configuration of cloudooo and start the application.
To start the application a number of params are required:
Keyword arguments:
debug_mode -- Mode as the application prints the messages.
e.g debug_mode=logging.DEBUG
path_dir_run_cloudooo -- Full path to create the environment of the processes.
e.g path_dir_run_cloudooo='/var/run/cloudooo'
virtual_display_port -- Port to start the Xvfb.
virtual_display_id -- Sets the display.
e.g virtual_display_id='99'
application_hostname -- Sets the host to Xvfb and Openoffice.
virtual_screen -- Use to define the screen to Xvfb
e.g virtual_screen='0'
number_instances_openoffice -- Defines a number of OOo Instances.
pool_port_range_start -- Initial number to start all OOo Instances.
e.g pool_port_range_start=4060
office_bin_path -- Full Path of the OOo executable.
e.g office_bin_path='/opt/openoffice.org3/program'
uno_path -- Full path to pyuno library.
e.g uno_path='/opt/openoffice.org/program'
"""
gc.enable()
debug_mode = convertStringToBool(local_config.get('debug_mode'))
configureLogger(debug_mode=debug_mode)
document_name = local_config.get("document_name")
# path of directory to run cloudooo
path_dir_run_cloudooo = local_config.get('path_dir_run_cloudooo')
cleanDirectory(path_dir_run_cloudooo, ignore_list=["tmp", document_name])
# directory to create temporary files
cloudooo_path_tmp_dir = path.join(path_dir_run_cloudooo, 'tmp')
cleanDirectory(cloudooo_path_tmp_dir)
# The Xvfb will run in the same local of the OpenOffice
application_hostname = local_config.get('application_hostname')
openoffice_port = int(local_config.get('openoffice_port'))
# Before start Xvfb, first loads the configuration
xvfb.loadSettings(application_hostname,
int(local_config.get('virtual_display_port')),
path_dir_run_cloudooo,
local_config.get('virtual_display_id'),