Enable Zope 2.12 instances with this buildout

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@34292 20353a03-c40f-0410-a6d1-a30d3c3de9de

buildout/README-2.12.txt

+===========================
+ERP5 buildout for Zope 2.12
+===========================
+
+Introduction
+============
+
+ERP5 Buildout provides a way to build and manage ERP5 software components
+with all needed dependencies.
+
+ERP5 Buildout also provides a way to manage separate installation instances of
+ERP5 to share non-data components of an ERP5 software installation from a
+single location, allowing for easy component upgrade.
+
+Software
+========
+
+Software part shall be system independent. In perfect world it shall depend
+only on:
+
+ * C compiler
+ * standard C and C++ library
+ * operating system kernel
+
+As the world is not yet perfect, some additional build time requirements are
+needed. See below for a way to acquire the list of system dependencies through
+helpers.
+
+Setup
+-----
+
+Checkout: https://svn.erp5.org/repos/public/erp5/trunk/buildout/
+For example:
+
+  svn co https://svn.erp5.org/repos/public/erp5/trunk/buildout/ ~/erp5.buildout
+
+Run the Zope 2.12 buildout:
+
+  $ cd ~/erp5.buildout
+  $ python2.6 bootstrap/bootstrap.py
+  $ bin/buildout -c buildout-2.12.cfg
+
+This will download and install the software components needed to run ERP5 on
+Zope 2.12 including Zope 2.12 plus dependencies (including
+Acquisition with _aq_dynamic patch) and CMF 2.2 plus dependencies.
+
+System dependency check
+-----------------------
+
+Each software component in this buildout might require some system
+dependencies, including development libraries and executables.
+To query what is required for all components, please run:
+
+  $ bin/buildout install show-requirements
+
+Minimal requirements
+--------------------
+
+At the very least, running buildout requires:
+
+ * Python 2.4 or later including development files (e.g. python2.4-devel or 
+   python2.4-dev package from your system package manager. A file like
+   /usr/lib*/python*/config/Makefile should be installed in the system.
+   XXX Since we compile our own python, are development files still necessary?)
+ * C development toolchain (Make, gcc, gpp, etc.) 
+ * subversion (svn) client, to check-out this buildout.
+
+Post-build check
+----------------
+
+There isn't yet a post-build check for running ERP5 on Zope 2.12.
+
+When it's ready, you can check if all components are working correctly by
+typing:
+
+  $ make assert
+
+Distribution helpers
+--------------------
+
+In the 'helpers' directory there are shell scripts to prepare different
+GNU/Linux distributions to run this buildout.
+
+For instance, to prepare Mandriva 2010.0 please type the following with root
+privileges:
+
+  helpers/mandriva2010.0.sh
+
+Please refer to the 'helpers' directory for other distributions.
+
+Instances
+=========
+
+Note: Instance generation is still a work in progress.
+
+After the software components are built, we can generate ERP5 instance
+buildouts from that software.
+
+Assuming the ERP5 software buildout is available in ~/erp5.buildout the
+following sequence of steps should result in a working "instance" buildout:
+
+$ mkdir ~/instances                     # 0
+$ cd ~/instances                        # 1
+$ ln -s ~/erp5.buildout/*profiles*      # 2
+$ cat > buildout.cfg                    # 3
+
+[buildout]
+extends-cache = instance-profiles/extends-cache
+extends =
+  profiles/development-2.12.cfg
+  instance-profiles/software-home.inc
+
+parts =
+  mysql-instance
+  oood-instance
+  supervisor-instance
+^D
+$ ~/erp5.buildout/bin/bootstrap2.6      # 4
+$ bin/buildout -ov                      # 5
+
+Notice how we managed to run buildout in "offline-mode" (-o). The software-home
+configuration (along with the 'extends-cache' in the 'instance-profiles'
+symlink) provides all the information and components that would otherwise have
+to be downloaded.
+
+The steps above generate instance configurations for mysql and the
+OpenOffice.org document conversion daemon. We need mysql, in particular,
+to be running before configuring an actual ERP5 instance, so we'll start
+supervisor:
+
+$ var/bin/supervisord                   # 6
+
+Now edit buildout.cfg and add "runUnitTest" (w/o quotes) to 'buildout:parts'.
+The "development-instance" part will be pulled in automatically as a
+dependency:
+
+$ $EDITOR buildout.cfg                  # 7
+
+Then run buildout again to finish the configuration
+
+$ bin/buildout -ov                      # 8
+
+Now a fully configured development instance will be available in the directory
+"var/development-instance", so you can do:
+
+ $ var/development-site/bin/zopectl fg
+
+And see an ERP5 instance running on "http://localhost:18080/". The port '18080'
+refers to the 'development-instance:http-address' setting in
+'instance-profiles/development-2.12.cfg'. The file 
+'instance-profiles/zope-2.12.cfg' provides the "Manager" credentials you should
+use (usually zope:zope), in the 'zope-instance-template:user' variable.
+
+You should also be able to run ERP5 unit tests like so:
+
+ $ bin/runUnitTest --erp5_sql_connection_string="test@127.0.0.1:10002 test" testBusinessTemplate
+
+The '127.0.0.1:10002' coordinate above refers to the address of the configured
+mysql instance, according to the settings 'configuration:mysql_host' and
+'configuration:mysql_port' in 'instance-profiles/mysql.cfg'.
+
+TODO
+====
+
+ * Adjust the 'runUnitTest' recipe to push the mysql server coordinates into
+   the 'bin/runUnitTest' script.
+
+ * Refactor the .cfg files to reduce duplication and so that only the
+   'instance-profiles' directory needs to be symlinked. Alternatively, push all
+   .cfg files into a single 'profiles'
+   directory.
+
+ * Combine steps 2, 3 and 4 into a single step by creating a more powerful
+   'bootstrap2.6' script.
+
+ * Running 'buildout' twice in the instance (once to configure 'supervisor',
+   'mysql' and 'oood' and once to setup the ZODB ERP5 instance) is confusing
+   and error-prone. A buildout shouldn't deal with persistent state, only with
+   file installation. Move the mysql database and ERP5 ZODB instance creation
+   procedures to dedicated scripts in 'bin/' instead of implicitly running them
+   in the (second) buildout run.
+
+ * Patch the SOAPpy package provided by Nexedi so it doesn't fail with a
+   SyntaxError on Python 2.6. Right now we're using a SOAPpy repackaging from
+   http://ibid.omnia.za.net/eggs/ .
+
+ * Synchronize the buildout behaviour for Zope 2.8 and 2.12 (i.e. allows Zope
+   2.8 to work with a single buildout check-out).
+
+ * See if we can use http://pypi.python.org/pypi/zc.sourcerelease/ to generate
+   a single (humongous) tarball with all needed software components for fully
+   offline operation.

buildout/buildout-2.12.cfg

 [buildout]
+extends-cache = instance-profiles/extends-cache
 extends = profiles/official-2.12.cfg

buildout/instance-profiles/zope-2.12.cfg

+# adapt https://svn.erp5.org/repos/public/experimental/erp5.buildout
+[buildout]
+parts = zope-instance
+
+[zope-instance-template]
+recipe = erp5.recipe.standaloneinstance
+user = zope:zope
+
+# modify this to reflect your local mysql configuration
+# Format:
+#     database[@host[:port]] [user [password [unix_socket]]]
+#   e.g "erp5db erp5user somepassword" or "erp5db erp5user"
+erp5_sql_connection_string = ${:mysql_database_name}@${:mysql_host}:${:mysql_port} ${:mysql_user} ${:mysql_password}
+
+zope_conf_template =
+  ${buildout:software_home}/templates/default-erp5-standalone-zope.conf.in
+
+portal_id = erp5
+force-zodb-update = false
+# zope.conf template part BEGIN
+# instancehome and http-address have to be defined
+# User modifable keys:
+environment =
+debug-mode = off
+ip-address = 0.0.0.0
+# Good defaults, which do not have to be touched:
+clienthome = ${:instancehome}/var
+lib_path = ${:instancehome}/lib/python
+effective-user =
+enable-product-installation = on
+locale =
+datetime-format = us
+zserver-threads = 4
+python-check-interval = 500
+zserver-read-only-mode = off
+pid-filename = ${:clienthome}/Z2.pid
+lock-filename = ${:clienthome}/Z2.lock
+mime-types =
+structured-text-header-level = 3
+rest-input-encoding =
+rest-output-encoding =
+rest-header-level = 3
+rest-language-code = en
+cgi-environment =
+dns-server =
+http-realm = Zope
+cgi-maxlen = 0
+http-header-max-length = 8192
+automatically-quote-dtml-request-data = on
+trusted-proxy =
+publisher-profile-file =
+security-policy-implementation = C
+skip-authentication-checking = off
+skip-ownership-checking = off
+verbose-security = off
+maximum-number-of-session-objects = 1000
+session-add-notify-script-path =
+session-delete-notify-script-path =
+session-timeout-minutes = 20
+session-resolution-seconds = 20
+suppress-all-access-rules = off
+suppress-all-site-roots = off
+database-quota-size =
+read-only-database = off
+eventlog-level = info
+eventlog-path = ${:instancehome}/log/event.log
+access-level = WARN
+access-path = ${:instancehome}/log/Z2.log
+access-format = %(message)s
+warnfilter =
+max-listen-sockets = 100
+port-base = 0
+large-file-threshold =
+default-zpublisher-encoding = utf8
+http-force-connection-close = off
+zodb-path = ${:instancehome}/var/Data.fs
+# zope.conf template part ENDS

buildout/profiles/common.cfg

 [buildout]
 develop +=
-  local-eggs/erp5.recipe.standaloneinstance
-  local-eggs/erp5.recipe.bt5checkout
-  local-eggs/erp5.recipe.mysqldatabase
-  local-eggs/erp5.recipe.mysqlserver
+  ${:software_home}/local-eggs/erp5.recipe.standaloneinstance
+  ${:software_home}/local-eggs/erp5.recipe.bt5checkout
+  ${:software_home}/local-eggs/erp5.recipe.mysqldatabase
+  ${:software_home}/local-eggs/erp5.recipe.mysqlserver
 
 find-links =
     http://download.zope.org/distribution/
@@ -11,6 +11,7 @@ find-links =
     http://download.hforge.org/itools/
     http://dist.repoze.org
     http://www.nexedi.org/static/packages/source/
+    http://ibid.omnia.za.net/eggs/
 
 [create-directories]
 recipe = plone.recipe.command

buildout/profiles/deployment-2.12.cfg

+[buildout]
+extends =
+  ../instance-profiles/mysql-database.cfg
+  ../instance-profiles/mysql.cfg
+  ../instance-profiles/oood.cfg
+  ../instance-profiles/supervisor.cfg
+  ../instance-profiles/zope-2.12.cfg
+  ../profiles/software-definition.cfg
+  ../profiles/versions-2.12.cfg
+  ../software-profiles/erp5-2.12.cfg
+  ../profiles/common.cfg
+
+parts =
+  eggs
+  mysql-instance
+  oood-instance
+  supervisor-instance
+
+python = software_definition
+versions = versions
+
+[eggs]
+scripts +=
+  mkzopeinstance
+  runzope
+
+[configuration]
+supervisor_programs =
+  10 mysql (autorestart=true) ${buildout:data-bin-directory}/pidproxy [ ${configuration:mysql_pid} ${buildout:data-bin-directory}/mysqld_safe ]
+  20 oood (autorestart=true) ${buildout:data-bin-directory}/pidproxy [ ${configuration:oood_pid} ${buildout:data-bin-directory}/oood-runserw --start ] ${buildout:etc-directory}
+
+#[software-links]
+#recipe = plone.recipe.command
+#command =
+#  ln -sf ${software_definition:software_home}/bin/python2.4 ${buildout:bin-directory}/python2.4
+#  ln -sf ${software_definition:software_home}/bin/ipython2.4 ${buildout:bin-directory}/ipython2.4
+#update-command = ${:command}

buildout/profiles/development-2.12.cfg

+[buildout]
+extends =
+  ../profiles/deployment-2.12.cfg
+
+parts +=
+  development-site
+
+[software_definition]
+software_home = ${buildout:directory}
+
+[runUnitTest]
+recipe = erp5.recipe.testrunner
+eggs = ${development-site:eggs}
+extra-paths = ${development-site:extra-paths}
+products =
+  ${development-site:products}
+
+# XXX have this list be populated by software_definition keys. Maybe adjust
+# erp5.recipe.testrunner to accept this parameter in multiple lines and convert
+# to this comma separated list.  
+bt5_path = ${development-site:bt5-path},${buildout:software_home}/parts/bt5-erp5,${buildout:software_home}/parts/bt5-erp5-extra
+
+instance-home = ${development-site:instancehome}
+initialization =
+  import os
+  os.environ['PATH'] = ':'.join(['${buildout:data-bin-directory}','${buildout:bin-directory}'] + os.environ.get('PATH','').split(':'))
+
+[development-site]
+<= zope-instance-template
+# this is provided by the enhanced [eggs] in deployment-2.12.cfg
+mkzopeinstance = ${buildout:directory}/bin/mkzopeinstance
+# developer by default want to have updatable Data.fs
+force-zodb-update = true
+bt5-path =
+bt5 =
+eggs =
+  Products.ExternalEditor
+  ${eggs:eggs}
+extra-paths = ${eggs:extra-paths}
+
+# MySQL
+mysql_database_name = development_site
+mysql_user = development_user
+mysql_password = development_password
+mysql_host = ${configuration:mysql_host}
+mysql_port = ${configuration:mysql_port}
+
+# create database
+mysql_create_database = true
+# below could be set in configuration
+mysql_superuser = root
+mysql_superpassword =
+
+# zope.conf template part BEGIN
+debug-mode = on
+instancehome = ${buildout:var-directory}/development-site
+http-address = 18080
+# until we will have more control over products we need to redefine to insert
+# local products to be able to override
+products =
+  ${:instancehome}/Products/
+  ${software_definition:products-erp5}
+  ${software_definition:products-erp5-development}
+  ${software_definition:products-deps}
+# zope.conf template part ENDS
+
+[requirements]
+binary +=
+  svn
+  svnadmin
+development +=
+  /usr/include.*/svn_client.h

buildout/profiles/official-2.12.cfg

@@ -22,10 +22,11 @@ extends =
   ../software-profiles/varnish.cfg
   ../profiles/common.cfg
 
-python = python2.6
+python = software_definition
 versions = versions
 
 parts =
+  itools-build
   eggs
   apache
   memcached
@@ -42,7 +43,9 @@ parts =
   products
 #  omelette
   bootstrap2.6
-#  software_home
+  bt5-erp5
+  bt5-erp5-extra
+  software_home
 
 [show-requirements]
 recipe = plone.recipe.command
@@ -53,9 +56,10 @@ update-command = ${:command}
 
 [software_definition]
 software_home = ${buildout:software_home}
+executable = ${python2.6:executable}
 
 [software_home]
 recipe = collective.recipe.template
 input = ${buildout:software_home}/templates/software-home.inc.in
-output = ${buildout:directory}/instance-profiles/software-home.inc
+output = ${buildout:software_home}/instance-profiles/software-home.inc
 

buildout/profiles/versions-2.12.cfg

 [buildout]
 
 extends =
-    http://svn.zope.org/repos/main/Zope/branches/2.12/versions.cfg
+  http://svn.zope.org/repos/main/Zope/tags/2.12.4/versions.cfg
 
 [versions]
 # XXX refactor parts shared with versions.cfg into a common file
@@ -15,7 +15,14 @@ numpy = 1.3.0
 plone.recipe.zope2instance = 3.6
 erp5.recipe.standaloneinstance >= 0.4
 pysvn = 1.7.2
-Zope2 = 2.12.3
+# Use SOAPpy from http://ibid.omnia.za.net/eggs/ instead of
+# http://www.nexedi.org/static/packages/source/ due to SyntaxError on
+# Python 2.6 
+SOAPpy = 0.12.0
+
+# clear the Acquisition version pin so our development checkout in local-eggs
+# is used instead
+Acquisition =
 
 erp5_products_revision =
 erp5_bt5_revision = ${:erp5_products_revision}

buildout/software-profiles/erp5-2.12.cfg

 # XXX refactor parts shared with erp5.cfg into a common file
 [buildout]
+extends =
+  mysql-tritonn-5.0.cfg
+  python-2.6.cfg
+
 parts =
   itools-build
   mysql-python
@@ -8,34 +12,46 @@ parts =
   products-erp5
 
 develop +=
+	${:software_home}/local-eggs/Products.DCWorkflow
 # The patched Acquisition
-	local-eggs/Acquisition
-	local-eggs/Products.DCWorkflow
+	${:software_home}/local-eggs/Acquisition
+
+# ugly itools building sections until itools and setuptools learn to get
+# along:
 
 [itools]
-recipe = hexagonit.recipe.download
-extra-paths = ${buildout:directory}/parts/site-packages
+# This part should not contain contain dependencies to other parts and should
+# contain only paths relative to software_home as it will be used directly by
+# instances. It should probably be moved to a .cfg file inside
+# "instance-profiles"
 pkgname = itools-0.50.8
-url = http://download.hforge.org/itools/0.50/${:pkgname}.tar.gz
+location = ${buildout:software_home}/parts/itools
+lib = ${:location}/lib
+source = ${:location}/src
+
+[itools-download]
+recipe = hexagonit.recipe.download
+strip-top-level-dir = true
+url = http://download.hforge.org/itools/0.50/${itools:pkgname}.tar.gz
+destination = ${itools:source}
 
 [itools-build]
 recipe = collective.recipe.cmd
-lib = ${itools:location}/lib
-source = ${itools:location}/${itools:pkgname}
 on_install = True
 on_update = False
+# reference itools-download:destination instead of itools:source to trigger its
+# execution
 cmds =
-    cd ${:source}
-#    ${python2.6:executable} setup.py install_lib --install-lib=${:lib}
-    ${python2.6:executable} setup.py install_lib -d ${:lib}
+    cd ${itools-download:destination}
+    ${python2.6:executable} setup.py install_lib -d ${itools:lib}
 
 uninstall_cmds =
-    cd ${:source}
-    ${buildout:executable} setup.py clean
-    rm -rf ${:lib}
+    cd ${itools-download:destination}
+    ${python2.6:executable} setup.py clean
+    rm -rf ${itools:lib}
 
 [mysql-python-env]
-PATH =${mysql-tritonn-5.0:location}/bin:%(PATH)s
+PATH =${software_definition:mysql_software}/bin:%(PATH)s
 
 [mysql-python]
 recipe = zc.recipe.egg:custom
@@ -105,18 +121,27 @@ eggs =
 # Other products
     Products.DCWorkflowGraph
     Products.MimetypesRegistry
+    Products.ExternalEditor
 # Currently forked in our repository
 #    Products.PortalTransforms
+# Dependency for our fork of PortalTransforms
+    StructuredText
 
 extra-paths =
-  ${itools-build:lib}
+  ${itools:lib}
 
 dependent-scripts = true
-python = python2.6
+# XXX getting python from [buildout] for the benefit of instance recipes
+# which should not attempt to compile python again
+#python = python2.6
+
+# parameterizing the version of the generated python interpreter name by the
+# python section version causes dependency between this egg section and the
+# installation of python, which we don't want on an instance
 interpreter = python2.6
 scripts =
-	python=python${python2.6:version}
-	ipython=ipython${python2.6:version}
+	python=${:interpreter}
+	ipython=i${:interpreter}
 
 [omelette]
 # XXX don't use this part until this omelette bug is fixed:
@@ -124,7 +149,7 @@ scripts =
 recipe = collective.recipe.omelette
 eggs = ${eggs:eggs}
 packages =
-      ${itools-build:lib} .
+      ${itools:lib} .
 products =
     ${products:paths}
 

buildout/templates/software-home.inc.in

 [buildout]
 software_home = ${buildout:software_home}
 eggs-directory = ${buildout:eggs-directory}
+develop-eggs-directory = ${buildout:develop-eggs-directory}
+
+[software_definition]
+software_home = ${buildout:software_home}
+executable = ${software_definition:executable}