Commit 7daada49 authored by jim's avatar jim

Factor out setuptools support from the buildout

git-svn-id: http://svn.zope.org/repos/main/zc.buildout/trunk@69872 62d5b8a3-27da-0310-9561-8e5933582275
parent 11dc43e6
=============
Zope Buildout
=============
.. contents::
The Zope Buildout project provides support for creating applications,
especially Python applications. It provides tools for assembling
applications from multiple parts, Python or otherwise. An application
may actually contain multiple programs, processes, and configuration
settings.
Here's an example of such an application that we built with an earlier
prototype of the buildout system. We have a Zope application consisting of:
- Multiple Zope instances
- 4 ZEO servers
- An LDAP server
- Cache-invalidation and Mail delivery servers
- Dozens of add-on packages
- Multiple test runners
- Multiple deployment modes, including dev, stage, and prod,
with prod deployment over multiple servers
Parts installed include:
- Application software installs, including Zope, ZEO and LDAP
software
- Add-on packages
- Bundles of configuration that define Zope, ZEO and LDAP instances
- Utility scripts such as test runners, server-control
scripts, cron jobs.
This is all defined using configuration files and recipes, which are
software that build and installs parts based on configuration data.
The prototype system has minimal documentation and no tests and
has no egg support. (It build on earlier make-based systems that had
no documentation or tests.)
This project provides a non-prototype implementation of the ideas and
knowledge gained from earlier efforts and leverages setuptools to make
recipe management cleaner and to provide better Python package and
script management.
The word "buildout" refers to a description of a set of parts and the
software to create and assemble them. It is often used informally to
refer to an installed system based on a buildout definition. For
example, if we are creating an application named "Foo", then "the Foo
buildout" is the collection of configuration and application-specific
software that allows an instance of the application to be created. We
may refer to such an instance of the application informally as "a Foo
buildout".
I expect that, for many Zope packages, we'll arrange the package
projects in subversion as buildouts. To work on the package, someone
will check the project out of Subversion and build it. Building it
will assemble all of packages and programs needed to work on it. For
example, a buildout for a project to provide a new security policy
will include the source of the policy and specifications to build the
application for working on it, including:
- a test runner
- a web server for running the user interface
- supporting packages
A buildout will typically contain a copy of bootstrap.py. When
someone checks out the project, they'll run bootstrap.py, which will
- create support directories, like bin, eggs, and work, as needed,
- download and install the zc.buildout and setuptools eggs,
- run bin/build (created by installing zc.buildout) to build the
application.
Buildouts are defined using configuration files. These files are
based on the Python ConfigParser module with some variable-definition
and substitution extensions.
Installation
============
There are two ways to install zc,buildout
1. Install it as an egg using `easy_install
<http://peak.telecommunity.com/DevCenter/EasyInstall>`_ into a
Python instaallation. Then just use the buildout script from your
Python bin or Scripts directory.
2. Use the `bootstrap script
<http://dev.zope.org/Buildout/bootstrap.py>`_ to install setuptools
and the buildout software into your buildout. Typically, you'll
check the bootstrap script into your project so that, whenever you
checkout your project, you can turn it into a buildout by just
running the bootstrap script.
More information
================
The detailed documentation for the various parts of buildout can be
found in the following files:
`buildout.txt <http://dev.zope.org/Buildout/buildout.html>`_
Describes how to define and run buildouts. It also describes how
to write recipes.
`easy_install.txt <http://dev.zope.org/Buildout/easy_install.html>`_
Describes an Python APIs for invoking easy_install for generation
of scripts with paths baked into them.
Download
========
You can download zc.buildout and many buildout recipes from the
`Python Package Index <http://www.python.org/pypi>`_.
Recipes
=======
Existing recipes include:
`zc.recipe.egg <http://dev.zope.org/Buildout/egg.html>`_
The egg recipe installes one or more eggs, with their
dependencies. It installs their console-script entry points with
the needed eggs included in their paths.
`zc.recipe.testrunner <http://dev.zope.org/Buildout/testrunner.html>`_
The testrunner egg installs creates a test runner script for one or
more eggs.
`zc.recipe.zope3checkout <http://dev.zope.org/Buildout/zope3checkout.html>`_
The zope3checkout recipe installs a Zope 3 checkout into a
buildout.
`zc.recipe.zope3instance <http://dev.zope.org/Buildout/zope3instance.html>`_
The zope3instance recipe sets up a Zope 3 instance.
`zc.recipe.filestorage <http://dev.zope.org/Buildout/filestorage.html>`_
The filestorage recipe sets up a ZODB file storage for use in a
Zope 3 instance creayed by the zope3instance recipe.
Buildout examples
=================
Some simple buildout examples:
`The zc.buildout project <http://svn.zope.org/zc.buildout/trunk>`_
This is the project for the buildout software itself, which is
developed as a buildout.
`The zc sharing project <http://svn.zope.org/zc.sharing/trunk>`_
This project illistrates using the buildout software with Zope 3.
Note that the bootstrap.py file is checked in so that a buildout
can be made when the project is checked out. The buildout.cfg
specified everything needed to create a Zope 3 installation with
the zc.sharing package installed in development mode.
Status
======
The buildout system is under active development. Some near term
priorities include:
- Better error reporing
- Windows support
- Handling of egg extras
- More recipes
Questions
=========
You can send questions to jim@zope.com.
Change History
==============
1.0.0b3
-------
- Added Windows support.
- Fixed some bugs in variable substitutions.
The characters "-", "." and " ", weren't allowed in section or
option names.
Substitutions with invalid names were ignored, which caused
missleading failures downstream.
- Improved error handling. No longer show tracebacks for user errors.
- Now require a recipe option (and therefore a section) for every part.
1.0.0b2
-------
Added support for specifying some build_ext options when installing eggs
from source distributions.
1.0.0b1
-------
- Changed the bootstrapping code to only install setuptools and
zc.buildout. The bootstrap code no-longer runs the buildout itself.
This was to fix a bug that caused parts to be recreated
unnecessarily because the recipe signature in the initial buildout
reflected temporary locations for setuptools and zc.buildout.
- Now create a minimal setup.py if it doesn't exist and issue a
warning that it is being created.
- Fixed bug in saving installed configuration data. %'s and extra
spaces weren't quoted.
1.0.0a1
-------
Initial public version
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap a buildout-based project
Simply run this script in a directory containing a buildout.cfg.
The script accepts buildout command-line options, so you can
use the -c option to specify an alternate configuration file.
$Id$
"""
import os, shutil, sys, tempfile, urllib2
tmpeggs = tempfile.mkdtemp()
ez = {}
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
).read() in ez
ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
import pkg_resources
cmd = 'from setuptools.command.easy_install import main; main()'
if sys.platform == 'win32':
cmd = '"%s"' % cmd # work around spawn lamosity on windows
ws = pkg_resources.working_set
assert os.spawnle(
os.P_WAIT, sys.executable, sys.executable,
'-c', cmd, '-mqNxd', tmpeggs, 'zc.buildout',
dict(os.environ,
'PYTHONPATH'=
ws.find(pkg_resources.Requirement.parse('setuptools')).location
),
) == 0
ws.add_entry(tmpeggs)
ws.require('zc.buildout')
import zc.buildout.buildout
zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap'])
shutil.rmtree(tmpeggs)
[buildout]
develop = zc.recipe.egg_ zc.recipe.testrunner zc.buildoutsupport
parts = test
# prevent slow access to cheeseshop:
index = http://download.zope.org
[test]
recipe = zc.recipe.testrunner
eggs =
zc.buildout
zc.recipe.egg
zc.recipe.testrunner
##############################################################################
#
# Copyright (c) 2005 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Bootstrap the buildout project itself.
This is different from a normal boostrapping process because the
buildout egg itself is installed as a develop egg.
$Id$
"""
import os, sys, urllib2
for d in 'eggs', 'develop-eggs', 'bin':
if not os.path.exists(d):
os.mkdir(d)
ez = {}
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py'
).read() in ez
ez['use_setuptools'](to_dir='eggs', download_delay=0)
import pkg_resources
os.spawnle(os.P_WAIT, sys.executable, sys.executable, 'setup.py',
'-q', 'develop', '-m', '-x', '-d', 'develop-eggs',
{'PYTHONPATH': os.path.dirname(pkg_resources.__file__)},
)
pkg_resources.working_set.add_entry('src')
import zc.buildout.easy_install
zc.buildout.easy_install.scripts(
['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin')
sys.exit(os.spawnl(os.P_WAIT, 'bin/buildout', 'bin/buildout'))
from setuptools import setup, find_packages
name = "zc.buildout"
setup(
name = name,
version = "1.0.0b2",
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "System for managing development buildouts",
long_description=open('README.txt').read(),
license = "ZPL 2.1",
keywords = "development build",
url='http://svn.zope.org/zc.buildout',
data_files = [('.', ['README.txt'])],
packages = ['zc', 'zc.buildout'],
package_dir = {'': 'src'},
namespace_packages = ['zc'],
install_requires = 'setuptools',
include_package_data = True,
tests_require = ['zope.testing'],
test_suite = name+'.tests.test_suite',
entry_points = {'console_scripts':
['buildout = %s.buildout:main' % name]},
dependency_links = ['http://download.zope.org/distribution/'],
)
This directory is (an experimental) place to manage Launchpad
specification bodies. Files here are referenced (via svn.zope.org
URLs) from the buildout project specifications in Launchpad,
https://features.launchpad.net/products/zc.buildout.
Repeatable (taggable) buildouts
===============================
It's important to be able to tag a buildout in a software repository
in such a way that, months, or even years later, the buildout tag can
be checked out and used to construct the same collection of parts,
with the same versions. (Note that parts could still behave
differently due to changes in parts of the environment, such as system
libraries, not controlled by the buildout.)
A feature of the buildout is it's use of eggs and the automatic
resolution of dependencies. The latest versions of dependencies are
automatically downloaded and installed. This is great during
development or when using the buildout for casual software
development, but it doesn't work very well for reproducing an old
buildout.
What's needed is some way to, when needed, record information about
the versions of eggs (and any other bits) who's versions are
determined dynamically.
Proposal
--------
We'll add a buildout option, create-repeatable. The option will
specify a file into which option information should be saved to create
a repeatable buildout. The data will be saved in a form that can be
used by the buildout or recipes in a later run. To make a tagged
buildout, a user would run the buildout with the create-repeatable
option set to a file name and then modify the buildout to be
extended-by this file.
Consider the following example buildout.cfg::
[buildout]
parts = foo
[foo]
recipe = zc.recipe.eggs
eggs = foo
eek
Now assume that:
- The current version of foo is 1.1
- Foo depends on bar =, which depends on baz. The current versions of
bar and bas are 1.1 and 2.1.
- The current version of eek is 1.5
- eek depends on ook, which is as version 1.3.
- zc.recipe.egg is at version 1.0b5
If we run the buildout with the command-line option::
buildout:create-repeatable=reapeatable.cfg
we'll get a repeatable.cfg file that looks something like::
[foo]
recipe = zc.recipe.eggs ==1.0b5
static = true
eggs = foo ==1.1
bar ==1.1
baz ==2.1
eek ==1.5
ook ==1.3
The file contains options for the foo part. The buildout software
itself added an entry for the recipe that fixes the recipe version
at the version used by the buildout.
The zc.recipe.eggs recipe added the eggs option that lists the
specifoc releases that were assembled.
Finally the buildout.cfg file can be modified to use the
repeatable.cfg file::
[buildout]
parts = foo
extended-by: repeatable.cfg
[foo]
recipe = zc.recipe.eggs
eggs = foo
eek
When the buildout is run, the options in repeatable.cfg will override
the onces in buildout.cfg, providing a repeatable buildout
Python API
----------
The recipe API will grow a repeatable method that is called after the
install method and is passed a dictionary that a recipe can store
option data in. A recipe instance will only be able to provide repeatable data
for it's part.
try:
__import__('pkg_resources').declare_namespace(__name__)
except:
# bootstrapping
pass
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
- Use setuptools PackageIndex objects to improve performance by
deciding whether we need to download anything without using
easy_install.
- tests
- distribution dependency links
- offline mode (there is an indirect test in the testrunner tests)
- Load from urls
- control python for develop (probbaly a new recipe)
- proper handling of extras
- Common recipes
- configure-make-make-install
- download, checkout
- Should it be possible to provide multiple recipies?
Or should recipies be combined through inheritence (or
composition)?
- Python
- Some way to freeze versions so we can have reproducable buildouts.
Maybe simple approach:
- Egg recipe outputs dependency info with debug logging
- Egg recipe has option to specify dependencies. When used,
don't automatically fetch newer data.
- Option to search python path for distros
- Part dependencies
- custom uninstall
- Fix develop so thet ordinary eggs fetched as dependencies end up
in eggs directory.
"fixed" except that fix is ineffective due to setuptools bug. :(
- spelling :)
- document recipe initialization order
Issues
- Should we include setuptools and buildout eggs for buildout process
in environment when searching for requirements?
- We don't want to look for new versions of setuptools all the time.
For now, we always use a local dist if there is one. Needs more
thought.
Buildout Egg-Installation Recipe
================================
The egg-installation recipe installes eggs into a buildout eggs
directory. It also generates scripts in a buildout bin directory with
egg paths baked into them.
The recipe provides the following options:
eggs
A list of eggs to install given as one ore more setuptools
requirement strings. Each string must be given on a separate
line.
find-links
One or more addresses of link servers to be searched for
distributions. This is optional. If not specified, links
specified in the buildout section will be used, if any.
index
The optional address of a distribution index server. If not
specified, then the option from the buildout section will be
used. If not specified in the part data or in the buildout
section, then http://www.python.org/pypi is used.
python
The name of a section defining the Python executable to use.
This defaults to buildout.
scripts
Control which scripts are generated. The value should be a list of
zero or more tokens. Each token is either a name, or a name
followed by an '=' and a new name. Only the named scripts are
generated. If no tokens are given, then script generation is
disabled. If the option isn't given at all, then all scripts
defined by the named eggs will be generated.
Custom eggs
-----------
The zc.recipe.egg:custom recipe supports building custom eggs,
currently with specialized options for building extensions.
extra-paths
Extra paths to include in a generates script.
To do
-----
- Some way to freeze the egg-versions used. This includes some way to
record which versions were selected dynamially and then a way to
require that the recorded versions be used in a later run.
- More control over script generation. In particular, some way to
specify data t be recored in the script.
Change History
==============
1.0.0a2
-------
Added a new recipe for building custom eggs from source distributions,
specifying custom distutils build_ext options.
1.0.0a1
-------
Initial public version
from setuptools import setup, find_packages
name = "zc.recipe.egg"
setup(
name = name,
version = "1.0.0a2",
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "Recipe for installing Python package distributions as eggs",
long_description = open('README.txt').read(),
license = "ZPL 2.1",
keywords = "development build",
url='http://svn.zope.org/zc.buildout',
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout', 'setuptools'],
tests_require = ['zope.testing'],
test_suite = name+'.tests.test_suite',
entry_points = {'zc.buildout': ['default = %s:Egg' % name,
'custom = %s:Custom' % name,
]
},
dependency_links = ['http://download.zope.org/distribution/'],
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
Installation of distributions as eggs
=====================================
The zc.recipe.egg recipe can be used to install various types if
distutils distributions as eggs. It takes a number of options:
eggs
A list of eggs to install given as one ore more setuptools
requirement strings. Each string must be given on a separate
line.
find-links
A list of URLs, files, or directories to search for distributions.
index
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index,
http://cheeseshop.python.org/pypi, is used. You can specify an
alternate index with this option. If you use the links option and
if the links point to the needed distributions, then the index can
be anything and will be largely ignored. In the examples, here,
we'll just point to an empty directory on our link server. This
will make our examples run a little bit faster.
python
The name of a section to get the Python executable from.
If not specified, then the buildout python option is used. The
Python executable is found in the executable option of the named
section.
scripts
Control which scripts are generated. The value should be a list of
zero or more tokens. Each token is either a name, or a name
followed by an '=' and a new name. Only the named scripts are
generated. If no tokens are given, then script generation is
disabled. If the option isn't given at all, then all scripts
defined by the named eggs will be generated.
extra-paths
Extra paths to include in a generates script.
We have a link server that has a number of eggs:
>>> print get(link_server),
<html><body>
<a href="demo-0.1-py2.3.egg">demo-0.1-py2.3.egg</a><br>
<a href="demo-0.2-py2.3.egg">demo-0.2-py2.3.egg</a><br>
<a href="demo-0.3-py2.3.egg">demo-0.3-py2.3.egg</a><br>
<a href="demoneeded-1.0-py2.3.egg">demoneeded-1.0-py2.3.egg</a><br>
<a href="demoneeded-1.1-py2.3.egg">demoneeded-1.1-py2.3.egg</a><br>
<a href="index/">index/</a><br>
<a href="other-1.0-py2.3.egg">other-1.0-py2.3.egg</a><br>
</body></html>
We have a sample buildout. Let's update it's configuration file to
install the demo package.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))
In this example, we limited ourself to revisions before 0.3. We also
specified where to find distributions using the find-links option.
Let's run the buildout:
>>> import os
>>> os.chdir(sample_buildout)
>>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(buildout),
Now, if we look at the buildout eggs directory:
>>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg
- demoneeded-1.1-py2.3.egg
- setuptools-0.6-py2.3.egg
- zc.buildout-1.0-py2.3.egg
We see that we got an egg for demo that met the requirement, as well
as the egg for demoneeded, wich demo requires. (We also see an egg
link for the recipe. This egg link was actually created as part of
the sample buildout setup. Normally, when using the recipe, you'll get
a regular egg installation.)
The demo egg also defined a script and we see that the script was
installed as well:
>>> ls(sample_buildout, 'bin')
- buildout
- demo
- py-demo
- py-zc.buildout
Here, in addition to the buildout script, we see the demo script,
demo, and we see a script, py-demo, for giving us a Python prompt with
the path for demo and any eggs it depends on included in sys.path.
This is useful for debugging and testing.
If we run the demo script, it prints out some minimal data:
>>> print system(os.path.join(sample_buildout, 'bin', 'demo')),
2 1
The value it prints out happens to be some values defined in the
modules installed.
We can also run the py-demo script. Here we'll just print out
the bits if the path added to reflect the eggs:
>>> print system(os.path.join(sample_buildout, 'bin', 'py-demo'),
... """import os, sys
... for p in sys.path:
... if 'demo' in p:
... print os.path.basename(p)
...
... """).replace('>>> ', '').replace('... ', ''),
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
demo-0.2-py2.4.egg
demoneeded-1.1-py2.4.egg
The recipe gets the most recent distribution that satisfies the
specification. For example, We remove the restriction on demo:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))
>>> print system(buildout),
Then we'll get a new demo egg:
>>> ls(sample_buildout, 'eggs')
- demo-0.2-py2.3.egg
- demo-0.3-py2.3.egg
- demoneeded-1.0-py2.3.egg
- setuptools-0.6-py2.4.egg
- zc.buildout-1.0-py2.4.egg
Note that we removed the eggs option, and the eggs
defaulted to the part name.
The script is updated too:
>>> print system(os.path.join(sample_buildout, 'bin', 'demo')),
3 1
You can control which scripts get generated using the scripts option.
For example, to suppress scripts, use the scripts option without any
arguments:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... find-links = %(server)s
... index = %(server)s/index
... scripts =
... """ % dict(server=link_server))
>>> print system(buildout),
>>> ls(sample_buildout, 'bin')
- buildout
- py-zc.buildout
You can also control the name used for scripts:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... find-links = %(server)s
... index = %(server)s/index
... scripts = demo=foo
... """ % dict(server=link_server))
>>> print system(buildout),
>>> ls(sample_buildout, 'bin')
- buildout
- foo
- py-zc.buildout
If we need to include extra paths in a script, we can use the
extra-paths option:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = zc.recipe.egg
... find-links = %(server)s
... index = %(server)s/index
... scripts = demo=foo
... extra-paths =
... /foo/bar
... /spam/eggs
... """ % dict(server=link_server))
>>> print system(buildout),
Let's look at the script that was generated:
>>> cat(sample_buildout, 'bin', 'foo') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.3
<BLANKLINE>
import sys
sys.path[0:0] = [
'/tmp/xyzsample-install/demo-0.3-py2.3.egg',
'/tmp/xyzsample-install/demoneeded-1.1-py2.3.egg',
'/foo/bar',
'/spam/eggs'
]
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main()
Offline mode
------------
If the buildout offline option is set to "true", then no attempt will
be made to contact an index server:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
... offline = true
...
... [demo]
... recipe = zc.recipe.egg
... index = eek!
... scripts = demo=foo
... """ % dict(server=link_server))
>>> print system(buildout),
from zc.recipe.egg.egg import Egg
from zc.recipe.egg.custom import Custom
Egg Recipe API for other Recipes
================================
It is common for recipes to accept a collection of egg specifications
and generate scripts based on the resulting working sets. The egg
recipe provides an API that other recipes can use.
A recipe can reuse the egg recipe, supporting the eggs, find-links,
index, and python options. This is done by creating an egg recipe
instance in a recipes's contructor. In the recipe's install script,
the egg-recipe instance's working_set method is used to collect the
requested eggs and working set.
To illustrate, we create a sample recipe that is a very thin layer
around the egg recipe:
>>> mkdir(sample_buildout, 'sample')
>>> write(sample_buildout, 'sample', 'sample.py',
... """
... import logging, os
... import zc.recipe.egg
...
... class Sample:
...
... def __init__(self, buildout, name, options):
... self.egg = zc.recipe.egg.Egg(buildout, name, options)
... self.name = name
... self.options = options
...
... def install(self):
... extras = self.options['extras'].split()
... requirements, ws = self.egg.working_set(extras)
... print 'Part:', self.name
... print 'Egg requirements:'
... for r in requirements:
... print r
... print 'Working set:'
... for d in ws:
... print d
... """)
Here we instantiated the egg recipe in the constructor, saving it in
an attribute. This also initialized the options dictionary.
In our install method, we called the working_set method on the
instance we saved. The working_set method takes an optional sequence
of extra requirements to be included in the working set.
>>> write(sample_buildout, 'sample', 'setup.py',
... """
... from setuptools import setup
...
... setup(
... name = "sample",
... entry_points = {'zc.buildout': ['default = sample:Sample']},
... install_requires = 'zc.recipe.egg',
... )
... """)
>>> write(sample_buildout, 'sample', 'README.txt', " ")
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... develop = sample
... parts = sample-part
...
... [sample-part]
... recipe = sample
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)sindex
... extras = other
... """ % dict(server=link_server))
>>> import os
>>> os.chdir(sample_buildout)
>>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(buildout),
Part: sample-part
Egg requirements:
demo<0.3
Working set:
demo 0.2
other 1.0
demoneeded 1.1
We can see that the options were augmented with additionl data
computed by the egg recipe by looking at .installed.cfg:
>>> cat(sample_buildout, '.installed.cfg')
[buildout]
parts = sample-part
<BLANKLINE>
[sample-part]
__buildout_installed__ =
__buildout_signature__ = sample-6aWMvV2EJ9Ijq+bR8ugArQ==
zc.recipe.egg-cAsnudgkduAa/Fd+WJIM6Q==
setuptools-0.6-py2.4.egg
zc.buildout-+rYeCcmFuD1K/aB77XTj5A==
_b = /tmp/tmpb7kP9bsample-buildout/bin
_d = /tmp/tmpb7kP9bsample-buildout/develop-eggs
_e = /tmp/tmpb7kP9bsample-buildout/eggs
eggs = demo<0.3
executable = /usr/local/bin/python2.3
extras = other
find-links = http://localhost:27071/
index = http://localhost:27071/index
recipe = sample
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Install packages as eggs
$Id$
"""
import os, re, zipfile
import zc.buildout.easy_install
class Custom:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
links = options.get('find-links',
buildout['buildout'].get('find-links'))
if links:
links = links.split()
options['find-links'] = '\n'.join(links)
else:
links = ()
self.links = links
index = options.get('index', buildout['buildout'].get('index'))
if index is not None:
options['index'] = index
self.index = index
options['_b'] = buildout['buildout']['bin-directory']
options['_e'] = buildout['buildout']['eggs-directory']
options['_d'] = buildout['buildout']['develop-eggs-directory']
assert options.get('unzip') in ('true', 'false', None)
python = options.get('python', buildout['buildout']['python'])
options['executable'] = buildout[python]['executable']
build_ext = {}
for be_option in ('include-dirs', 'library-dirs', 'rpath'):
value = options.get(be_option)
if value is None:
continue
value = [
os.path.join(
buildout['buildout']['directory'],
v.strip()
)
for v in value.strip().split('\n')
if v.strip()
]
build_ext[be_option] = ':'.join(value)
options[be_option] = ':'.join(value)
self.build_ext = build_ext
def install(self):
if self.buildout['buildout'].get('offline') == 'true':
return
options = self.options
distribution = options.get('eggs', self.name).strip()
build_ext = dict([
(k, options[k])
for k in ('include-dirs', 'library-dirs', 'rpath')
if k in options
])
zc.buildout.easy_install.build(
distribution, options['_d'], self.build_ext,
self.links, self.index, options['executable'], [options['_e']],
)
Custon eggs
===========
Sometimes, It's necessary to provide extra control over how an egg is
created. This is commonly true for eggs with extension modules that
need to access libraries or include files.
The zc.recipe.egg:custom recipe can be used to define an egg with
custom build parameters. The currently defined parameters are:
include-dirs
A new-line separated list of directories to search for include
files.
library-dirs
A new-line separated list of directories to search for libraries
to link with.
rpath
A new-line separated list of directories to search for dynamic libraries
at run time.
In addition, the following options can be used to specify the egg:
egg
An eggs to install given as a setuptools requirement string.
This defaults to the part name.
find-links
A list of URLs, files, or directories to search for distributions.
index
The URL of an index server, or almost any other valid URL. :)
If not specified, the Python Package Index,
http://cheeseshop.python.org/pypi, is used. You can specify an
alternate index with this option. If you use the links option and
if the links point to the needed distributions, then the index can
be anything and will be largely ignored. In the examples, here,
we'll just point to an empty directory on our link server. This
will make our examples run a little bit faster.
python
The name of a section to get the Python executable from.
If not specified, then the buildout python option is used. The
Python executable is found in the executable option of the named
section.
To illustrate this, we'll define a buildout that builds an egg for a
package that has a simple extension module::
#include <Python.h>
#include <extdemo.h>
static PyMethodDef methods[] = {};
PyMODINIT_FUNC
initextdemo(void)
{
PyObject *d;
d = Py_InitModule3("extdemo", methods, "");
PyDict_SetItemString(d, "val", PyInt_FromLong(EXTDEMO));
}
The extension depends on a system-dependnt include file, extdemo.h,
that defines a constant, EXTDEMO, that is exposed by the extension.
The extension module is available as a source distribution,
extdemo-1.4.tar.gz, on a distribution server.
We have a sample buildout that we'll add an include directory to with
the necessary include file:
>>> mkdir(sample_buildout, 'include')
>>> import os
>>> open(os.path.join(sample_buildout, 'include', 'extdemo.h'), 'w').write(
... "#define EXTDEMO 42\n")
We'll also update the buildout configuration file to define a part for
the egg:
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = extdemo
...
... [extdemo]
... recipe = zc.recipe.egg:custom
... find-links = %(server)s
... index = %(server)s/index
... include-dirs = include
... """ % dict(server=link_server))
>>> os.chdir(sample_buildout)
>>> buildout = os.path.join(sample_buildout, 'bin', 'buildout')
>>> print system(buildout),
zip_safe flag not set; analyzing archive contents...
We got the zip_safe warning because the source distribution we used
wasn't setuptools based and thus didn't set the option.
The egg is created in the develop-eggs directory *not* the eggs
directory because it depends on buildout-specific parameters and the
eggs directory can be shared across multiple buildouts.
>>> ls(sample_buildout, 'develop-eggs')
d extdemo-1.4-py2.4-unix-i686.egg
- zc.recipe.egg.egg-link
Note that no scripts or dependencies are installed. To install
dependencies or scripts for a custom egg, define another part and use
the zc.recipe.egg recipe, listing the custom egg as one of the eggs to
be installed. The zc.recipe.egg recipe will use the installed egg.
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Install packages as eggs
$Id$
"""
import os, re, zipfile
import zc.buildout.easy_install
class Egg:
def __init__(self, buildout, name, options):
self.buildout = buildout
self.name = name
self.options = options
links = options.get('find-links',
buildout['buildout'].get('find-links'))
if links:
links = links.split()
options['find-links'] = '\n'.join(links)
else:
links = ()
self.links = links
index = options.get('index', buildout['buildout'].get('index'))
if index is not None:
options['index'] = index
self.index = index
self.extra_paths = [
os.path.join(buildout['buildout']['directory'], p.strip())
for p in options.get('extra-paths', '').split('\n')
if p.strip()
]
if self.extra_paths:
options['extra-paths'] = '\n'.join(self.extra_paths)
options['_b'] = buildout['buildout']['bin-directory']
options['_e'] = buildout['buildout']['eggs-directory']
options['_d'] = buildout['buildout']['develop-eggs-directory']
assert options.get('unzip') in ('true', 'false', None)
python = options.get('python', buildout['buildout']['python'])
options['executable'] = buildout[python]['executable']
def working_set(self, extra=()):
"""Separate method to just get the working set
This is intended for reuse by similar recipes.
"""
options = self.options
distributions = [
r.strip()
for r in options.get('eggs', self.name).split('\n')
if r.strip()]
orig_distributions = distributions[:]
distributions.extend(extra)
if self.buildout['buildout'].get('offline') == 'true':
ws = zc.buildout.easy_install.working_set(
distributions, options['executable'],
[options['_d'], options['_e']]
)
else:
ws = zc.buildout.easy_install.install(
distributions, options['_e'],
links = self.links,
index = self.index,
executable = options['executable'],
always_unzip=options.get('unzip') == 'true',
path=[options['_d']]
)
return orig_distributions, ws
def install(self):
distributions, ws = self.working_set()
options = self.options
scripts = options.get('scripts')
if scripts or scripts is None:
if scripts is not None:
scripts = scripts.split()
scripts = dict([
('=' in s) and s.split('=', 1) or (s, s)
for s in scripts
])
return zc.buildout.easy_install.scripts(
distributions, ws, options['executable'],
options['_b'],
scripts=scripts,
extra_paths=self.extra_paths)
This diff is collapsed.
Test-Runner Recipe
==================
This recipe generates zope.testing test-runenr scripts for testing a
collection of eggs. The eggs must already be installed (using the
zc.recipe.egg recipe)
The test-runner recipe has 2 options:
- The eggs option takes the names of the eggs to be
tested. These are not installed by the recipe. They must be
installed by some other recipe (or using the buildout develop
option). The distributions are in the form os setuptools
requirements. Multiple distributions must be listed on separate
lines. This option is required.
- The script option gives the name of the script to generate, in the
buildout bin directory. Of the option isn't used, the part name
will be used.
To do
-----
- Support specifying testrunner defaults (e.g. verbosity, test file
patterns, etc.)
from setuptools import setup, find_packages
name = "zc.recipe.testrunner"
setup(
name = name,
version = "1.0.0a1",
author = "Jim Fulton",
author_email = "jim@zope.com",
description = "ZC Buildout recipe for creating test runners",
long_description=open('README.txt').read(),
license = "ZPL 2.1",
keywords = "development build testing",
url='http://svn.zope.org/zc.buildout',
packages = find_packages('src'),
include_package_data = True,
package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'],
install_requires = ['zc.buildout', 'zope.testing', 'setuptools',
'zc.recipe.egg',
],
test_suite = name+'.tests.test_suite',
entry_points = {'zc.buildout': ['default = %s:TestRunner' % name]},
dependency_links = ['http://download.zope.org/distribution/'],
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment