Commit 96c1abf9 authored by gary's avatar gary

merge lp:~gary/zc.buildout/python-support (also available as...

merge lp:~gary/zc.buildout/python-support (also available as svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-launchpad).  This adds support for system Pythons, as well as other changes, such as improved Distribute support and some cleanups.

git-svn-id: http://svn.zope.org/repos/main/zc.buildout/trunk@111588 62d5b8a3-27da-0310-9561-8e5933582275
parent 9a39ba18
.installed.cfg
bin
build
develop-eggs
eggs
parts
src/zc.buildout.egg-info
z3c.recipe.scripts_/src/z3c.recipe.scripts.egg-info
zc.recipe.egg_/src/zc.recipe.egg.egg-info
Change History Change History
************** **************
1.4.4 (?) 1.5.0 (201?-??-??)
========= ==================
New Features:
- Added buildout:socket-timout option so that socket timeout can be configured
both from command line and from config files. (gotcha)
- Buildout can be safely used with a system Python (or any Python with code
in site-packages), as long as you use the new z3c.recipe.scripts
recipe to generate scripts and interpreters, rather than zc.recipe.egg.
zc.recipe.egg is still a fully supported, and simpler, way of
generating scripts and interpreters if you are using a "clean" Python,
without code installed in site-packages. It keeps its previous behavior in
order to provide backwards compatibility.
The z3c.recipe.scripts recipe allows you to control how you use the
code in site-packages. You can exclude it entirely; allow eggs in it
to fulfill package dependencies declared in setup.py and buildout
configuration; allow it to be available but not used to fulfill
dependencies declared in setup.py or buildout configuration; or only
allow certain eggs in site-packages to fulfill dependencies.
- Added new function, ``zc.buildout.easy_install.sitepackage_safe_scripts``,
to generate scripts and interpreter. It produces a full-featured
interpreter (all command-line options supported) and the ability to
safely let scripts include site packages, such as with a system
Python. The ``z3c.recipe.scripts`` recipe uses this new function.
- Improve bootstrap.
* New options let you specify where to find ez_setup.py and where to find
a download cache. These options can keep bootstrap from going over the
network.
* Another new option lets you specify where to put generated eggs.
* The buildout script generated by bootstrap honors more of the settings
in the designated configuration file (e.g., buildout.cfg).
- You can develop zc.buildout using Distribute instead of Setuptools. Use
the --distribute option on the dev.py script. (Releases should be tested
with both Distribute and Setuptools.)
- The ``distribute-version`` now works in the [buildout] section, mirroring
the ``setuptools-version`` option (this is for consistency; using the
general-purpose ``versions`` option is preferred).
Bugs fixed:
- Using Distribute with the ``allow-picked-versions = false`` buildout
option no longer causes an error.
- The handling and documenting of default buildout options was normalized.
This means, among other things, that ``bin/buildout -vv`` and
``bin/buildout annotate`` correctly list more of the options.
- Installing a namespace package using a Python that already has a package
in the same namespace (e.g., in the Python's site-packages) failed in
some cases. It is now handled correctly.
- Another variation of this error showed itself when at least two
dependencies were in a shared location like site-packages, and the
first one met the "versions" setting. The first dependency would be
added, but subsequent dependencies from the same location (e.g.,
site-packages) would use the version of the package found in the
shared location, ignoring the version setting. This is also now
handled correctly.
1.4.3 (2009-12-10) 1.4.3 (2009-12-10)
================== ==================
......
...@@ -35,7 +35,15 @@ Existing recipes include: ...@@ -35,7 +35,15 @@ Existing recipes include:
`zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_ `zc.recipe.egg <http://pypi.python.org/pypi/zc.recipe.egg>`_
The egg recipe installes one or more eggs, with their The egg recipe installes one or more eggs, with their
dependencies. It installs their console-script entry points with dependencies. It installs their console-script entry points with
the needed eggs included in their paths. the needed eggs included in their paths. It is suitable for use with
a "clean" Python: one without packages installed in site-packages.
`z3c.recipe.scripts <http://pypi.python.org/pypi/z3c.recipe.scripts>`_
Like zc.recipe.egg, this recipe builds interpreter scripts and entry
point scripts based on eggs. It can be used with a Python that has
packages installed in site-packages, such as a system Python. The
interpreter also has more features than the one offered by
zc.recipe.egg.
`zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_ `zc.recipe.testrunner <http://pypi.python.org/pypi/zc.recipe.testrunner>`_
The testrunner egg creates a test runner script for one or The testrunner egg creates a test runner script for one or
......
...@@ -20,102 +20,187 @@ use the -c option to specify an alternate configuration file. ...@@ -20,102 +20,187 @@ use the -c option to specify an alternate configuration file.
$Id$ $Id$
""" """
import os, shutil, sys, tempfile, urllib2 import os, shutil, sys, tempfile, textwrap, urllib, urllib2
from optparse import OptionParser from optparse import OptionParser
tmpeggs = tempfile.mkdtemp() if sys.platform == 'win32':
def quote(c):
if ' ' in c:
return '"%s"' % c # work around spawn lamosity on windows
else:
return c
else:
quote = str
# In order to be more robust in the face of system Pythons, we want to
# run without site-packages loaded. This is somewhat tricky, in
# particular because Python 2.6's distutils imports site, so starting
# with the -S flag is not sufficient. However, we'll start with that:
if 'site' in sys.modules:
# We will restart with python -S.
args = sys.argv[:]
args[0:0] = [sys.executable, '-S']
args = map(quote, args)
os.execv(sys.executable, args)
# Now we are running with -S. We'll get the clean sys.path, import site
# because distutils will do it later, and then reset the path and clean
# out any namespace packages from site-packages that might have been
# loaded by .pth files.
clean_path = sys.path[:]
import site
sys.path[:] = clean_path
for k, v in sys.modules.items():
if (hasattr(v, '__path__') and
len(v.__path__)==1 and
not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
# This is a namespace package. Remove it.
sys.modules.pop(k)
is_jython = sys.platform.startswith('java') is_jython = sys.platform.startswith('java')
setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
distribute_source = 'http://python-distribute.org/distribute_setup.py'
# parsing arguments # parsing arguments
parser = OptionParser() def normalize_to_url(option, opt_str, value, parser):
if value:
if '://' not in value: # It doesn't smell like a URL.
value = 'file://%s' % (
urllib.pathname2url(
os.path.abspath(os.path.expanduser(value))),)
if opt_str == '--download-base' and not value.endswith('/'):
# Download base needs a trailing slash to make the world happy.
value += '/'
else:
value = None
name = opt_str[2:].replace('-', '_')
setattr(parser.values, name, value)
usage = '''\
[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
Bootstraps a buildout-based project.
Simply run this script in a directory containing a buildout.cfg, using the
Python that you want bin/buildout to use.
Note that by using --setup-source and --download-base to point to
local resources, you can keep this script from going over the network.
'''
parser = OptionParser(usage=usage)
parser.add_option("-v", "--version", dest="version", parser.add_option("-v", "--version", dest="version",
help="use a specific zc.buildout version") help="use a specific zc.buildout version")
parser.add_option("-d", "--distribute", parser.add_option("-d", "--distribute",
action="store_true", dest="distribute", default=False, action="store_true", dest="use_distribute", default=False,
help="Use Distribute rather than Setuptools.") help="Use Distribute rather than Setuptools.")
parser.add_option("--setup-source", action="callback", dest="setup_source",
callback=normalize_to_url, nargs=1, type="string",
help=("Specify a URL or file location for the setup file. "
"If you use Setuptools, this will default to " +
setuptools_source + "; if you use Distribute, this "
"will default to " + distribute_source +"."))
parser.add_option("--download-base", action="callback", dest="download_base",
callback=normalize_to_url, nargs=1, type="string",
help=("Specify a URL or directory for downloading "
"zc.buildout and either Setuptools or Distribute. "
"Defaults to PyPI."))
parser.add_option("--eggs",
help=("Specify a directory for storing eggs. Defaults to "
"a temporary directory that is deleted when the "
"bootstrap script completes."))
parser.add_option("-c", None, action="store", dest="config_file", parser.add_option("-c", None, action="store", dest="config_file",
help=("Specify the path to the buildout configuration " help=("Specify the path to the buildout configuration "
"file to be used.")) "file to be used."))
options, args = parser.parse_args() options, args = parser.parse_args()
# if -c was provided, we push it back into args for buildout' main function # if -c was provided, we push it back into args for buildout's main function
if options.config_file is not None: if options.config_file is not None:
args += ['-c', options.config_file] args += ['-c', options.config_file]
if options.version is not None: if options.eggs:
VERSION = '==%s' % options.version eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
else: else:
VERSION = '' eggs_dir = tempfile.mkdtemp()
if options.setup_source is None:
if options.use_distribute:
options.setup_source = distribute_source
else:
options.setup_source = setuptools_source
USE_DISTRIBUTE = options.distribute
args = args + ['bootstrap'] args = args + ['bootstrap']
to_reload = False
try: try:
to_reload = False
import pkg_resources import pkg_resources
if not hasattr(pkg_resources, '_distribute'):
to_reload = True to_reload = True
if not hasattr(pkg_resources, '_distribute'):
raise ImportError raise ImportError
import setuptools # A flag. Sometimes pkg_resources is installed alone.
except ImportError: except ImportError:
ez_code = urllib2.urlopen(
options.setup_source).read().replace('\r\n', '\n')
ez = {} ez = {}
if USE_DISTRIBUTE: exec ez_code in ez
exec urllib2.urlopen('http://python-distribute.org/distribute_setup.py' setup_args = dict(to_dir=eggs_dir, download_delay=0)
).read() in ez if options.download_base:
ez['use_setuptools'](to_dir=tmpeggs, download_delay=0, no_fake=True) setup_args['download_base'] = options.download_base
else: if options.use_distribute:
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py' setup_args['no_fake'] = True
).read() in ez ez['use_setuptools'](**setup_args)
ez['use_setuptools'](to_dir=tmpeggs, download_delay=0)
if to_reload: if to_reload:
reload(pkg_resources) reload(pkg_resources)
else: else:
import pkg_resources import pkg_resources
# This does not (always?) update the default working set. We will
if sys.platform == 'win32': # do it.
def quote(c): for path in sys.path:
if ' ' in c: if path not in pkg_resources.working_set.entries:
return '"%s"' % c # work around spawn lamosity on windows pkg_resources.working_set.add_entry(path)
else:
return c cmd = [quote(sys.executable),
'-c',
quote('from setuptools.command.easy_install import main; main()'),
'-mqNxd',
quote(eggs_dir)]
if options.download_base:
cmd.extend(['-f', quote(options.download_base)])
requirement = 'zc.buildout'
if options.version:
requirement = '=='.join((requirement, options.version))
cmd.append(requirement)
if options.use_distribute:
setup_requirement = 'distribute'
else: else:
def quote (c): setup_requirement = 'setuptools'
return c
cmd = 'from setuptools.command.easy_install import main; main()'
ws = pkg_resources.working_set ws = pkg_resources.working_set
env = dict(
if USE_DISTRIBUTE: os.environ,
requirement = 'distribute' PYTHONPATH=ws.find(
else: pkg_resources.Requirement.parse(setup_requirement)).location)
requirement = 'setuptools'
if is_jython: if is_jython:
import subprocess import subprocess
exitcode = subprocess.Popen(cmd, env=env).wait()
assert subprocess.Popen([sys.executable] + ['-c', quote(cmd), '-mqNxd', else: # Windows prefers this, apparently; otherwise we would prefer subprocess
quote(tmpeggs), 'zc.buildout' + VERSION], exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
env=dict(os.environ, if exitcode != 0:
PYTHONPATH= sys.stdout.flush()
ws.find(pkg_resources.Requirement.parse(requirement)).location sys.stderr.flush()
), print ("An error occured when trying to install zc.buildout. "
).wait() == 0 "Look above this message for any errors that "
"were output by easy_install.")
else: sys.exit(exitcode)
assert os.spawnle(
os.P_WAIT, sys.executable, quote (sys.executable), ws.add_entry(eggs_dir)
'-c', quote (cmd), '-mqNxd', quote (tmpeggs), 'zc.buildout' + VERSION, ws.require(requirement)
dict(os.environ,
PYTHONPATH=
ws.find(pkg_resources.Requirement.parse(requirement)).location
),
) == 0
ws.add_entry(tmpeggs)
ws.require('zc.buildout' + VERSION)
import zc.buildout.buildout import zc.buildout.buildout
zc.buildout.buildout.main(args) zc.buildout.buildout.main(args)
shutil.rmtree(tmpeggs) if not options.eggs: # clean up temporary egg directory
shutil.rmtree(eggs_dir)
[buildout] [buildout]
develop = zc.recipe.egg_ . develop = zc.recipe.egg_ z3c.recipe.scripts_ .
parts = test oltest py parts = test oltest py
[py] [py]
recipe = zc.recipe.egg recipe = z3c.recipe.scripts
eggs = zc.buildout eggs = zc.buildout
zope.testing zope.testing
interpreter = py interpreter = py
...@@ -13,6 +13,7 @@ recipe = zc.recipe.testrunner ...@@ -13,6 +13,7 @@ recipe = zc.recipe.testrunner
eggs = eggs =
zc.buildout zc.buildout
zc.recipe.egg zc.recipe.egg
z3c.recipe.scripts
# Tests that can be run wo a network # Tests that can be run wo a network
[oltest] [oltest]
...@@ -20,6 +21,7 @@ recipe = zc.recipe.testrunner ...@@ -20,6 +21,7 @@ recipe = zc.recipe.testrunner
eggs = eggs =
zc.buildout zc.buildout
zc.recipe.egg zc.recipe.egg
z3c.recipe.scripts
defaults = defaults =
[ [
'-t', '-t',
......
...@@ -13,43 +13,126 @@ ...@@ -13,43 +13,126 @@
############################################################################## ##############################################################################
"""Bootstrap the buildout project itself. """Bootstrap the buildout project itself.
This is different from a normal boostrapping process because the This is different from a normal bootstrapping process because the
buildout egg itself is installed as a develop egg. buildout egg itself is installed as a develop egg.
$Id$ $Id$
""" """
import os, shutil, sys, subprocess, urllib2 import os, shutil, sys, subprocess, urllib2
from optparse import OptionParser
if sys.platform == 'win32':
def quote(c):
if ' ' in c:
return '"%s"' % c # work around spawn lamosity on windows
else:
return c
else:
quote = str
# In order to be more robust in the face of system Pythons, we want to
# run without site-packages loaded. This is somewhat tricky, in
# particular because Python 2.6's distutils imports site, so starting
# with the -S flag is not sufficient. However, we'll start with that:
if 'site' in sys.modules:
# We will restart with python -S.
args = sys.argv[:]
args[0:0] = [sys.executable, '-S']
args = map(quote, args)
os.execv(sys.executable, args)
# Now we are running with -S. We'll get the clean sys.path, import site
# because distutils will do it later, and then reset the path and clean
# out any namespace packages from site-packages that might have been
# loaded by .pth files.
clean_path = sys.path[:]
import site
sys.path[:] = clean_path
for k, v in sys.modules.items():
if (hasattr(v, '__path__') and
len(v.__path__)==1 and
not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
# This is a namespace package. Remove it.
sys.modules.pop(k)
is_jython = sys.platform.startswith('java') is_jython = sys.platform.startswith('java')
setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
distribute_source = 'http://python-distribute.org/distribute_setup.py'
usage = '''\
[DESIRED PYTHON FOR DEVELOPING BUILDOUT] dev.py [options]
Bootstraps buildout itself for development.
This is different from a normal bootstrapping process because the
buildout egg itself is installed as a develop egg.
'''
parser = OptionParser(usage=usage)
parser.add_option("-d", "--distribute",
action="store_true", dest="use_distribute", default=False,
help="Use Distribute rather than Setuptools.")
options, args = parser.parse_args()
if args:
parser.error('This script accepts no arguments other than its options.')
if options.use_distribute:
setup_source = distribute_source
else:
setup_source = setuptools_source
for d in 'eggs', 'develop-eggs', 'bin': for d in 'eggs', 'develop-eggs', 'bin':
if not os.path.exists(d): if not os.path.exists(d):
os.mkdir(d) os.mkdir(d)
if os.path.isdir('build'): if os.path.isdir('build'):
shutil.rmtree('build') shutil.rmtree('build')
try: try:
to_reload = False
import pkg_resources import pkg_resources
to_reload = True
if not hasattr(pkg_resources, '_distribute'):
raise ImportError
import setuptools # A flag. Sometimes pkg_resources is installed alone.
except ImportError: except ImportError:
ez_code = urllib2.urlopen(setup_source).read().replace('\r\n', '\n')
ez = {} ez = {}
exec urllib2.urlopen('http://peak.telecommunity.com/dist/ez_setup.py' exec ez_code in ez
).read() in ez setup_args = dict(to_dir='eggs', download_delay=0)
ez['use_setuptools'](to_dir='eggs', download_delay=0) if options.use_distribute:
setup_args['no_fake'] = True
ez['use_setuptools'](**setup_args)
if to_reload:
reload(pkg_resources)
else:
import pkg_resources import pkg_resources
# This does not (always?) update the default working set. We will
# do it.
for path in sys.path:
if path not in pkg_resources.working_set.entries:
pkg_resources.working_set.add_entry(path)
env = os.environ.copy() # Windows needs yet-to-be-determined values from this.
env['PYTHONPATH'] = os.path.dirname(pkg_resources.__file__)
subprocess.Popen( subprocess.Popen(
[sys.executable] + [sys.executable] +
['setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'], ['setup.py', '-q', 'develop', '-m', '-x', '-d', 'develop-eggs'],
env = {'PYTHONPATH': os.path.dirname(pkg_resources.__file__)}).wait() env=env).wait()
pkg_resources.working_set.add_entry('src') pkg_resources.working_set.add_entry('src')
import zc.buildout.easy_install import zc.buildout.easy_install
zc.buildout.easy_install.scripts( if not os.path.exists('parts'):
['zc.buildout'], pkg_resources.working_set , sys.executable, 'bin') os.mkdir('parts')
partsdir = os.path.join('parts', 'buildout')
if not os.path.exists(partsdir):
os.mkdir(partsdir)
zc.buildout.easy_install.sitepackage_safe_scripts(
'bin', pkg_resources.working_set, sys.executable, partsdir,
reqs=['zc.buildout'])
bin_buildout = os.path.join('bin', 'buildout') bin_buildout = os.path.join('bin', 'buildout')
...@@ -57,4 +140,5 @@ if is_jython: ...@@ -57,4 +140,5 @@ if is_jython:
# Jython needs the script to be called twice via sys.executable # Jython needs the script to be called twice via sys.executable
assert subprocess.Popen([sys.executable] + [bin_buildout]).wait() == 0 assert subprocess.Popen([sys.executable] + [bin_buildout]).wait() == 0
sys.exit(subprocess.Popen(bin_buildout).wait()) sys.exit(subprocess.Popen(bin_buildout).wait())
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
# #
############################################################################## ##############################################################################
name = "zc.buildout" name = "zc.buildout"
version = "1.4.4dev" version = "1.5.0dev"
import os import os
from setuptools import setup from setuptools import setup
......
...@@ -47,7 +47,7 @@ Make sure the bootstrap script actually works:: ...@@ -47,7 +47,7 @@ Make sure the bootstrap script actually works::
X... X...
d zc.buildout-...egg d zc.buildout-...egg
Now trying the `--version` option, that let you define a version for Now we will try the `--version` option, which lets you define a version for
`zc.buildout`. If not provided, bootstrap will look for the latest one. `zc.buildout`. If not provided, bootstrap will look for the latest one.
Let's try with an unknown version:: Let's try with an unknown version::
...@@ -57,7 +57,7 @@ Let's try with an unknown version:: ...@@ -57,7 +57,7 @@ Let's try with an unknown version::
... 'bootstrap.py --version UNKNOWN'); print 'X' # doctest: +ELLIPSIS ... 'bootstrap.py --version UNKNOWN'); print 'X' # doctest: +ELLIPSIS
... ...
X X
No local packages or download links found for zc.buildout==UNKNOWN No local packages or download links found for zc.buildout==UNKNOWN...
... ...
Now let's try with `1.1.2`, which happens to exist:: Now let's try with `1.1.2`, which happens to exist::
...@@ -119,8 +119,8 @@ Let's make sure the generated `buildout` script uses it:: ...@@ -119,8 +119,8 @@ Let's make sure the generated `buildout` script uses it::
zc.buildout.buildout.main() zc.buildout.buildout.main()
<BLANKLINE> <BLANKLINE>
`zc.buildout` now can also run with `Distribute` with the `--distribute` option:: `zc.buildout` now can also run with `Distribute` with the `--distribute`
option::
>>> print 'X'; print system( >>> print 'X'; print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
...@@ -153,7 +153,8 @@ Make sure both options can be used together:: ...@@ -153,7 +153,8 @@ Make sure both options can be used together::
>>> print 'X'; print system( >>> print 'X'; print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+ ... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
... 'bootstrap.py --distribute --version 1.2.1'); print 'X' # doctest: +ELLIPSIS ... 'bootstrap.py --distribute --version 1.2.1'); print 'X'
... # doctest: +ELLIPSIS
... ...
X X
... ...
...@@ -161,7 +162,8 @@ Make sure both options can be used together:: ...@@ -161,7 +162,8 @@ Make sure both options can be used together::
<BLANKLINE> <BLANKLINE>
X X
Let's make sure the generated `buildout` script uses ``Distribute`` *and* ``zc.buildout-1.2.1``:: Let's make sure the generated `buildout` script uses ``Distribute`` *and*
``zc.buildout-1.2.1``::
>>> print open(buildout_script).read() # doctest: +ELLIPSIS >>> print open(buildout_script).read() # doctest: +ELLIPSIS
#... #...
...@@ -194,4 +196,70 @@ Last, the -c option needs to work on bootstrap.py:: ...@@ -194,4 +196,70 @@ Last, the -c option needs to work on bootstrap.py::
<BLANKLINE> <BLANKLINE>
X X
You can specify a location of ez_setup.py or distribute_setup, so you
can rely on a local or remote location. We'll write our own ez_setup.py
that we will also use to test some other bootstrap options.
>>> write('ez_setup.py', '''\
... def use_setuptools(**kwargs):
... import sys, pprint
... pprint.pprint(kwargs, width=40)
... sys.exit()
... ''')
>>> print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
... 'bootstrap.py --setup-source=./ez_setup.py')
... # doctest: +ELLIPSIS
{'download_delay': 0,
'to_dir': '...'}
<BLANKLINE>
You can also pass a download-cache, and a place in which eggs should be stored
(they are normally stored in a temporary directory).
>>> print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
... 'bootstrap.py --setup-source=./ez_setup.py '+
... '--download-base=./download-cache --eggs=eggs')
... # doctest: +ELLIPSIS
{'download_base': '/sample/download-cache/',
'download_delay': 0,
'to_dir': '/sample/eggs'}
<BLANKLINE>
Here's the entire help text.
>>> print system(
... zc.buildout.easy_install._safe_arg(sys.executable)+' '+
... 'bootstrap.py --help'),
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
Usage: [DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
<BLANKLINE>
Bootstraps a buildout-based project.
<BLANKLINE>
Simply run this script in a directory containing a buildout.cfg, using the
Python that you want bin/buildout to use.
<BLANKLINE>
Note that by using --setup-source and --download-base to point to
local resources, you can keep this script from going over the network.
<BLANKLINE>
<BLANKLINE>
Options:
-h, --help show this help message and exit
-v VERSION, --version=VERSION
use a specific zc.buildout version
-d, --distribute Use Distribute rather than Setuptools.
--setup-source=SETUP_SOURCE
Specify a URL or file location for the setup file. If
you use Setuptools, this will default to
http://peak.telecommunity.com/dist/ez_setup.py; if you
use Distribute, this will default to http://python-
distribute.org/distribute_setup.py.
--download-base=DOWNLOAD_BASE
Specify a URL or directory for downloading zc.buildout
and either Setuptools or Distribute. Defaults to PyPI.
--eggs=EGGS Specify a directory for storing eggs. Defaults to a
temporary directory that is deleted when the bootstrap
script completes.
-c CONFIG_FILE Specify the path to the buildout configuration file to
be used.
...@@ -112,15 +112,26 @@ def _unannotate(data): ...@@ -112,15 +112,26 @@ def _unannotate(data):
return data return data
_buildout_default_options = _annotate_section({ _buildout_default_options = _annotate_section({
'eggs-directory': 'eggs', 'allow-hosts': '*',
'develop-eggs-directory': 'develop-eggs', 'allow-picked-versions': 'true',
'bin-directory': 'bin', 'bin-directory': 'bin',
'parts-directory': 'parts', 'develop-eggs-directory': 'develop-eggs',
'installed': '.installed.cfg', 'eggs-directory': 'eggs',
'python': 'buildout',
'executable': sys.executable, 'executable': sys.executable,
'log-level': 'INFO', 'find-links': '',
'install-from-cache': 'false',
'installed': '.installed.cfg',
'log-format': '', 'log-format': '',
'log-level': 'INFO',
'newest': 'true',
'offline': 'false',
'parts-directory': 'parts',
'prefer-final': 'false',
'python': 'buildout',
'relative-paths': 'false',
'socket-timeout': '',
'unzip': 'false',
'use-dependency-links': 'true',
}, 'DEFAULT_VALUE') }, 'DEFAULT_VALUE')
...@@ -191,7 +202,7 @@ class Buildout(UserDict.DictMixin): ...@@ -191,7 +202,7 @@ class Buildout(UserDict.DictMixin):
# provide some defaults before options are parsed # provide some defaults before options are parsed
# because while parsing options those attributes might be # because while parsing options those attributes might be
# used already (Gottfried Ganssauge) # used already (Gottfried Ganssauge)
buildout_section = data.get('buildout') buildout_section = data['buildout']
# Try to make sure we have absolute paths for standard # Try to make sure we have absolute paths for standard
# directories. We do this before doing substitutions, in case # directories. We do this before doing substitutions, in case
...@@ -204,22 +215,28 @@ class Buildout(UserDict.DictMixin): ...@@ -204,22 +215,28 @@ class Buildout(UserDict.DictMixin):
d = self._buildout_path(buildout_section[name+'-directory']) d = self._buildout_path(buildout_section[name+'-directory'])
buildout_section[name+'-directory'] = d buildout_section[name+'-directory'] = d
links = buildout_section and buildout_section.get('find-links', '') # Attributes on this buildout object shouldn't be used by
# recipes in their __init__. It can cause bugs, because the
# recipes will be instantiated below (``options = self['buildout']``)
# before this has completed initializing. These attributes are
# left behind for legacy support but recipe authors should
# beware of using them. A better practice is for a recipe to
# use the buildout['buildout'] options.
links = buildout_section['find-links']
self._links = links and links.split() or () self._links = links and links.split() or ()
allow_hosts = buildout_section['allow-hosts'].split('\n')
allow_hosts = buildout_section and buildout_section.get(
'allow-hosts', '*').split('\n')
self._allow_hosts = tuple([host.strip() for host in allow_hosts self._allow_hosts = tuple([host.strip() for host in allow_hosts
if host.strip() != '']) if host.strip() != ''])
self._logger = logging.getLogger('zc.buildout') self._logger = logging.getLogger('zc.buildout')
self.offline = False self.offline = (buildout_section['offline'] == 'true')
self.newest = True self.newest = (buildout_section['newest'] == 'true')
################################################################## ##################################################################
## WARNING!!! ## WARNING!!!
## ALL ATTRIBUTES MUST HAVE REASONABLE DEFAULTS AT THIS POINT ## ALL ATTRIBUTES MUST HAVE REASONABLE DEFAULTS AT THIS POINT
## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME ## OTHERWISE ATTRIBUTEERRORS MIGHT HAPPEN ANY TIME FROM RECIPES.
## RECIPES SHOULD GENERALLY USE buildout['buildout'] OPTIONS, NOT
## BUILDOUT ATTRIBUTES.
################################################################## ##################################################################
# initialize some attrs and buildout directories. # initialize some attrs and buildout directories.
options = self['buildout'] options = self['buildout']
...@@ -228,7 +245,7 @@ class Buildout(UserDict.DictMixin): ...@@ -228,7 +245,7 @@ class Buildout(UserDict.DictMixin):
links = options.get('find-links', '') links = options.get('find-links', '')
self._links = links and links.split() or () self._links = links and links.split() or ()
allow_hosts = options.get('allow-hosts', '*').split('\n') allow_hosts = options['allow-hosts'].split('\n')
self._allow_hosts = tuple([host.strip() for host in allow_hosts self._allow_hosts = tuple([host.strip() for host in allow_hosts
if host.strip() != '']) if host.strip() != ''])
...@@ -246,44 +263,42 @@ class Buildout(UserDict.DictMixin): ...@@ -246,44 +263,42 @@ class Buildout(UserDict.DictMixin):
self._setup_logging() self._setup_logging()
offline = options.get('offline', 'false') offline = options['offline']
if offline not in ('true', 'false'): if offline not in ('true', 'false'):
self._error('Invalid value for offline option: %s', offline) self._error('Invalid value for offline option: %s', offline)
options['offline'] = offline self.offline = (offline == 'true')
self.offline = offline == 'true'
if self.offline: if self.offline:
newest = options['newest'] = 'false' newest = options['newest'] = 'false'
else: else:
newest = options.get('newest', 'true') newest = options['newest']
if newest not in ('true', 'false'): if newest not in ('true', 'false'):
self._error('Invalid value for newest option: %s', newest) self._error('Invalid value for newest option: %s', newest)
options['newest'] = newest self.newest = (newest == 'true')
self.newest = newest == 'true'
versions = options.get('versions') versions = options.get('versions')
if versions: if versions:
zc.buildout.easy_install.default_versions(dict(self[versions])) zc.buildout.easy_install.default_versions(dict(self[versions]))
prefer_final = options.get('prefer-final', 'false') prefer_final = options['prefer-final']
if prefer_final not in ('true', 'false'): if prefer_final not in ('true', 'false'):
self._error('Invalid value for prefer-final option: %s', self._error('Invalid value for prefer-final option: %s',
prefer_final) prefer_final)
zc.buildout.easy_install.prefer_final(prefer_final=='true') zc.buildout.easy_install.prefer_final(prefer_final=='true')
use_dependency_links = options.get('use-dependency-links', 'true') use_dependency_links = options['use-dependency-links']
if use_dependency_links not in ('true', 'false'): if use_dependency_links not in ('true', 'false'):
self._error('Invalid value for use-dependency-links option: %s', self._error('Invalid value for use-dependency-links option: %s',
use_dependency_links) use_dependency_links)
zc.buildout.easy_install.use_dependency_links( zc.buildout.easy_install.use_dependency_links(
use_dependency_links == 'true') use_dependency_links == 'true')
allow_picked_versions = options.get('allow-picked-versions', 'true') allow_picked_versions = options['allow-picked-versions']
if allow_picked_versions not in ('true', 'false'): if allow_picked_versions not in ('true', 'false'):
self._error('Invalid value for allow-picked-versions option: %s', self._error('Invalid value for allow-picked-versions option: %s',
allow_picked_versions) allow_picked_versions)
zc.buildout.easy_install.allow_picked_versions( zc.buildout.easy_install.allow_picked_versions(
allow_picked_versions=='true') allow_picked_versions == 'true')
download_cache = options.get('download-cache') download_cache = options.get('download-cache')
if download_cache: if download_cache:
...@@ -300,22 +315,18 @@ class Buildout(UserDict.DictMixin): ...@@ -300,22 +315,18 @@ class Buildout(UserDict.DictMixin):
zc.buildout.easy_install.download_cache(download_cache) zc.buildout.easy_install.download_cache(download_cache)
install_from_cache = options.get('install-from-cache') install_from_cache = options['install-from-cache']
if install_from_cache:
if install_from_cache not in ('true', 'false'): if install_from_cache not in ('true', 'false'):
self._error('Invalid value for install-from-cache option: %s', self._error('Invalid value for install-from-cache option: %s',
install_from_cache) install_from_cache)
if install_from_cache == 'true': zc.buildout.easy_install.install_from_cache(
zc.buildout.easy_install.install_from_cache(True) install_from_cache=='true')
always_unzip = options['unzip']
always_unzip = options.get('unzip')
if always_unzip:
if always_unzip not in ('true', 'false'): if always_unzip not in ('true', 'false'):
self._error('Invalid value for unzip option: %s', self._error('Invalid value for unzip option: %s',
always_unzip) always_unzip)
if always_unzip == 'true': zc.buildout.easy_install.always_unzip(always_unzip=='true')
zc.buildout.easy_install.always_unzip(True)
# "Use" each of the defaults so they aren't reported as unused options. # "Use" each of the defaults so they aren't reported as unused options.
for name in _buildout_default_options: for name in _buildout_default_options:
...@@ -338,11 +349,35 @@ class Buildout(UserDict.DictMixin): ...@@ -338,11 +349,35 @@ class Buildout(UserDict.DictMixin):
self._setup_directories() self._setup_directories()
options = self['buildout']
# Get a base working set for our distributions that corresponds to the
# stated desires in the configuration.
distributions = ['setuptools', 'zc.buildout']
if options.get('offline') == 'true':
ws = zc.buildout.easy_install.working_set(
distributions, options['executable'],
[options['develop-eggs-directory'],
options['eggs-directory']],
include_site_packages=False,
)
else:
ws = zc.buildout.easy_install.install(
distributions, options['eggs-directory'],
links=self._links,
index=options.get('index'),
executable=options['executable'],
path=[options['develop-eggs-directory']],
newest=self.newest,
allow_hosts=self._allow_hosts,
include_site_packages=False,
)
# Now copy buildout and setuptools eggs, and record destination eggs: # Now copy buildout and setuptools eggs, and record destination eggs:
entries = [] entries = []
for name in 'setuptools', 'zc.buildout': for name in 'setuptools', 'zc.buildout':
r = pkg_resources.Requirement.parse(name) r = pkg_resources.Requirement.parse(name)
dist = pkg_resources.working_set.find(r) dist = ws.find(r)
if dist.precedence == pkg_resources.DEVELOP_DIST: if dist.precedence == pkg_resources.DEVELOP_DIST:
dest = os.path.join(self['buildout']['develop-eggs-directory'], dest = os.path.join(self['buildout']['develop-eggs-directory'],
name+'.egg-link') name+'.egg-link')
...@@ -361,9 +396,19 @@ class Buildout(UserDict.DictMixin): ...@@ -361,9 +396,19 @@ class Buildout(UserDict.DictMixin):
# Create buildout script # Create buildout script
ws = pkg_resources.WorkingSet(entries) ws = pkg_resources.WorkingSet(entries)
ws.require('zc.buildout') ws.require('zc.buildout')
zc.buildout.easy_install.scripts( partsdir = os.path.join(options['parts-directory'], 'buildout')
['zc.buildout'], ws, sys.executable, if not os.path.exists(partsdir):
self['buildout']['bin-directory']) os.mkdir(partsdir)
# (Honor the relative-paths option.)
relative_paths = options.get('relative-paths', 'false')
if relative_paths == 'true':
relative_paths = options['directory']
else:
assert relative_paths == 'false'
relative_paths = ''
zc.buildout.easy_install.sitepackage_safe_scripts(
options['bin-directory'], ws, options['executable'], partsdir,
reqs=['zc.buildout'], relative_paths=relative_paths)
init = bootstrap init = bootstrap
...@@ -533,7 +578,7 @@ class Buildout(UserDict.DictMixin): ...@@ -533,7 +578,7 @@ class Buildout(UserDict.DictMixin):
if installed_files is None: if installed_files is None:
self._logger.warning( self._logger.warning(
"The %s install returned None. A path or " "The %s install returned None. A path or "
"iterable os paths should be returned.", "iterable of paths should be returned.",
part) part)
installed_files = () installed_files = ()
elif isinstance(installed_files, str): elif isinstance(installed_files, str):
...@@ -792,16 +837,24 @@ class Buildout(UserDict.DictMixin): ...@@ -792,16 +837,24 @@ class Buildout(UserDict.DictMixin):
if not self.newest: if not self.newest:
return return
options = self['buildout']
specs = ['zc.buildout']
if zc.buildout.easy_install.is_distribute:
specs.append('distribute')
else:
specs.append('setuptools')
ws = zc.buildout.easy_install.install( ws = zc.buildout.easy_install.install(
[ [
(spec + ' ' + self['buildout'].get(spec+'-version', '')).strip() (spec + ' ' + options.get(spec+'-version', '')).strip()
for spec in ('zc.buildout', 'setuptools') for spec in specs
], ],
self['buildout']['eggs-directory'], options['eggs-directory'],
links = self['buildout'].get('find-links', '').split(), links = options.get('find-links', '').split(),
index = self['buildout'].get('index'), index = options.get('index'),
path = [self['buildout']['develop-eggs-directory']], path = [options['develop-eggs-directory']],
allow_hosts = self._allow_hosts allow_hosts = self._allow_hosts,
include_site_packages=False
) )
upgraded = [] upgraded = []
...@@ -817,7 +870,7 @@ class Buildout(UserDict.DictMixin): ...@@ -817,7 +870,7 @@ class Buildout(UserDict.DictMixin):
__doing__ = 'Upgrading.' __doing__ = 'Upgrading.'
should_run = realpath( should_run = realpath(
os.path.join(os.path.abspath(self['buildout']['bin-directory']), os.path.join(os.path.abspath(options['bin-directory']),
'buildout') 'buildout')
) )
if sys.platform == 'win32': if sys.platform == 'win32':
...@@ -849,21 +902,34 @@ class Buildout(UserDict.DictMixin): ...@@ -849,21 +902,34 @@ class Buildout(UserDict.DictMixin):
# the new dist is different, so we've upgraded. # the new dist is different, so we've upgraded.
# Update the scripts and return True # Update the scripts and return True
zc.buildout.easy_install.scripts( partsdir = os.path.join(options['parts-directory'], 'buildout')
['zc.buildout'], ws, sys.executable, if os.path.exists(partsdir):
self['buildout']['bin-directory'], # This is primarily for unit tests, in which .py files change too
) # fast for Python to know to regenerate the .pyc/.pyo files.
shutil.rmtree(partsdir)
os.mkdir(partsdir)
zc.buildout.easy_install.sitepackage_safe_scripts(
options['bin-directory'], ws, sys.executable, partsdir,
reqs=['zc.buildout'])
# Restart # Restart
args = map(zc.buildout.easy_install._safe_arg, sys.argv) args = map(zc.buildout.easy_install._safe_arg, sys.argv)
if not __debug__: if not __debug__:
args.insert(0, '-O') args.insert(0, '-O')
args.insert(0, zc.buildout.easy_install._safe_arg (sys.executable)) args.insert(0, zc.buildout.easy_install._safe_arg(sys.executable))
# We want to make sure that our new site.py is used for rerunning
# buildout, so we put the partsdir in PYTHONPATH for our restart.
# This overrides any set PYTHONPATH, but since we generally are
# trying to run with a completely "clean" python (only the standard
# library) then that should be fine.
env = os.environ.copy()
env['PYTHONPATH'] = partsdir
if is_jython: if is_jython:
sys.exit(subprocess.Popen([sys.executable] + list(args)).wait()) sys.exit(
subprocess.Popen(
[sys.executable] + list(args), env=env).wait())
else: else:
sys.exit(os.spawnv(os.P_WAIT, sys.executable, args)) sys.exit(os.spawnve(os.P_WAIT, sys.executable, args, env))
def _load_extensions(self): def _load_extensions(self):
__doing__ = 'Loading extensions.' __doing__ = 'Loading extensions.'
...@@ -884,7 +950,8 @@ class Buildout(UserDict.DictMixin): ...@@ -884,7 +950,8 @@ class Buildout(UserDict.DictMixin):
working_set=pkg_resources.working_set, working_set=pkg_resources.working_set,
links = self['buildout'].get('find-links', '').split(), links = self['buildout'].get('find-links', '').split(),
index = self['buildout'].get('index'), index = self['buildout'].get('index'),
newest=self.newest, allow_hosts=self._allow_hosts) newest=self.newest, allow_hosts=self._allow_hosts,
include_site_packages=False)
# Clear cache because extensions might now let us read pages we # Clear cache because extensions might now let us read pages we
# couldn't read before. # couldn't read before.
...@@ -1001,7 +1068,8 @@ def _install_and_load(spec, group, entry, buildout): ...@@ -1001,7 +1068,8 @@ def _install_and_load(spec, group, entry, buildout):
path=path, path=path,
working_set=pkg_resources.working_set, working_set=pkg_resources.working_set,
newest=buildout.newest, newest=buildout.newest,
allow_hosts=buildout._allow_hosts allow_hosts=buildout._allow_hosts,
include_site_packages=False,
) )
__doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry
......
...@@ -56,10 +56,9 @@ The bin directory contains scripts. ...@@ -56,10 +56,9 @@ The bin directory contains scripts.
- setuptools-0.6-py2.4.egg - setuptools-0.6-py2.4.egg
- zc.buildout-1.0-py2.4.egg - zc.buildout-1.0-py2.4.egg
The develop-eggs and parts directories are initially empty: The develop-eggs directory is initially empty:
>>> ls(sample_buildout, 'develop-eggs') >>> ls(sample_buildout, 'develop-eggs')
>>> ls(sample_buildout, 'parts')
The develop-eggs directory holds egg links for software being The develop-eggs directory holds egg links for software being
developed in the buildout. We separate develop-eggs and other eggs to developed in the buildout. We separate develop-eggs and other eggs to
...@@ -69,6 +68,12 @@ directory in their home that all non-develop eggs are stored in. This ...@@ -69,6 +68,12 @@ directory in their home that all non-develop eggs are stored in. This
allows larger buildouts to be set up much more quickly and saves disk allows larger buildouts to be set up much more quickly and saves disk
space. space.
The parts directory just contains some helpers for the buildout script
itself.
>>> ls(sample_buildout, 'parts')
d buildout
The parts directory provides an area where recipes can install The parts directory provides an area where recipes can install
part data. For example, if we built a custom Python, we would part data. For example, if we built a custom Python, we would
install it in the part directory. Part data is stored in a install it in the part directory. Part data is stored in a
...@@ -576,7 +581,7 @@ When we rerun the buildout: ...@@ -576,7 +581,7 @@ When we rerun the buildout:
.. Wait for the file to really disappear. My linux is weird. .. Wait for the file to really disappear. My linux is weird.
>>> wait_until("foo goes away", lambda : not os.path.exists('foo'), >>> wait_until("foo goes away", lambda : not os.path.exists('foo'),
... timeout=100) ... timeout=200)
we get the same error, but we don't get the directory left behind: we get the same error, but we don't get the directory left behind:
...@@ -724,6 +729,10 @@ COMMAND_LINE_VALUE). ...@@ -724,6 +729,10 @@ COMMAND_LINE_VALUE).
================== ==================
<BLANKLINE> <BLANKLINE>
[buildout] [buildout]
allow-hosts= *
DEFAULT_VALUE
allow-picked-versions= true
DEFAULT_VALUE
bin-directory= bin bin-directory= bin
DEFAULT_VALUE DEFAULT_VALUE
develop= recipes develop= recipes
...@@ -736,18 +745,36 @@ COMMAND_LINE_VALUE). ...@@ -736,18 +745,36 @@ COMMAND_LINE_VALUE).
DEFAULT_VALUE DEFAULT_VALUE
executable= ... executable= ...
DEFAULT_VALUE DEFAULT_VALUE
find-links=
DEFAULT_VALUE
install-from-cache= false
DEFAULT_VALUE
installed= .installed.cfg installed= .installed.cfg
DEFAULT_VALUE DEFAULT_VALUE
log-format= log-format=
DEFAULT_VALUE DEFAULT_VALUE
log-level= INFO log-level= INFO
DEFAULT_VALUE DEFAULT_VALUE
newest= true
DEFAULT_VALUE
offline= false
DEFAULT_VALUE
parts= data-dir parts= data-dir
/sample-buildout/buildout.cfg /sample-buildout/buildout.cfg
parts-directory= parts parts-directory= parts
DEFAULT_VALUE DEFAULT_VALUE
prefer-final= false
DEFAULT_VALUE
python= buildout python= buildout
DEFAULT_VALUE DEFAULT_VALUE
relative-paths= false
DEFAULT_VALUE
socket-timeout=
DEFAULT_VALUE
unzip= false
DEFAULT_VALUE
use-dependency-links= true
DEFAULT_VALUE
<BLANKLINE> <BLANKLINE>
[data-dir] [data-dir]
path= foo bins path= foo bins
...@@ -2194,17 +2221,21 @@ database is shown. ...@@ -2194,17 +2221,21 @@ database is shown.
>>> print system(buildout+' -vv'), # doctest: +NORMALIZE_WHITESPACE >>> print system(buildout+' -vv'), # doctest: +NORMALIZE_WHITESPACE
Installing 'zc.buildout', 'setuptools'. Installing 'zc.buildout', 'setuptools'.
We have a develop egg: zc.buildout 1.0.0. We have a develop egg: zc.buildout X.X.
We have the best distribution that satisfies 'setuptools'. We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = 0.6 Picked: setuptools = V.V
<BLANKLINE> <BLANKLINE>
Configuration data: Configuration data:
[buildout] [buildout]
allow-hosts = *
allow-picked-versions = true
bin-directory = /sample-buildout/bin bin-directory = /sample-buildout/bin
develop-eggs-directory = /sample-buildout/develop-eggs develop-eggs-directory = /sample-buildout/develop-eggs
directory = /sample-buildout directory = /sample-buildout
eggs-directory = /sample-buildout/eggs eggs-directory = /sample-buildout/eggs
executable = /usr/local/bin/python2.3 executable = python
find-links =
install-from-cache = false
installed = /sample-buildout/.installed.cfg installed = /sample-buildout/.installed.cfg
log-format = log-format =
log-level = INFO log-level = INFO
...@@ -2212,7 +2243,12 @@ database is shown. ...@@ -2212,7 +2243,12 @@ database is shown.
offline = false offline = false
parts = parts =
parts-directory = /sample-buildout/parts parts-directory = /sample-buildout/parts
prefer-final = false
python = buildout python = buildout
relative-paths = false
socket-timeout =
unzip = false
use-dependency-links = true
verbosity = 20 verbosity = 20
<BLANKLINE> <BLANKLINE>
...@@ -2273,6 +2309,33 @@ python ...@@ -2273,6 +2309,33 @@ python
Python executable. By default, the buildout section defines the Python executable. By default, the buildout section defines the
default Python as the Python used to run the buildout. default Python as the Python used to run the buildout.
relative-paths
The paths generated by zc.buildout are absolute by default, and this
option is ``false``. However, if you set this value to be ``true``,
bin/buildout will be generated with code that makes the paths relative.
Some recipes, such as zc.recipe.egg and z3c.recipe.scripts, honor this
value as well.
unzip
By default, zc.buildout doesn't unzip zip-safe eggs ("unzip = false").
This follows the policy followed by setuptools itself. Experience shows
this policy to to be inconvenient. Zipped eggs make debugging more
difficult and often import more slowly. You can include an unzip option in
the buildout section to change the default unzipping policy ("unzip =
true").
use-dependency-links
By default buildout will obey the setuptools dependency_links metadata
when it looks for dependencies. This behavior can be controlled with
the use-dependency-links buildout option::
[buildout]
...
use-dependency-links = false
The option defaults to true. If you set it to false, then dependency
links are only looked for in the locations specified by find-links.
verbosity verbosity
A log-level adjustment. Typically, this is set via the -q and -v A log-level adjustment. Typically, this is set via the -q and -v
command-line options. command-line options.
...@@ -2319,8 +2382,56 @@ buildout or setuptools egg could be installed in the develop-eggs ...@@ -2319,8 +2382,56 @@ buildout or setuptools egg could be installed in the develop-eggs
directory if the original buildout had develop eggs for either directory if the original buildout had develop eggs for either
buildout or setuptools.) buildout or setuptools.)
Note that the buildout script was installed but not run. To run If relative-paths is ``true``, the buildout script uses relative paths.
the buildout, we'd have to run the installed buildout script.
>>> write(sample_bootstrapped, 'setup.cfg',
... '''
... [buildout]
... relative-paths = true
... parts =
... ''')
>>> print system(buildout
... +' -c'+os.path.join(sample_bootstrapped, 'setup.cfg')
... +' bootstrap'),
Generated script '/sample-bootstrapped/bin/buildout'.
>>> buildout_script = join(sample_bootstrapped, 'bin', 'buildout')
>>> import sys
>>> if sys.platform.startswith('win'):
... buildout_script += '-script.py'
>>> print open(buildout_script).read() # doctest: +ELLIPSIS
#!... -S
<BLANKLINE>
import os
<BLANKLINE>
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
<BLANKLINE>
import sys
sys.path[0:0] = [
join(base, 'parts/buildout'),
]
<BLANKLINE>
<BLANKLINE>
import os
path = sys.path[0]
if os.environ.get('PYTHONPATH'):
path = os.pathsep.join([path, os.environ['PYTHONPATH']])
os.environ['PYTHONPATH'] = path
import site # imports custom buildout-generated site.py
<BLANKLINE>
import zc.buildout.buildout
<BLANKLINE>
if __name__ == '__main__':
zc.buildout.buildout.main()
<BLANKLINE>
Note that, in the above two examples, the buildout script was installed
but not run. To run the buildout, we'd have to run the installed
buildout script.
If we have an existing buildout that already has a buildout.cfg, we'll If we have an existing buildout that already has a buildout.cfg, we'll
normally use the bootstrap command instead of init. It will complain normally use the bootstrap command instead of init. It will complain
......
...@@ -19,6 +19,7 @@ installed. ...@@ -19,6 +19,7 @@ installed.
""" """
import distutils.errors import distutils.errors
import fnmatch
import glob import glob
import logging import logging
import os import os
...@@ -50,6 +51,8 @@ url_match = re.compile('[a-z0-9+.-]+://').match ...@@ -50,6 +51,8 @@ url_match = re.compile('[a-z0-9+.-]+://').match
is_win32 = sys.platform == 'win32' is_win32 = sys.platform == 'win32'
is_jython = sys.platform.startswith('java') is_jython = sys.platform.startswith('java')
is_distribute = (
pkg_resources.Requirement.parse('setuptools').key=='distribute')
if is_jython: if is_jython:
import java.lang.System import java.lang.System
...@@ -60,12 +63,75 @@ setuptools_loc = pkg_resources.working_set.find( ...@@ -60,12 +63,75 @@ setuptools_loc = pkg_resources.working_set.find(
pkg_resources.Requirement.parse('setuptools') pkg_resources.Requirement.parse('setuptools')
).location ).location
# Include buildout and setuptools eggs in paths # Include buildout and setuptools eggs in paths. We prevent dupes just to
buildout_and_setuptools_path = [ # keep from duplicating any log messages about them.
setuptools_loc, buildout_loc = pkg_resources.working_set.find(
pkg_resources.working_set.find( pkg_resources.Requirement.parse('zc.buildout')).location
pkg_resources.Requirement.parse('zc.buildout')).location, buildout_and_setuptools_path = [setuptools_loc]
] if os.path.normpath(setuptools_loc) != os.path.normpath(buildout_loc):
buildout_and_setuptools_path.append(buildout_loc)
def _get_system_paths(executable):
"""Return lists of standard lib and site paths for executable.
"""
# We want to get a list of the site packages, which is not easy.
# The canonical way to do this is to use
# distutils.sysconfig.get_python_lib(), but that only returns a
# single path, which does not reflect reality for many system
# Pythons, which have multiple additions. Instead, we start Python
# with -S, which does not import site.py and set up the extra paths
# like site-packages or (Ubuntu/Debian) dist-packages and
# python-support. We then compare that sys.path with the normal one
# (minus user packages if this is Python 2.6, because we don't
# support those (yet?). The set of the normal one minus the set of
# the ones in ``python -S`` is the set of packages that are
# effectively site-packages.
#
# The given executable might not be the current executable, so it is
# appropriate to do another subprocess to figure out what the
# additional site-package paths are. Moreover, even if this
# executable *is* the current executable, this code might be run in
# the context of code that has manipulated the sys.path--for
# instance, to add local zc.buildout or setuptools eggs.
def get_sys_path(*args, **kwargs):
cmd = [executable]
cmd.extend(args)
cmd.extend([
"-c", "import sys, os;"
"print repr([os.path.normpath(p) for p in sys.path if p])"])
# Windows needs some (as yet to be determined) part of the real env.
env = os.environ.copy()
# We need to make sure that PYTHONPATH, which will often be set
# to include a custom buildout-generated site.py, is not set, or
# else we will not get an accurate sys.path for the executable.
env.pop('PYTHONPATH', None)
env.update(kwargs)
_proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
stdout, stderr = _proc.communicate();
if _proc.returncode:
raise RuntimeError(
'error trying to get system packages:\n%s' % (stderr,))
res = eval(stdout.strip())
try:
res.remove('.')
except ValueError:
pass
return res
stdlib = get_sys_path('-S') # stdlib only
no_user_paths = get_sys_path(PYTHONNOUSERSITE='x')
site_paths = [p for p in no_user_paths if p not in stdlib]
return (stdlib, site_paths)
def _get_version_info(executable):
cmd = [executable, '-Sc', 'import sys; print repr(sys.version_info)']
_proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = _proc.communicate();
if _proc.returncode:
raise RuntimeError(
'error trying to get system packages:\n%s' % (stderr,))
return eval(stdout.strip())
class IncompatibleVersionError(zc.buildout.UserError): class IncompatibleVersionError(zc.buildout.UserError):
...@@ -109,7 +175,12 @@ class AllowHostsPackageIndex(setuptools.package_index.PackageIndex): ...@@ -109,7 +175,12 @@ class AllowHostsPackageIndex(setuptools.package_index.PackageIndex):
_indexes = {} _indexes = {}
def _get_index(executable, index_url, find_links, allow_hosts=('*',)): def _get_index(executable, index_url, find_links, allow_hosts=('*',),
path=None):
# If path is None, the index will use sys.path. If you provide an empty
# path ([]), it will complain uselessly about missing index pages for
# packages found in the paths that you expect to use. Therefore, this path
# is always the same as the _env path in the Installer.
key = executable, index_url, tuple(find_links) key = executable, index_url, tuple(find_links)
index = _indexes.get(key) index = _indexes.get(key)
if index is not None: if index is not None:
...@@ -118,7 +189,8 @@ def _get_index(executable, index_url, find_links, allow_hosts=('*',)): ...@@ -118,7 +189,8 @@ def _get_index(executable, index_url, find_links, allow_hosts=('*',)):
if index_url is None: if index_url is None:
index_url = default_index_url index_url = default_index_url
index = AllowHostsPackageIndex( index = AllowHostsPackageIndex(
index_url, hosts=allow_hosts, python=_get_version(executable) index_url, hosts=allow_hosts, search_path=path,
python=_get_version(executable)
) )
if find_links: if find_links:
...@@ -131,15 +203,79 @@ clear_index_cache = _indexes.clear ...@@ -131,15 +203,79 @@ clear_index_cache = _indexes.clear
if is_win32: if is_win32:
# work around spawn lamosity on windows # work around spawn lamosity on windows
# XXX need safe quoting (see the subproces.list2cmdline) and test # XXX need safe quoting (see the subprocess.list2cmdline) and test
def _safe_arg(arg): def _safe_arg(arg):
return '"%s"' % arg return '"%s"' % arg
else: else:
_safe_arg = str _safe_arg = str
_easy_install_cmd = _safe_arg( # The following string is used to run easy_install in
'from setuptools.command.easy_install import main; main()' # Installer._call_easy_install. It is started with python -S (that is,
) # don't import site at start). That flag, and all of the code in this
# snippet above the last two lines, exist to work around a relatively rare
# problem. If
#
# - your buildout configuration is trying to install a package that is within
# a namespace package, and
#
# - you use a Python that has a different version of this package
# installed in in its site-packages using
# --single-version-externally-managed (that is, using the mechanism
# sometimes used by system packagers:
# http://peak.telecommunity.com/DevCenter/setuptools#install-command ), and
#
# - the new package tries to do sys.path tricks in the setup.py to get a
# __version__,
#
# then the older package will be loaded first, making the setup version
# the wrong number. While very arguably packages simply shouldn't do
# the sys.path tricks, some do, and we don't want buildout to fall over
# when they do.
#
# The namespace packages installed in site-packages with
# --single-version-externally-managed use a mechanism that cause them to
# be processed when site.py is imported (see
# http://mail.python.org/pipermail/distutils-sig/2009-May/011730.html
# for another description of the problem). Simply starting Python with
# -S addresses the problem in Python 2.4 and 2.5, but Python 2.6's
# distutils imports a value from the site module, so we unfortunately
# have to do more drastic surgery in the _easy_install_cmd code below.
#
# Here's an example of the .pth files created by setuptools when using that
# flag:
#
# import sys,new,os;
# p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('<NAMESPACE>',));
# ie = os.path.exists(os.path.join(p,'__init__.py'));
# m = not ie and sys.modules.setdefault('<NAMESPACE>',new.module('<NAMESPACE>'));
# mp = (m or []) and m.__dict__.setdefault('__path__',[]);
# (p not in mp) and mp.append(p)
#
# The code, below, then, runs under -S, indicating that site.py should
# not be loaded initially. It gets the initial sys.path under these
# circumstances, and then imports site (because Python 2.6's distutils
# will want it, as mentioned above). It then reinstates the old sys.path
# value. Then it removes namespace packages (created by the setuptools
# code above) from sys.modules. It identifies namespace packages by
# iterating over every loaded module. It first looks if there is a
# __path__, so it is a package; and then it sees if that __path__ does
# not have an __init__.py. (Note that PEP 382,
# http://www.python.org/dev/peps/pep-0382, makes it possible to have a
# namespace package that has an __init__.py, but also should make it
# unnecessary for site.py to preprocess these packages, so it should be
# fine, as far as can be guessed as of this writing.) Finally, it
# imports easy_install and runs it.
_easy_install_cmd = _safe_arg('''\
import sys,os;\
p = sys.path[:];\
import site;\
sys.path[:] = p;\
[sys.modules.pop(k) for k, v in sys.modules.items()\
if hasattr(v, '__path__') and len(v.__path__)==1 and\
not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))];\
from setuptools.command.easy_install import main;\
main()''')
class Installer: class Installer:
...@@ -151,6 +287,8 @@ class Installer: ...@@ -151,6 +287,8 @@ class Installer:
_use_dependency_links = True _use_dependency_links = True
_allow_picked_versions = True _allow_picked_versions = True
_always_unzip = False _always_unzip = False
_include_site_packages = True
_allowed_eggs_from_site_packages = ('*',)
def __init__(self, def __init__(self,
dest=None, dest=None,
...@@ -162,7 +300,9 @@ class Installer: ...@@ -162,7 +300,9 @@ class Installer:
newest=True, newest=True,
versions=None, versions=None,
use_dependency_links=None, use_dependency_links=None,
allow_hosts=('*',) allow_hosts=('*',),
include_site_packages=None,
allowed_eggs_from_site_packages=None
): ):
self._dest = dest self._dest = dest
self._allow_hosts = allow_hosts self._allow_hosts = allow_hosts
...@@ -184,7 +324,25 @@ class Installer: ...@@ -184,7 +324,25 @@ class Installer:
self._executable = executable self._executable = executable
if always_unzip is not None: if always_unzip is not None:
self._always_unzip = always_unzip self._always_unzip = always_unzip
path = (path and path[:] or []) + buildout_and_setuptools_path path = (path and path[:] or [])
if include_site_packages is not None:
self._include_site_packages = include_site_packages
if allowed_eggs_from_site_packages is not None:
self._allowed_eggs_from_site_packages = tuple(
allowed_eggs_from_site_packages)
stdlib, self._site_packages = _get_system_paths(executable)
version_info = _get_version_info(executable)
if version_info == sys.version_info:
# Maybe we can add the buildout and setuptools path. If we
# are including site_packages, we only have to include the extra
# bits here, so we don't duplicate. On the other hand, if we
# are not including site_packages, we only want to include the
# parts that are not in site_packages, so the code is the same.
path.extend(
set(buildout_and_setuptools_path).difference(
self._site_packages))
if self._include_site_packages:
path.extend(self._site_packages)
if dest is not None and dest not in path: if dest is not None and dest not in path:
path.insert(0, dest) path.insert(0, dest)
self._path = path self._path = path
...@@ -193,13 +351,42 @@ class Installer: ...@@ -193,13 +351,42 @@ class Installer:
self._newest = newest self._newest = newest
self._env = pkg_resources.Environment(path, self._env = pkg_resources.Environment(path,
python=_get_version(executable)) python=_get_version(executable))
self._index = _get_index(executable, index, links, self._allow_hosts) self._index = _get_index(executable, index, links, self._allow_hosts,
self._path)
if versions is not None: if versions is not None:
self._versions = versions self._versions = versions
_allowed_eggs_from_site_packages_regex = None
def allow_site_package_egg(self, name):
if (not self._include_site_packages or
not self._allowed_eggs_from_site_packages):
# If the answer is a blanket "no," perform a shortcut.
return False
if self._allowed_eggs_from_site_packages_regex is None:
pattern = '(%s)' % (
'|'.join(
fnmatch.translate(name)
for name in self._allowed_eggs_from_site_packages),
)
self._allowed_eggs_from_site_packages_regex = re.compile(pattern)
return bool(self._allowed_eggs_from_site_packages_regex.match(name))
def _satisfied(self, req, source=None): def _satisfied(self, req, source=None):
dists = [dist for dist in self._env[req.project_name] if dist in req] # We get all distributions that match the given requirement. If we are
# not supposed to include site-packages for the given egg, we also
# filter those out. Even if include_site_packages is False and so we
# have excluded site packages from the _env's paths (see
# Installer.__init__), we need to do the filtering here because an
# .egg-link, such as one for setuptools or zc.buildout installed by
# zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
# path in our _site_packages.
dists = [dist for dist in self._env[req.project_name] if (
dist in req and (
dist.location not in self._site_packages or
self.allow_site_package_egg(dist.project_name))
)
]
if not dists: if not dists:
logger.debug('We have no distributions for %s that satisfies %r.', logger.debug('We have no distributions for %s that satisfies %r.',
req.project_name, str(req)) req.project_name, str(req))
...@@ -301,7 +488,7 @@ class Installer: ...@@ -301,7 +488,7 @@ class Installer:
try: try:
path = setuptools_loc path = setuptools_loc
args = ('-c', _easy_install_cmd, '-mUNxd', _safe_arg(tmp)) args = ('-Sc', _easy_install_cmd, '-mUNxd', _safe_arg(tmp))
if self._always_unzip: if self._always_unzip:
args += ('-Z', ) args += ('-Z', )
level = logger.getEffectiveLevel() level = logger.getEffectiveLevel()
...@@ -400,13 +587,21 @@ class Installer: ...@@ -400,13 +587,21 @@ class Installer:
# Nothing is available. # Nothing is available.
return None return None
# Filter the available dists for the requirement and source flag # Filter the available dists for the requirement and source flag. If
dists = [dist for dist in index[requirement.project_name] # we are not supposed to include site-packages for the given egg, we
if ((dist in requirement) # also filter those out. Even if include_site_packages is False and so
and # we have excluded site packages from the _env's paths (see
((not source) or # Installer.__init__), we need to do the filtering here because an
(dist.precedence == pkg_resources.SOURCE_DIST) # .egg-link, such as one for setuptools or zc.buildout installed by
) # zc.buildout.buildout.Buildout.bootstrap, can indirectly include a
# path in our _site_packages.
dists = [dist for dist in index[requirement.project_name] if (
dist in requirement and (
dist.location not in self._site_packages or
self.allow_site_package_egg(dist.project_name))
and (
(not source) or
(dist.precedence == pkg_resources.SOURCE_DIST))
) )
] ]
...@@ -567,7 +762,7 @@ class Installer: ...@@ -567,7 +762,7 @@ class Installer:
self._links.append(link) self._links.append(link)
self._index = _get_index(self._executable, self._index = _get_index(self._executable,
self._index_url, self._links, self._index_url, self._links,
self._allow_hosts) self._allow_hosts, self._path)
for dist in dists: for dist in dists:
# Check whether we picked a version and, if we did, report it: # Check whether we picked a version and, if we did, report it:
...@@ -609,6 +804,8 @@ class Installer: ...@@ -609,6 +804,8 @@ class Installer:
def _constrain(self, requirement): def _constrain(self, requirement):
if is_distribute and requirement.key == 'setuptools':
requirement = pkg_resources.Requirement.parse('distribute')
version = self._versions.get(requirement.project_name) version = self._versions.get(requirement.project_name)
if version: if version:
if version not in requirement: if version not in requirement:
...@@ -648,35 +845,52 @@ class Installer: ...@@ -648,35 +845,52 @@ class Installer:
self._maybe_add_setuptools(ws, dist) self._maybe_add_setuptools(ws, dist)
# OK, we have the requested distributions and they're in the working # OK, we have the requested distributions and they're in the working
# set, but they may have unmet requirements. We'll simply keep # set, but they may have unmet requirements. We'll resolve these
# trying to resolve requirements, adding missing requirements as they # requirements. This is code modified from
# are reported. # pkg_resources.WorkingSet.resolve. We can't reuse that code directly
# # because we have to constrain our requirements (see
# Note that we don't pass in the environment, because we want # versions_section_ignored_for_dependency_in_favor_of_site_packages in
# zc.buildout.tests).
requirements.reverse() # Set up the stack.
processed = {} # This is a set of processed requirements.
best = {} # This is a mapping of key -> dist.
# Note that we don't use the existing environment, because we want
# to look for new eggs unless what we have is the best that # to look for new eggs unless what we have is the best that
# matches the requirement. # matches the requirement.
while 1: env = pkg_resources.Environment(ws.entries)
while requirements:
# Process dependencies breadth-first.
req = self._constrain(requirements.pop(0))
if req in processed:
# Ignore cyclic or redundant dependencies.
continue
dist = best.get(req.key)
if dist is None:
# Find the best distribution and add it to the map.
dist = ws.by_key.get(req.key)
if dist is None:
try: try:
ws.resolve(requirements) dist = best[req.key] = env.best_match(req, ws)
except pkg_resources.DistributionNotFound, err: except pkg_resources.VersionConflict, err:
[requirement] = err raise VersionConflict(err, ws)
requirement = self._constrain(requirement) if dist is None:
if destination: if destination:
logger.debug('Getting required %r', str(requirement)) logger.debug('Getting required %r', str(req))
else: else:
logger.debug('Adding required %r', str(requirement)) logger.debug('Adding required %r', str(req))
_log_requirement(ws, requirement) _log_requirement(ws, req)
for dist in self._get_dist(req,
for dist in self._get_dist(requirement, ws, self._always_unzip ws, self._always_unzip):
):
ws.add(dist) ws.add(dist)
self._maybe_add_setuptools(ws, dist) self._maybe_add_setuptools(ws, dist)
except pkg_resources.VersionConflict, err: if dist not in req:
raise VersionConflict(err, ws) # Oops, the "best" so far conflicts with a dependency.
else: raise VersionConflict(
break pkg_resources.VersionConflict(dist, req), ws)
requirements.extend(dist.requires(req.extras)[::-1])
processed[req] = True
if dist.location in self._site_packages:
logger.debug('Egg from site-packages: %s', dist)
return ws return ws
def build(self, spec, build_ext): def build(self, spec, build_ext):
...@@ -771,6 +985,18 @@ def prefer_final(setting=None): ...@@ -771,6 +985,18 @@ def prefer_final(setting=None):
Installer._prefer_final = bool(setting) Installer._prefer_final = bool(setting)
return old return old
def include_site_packages(setting=None):
old = Installer._include_site_packages
if setting is not None:
Installer._include_site_packages = bool(setting)
return old
def allowed_eggs_from_site_packages(setting=None):
old = Installer._allowed_eggs_from_site_packages
if setting is not None:
Installer._allowed_eggs_from_site_packages = tuple(setting)
return old
def use_dependency_links(setting=None): def use_dependency_links(setting=None):
old = Installer._use_dependency_links old = Installer._use_dependency_links
if setting is not None: if setting is not None:
...@@ -793,19 +1019,27 @@ def install(specs, dest, ...@@ -793,19 +1019,27 @@ def install(specs, dest,
links=(), index=None, links=(), index=None,
executable=sys.executable, always_unzip=None, executable=sys.executable, always_unzip=None,
path=None, working_set=None, newest=True, versions=None, path=None, working_set=None, newest=True, versions=None,
use_dependency_links=None, allow_hosts=('*',)): use_dependency_links=None, allow_hosts=('*',),
include_site_packages=None, allowed_eggs_from_site_packages=None):
installer = Installer(dest, links, index, executable, always_unzip, path, installer = Installer(dest, links, index, executable, always_unzip, path,
newest, versions, use_dependency_links, newest, versions, use_dependency_links,
allow_hosts=allow_hosts) allow_hosts=allow_hosts,
include_site_packages=include_site_packages,
allowed_eggs_from_site_packages=
allowed_eggs_from_site_packages)
return installer.install(specs, working_set) return installer.install(specs, working_set)
def build(spec, dest, build_ext, def build(spec, dest, build_ext,
links=(), index=None, links=(), index=None,
executable=sys.executable, executable=sys.executable,
path=None, newest=True, versions=None, allow_hosts=('*',)): path=None, newest=True, versions=None, allow_hosts=('*',),
include_site_packages=None, allowed_eggs_from_site_packages=None):
installer = Installer(dest, links, index, executable, True, path, newest, installer = Installer(dest, links, index, executable, True, path, newest,
versions, allow_hosts=allow_hosts) versions, allow_hosts=allow_hosts,
include_site_packages=include_site_packages,
allowed_eggs_from_site_packages=
allowed_eggs_from_site_packages)
return installer.build(spec, build_ext) return installer.build(spec, build_ext)
...@@ -900,9 +1134,15 @@ def develop(setup, dest, ...@@ -900,9 +1134,15 @@ def develop(setup, dest,
undo.reverse() undo.reverse()
[f() for f in undo] [f() for f in undo]
def working_set(specs, executable, path, include_site_packages=None,
allowed_eggs_from_site_packages=None):
return install(
specs, None, executable=executable, path=path,
include_site_packages=include_site_packages,
allowed_eggs_from_site_packages=allowed_eggs_from_site_packages)
def working_set(specs, executable, path): ############################################################################
return install(specs, None, executable=executable, path=path) # Script generation functions
def scripts(reqs, working_set, executable, dest, def scripts(reqs, working_set, executable, dest,
scripts=None, scripts=None,
...@@ -912,20 +1152,95 @@ def scripts(reqs, working_set, executable, dest, ...@@ -912,20 +1152,95 @@ def scripts(reqs, working_set, executable, dest,
initialization='', initialization='',
relative_paths=False, relative_paths=False,
): ):
"""Generate scripts and/or an interpreter.
path = [dist.location for dist in working_set] See sitepackage_safe_scripts for a version that can be used with a Python
path.extend(extra_paths) that has code installed in site-packages. It has more options and a
path = map(realpath, path) different approach.
"""
path = _get_path(working_set, extra_paths)
if initialization:
initialization = '\n'+initialization+'\n'
generated = _generate_scripts(
reqs, working_set, dest, path, scripts, relative_paths,
initialization, executable, arguments)
if interpreter:
sname = os.path.join(dest, interpreter)
spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
generated.extend(
_pyscript(spath, sname, executable, rpsetup))
return generated
def sitepackage_safe_scripts(
dest, working_set, executable, site_py_dest,
reqs=(), scripts=None, interpreter=None, extra_paths=(),
initialization='', include_site_packages=False, exec_sitecustomize=False,
relative_paths=False, script_arguments='', script_initialization=''):
"""Generate scripts and/or an interpreter from a system Python.
This accomplishes the same job as the ``scripts`` function, above,
but it does so in an alternative way that allows safely including
Python site packages, if desired, and choosing to execute the Python's
sitecustomize.
"""
generated = [] generated = []
generated.append(_generate_sitecustomize(
site_py_dest, executable, initialization, exec_sitecustomize))
generated.append(_generate_site(
site_py_dest, working_set, executable, extra_paths,
include_site_packages, relative_paths))
script_initialization = _script_initialization_template % dict(
site_py_dest=site_py_dest,
script_initialization=script_initialization)
if not script_initialization.endswith('\n'):
script_initialization += '\n'
generated.extend(_generate_scripts(
reqs, working_set, dest, [site_py_dest], scripts, relative_paths,
script_initialization, executable, script_arguments, block_site=True))
if interpreter:
generated.extend(_generate_interpreter(
interpreter, dest, executable, site_py_dest, relative_paths))
return generated
_script_initialization_template = '''
import os
path = sys.path[0]
if os.environ.get('PYTHONPATH'):
path = os.pathsep.join([path, os.environ['PYTHONPATH']])
os.environ['PYTHONPATH'] = path
import site # imports custom buildout-generated site.py
%(script_initialization)s'''
# Utilities for the script generation functions.
# These are shared by both ``scripts`` and ``sitepackage_safe_scripts``
def _get_path(working_set, extra_paths=()):
"""Given working set and extra paths, return a normalized path list."""
path = [dist.location for dist in working_set]
path.extend(extra_paths)
return map(realpath, path)
def _generate_scripts(reqs, working_set, dest, path, scripts, relative_paths,
initialization, executable, arguments,
block_site=False):
"""Generate scripts for the given requirements.
- reqs is an iterable of string requirements or entry points.
- The requirements must be findable in the given working_set.
- The dest is the directory in which the scripts should be created.
- The path is a list of paths that should be added to sys.path.
- The scripts is an optional dictionary. If included, the keys should be
the names of the scripts that should be created, as identified in their
entry points; and the values should be the name the script should
actually be created with.
- relative_paths, if given, should be the path that is the root of the
buildout (the common path that should be the root of what is relative).
"""
if isinstance(reqs, str): if isinstance(reqs, str):
raise TypeError('Expected iterable of requirements or entry points,' raise TypeError('Expected iterable of requirements or entry points,'
' got string.') ' got string.')
generated = []
if initialization:
initialization = '\n'+initialization+'\n'
entry_points = [] entry_points = []
for req in reqs: for req in reqs:
if isinstance(req, str): if isinstance(req, str):
...@@ -939,7 +1254,6 @@ def scripts(reqs, working_set, executable, dest, ...@@ -939,7 +1254,6 @@ def scripts(reqs, working_set, executable, dest,
) )
else: else:
entry_points.append(req) entry_points.append(req)
for name, module_name, attrs in entry_points: for name, module_name, attrs in entry_points:
if scripts is not None: if scripts is not None:
sname = scripts.get(name) sname = scripts.get(name)
...@@ -947,40 +1261,51 @@ def scripts(reqs, working_set, executable, dest, ...@@ -947,40 +1261,51 @@ def scripts(reqs, working_set, executable, dest,
continue continue
else: else:
sname = name sname = name
sname = os.path.join(dest, sname) sname = os.path.join(dest, sname)
spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths) spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
generated.extend( generated.extend(
_script(module_name, attrs, spath, sname, executable, arguments, _script(sname, executable, rpsetup, spath, initialization,
initialization, rpsetup) module_name, attrs, arguments, block_site=block_site))
)
if interpreter:
sname = os.path.join(dest, interpreter)
spath, rpsetup = _relative_path_and_setup(sname, path, relative_paths)
generated.extend(_pyscript(spath, sname, executable, rpsetup))
return generated return generated
def _relative_path_and_setup(sname, path, relative_paths): def _relative_path_and_setup(sname, path,
relative_paths=False, indent_level=1,
omit_os_import=False):
"""Return a string of code of paths and of setup if appropriate.
- sname is the full path to the script name to be created.
- path is the list of paths to be added to sys.path.
- relative_paths, if given, should be the path that is the root of the
buildout (the common path that should be the root of what is relative).
- indent_level is the number of four-space indents that the path should
insert before each element of the path.
"""
if relative_paths: if relative_paths:
relative_paths = os.path.normcase(relative_paths) relative_paths = os.path.normcase(relative_paths)
sname = os.path.normcase(os.path.abspath(sname)) sname = os.path.normcase(os.path.abspath(sname))
spath = ',\n '.join( spath = _format_paths(
[_relativitize(os.path.normcase(path_item), sname, relative_paths) [_relativitize(os.path.normcase(path_item), sname, relative_paths)
for path_item in path] for path_item in path], indent_level=indent_level)
)
rpsetup = relative_paths_setup rpsetup = relative_paths_setup
if not omit_os_import:
rpsetup = '\n\nimport os\n' + rpsetup
for i in range(_relative_depth(relative_paths, sname)): for i in range(_relative_depth(relative_paths, sname)):
rpsetup += "base = os.path.dirname(base)\n" rpsetup += "\nbase = os.path.dirname(base)"
else: else:
spath = repr(path)[1:-1].replace(', ', ',\n ') spath = _format_paths((repr(p) for p in path),
indent_level=indent_level)
rpsetup = '' rpsetup = ''
return spath, rpsetup return spath, rpsetup
def _relative_depth(common, path): def _relative_depth(common, path):
"""Return number of dirs separating ``path`` from ancestor, ``common``.
For instance, if path is /foo/bar/baz/bing, and common is /foo, this will
return 2--in UNIX, the number of ".." to get from bing's directory
to foo.
This is a helper for _relative_path_and_setup.
"""
n = 0 n = 0
while 1: while 1:
dirname = os.path.dirname(path) dirname = os.path.dirname(path)
...@@ -993,6 +1318,11 @@ def _relative_depth(common, path): ...@@ -993,6 +1318,11 @@ def _relative_depth(common, path):
return n return n
def _relative_path(common, path): def _relative_path(common, path):
"""Return the relative path from ``common`` to ``path``.
This is a helper for _relativitize, which is a helper to
_relative_path_and_setup.
"""
r = [] r = []
while 1: while 1:
dirname, basename = os.path.split(path) dirname, basename = os.path.split(path)
...@@ -1006,6 +1336,11 @@ def _relative_path(common, path): ...@@ -1006,6 +1336,11 @@ def _relative_path(common, path):
return os.path.join(*r) return os.path.join(*r)
def _relativitize(path, script, relative_paths): def _relativitize(path, script, relative_paths):
"""Return a code string for the given path.
Path is relative to the base path ``relative_paths``if the common prefix
between ``path`` and ``script`` starts with ``relative_paths``.
"""
if path == script: if path == script:
raise AssertionError("path == script") raise AssertionError("path == script")
common = os.path.dirname(os.path.commonprefix([path, script])) common = os.path.dirname(os.path.commonprefix([path, script]))
...@@ -1016,66 +1351,82 @@ def _relativitize(path, script, relative_paths): ...@@ -1016,66 +1351,82 @@ def _relativitize(path, script, relative_paths):
else: else:
return repr(path) return repr(path)
relative_paths_setup = """ relative_paths_setup = """
import os
join = os.path.join join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__))) base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))"""
"""
def _script(module_name, attrs, path, dest, executable, arguments,
initialization, rsetup):
generated = []
script = dest
if is_win32:
dest += '-script.py'
contents = script_template % dict( def _write_script(full_name, contents, logged_type):
python = _safe_arg(executable), """Write contents of script in full_name, logging the action.
path = path,
module_name = module_name,
attrs = attrs,
arguments = arguments,
initialization = initialization,
relative_paths_setup = rsetup,
)
changed = not (os.path.exists(dest) and open(dest).read() == contents)
The only tricky bit in this function is that it supports Windows by
creating exe files using a pkg_resources helper.
"""
generated = []
script_name = full_name
if is_win32: if is_win32:
# generate exe file and give the script a magic name: script_name += '-script.py'
exe = script+'.exe' # Generate exe file and give the script a magic name.
exe = full_name + '.exe'
new_data = pkg_resources.resource_string('setuptools', 'cli.exe') new_data = pkg_resources.resource_string('setuptools', 'cli.exe')
if not os.path.exists(exe) or (open(exe, 'rb').read() != new_data): if not os.path.exists(exe) or (open(exe, 'rb').read() != new_data):
# Only write it if it's different. # Only write it if it's different.
open(exe, 'wb').write(new_data) open(exe, 'wb').write(new_data)
generated.append(exe) generated.append(exe)
changed = not (os.path.exists(script_name) and
open(script_name).read() == contents)
if changed: if changed:
open(dest, 'w').write(contents) open(script_name, 'w').write(contents)
logger.info("Generated script %r.", script)
try: try:
os.chmod(dest, 0755) os.chmod(script_name, 0755)
except (AttributeError, os.error): except (AttributeError, os.error):
pass pass
logger.info("Generated %s %r.", logged_type, full_name)
generated.append(dest) generated.append(script_name)
return generated return generated
def _format_paths(paths, indent_level=1):
"""Format paths for inclusion in a script."""
separator = ',\n' + indent_level * ' '
return separator.join(paths)
def _script(dest, executable, relative_paths_setup, path, initialization,
module_name, attrs, arguments, block_site=False):
if block_site:
dash_S = ' -S'
else:
dash_S = ''
contents = script_template % dict(
python=_safe_arg(executable),
dash_S=dash_S,
path=path,
module_name=module_name,
attrs=attrs,
arguments=arguments,
initialization=initialization,
relative_paths_setup=relative_paths_setup,
)
return _write_script(dest, contents, 'script')
if is_jython and jython_os_name == 'linux': if is_jython and jython_os_name == 'linux':
script_header = '#!/usr/bin/env %(python)s' script_header = '#!/usr/bin/env %(python)s%(dash_S)s'
else: else:
script_header = '#!%(python)s' script_header = '#!%(python)s%(dash_S)s'
sys_path_template = '''\
import sys
sys.path[0:0] = [
%s,
]
'''
script_template = script_header + '''\ script_template = script_header + '''\
%(relative_paths_setup)s %(relative_paths_setup)s
import sys import sys
sys.path[0:0] = [ sys.path[0:0] = [
%(path)s, %(path)s,
] ]
%(initialization)s %(initialization)s
import %(module_name)s import %(module_name)s
...@@ -1083,42 +1434,20 @@ if __name__ == '__main__': ...@@ -1083,42 +1434,20 @@ if __name__ == '__main__':
%(module_name)s.%(attrs)s(%(arguments)s) %(module_name)s.%(attrs)s(%(arguments)s)
''' '''
# These are used only by the older ``scripts`` function.
def _pyscript(path, dest, executable, rsetup): def _pyscript(path, dest, executable, rsetup):
generated = []
script = dest
if is_win32:
dest += '-script.py'
contents = py_script_template % dict( contents = py_script_template % dict(
python = _safe_arg(executable), python=_safe_arg(executable),
path = path, dash_S='',
relative_paths_setup = rsetup, path=path,
relative_paths_setup=rsetup,
) )
changed = not (os.path.exists(dest) and open(dest).read() == contents) return _write_script(dest, contents, 'interpreter')
if is_win32:
# generate exe file and give the script a magic name:
exe = script + '.exe'
open(exe, 'wb').write(
pkg_resources.resource_string('setuptools', 'cli.exe')
)
generated.append(exe)
if changed:
open(dest, 'w').write(contents)
try:
os.chmod(dest,0755)
except (AttributeError, os.error):
pass
logger.info("Generated interpreter %r.", script)
generated.append(dest)
return generated
py_script_template = script_header + '''\ py_script_template = script_header + '''\
%(relative_paths_setup)s %(relative_paths_setup)s
import sys import sys
sys.path[0:0] = [ sys.path[0:0] = [
...@@ -1151,6 +1480,235 @@ if _interactive: ...@@ -1151,6 +1480,235 @@ if _interactive:
__import__("code").interact(banner="", local=globals()) __import__("code").interact(banner="", local=globals())
''' '''
# These are used only by the newer ``sitepackage_safe_scripts`` function.
def _get_module_file(executable, name):
"""Return a module's file path.
- executable is a path to the desired Python executable.
- name is the name of the (pure, not C) Python module.
"""
cmd = [executable, "-Sc",
"import imp; "
"fp, path, desc = imp.find_module(%r); "
"fp.close; "
"print path" % (name,)]
env = os.environ.copy()
# We need to make sure that PYTHONPATH, which will often be set to
# include a custom buildout-generated site.py, is not set, or else
# we will not get an accurate value for the "real" site.py and
# sitecustomize.py.
env.pop('PYTHONPATH', None)
_proc = subprocess.Popen(
cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
stdout, stderr = _proc.communicate();
if _proc.returncode:
logger.info(
'Could not find file for module %s:\n%s', name, stderr)
return None
# else: ...
res = stdout.strip()
if res.endswith('.pyc') or res.endswith('.pyo'):
raise RuntimeError('Cannot find uncompiled version of %s' % (name,))
if not os.path.exists(res):
raise RuntimeError(
'File does not exist for module %s:\n%s' % (name, res))
return res
def _generate_sitecustomize(dest, executable, initialization='',
exec_sitecustomize=False):
"""Write a sitecustomize file with optional custom initialization.
The created script will execute the underlying Python's
sitecustomize if exec_sitecustomize is True.
"""
sitecustomize_path = os.path.join(dest, 'sitecustomize.py')
sitecustomize = open(sitecustomize_path, 'w')
if initialization:
sitecustomize.write(initialization + '\n')
if exec_sitecustomize:
real_sitecustomize_path = _get_module_file(
executable, 'sitecustomize')
if real_sitecustomize_path:
real_sitecustomize = open(real_sitecustomize_path, 'r')
sitecustomize.write(
'\n# The following is from\n# %s\n' %
(real_sitecustomize_path,))
sitecustomize.write(real_sitecustomize.read())
real_sitecustomize.close()
sitecustomize.close()
return sitecustomize_path
def _generate_site(dest, working_set, executable, extra_paths=(),
include_site_packages=False, relative_paths=False):
"""Write a site.py file with eggs from working_set.
extra_paths will be added to the path. If include_site_packages is True,
paths from the underlying Python will be added.
"""
path = _get_path(working_set, extra_paths)
site_path = os.path.join(dest, 'site.py')
egg_path_string, preamble = _relative_path_and_setup(
site_path, path, relative_paths, indent_level=2, omit_os_import=True)
if preamble:
preamble = '\n'.join(
[(line and ' %s' % (line,) or line)
for line in preamble.split('\n')])
original_path_setup = ''
if include_site_packages:
stdlib, site_paths = _get_system_paths(executable)
original_path_setup = original_path_snippet % (
_format_paths((repr(p) for p in site_paths), 2),)
distribution = working_set.find(
pkg_resources.Requirement.parse('setuptools'))
if distribution is not None:
# We need to worry about namespace packages.
if relative_paths:
location = _relativitize(
distribution.location,
os.path.normcase(os.path.abspath(site_path)),
relative_paths)
else:
location = repr(distribution.location)
preamble += namespace_include_site_packages_setup % (location,)
original_path_setup = (
addsitedir_namespace_originalpackages_snippet +
original_path_setup)
addsitepackages_marker = 'def addsitepackages('
enableusersite_marker = 'ENABLE_USER_SITE = '
successful_rewrite = False
real_site_path = _get_module_file(executable, 'site')
real_site = open(real_site_path, 'r')
site = open(site_path, 'w')
try:
for line in real_site.readlines():
if line.startswith(enableusersite_marker):
site.write(enableusersite_marker)
site.write('False # buildout does not support user sites.\n')
elif line.startswith(addsitepackages_marker):
site.write(addsitepackages_script % (
preamble, egg_path_string, original_path_setup))
site.write(line[len(addsitepackages_marker):])
successful_rewrite = True
else:
site.write(line)
finally:
site.close()
real_site.close()
if not successful_rewrite:
raise RuntimeError(
'Buildout did not successfully rewrite %s to %s' %
(real_site_path, site_path))
return site_path
namespace_include_site_packages_setup = '''
setuptools_path = %s
sys.path.append(setuptools_path)
known_paths.add(os.path.normcase(setuptools_path))
import pkg_resources'''
addsitedir_namespace_originalpackages_snippet = '''
pkg_resources.working_set.add_entry(sitedir)'''
original_path_snippet = '''
sys.__egginsert = len(buildout_paths) # Support distribute.
original_paths = [
%s
]
for path in original_paths:
addsitedir(path, known_paths)'''
addsitepackages_script = '''\
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
See original_addsitepackages, below, for the original version."""%s
buildout_paths = [
%s
]
for path in buildout_paths:
sitedir, sitedircase = makepath(path)
if not sitedircase in known_paths and os.path.exists(sitedir):
sys.path.append(sitedir)
known_paths.add(sitedircase)%s
return known_paths
def original_addsitepackages('''
def _generate_interpreter(name, dest, executable, site_py_dest,
relative_paths=False):
"""Write an interpreter script, using the site.py approach."""
full_name = os.path.join(dest, name)
site_py_dest_string, rpsetup = _relative_path_and_setup(
full_name, [site_py_dest], relative_paths, omit_os_import=True)
if rpsetup:
rpsetup += "\n"
if sys.platform == 'win32':
windows_import = '\nimport subprocess'
# os.exec* is a mess on Windows, particularly if the path
# to the executable has spaces and the Python is using MSVCRT.
# The standard fix is to surround the executable's path with quotes,
# but that has been unreliable in testing.
#
# Here's a demonstration of the problem. Given a Python
# compiled with a MSVCRT-based compiler, such as the free Visual
# C++ 2008 Express Edition, and an executable path with spaces
# in it such as the below, we see the following.
#
# >>> import os
# >>> p0 = 'C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe'
# >>> os.path.exists(p0)
# True
# >>> os.execv(p0, [])
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# OSError: [Errno 22] Invalid argument
#
# That seems like a standard problem. The standard solution is
# to quote the path (see, for instance
# http://bugs.python.org/issue436259). However, this solution,
# and other variations, fail:
#
# >>> p1 = '"C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\Python-2.6.4\\PCbuild\\python.exe"'
# >>> os.execv(p1, [])
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# OSError: [Errno 22] Invalid argument
#
# We simply use subprocess instead, since it handles everything
# nicely, and the transparency of exec* (that is, not running,
# perhaps unexpectedly, in a subprocess) is arguably not a
# necessity, at least for many use cases.
execute = 'subprocess.call(argv, env=environ)'
else:
windows_import = ''
execute = 'os.execve(sys.executable, argv, environ)'
contents = interpreter_template % dict(
python=_safe_arg(executable),
dash_S=' -S',
site_dest=site_py_dest_string,
relative_paths_setup=rpsetup,
windows_import=windows_import,
execute=execute,
)
return _write_script(full_name, contents, 'interpreter')
interpreter_template = script_header + '''
import os
import sys%(windows_import)s
%(relative_paths_setup)s
argv = [sys.executable] + sys.argv[1:]
environ = os.environ.copy()
path = %(site_dest)s
if environ.get('PYTHONPATH'):
path = os.pathsep.join([path, environ['PYTHONPATH']])
environ['PYTHONPATH'] = path
%(execute)s
'''
# End of script generation code.
############################################################################
runsetup_template = """ runsetup_template = """
import sys import sys
sys.path.insert(0, %(setupdir)r) sys.path.insert(0, %(setupdir)r)
......
...@@ -89,6 +89,14 @@ use_dependency_links ...@@ -89,6 +89,14 @@ use_dependency_links
for using dependency_links in preference to other for using dependency_links in preference to other
locations. Defaults to true. locations. Defaults to true.
include_site_packages
A flag indicating whether Python's non-standard-library packages should
be available for finding dependencies. Defaults to true.
Paths outside of Python's standard library--or more precisely, those that
are not included when Python is started with the -S argument--are loosely
referred to as "site-packages" here.
relative_paths relative_paths
Adjust egg paths so they are relative to the script path. This Adjust egg paths so they are relative to the script path. This
allows scripts to work when scripts and eggs are moved, as long as allows scripts to work when scripts and eggs are moved, as long as
...@@ -214,7 +222,9 @@ dependencies. We might do this to specify a specific version. ...@@ -214,7 +222,9 @@ dependencies. We might do this to specify a specific version.
d other-1.0-py2.4.egg d other-1.0-py2.4.egg
We can request that eggs be unzipped even if they are zip safe. This We can request that eggs be unzipped even if they are zip safe. This
can be useful when debugging. can be useful when debugging. (Note that Distribute will unzip eggs by
default, so if you are using Distribute, most or all eggs will already be
unzipped without this flag.)
>>> rmdir(dest) >>> rmdir(dest)
>>> dest = tmpdir('sample-install') >>> dest = tmpdir('sample-install')
...@@ -399,6 +409,68 @@ dictionary: ...@@ -399,6 +409,68 @@ dictionary:
>>> [d.version for d in ws] >>> [d.version for d in ws]
['0.3', '1.1'] ['0.3', '1.1']
Dependencies in Site Packages
-----------------------------
Paths outside of Python's standard library--or more precisely, those that are
not included when Python is started with the -S argument--are loosely referred
to as "site-packages" here. These site-packages are searched by default for
distributions. This can be disabled, so that, for instance, a system Python
can be used with buildout, cleaned of any packages installed by a user or
system package manager.
The default behavior can be controlled and introspected using
zc.buildout.easy_install.include_site_packages.
>>> zc.buildout.easy_install.include_site_packages()
True
Here's an example of using a Python executable that includes our dependencies.
Our "py_path" will have the "demoneeded," and "demo" packages available.
We'll simply be asking for "demoneeded" here, but without any external
index or links.
>>> from zc.buildout.tests import create_sample_sys_install
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None)
>>> [dist.project_name for dist in workingset]
['demoneeded']
That worked fine. Let's try again with site packages not allowed. We'll
change the policy by changing the default. Notice that the function for
changing the default value returns the previous value.
>>> zc.buildout.easy_install.include_site_packages(False)
True
>>> zc.buildout.easy_install.include_site_packages()
False
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None)
Traceback (most recent call last):
...
MissingDistribution: Couldn't find a distribution for 'demoneeded'.
>>> zc.buildout.easy_install.clear_index_cache()
Now we'll reset the default.
>>> zc.buildout.easy_install.include_site_packages(True)
False
>>> zc.buildout.easy_install.include_site_packages()
True
Dependency links Dependency links
---------------- ----------------
...@@ -521,25 +593,38 @@ To return the dependency_links behavior to normal call the function again. ...@@ -521,25 +593,38 @@ To return the dependency_links behavior to normal call the function again.
Script generation Script generation
----------------- -----------------
The easy_install module provides support for creating scripts from The easy_install module provides support for creating scripts from eggs.
eggs. It provides a function similar to setuptools except that it It provides two competing functions. One, ``scripts``, is a
provides facilities for baking a script's path into the script. This well-established approach to generating reliable scripts with a "clean"
has two advantages: Python--e.g., one that does not have any packages in its site-packages.
The other, ``sitepackage_safe_scripts``, is newer, a bit trickier, and is
designed to work with a Python that has code in its site-packages, such
as a system Python.
Both are similar to setuptools except that they provides facilities for
baking a script's path into the script. This has two advantages:
- The eggs to be used by a script are not chosen at run time, making - The eggs to be used by a script are not chosen at run time, making
startup faster and, more importantly, deterministic. startup faster and, more importantly, deterministic.
- The script doesn't have to import pkg_resources because the logic - The script doesn't have to import pkg_resources because the logic that
that pkg_resources would execute at run time is executed at pkg_resources would execute at run time is executed at script-creation
script-creation time. time. (There is an exception in ``sitepackage_safe_scripts`` if you
want to have your Python's site packages available, as discussed
below, but even in that case pkg_resources is only partially
activated, which can be a significant time savings.)
The scripts method can be used to generate scripts. Let's create a
destination directory for it to place them in:
>>> import tempfile The ``scripts`` function
~~~~~~~~~~~~~~~~~~~~~~~~
The ``scripts`` function is the first way to generate scripts that we'll
examine. It is the earlier approach that the package offered. Let's
create a destination directory for it to place them in:
>>> bin = tmpdir('bin') >>> bin = tmpdir('bin')
Now, we'll use the scripts method to generate scripts in this directory Now, we'll use the scripts function to generate scripts in this directory
from the demo egg: from the demo egg:
>>> import sys >>> import sys
...@@ -736,8 +821,8 @@ original script names to new script names. ...@@ -736,8 +821,8 @@ original script names to new script names.
>>> print system(os.path.join(bin, 'run')), >>> print system(os.path.join(bin, 'run')),
3 1 3 1
Including extra paths in scripts The ``scripts`` function: Including extra paths in scripts
-------------------------------- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We can pass a keyword argument, extra paths, to cause additional paths We can pass a keyword argument, extra paths, to cause additional paths
to be included in the a generated script: to be included in the a generated script:
...@@ -762,8 +847,8 @@ to be included in the a generated script: ...@@ -762,8 +847,8 @@ to be included in the a generated script:
if __name__ == '__main__': if __name__ == '__main__':
eggrecipedemo.main() eggrecipedemo.main()
Providing script arguments The ``scripts`` function: Providing script arguments
-------------------------- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An "argument" keyword argument can be used to pass arguments to an An "argument" keyword argument can be used to pass arguments to an
entry point. The value passed is a source string to be placed between the entry point. The value passed is a source string to be placed between the
...@@ -786,8 +871,8 @@ parentheses in the call: ...@@ -786,8 +871,8 @@ parentheses in the call:
if __name__ == '__main__': if __name__ == '__main__':
eggrecipedemo.main(1, 2) eggrecipedemo.main(1, 2)
Passing initialization code The ``scripts`` function: Passing initialization code
--------------------------- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can also pass script initialization code: You can also pass script initialization code:
...@@ -812,8 +897,8 @@ You can also pass script initialization code: ...@@ -812,8 +897,8 @@ You can also pass script initialization code:
if __name__ == '__main__': if __name__ == '__main__':
eggrecipedemo.main(1, 2) eggrecipedemo.main(1, 2)
Relative paths The ``scripts`` function: Relative paths
-------------- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes, you want to be able to move a buildout directory around and Sometimes, you want to be able to move a buildout directory around and
have scripts still work without having to rebuild them. We can have scripts still work without having to rebuild them. We can
...@@ -836,7 +921,7 @@ to pass a common base directory of the scripts and eggs: ...@@ -836,7 +921,7 @@ to pass a common base directory of the scripts and eggs:
... interpreter='py', ... interpreter='py',
... relative_paths=bo) ... relative_paths=bo)
>>> cat(bo, 'bin', 'run') >>> cat(bo, 'bin', 'run') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4 #!/usr/local/bin/python2.4
<BLANKLINE> <BLANKLINE>
import os import os
...@@ -868,7 +953,7 @@ Of course, running the script works: ...@@ -868,7 +953,7 @@ Of course, running the script works:
We specified an interpreter and its paths are adjusted too: We specified an interpreter and its paths are adjusted too:
>>> cat(bo, 'bin', 'py') >>> cat(bo, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4 #!/usr/local/bin/python2.4
<BLANKLINE> <BLANKLINE>
import os import os
...@@ -911,6 +996,570 @@ We specified an interpreter and its paths are adjusted too: ...@@ -911,6 +996,570 @@ We specified an interpreter and its paths are adjusted too:
del _interactive del _interactive
__import__("code").interact(banner="", local=globals()) __import__("code").interact(banner="", local=globals())
The ``sitepackage_safe_scripts`` function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The newer function for creating scripts is ``sitepackage_safe_scripts``.
It has the same basic functionality as the ``scripts`` function: it can
create scripts to run arbitrary entry points, and to run a Python
interpreter. The following are the differences from a user's
perspective.
- It can be used safely with a Python that has packages installed itself,
such as a system-installed Python.
- In contrast to the interpreter generated by the ``scripts`` method, which
supports only a small subset of the usual Python executable's options,
the interpreter generated by ``sitepackage_safe_scripts`` supports all
of them. This makes it possible to use as full Python replacement for
scripts that need the distributions specified in your buildout.
- Both the interpreter and the entry point scripts allow you to include the
site packages, and/or the sitecustomize, of the Python executable, if
desired.
It works by creating site.py and sitecustomize.py files that set up the
desired paths and initialization. These must be placed within an otherwise
empty directory. Typically this is in a recipe's parts directory.
Here's the simplest example, building an interpreter script.
>>> interpreter_dir = tmpdir('interpreter')
>>> interpreter_parts_dir = os.path.join(
... interpreter_dir, 'parts', 'interpreter')
>>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
>>> mkdir(interpreter_bin_dir)
>>> mkdir(interpreter_dir, 'eggs')
>>> mkdir(interpreter_dir, 'parts')
>>> mkdir(interpreter_parts_dir)
>>> ws = zc.buildout.easy_install.install(
... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
... index=link_server+'index/')
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py')
Depending on whether the machine being used is running Windows or not, this
produces either three or four files. In both cases, we have site.py and
sitecustomize.py generated in the parts/interpreter directory. For Windows,
we have py.exe and py-script.py; for other operating systems, we have py.
>>> sitecustomize_path = os.path.join(
... interpreter_parts_dir, 'sitecustomize.py')
>>> site_path = os.path.join(interpreter_parts_dir, 'site.py')
>>> interpreter_path = os.path.join(interpreter_bin_dir, 'py')
>>> if sys.platform == 'win32':
... py_path = os.path.join(interpreter_bin_dir, 'py-script.py')
... expected = [sitecustomize_path,
... site_path,
... os.path.join(interpreter_bin_dir, 'py.exe'),
... py_path]
... else:
... py_path = interpreter_path
... expected = [sitecustomize_path, site_path, py_path]
...
>>> assert generated == expected, repr((generated, expected))
We didn't ask for any initialization, and we didn't ask to use the underlying
sitecustomization, so sitecustomize.py is empty.
>>> cat(sitecustomize_path)
The interpreter script is simple. It puts the directory with the
site.py and sitecustomize.py on the PYTHONPATH and (re)starts Python.
>>> cat(py_path)
#!/usr/bin/python -S
import os
import sys
<BLANKLINE>
argv = [sys.executable] + sys.argv[1:]
environ = os.environ.copy()
path = '/interpreter/parts/interpreter'
if environ.get('PYTHONPATH'):
path = os.pathsep.join([path, environ['PYTHONPATH']])
environ['PYTHONPATH'] = path
os.execve(sys.executable, argv, environ)
The site.py file is a modified version of the underlying Python's site.py.
The most important modification is that it has a different version of the
addsitepackages function. It sets up the Python path, similarly to the
behavior of the function it replaces. The following shows the part that
buildout inserts, in the simplest case.
>>> sys.stdout.write('#\n'); cat(site_path)
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
#...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE>
See original_addsitepackages, below, for the original version."""
buildout_paths = [
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
]
for path in buildout_paths:
sitedir, sitedircase = makepath(path)
if not sitedircase in known_paths and os.path.exists(sitedir):
sys.path.append(sitedir)
known_paths.add(sitedircase)
return known_paths
<BLANKLINE>
def original_addsitepackages(known_paths):...
Here are some examples of the interpreter in use.
>>> print call_py(interpreter_path, "print 16+26")
42
<BLANKLINE>
>>> res = call_py(interpreter_path, "import sys; print sys.path")
>>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
['',
'/interpreter/parts/interpreter',
...,
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
<BLANKLINE>
>>> clean_paths = eval(res.strip()) # This is used later for comparison.
If you provide initialization, it goes in sitecustomize.py.
>>> def reset_interpreter():
... # This is necessary because, in our tests, the timestamps of the
... # .pyc files are not outdated when we want them to be.
... rmdir(interpreter_bin_dir)
... mkdir(interpreter_bin_dir)
... rmdir(interpreter_parts_dir)
... mkdir(interpreter_parts_dir)
...
>>> reset_interpreter()
>>> initialization_string = """\
... import os
... os.environ['FOO'] = 'bar baz bing shazam'"""
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py', initialization=initialization_string)
>>> cat(sitecustomize_path)
import os
os.environ['FOO'] = 'bar baz bing shazam'
>>> print call_py(interpreter_path, "import os; print os.environ['FOO']")
bar baz bing shazam
<BLANKLINE>
If you use relative paths, this affects the interpreter and site.py. (This is
again the UNIX version; the Windows version uses subprocess instead of
os.execve.)
>>> reset_interpreter()
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py', relative_paths=interpreter_dir)
>>> cat(py_path)
#!/usr/bin/python -S
import os
import sys
<BLANKLINE>
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
<BLANKLINE>
argv = [sys.executable] + sys.argv[1:]
environ = os.environ.copy()
path = join(base, 'parts/interpreter')
if environ.get('PYTHONPATH'):
path = os.pathsep.join([path, environ['PYTHONPATH']])
environ['PYTHONPATH'] = path
os.execve(sys.executable, argv, environ)
For site.py, we again show only the pertinent parts. Notice that the egg
paths join a base to a path, as with the use of this argument in the
``scripts`` function.
>>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
#...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE>
See original_addsitepackages, below, for the original version."""
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
base = os.path.dirname(base)
buildout_paths = [
join(base, 'eggs/demo-0.3-pyN.N.egg'),
join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
]...
The paths resolve in practice as you would expect.
>>> print call_py(interpreter_path,
... "import sys, pprint; pprint.pprint(sys.path)")
... # doctest: +ELLIPSIS
['',
'/interpreter/parts/interpreter',
...,
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
<BLANKLINE>
The ``extra_paths`` argument affects the path in site.py. Notice that
/interpreter/other is added after the eggs.
>>> reset_interpreter()
>>> mkdir(interpreter_dir, 'other')
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py', extra_paths=[join(interpreter_dir, 'other')])
>>> sys.stdout.write('#\n'); cat(site_path) # doctest: +ELLIPSIS
#...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE>
See original_addsitepackages, below, for the original version."""
buildout_paths = [
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
'/interpreter/other'
]...
>>> print call_py(interpreter_path,
... "import sys, pprint; pprint.pprint(sys.path)")
... # doctest: +ELLIPSIS
['',
'/interpreter/parts/interpreter',
...,
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
'/interpreter/other']
<BLANKLINE>
The ``sitepackage_safe_scripts`` function: using site-packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``sitepackage_safe_scripts`` function supports including site
packages. This has some advantages and some serious dangers.
A typical reason to include site-packages is that it is easier to
install one or more dependencies in your Python than it is with
buildout. Some packages, such as lxml or Python PostgreSQL integration,
have dependencies that can be much easier to build and/or install using
other mechanisms, such as your operating system's package manager. By
installing some core packages into your Python's site-packages, this can
significantly simplify some application installations.
However, doing this has a significant danger. One of the primary goals
of buildout is to provide repeatability. Some packages (one of the
better known Python openid packages, for instance) change their behavior
depending on what packages are available. If Python curl bindings are
available, these may be preferred by the library. If a certain XML
package is installed, it may be preferred by the library. These hidden
choices may cause small or large behavior differences. The fact that
they can be rarely encountered can actually make it worse: you forget
that this might be a problem, and debugging the differences can be
difficult. If you allow site-packages to be included in your buildout,
and the Python you use is not managed precisely by your application (for
instance, it is a system Python), you open yourself up to these
possibilities. Don't be unaware of the dangers.
That explained, let's see how it works. If you don't use namespace packages,
this is very straightforward.
>>> reset_interpreter()
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py', include_site_packages=True)
>>> sys.stdout.write('#\n'); cat(site_path)
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
#...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE>
See original_addsitepackages, below, for the original version."""
buildout_paths = [
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
]
for path in buildout_paths:
sitedir, sitedircase = makepath(path)
if not sitedircase in known_paths and os.path.exists(sitedir):
sys.path.append(sitedir)
known_paths.add(sitedircase)
sys.__egginsert = len(buildout_paths) # Support distribute.
original_paths = [
...
]
for path in original_paths:
addsitedir(path, known_paths)
return known_paths
<BLANKLINE>
def original_addsitepackages(known_paths):...
It simply adds the original paths using addsitedir after the code to add the
buildout paths.
Here's an example of the new script in use. Other documents and tests in
this package give the feature a more thorough workout, but this should
give you an idea of the feature.
>>> res = call_py(interpreter_path, "import sys; print sys.path")
>>> print res # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
['',
'/interpreter/parts/interpreter',
...,
'/interpreter/eggs/demo-0.3-py2.4.egg',
'/interpreter/eggs/demoneeded-1.1-py2.4.egg',
...]
<BLANKLINE>
The clean_paths gathered earlier is a subset of this full list of paths.
>>> full_paths = eval(res.strip())
>>> len(clean_paths) < len(full_paths)
True
>>> set(os.path.normpath(p) for p in clean_paths).issubset(
... os.path.normpath(p) for p in full_paths)
True
Unfortunately, because of how setuptools namespace packages are implemented
differently for operating system packages (debs or rpms) as opposed to
standard setuptools installation, there's a slightly trickier dance if you
use them. To show this we'll needs some extra eggs that use namespaces.
We'll use the ``tellmy.fortune`` package, which we'll need to make an initial
call to another text fixture to create.
>>> from zc.buildout.tests import create_sample_namespace_eggs
>>> namespace_eggs = tmpdir('namespace_eggs')
>>> create_sample_namespace_eggs(namespace_eggs)
>>> reset_interpreter()
>>> ws = zc.buildout.easy_install.install(
... ['demo', 'tellmy.fortune'], join(interpreter_dir, 'eggs'),
... links=[link_server, namespace_eggs], index=link_server+'index/')
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py', include_site_packages=True)
>>> sys.stdout.write('#\n'); cat(site_path)
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
#...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE>
See original_addsitepackages, below, for the original version."""
setuptools_path = '...setuptools...'
sys.path.append(setuptools_path)
known_paths.add(os.path.normcase(setuptools_path))
import pkg_resources
buildout_paths = [
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
'...setuptools...',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg'
]
for path in buildout_paths:
sitedir, sitedircase = makepath(path)
if not sitedircase in known_paths and os.path.exists(sitedir):
sys.path.append(sitedir)
known_paths.add(sitedircase)
pkg_resources.working_set.add_entry(sitedir)
sys.__egginsert = len(buildout_paths) # Support distribute.
original_paths = [
...
]
for path in original_paths:
addsitedir(path, known_paths)
return known_paths
<BLANKLINE>
def original_addsitepackages(known_paths):...
>>> print call_py(interpreter_path, "import sys; print sys.path")
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
['',
'/interpreter/parts/interpreter',
...,
'...setuptools...',
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
...]
As you can see, the script now first imports pkg_resources. Then we
need to process egg files specially to look for namespace packages there
*before* we process process lines in .pth files that use the "import"
feature--lines that might be part of the setuptools namespace package
implementation for system packages, as mentioned above, and that must
come after processing egg namespaces.
The most complex that this function gets is if you use namespace packages,
include site-packages, and use relative paths. For completeness, we'll look
at that result.
>>> reset_interpreter()
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... interpreter='py', include_site_packages=True,
... relative_paths=interpreter_dir)
>>> sys.stdout.write('#\n'); cat(site_path)
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
#...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE>
See original_addsitepackages, below, for the original version."""
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
base = os.path.dirname(base)
setuptools_path = '...setuptools...'
sys.path.append(setuptools_path)
known_paths.add(os.path.normcase(setuptools_path))
import pkg_resources
buildout_paths = [
join(base, 'eggs/demo-0.3-pyN.N.egg'),
join(base, 'eggs/tellmy.fortune-1.0-pyN.N.egg'),
'...setuptools...',
join(base, 'eggs/demoneeded-1.1-pyN.N.egg')
]
for path in buildout_paths:
sitedir, sitedircase = makepath(path)
if not sitedircase in known_paths and os.path.exists(sitedir):
sys.path.append(sitedir)
known_paths.add(sitedircase)
pkg_resources.working_set.add_entry(sitedir)
sys.__egginsert = len(buildout_paths) # Support distribute.
original_paths = [
...
]
for path in original_paths:
addsitedir(path, known_paths)
return known_paths
<BLANKLINE>
def original_addsitepackages(known_paths):...
>>> print call_py(interpreter_path, "import sys; print sys.path")
... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
['',
'/interpreter/parts/interpreter',
...,
'...setuptools...',
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/tellmy.fortune-1.0-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg',
...]
The ``exec_sitecustomize`` argument does the same thing for the
sitecustomize module--it allows you to include the code from the
sitecustomize module in the underlying Python if you set the argument to
True. The z3c.recipe.scripts package sets up the full environment necessary
to demonstrate this piece.
The ``sitepackage_safe_scripts`` function: writing scripts for entry points
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All of the examples so far for this function have been creating
interpreters. The function can also write scripts for entry
points. They are almost identical to the scripts that we saw for the
``scripts`` function except that they ``import site`` after setting the
sys.path to include our custom site.py and sitecustomize.py files. These
files then initialize the Python environment as we have already seen. Let's
see a simple example.
>>> reset_interpreter()
>>> ws = zc.buildout.easy_install.install(
... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
... index=link_server+'index/')
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... reqs=['demo'])
As before, in Windows, 2 files are generated for each script. A script
file, ending in '-script.py', and an exe file that allows the script
to be invoked directly without having to specify the Python
interpreter and without having to provide a '.py' suffix. This is in addition
to the site.py and sitecustomize.py files that are generated as with our
interpreter examples above.
>>> if sys.platform == 'win32':
... demo_path = os.path.join(interpreter_bin_dir, 'demo-script.py')
... expected = [sitecustomize_path,
... site_path,
... os.path.join(interpreter_bin_dir, 'demo.exe'),
... demo_path]
... else:
... demo_path = os.path.join(interpreter_bin_dir, 'demo')
... expected = [sitecustomize_path, site_path, demo_path]
...
>>> assert generated == expected, repr((generated, expected))
The demo script runs the entry point defined in the demo egg:
>>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4 -S
<BLANKLINE>
import sys
sys.path[0:0] = [
'/interpreter/parts/interpreter',
]
<BLANKLINE>
<BLANKLINE>
import os
path = sys.path[0]
if os.environ.get('PYTHONPATH'):
path = os.pathsep.join([path, os.environ['PYTHONPATH']])
os.environ['PYTHONPATH'] = path
import site # imports custom buildout-generated site.py
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main()
>>> demo_call = join(interpreter_bin_dir, 'demo')
>>> if sys.platform == 'win32':
... demo_call = '"%s"' % demo_call
>>> print system(demo_call)
3 1
<BLANKLINE>
There are a few differences from the ``scripts`` function. First, the
``reqs`` argument (an iterable of string requirements or entry point
tuples) is a keyword argument here. We see that in the example above.
Second, the ``arguments`` argument is now named ``script_arguments`` to
try and clarify that it does not affect interpreters. While the
``initialization`` argument continues to affect both the interpreters
and the entry point scripts, if you have initialization that is only
pertinent to the entry point scripts, you can use the
``script_initialization`` argument.
Let's see ``script_arguments`` and ``script_initialization`` in action.
>>> reset_interpreter()
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... reqs=['demo'], script_arguments='1, 2',
... script_initialization='import os\nos.chdir("foo")')
>>> cat(demo_path) # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4 -S
import sys
sys.path[0:0] = [
'/interpreter/parts/interpreter',
]
<BLANKLINE>
import os
path = sys.path[0]
if os.environ.get('PYTHONPATH'):
path = os.pathsep.join([path, os.environ['PYTHONPATH']])
os.environ['PYTHONPATH'] = path
import site # imports custom buildout-generated site.py
import os
os.chdir("foo")
<BLANKLINE>
import eggrecipedemo
<BLANKLINE>
if __name__ == '__main__':
eggrecipedemo.main(1, 2)
Handling custom build options for extensions provided in source distributions Handling custom build options for extensions provided in source distributions
----------------------------------------------------------------------------- -----------------------------------------------------------------------------
......
...@@ -28,6 +28,7 @@ import socket ...@@ -28,6 +28,7 @@ import socket
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
import textwrap
import threading import threading
import time import time
import urllib2 import urllib2
...@@ -109,6 +110,16 @@ def system(command, input=''): ...@@ -109,6 +110,16 @@ def system(command, input=''):
e.close() e.close()
return result return result
def call_py(interpreter, cmd, flags=None):
if sys.platform == 'win32':
args = ['"%s"' % arg for arg in (interpreter, flags, cmd) if arg]
args.insert(-1, '"-c"')
return system('"%s"' % ' '.join(args))
else:
cmd = repr(cmd)
return system(
' '.join(arg for arg in (interpreter, flags, '-c', cmd) if arg))
def get(url): def get(url):
return urllib2.urlopen(url).read() return urllib2.urlopen(url).read()
...@@ -120,7 +131,11 @@ def _runsetup(setup, executable, *args): ...@@ -120,7 +131,11 @@ def _runsetup(setup, executable, *args):
args = [zc.buildout.easy_install._safe_arg(arg) args = [zc.buildout.easy_install._safe_arg(arg)
for arg in args] for arg in args]
args.insert(0, '-q') args.insert(0, '-q')
args.append(dict(os.environ, PYTHONPATH=setuptools_location)) env = dict(os.environ)
if executable == sys.executable:
env['PYTHONPATH'] = setuptools_location
# else pass an executable that has setuptools! See testselectingpython.py.
args.append(env)
here = os.getcwd() here = os.getcwd()
try: try:
...@@ -139,6 +154,11 @@ def sdist(setup, dest): ...@@ -139,6 +154,11 @@ def sdist(setup, dest):
def bdist_egg(setup, executable, dest): def bdist_egg(setup, executable, dest):
_runsetup(setup, executable, 'bdist_egg', '-d', dest) _runsetup(setup, executable, 'bdist_egg', '-d', dest)
def sys_install(setup, dest):
_runsetup(setup, sys.executable, 'install', '--install-purelib', dest,
'--record', os.path.join(dest, '__added_files__'),
'--single-version-externally-managed')
def find_python(version): def find_python(version):
e = os.environ.get('PYTHON%s' % version) e = os.environ.get('PYTHON%s' % version)
if e is not None: if e is not None:
...@@ -206,14 +226,64 @@ def wait_until(label, func, *args, **kw): ...@@ -206,14 +226,64 @@ def wait_until(label, func, *args, **kw):
time.sleep(0.01) time.sleep(0.01)
raise ValueError('Timed out waiting for: '+label) raise ValueError('Timed out waiting for: '+label)
def get_installer_values():
"""Get the current values for the easy_install module.
This is necessary because instantiating a Buildout will force the
Buildout's values on the installer.
Returns a dict of names-values suitable for set_installer_values."""
names = ('default_versions', 'download_cache', 'install_from_cache',
'prefer_final', 'include_site_packages',
'allowed_eggs_from_site_packages', 'use_dependency_links',
'allow_picked_versions', 'always_unzip'
)
values = {}
for name in names:
values[name] = getattr(zc.buildout.easy_install, name)()
return values
def set_installer_values(values):
"""Set the given values on the installer."""
for name, value in values.items():
getattr(zc.buildout.easy_install, name)(value)
def make_buildout(executable=None):
"""Make a buildout that uses this version of zc.buildout."""
# Create a basic buildout.cfg to avoid a warning from buildout.
open('buildout.cfg', 'w').write(
"[buildout]\nparts =\n"
)
# Get state of installer defaults so we can reinstate them (instantiating
# a Buildout will force the Buildout's defaults on the installer).
installer_values = get_installer_values()
# Use the buildout bootstrap command to create a buildout
config = [
('buildout', 'log-level', 'WARNING'),
# trick bootstrap into putting the buildout develop egg
# in the eggs dir.
('buildout', 'develop-eggs-directory', 'eggs'),
]
if executable is not None:
config.append(('buildout', 'executable', executable))
zc.buildout.buildout.Buildout(
'buildout.cfg', config,
user_defaults=False,
).bootstrap([])
# Create the develop-eggs dir, which didn't get created the usual
# way due to the trick above:
os.mkdir('develop-eggs')
# Reinstate the default values of the installer.
set_installer_values(installer_values)
def buildoutSetUp(test): def buildoutSetUp(test):
test.globs['__tear_downs'] = __tear_downs = [] test.globs['__tear_downs'] = __tear_downs = []
test.globs['register_teardown'] = register_teardown = __tear_downs.append test.globs['register_teardown'] = register_teardown = __tear_downs.append
prefer_final = zc.buildout.easy_install.prefer_final() installer_values = get_installer_values()
register_teardown( register_teardown(
lambda: zc.buildout.easy_install.prefer_final(prefer_final) lambda: set_installer_values(installer_values)
) )
here = os.getcwd() here = os.getcwd()
...@@ -259,27 +329,7 @@ def buildoutSetUp(test): ...@@ -259,27 +329,7 @@ def buildoutSetUp(test):
sample = tmpdir('sample-buildout') sample = tmpdir('sample-buildout')
os.chdir(sample) os.chdir(sample)
make_buildout()
# Create a basic buildout.cfg to avoid a warning from buildout:
open('buildout.cfg', 'w').write(
"[buildout]\nparts =\n"
)
# Use the buildout bootstrap command to create a buildout
zc.buildout.buildout.Buildout(
'buildout.cfg',
[('buildout', 'log-level', 'WARNING'),
# trick bootstrap into putting the buildout develop egg
# in the eggs dir.
('buildout', 'develop-eggs-directory', 'eggs'),
]
).bootstrap([])
# Create the develop-eggs dir, which didn't get created the usual
# way due to the trick above:
os.mkdir('develop-eggs')
def start_server(path): def start_server(path):
port, thread = _start_server(path, name=path) port, thread = _start_server(path, name=path)
...@@ -287,6 +337,50 @@ def buildoutSetUp(test): ...@@ -287,6 +337,50 @@ def buildoutSetUp(test):
register_teardown(lambda: stop_server(url, thread)) register_teardown(lambda: stop_server(url, thread))
return url return url
def make_py(initialization=''):
"""Returns paths to new executable and to its site-packages.
"""
buildout = tmpdir('executable_buildout')
site_packages_dir = os.path.join(buildout, 'site-packages')
mkdir(site_packages_dir)
old_wd = os.getcwd()
os.chdir(buildout)
make_buildout()
# Normally we don't process .pth files in extra-paths. We want to
# in this case so that we can test with setuptools system installs
# (--single-version-externally-managed), which use .pth files.
initialization = (
('import sys\n'
'import site\n'
'known_paths = set(sys.path)\n'
'site_packages_dir = %r\n'
'site.addsitedir(site_packages_dir, known_paths)\n'
) % (site_packages_dir,)) + initialization
initialization = '\n'.join(
' ' + line for line in initialization.split('\n'))
install_develop(
'zc.recipe.egg', os.path.join(buildout, 'develop-eggs'))
install_develop(
'z3c.recipe.scripts', os.path.join(buildout, 'develop-eggs'))
write('buildout.cfg', textwrap.dedent('''\
[buildout]
parts = py
[py]
recipe = z3c.recipe.scripts
interpreter = py
initialization =
%(initialization)s
extra-paths = %(site-packages)s
eggs = setuptools
''') % {
'initialization': initialization,
'site-packages': site_packages_dir})
system(os.path.join(buildout, 'bin', 'buildout'))
os.chdir(old_wd)
return (
os.path.join(buildout, 'bin', 'py'), site_packages_dir)
test.globs.update(dict( test.globs.update(dict(
sample_buildout = sample, sample_buildout = sample,
ls = ls, ls = ls,
...@@ -297,6 +391,7 @@ def buildoutSetUp(test): ...@@ -297,6 +391,7 @@ def buildoutSetUp(test):
tmpdir = tmpdir, tmpdir = tmpdir,
write = write, write = write,
system = system, system = system,
call_py = call_py,
get = get, get = get,
cd = (lambda *path: os.chdir(os.path.join(*path))), cd = (lambda *path: os.chdir(os.path.join(*path))),
join = os.path.join, join = os.path.join,
...@@ -305,10 +400,9 @@ def buildoutSetUp(test): ...@@ -305,10 +400,9 @@ def buildoutSetUp(test):
start_server = start_server, start_server = start_server,
buildout = os.path.join(sample, 'bin', 'buildout'), buildout = os.path.join(sample, 'bin', 'buildout'),
wait_until = wait_until, wait_until = wait_until,
make_py = make_py
)) ))
zc.buildout.easy_install.prefer_final(prefer_final)
def buildoutTearDown(test): def buildoutTearDown(test):
for f in test.globs['__tear_downs']: for f in test.globs['__tear_downs']:
f() f()
......
...@@ -53,6 +53,7 @@ We should be able to deal with setup scripts that aren't setuptools based. ...@@ -53,6 +53,7 @@ We should be able to deal with setup scripts that aren't setuptools based.
>>> ls('develop-eggs') >>> ls('develop-eggs')
- foo.egg-link - foo.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
""" """
...@@ -84,6 +85,7 @@ We should be able to deal with setup scripts that aren't setuptools based. ...@@ -84,6 +85,7 @@ We should be able to deal with setup scripts that aren't setuptools based.
>>> ls('develop-eggs') >>> ls('develop-eggs')
- foo.egg-link - foo.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
>>> print system(join('bin', 'buildout')+' -vvv'), # doctest: +ELLIPSIS >>> print system(join('bin', 'buildout')+' -vvv'), # doctest: +ELLIPSIS
...@@ -383,6 +385,64 @@ buildout will tell us who's asking for something that we can't find. ...@@ -383,6 +385,64 @@ buildout will tell us who's asking for something that we can't find.
Error: Couldn't find a distribution for 'demoneeded'. Error: Couldn't find a distribution for 'demoneeded'.
""" """
def show_eggs_from_site_packages():
"""
Sometimes you want to know what eggs are coming from site-packages. This
might be for a diagnostic, or so that you can get a starting value for the
allowed-eggs-from-site-packages option. The -v flag will also include this
information.
Our "py_path" has the "demoneeded," "demo"
packages available. We'll ask for "bigdemo," which will get both of them.
Here's our set up.
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... prefer-final = true
... find-links = %(link_server)s
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... python = primed_python
... eggs = bigdemo
... ''' % globals())
Now here is the output. The lines that begin with "Egg from site-packages:"
indicate the eggs from site-packages that have been selected. You'll see
we have two: demo 0.3 and demoneeded 1.1.
>>> print system(buildout+" -v")
Installing 'zc.buildout', 'setuptools'.
We have a develop egg: zc.buildout V
We have the best distribution that satisfies 'setuptools'.
Picked: setuptools = V
Installing 'zc.recipe.egg'.
We have a develop egg: zc.recipe.egg V
Installing eggs.
Installing 'bigdemo'.
We have no distributions for bigdemo that satisfies 'bigdemo'.
Getting distribution for 'bigdemo'.
Got bigdemo 0.1.
Picked: bigdemo = 0.1
Getting required 'demo'
required by bigdemo 0.1.
We have a develop egg: demo V
Egg from site-packages: demo 0.3
Getting required 'demoneeded'
required by demo 0.3.
We have a develop egg: demoneeded V
Egg from site-packages: demoneeded 1.1
<BLANKLINE>
"""
def test_comparing_saved_options_with_funny_characters(): def test_comparing_saved_options_with_funny_characters():
""" """
...@@ -668,6 +728,7 @@ Create a develop egg: ...@@ -668,6 +728,7 @@ Create a develop egg:
>>> ls('develop-eggs') >>> ls('develop-eggs')
- foox.egg-link - foox.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
Create another: Create another:
...@@ -692,6 +753,7 @@ Create another: ...@@ -692,6 +753,7 @@ Create another:
>>> ls('develop-eggs') >>> ls('develop-eggs')
- foox.egg-link - foox.egg-link
- fooy.egg-link - fooy.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
Remove one: Remove one:
...@@ -709,6 +771,7 @@ It is gone ...@@ -709,6 +771,7 @@ It is gone
>>> ls('develop-eggs') >>> ls('develop-eggs')
- fooy.egg-link - fooy.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
Remove the other: Remove the other:
...@@ -723,6 +786,7 @@ Remove the other: ...@@ -723,6 +786,7 @@ Remove the other:
All gone All gone
>>> ls('develop-eggs') >>> ls('develop-eggs')
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
''' '''
...@@ -797,9 +861,11 @@ On the other hand, if we have a regular egg, rather than a develop egg: ...@@ -797,9 +861,11 @@ On the other hand, if we have a regular egg, rather than a develop egg:
... + join(sample_buildout, 'eggs')) ... + join(sample_buildout, 'eggs'))
>>> ls('develop-eggs') >>> ls('develop-eggs')
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
>>> ls('eggs') # doctest: +ELLIPSIS >>> print 'START ->'; ls('eggs') # doctest: +ELLIPSIS
START...
- foox-0.0.0-py2.4.egg - foox-0.0.0-py2.4.egg
... ...
...@@ -1769,6 +1835,658 @@ def bug_105081_Specific_egg_versions_are_ignored_when_newer_eggs_are_around(): ...@@ -1769,6 +1835,658 @@ def bug_105081_Specific_egg_versions_are_ignored_when_newer_eggs_are_around():
1 2 1 2
""" """
def versions_section_ignored_for_dependency_in_favor_of_site_packages():
r"""
This is a test for a bugfix.
The error showed itself when at least two dependencies were in a shared
location like site-packages, and the first one met the "versions" setting. The
first dependency would be added, but subsequent dependencies from the same
location (e.g., site-packages) would use the version of the package found in
the shared location, ignoring the version setting.
We begin with a Python that has demoneeded version 1.1 installed and a
demo version 0.3, all in a site-packages-like shared directory. We need
to create this. ``eggrecipedemo.main()`` shows the number after the dot
(that is, ``X`` in ``1.X``), for the demo package and the demoneeded
package, so this demonstrates that our Python does in fact have demo
version 0.3 and demoneeded version 1.1.
>>> py_path = make_py_with_system_install(make_py, sample_eggs)
>>> print call_py(
... py_path,
... "import tellmy.version; print tellmy.version.__version__"),
1.1
Now here's a setup that would expose the bug, using the
zc.buildout.easy_install API.
>>> example_dest = tmpdir('example_dest')
>>> workingset = zc.buildout.easy_install.install(
... ['tellmy.version'], example_dest, links=[sample_eggs],
... executable=py_path,
... index=None,
... versions={'tellmy.version': '1.0'})
>>> for dist in workingset:
... res = str(dist)
... if res.startswith('tellmy.version'):
... print res
... break
tellmy.version 1.0
Before the bugfix, the desired tellmy.version distribution would have
been blocked the one in site-packages.
"""
def handle_namespace_package_in_both_site_packages_and_buildout_eggs():
r"""
If you have the same namespace package in both site-packages and in
buildout, we need to be very careful that faux-Python-executables and
scripts generated by easy_install.sitepackage_safe_scripts correctly
combine the two. We show this with the local recipe that uses the
function, z3c.recipe.scripts.
To demonstrate this, we will create three packages: tellmy.version 1.0,
tellmy.version 1.1, and tellmy.fortune 1.0. tellmy.version 1.1 is installed.
>>> py_path = make_py_with_system_install(make_py, sample_eggs)
>>> print call_py(
... py_path,
... "import tellmy.version; print tellmy.version.__version__")
1.1
<BLANKLINE>
Now we will create a buildout that creates a script and a faux-Python script.
We want to see that both can successfully import the specified versions of
tellmy.version and tellmy.fortune.
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links = %(link_server)s
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... python = primed_python
... interpreter = py
... include-site-packages = true
... eggs = tellmy.version == 1.0
... tellmy.fortune == 1.0
... demo
... script-initialization =
... import tellmy.version
... print tellmy.version.__version__
... import tellmy.fortune
... print tellmy.fortune.__version__
... ''' % globals())
>>> print system(buildout)
Installing eggs.
Getting distribution for 'tellmy.version==1.0'.
Got tellmy.version 1.0.
Getting distribution for 'tellmy.fortune==1.0'.
Got tellmy.fortune 1.0.
Getting distribution for 'demo'.
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.
<BLANKLINE>
Finally, we are ready to see if it worked. Prior to the bug fix that
this tests, the results of both calls below was the following::
1.1
Traceback (most recent call last):
...
ImportError: No module named fortune
<BLANKLINE>
In other words, we got the site-packages version of tellmy.version, and
we could not import tellmy.fortune at all. The following are the correct
results for the interpreter and for the script.
>>> print call_py(
... join('bin', 'py'),
... "import tellmy.version; " +
... "print tellmy.version.__version__; " +
... "import tellmy.fortune; " +
... "print tellmy.fortune.__version__") # doctest: +ELLIPSIS
1.0
1.0...
>>> print system(join('bin', 'demo'))
1.0
1.0
4 2
<BLANKLINE>
"""
def handle_sys_path_version_hack():
r"""
This is a test for a bugfix.
If you use a Python that has a different version of one of your
dependencies, and the new package tries to do sys.path tricks in the
setup.py to get a __version__, and it uses namespace packages, the older
package will be loaded first, making the setup version the wrong number.
While very arguably packages simply shouldn't do this, some do, and we
don't want buildout to fall over when they do.
To demonstrate this, we will need to create a distribution that has one of
these unpleasant tricks, and a Python that has an older version installed.
>>> py_path, site_packages_path = make_py()
>>> for version in ('1.0', '1.1'):
... tmp = tempfile.mkdtemp()
... try:
... write(tmp, 'README.txt', '')
... mkdir(tmp, 'src')
... mkdir(tmp, 'src', 'tellmy')
... write(tmp, 'src', 'tellmy', '__init__.py',
... "__import__("
... "'pkg_resources').declare_namespace(__name__)\n")
... mkdir(tmp, 'src', 'tellmy', 'version')
... write(tmp, 'src', 'tellmy', 'version',
... '__init__.py', '__version__=%r\n' % version)
... write(
... tmp, 'setup.py',
... "from setuptools import setup\n"
... "import sys\n"
... "sys.path.insert(0, 'src')\n"
... "from tellmy.version import __version__\n"
... "setup(\n"
... " name='tellmy.version',\n"
... " package_dir = {'': 'src'},\n"
... " packages = ['tellmy', 'tellmy.version'],\n"
... " install_requires = ['setuptools'],\n"
... " namespace_packages=['tellmy'],\n"
... " zip_safe=True, version=__version__,\n"
... " author='bob', url='bob', author_email='bob')\n"
... )
... zc.buildout.testing.sdist(tmp, sample_eggs)
... if version == '1.0':
... # We install the 1.0 version in site packages the way a
... # system packaging system (debs, rpms) would do it.
... zc.buildout.testing.sys_install(tmp, site_packages_path)
... finally:
... shutil.rmtree(tmp)
>>> print call_py(
... py_path,
... "import tellmy.version; print tellmy.version.__version__")
1.0
<BLANKLINE>
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links = %(sample_eggs)s
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... python = primed_python
... eggs = tellmy.version == 1.1
... ''' % globals())
Before the bugfix, running this buildout would generate this error:
Installing eggs.
Getting distribution for 'tellmy.version==1.1'.
Installing tellmy.version 1.1
Caused installation of a distribution:
tellmy.version 1.0
with a different version.
Got None.
While:
Installing eggs.
Error: There is a version conflict.
We already have: tellmy.version 1.0
<BLANKLINE>
You can see the copiously commented fix for this in easy_install.py (see
zc.buildout.easy_install.Installer._call_easy_install and particularly
the comment leading up to zc.buildout.easy_install._easy_install_cmd).
Now the install works correctly, as seen here.
>>> print system(buildout)
Installing eggs.
Getting distribution for 'tellmy.version==1.1'.
Got tellmy.version 1.1.
<BLANKLINE>
"""
def isolated_include_site_packages():
"""
This is an isolated test of the include_site_packages functionality, passing
the argument directly to install, overriding a default.
Our "py_path" has the "demoneeded" and "demo" packages available. We'll
simply be asking for "demoneeded" here.
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> zc.buildout.easy_install.include_site_packages(False)
True
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None, include_site_packages=True)
>>> [dist.project_name for dist in workingset]
['demoneeded']
That worked fine. Let's try again with site packages not allowed (and
reversing the default).
>>> zc.buildout.easy_install.include_site_packages(True)
False
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None, include_site_packages=False)
Traceback (most recent call last):
...
MissingDistribution: Couldn't find a distribution for 'demoneeded'.
That's a failure, as expected.
Now we explore an important edge case.
Some system Pythons include setuptools (and other Python packages) in their
site-packages (or equivalent) using a .egg-info directory. The pkg_resources
module (from setuptools) considers a package installed using .egg-info to be a
develop egg.
zc.buildout.buildout.Buildout.bootstrap will make setuptools and zc.buildout
available to the buildout via the eggs directory, for normal eggs; or the
develop-eggs directory, for develop-eggs.
If setuptools or zc.buildout is found in site-packages and considered by
pkg_resources to be a develop egg, then the bootstrap code will use a .egg-link
in the local develop-eggs, pointing to site-packages, in its entirety. Because
develop-eggs must always be available for searching for distributions, this
indirectly brings site-packages back into the search path for distributions.
Because of this, we have to take special care that we still exclude
site-packages even in this case. See the comments about site packages in the
Installer._satisfied and Installer._obtain methods for the implementation
(as of this writing).
In this demonstration, we insert a link to the "demoneeded" distribution
in our develop-eggs, which would bring the package back in, except for
the special care we have taken to exclude it.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> mkdir(example_dest, 'develop-eggs')
>>> write(example_dest, 'develop-eggs', 'demoneeded.egg-link',
... site_packages_path)
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[],
... path=[join(example_dest, 'develop-eggs')],
... executable=py_path,
... index=None, include_site_packages=False)
Traceback (most recent call last):
...
MissingDistribution: Couldn't find a distribution for 'demoneeded'.
The MissingDistribution error shows that buildout correctly excluded the
"site-packages" source even though it was indirectly included in the path
via a .egg-link file.
"""
def allowed_eggs_from_site_packages():
"""
Sometimes you need or want to control what eggs from site-packages are used.
The allowed-eggs-from-site-packages option allows you to specify a whitelist of
project names that may be included from site-packages. You can use globs to
specify the value. It defaults to a single value of '*', indicating that any
package may come from site-packages.
This option interacts with include-site-packages in the following ways.
If include-site-packages is true, then allowed-eggs-from-site-packages filters
what eggs from site-packages may be chosen. If allowed-eggs-from-site-packages
is an empty list, then no eggs from site-packages are chosen, but site-packages
will still be included at the end of path lists.
If include-site-packages is false, allowed-eggs-from-site-packages is
irrelevant.
This test shows the interaction with the zc.buildout.easy_install API. Another
test below (allow_site_package_eggs_option) shows using it with a buildout.cfg.
Our "py_path" has the "demoneeded" and "demo" packages available. We'll
simply be asking for "demoneeded" here.
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None,
... allowed_eggs_from_site_packages=['demoneeded', 'other'])
>>> [dist.project_name for dist in workingset]
['demoneeded']
That worked fine. It would work fine for a glob too.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None,
... allowed_eggs_from_site_packages=['?emon*', 'other'])
>>> [dist.project_name for dist in workingset]
['demoneeded']
But now let's try again with 'demoneeded' not allowed.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None,
... allowed_eggs_from_site_packages=['demo'])
Traceback (most recent call last):
...
MissingDistribution: Couldn't find a distribution for 'demoneeded'.
Here's the same, but with an empty list.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None,
... allowed_eggs_from_site_packages=[])
Traceback (most recent call last):
...
MissingDistribution: Couldn't find a distribution for 'demoneeded'.
Of course, this doesn't stop us from getting a package from elsewhere. Here,
we add a link server.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, executable=py_path,
... links=[link_server], index=link_server+'index/',
... allowed_eggs_from_site_packages=['other'])
>>> [dist.project_name for dist in workingset]
['demoneeded']
>>> [dist.location for dist in workingset]
['/site-packages-example-install/demoneeded-1.1-py2.6.egg']
Finally, here's an example of an interaction: we say that it is OK to
allow the "demoneeded" egg to come from site-packages, but we don't
include-site-packages.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir(example_dest)
>>> example_dest = tmpdir('site-packages-example-install')
>>> workingset = zc.buildout.easy_install.install(
... ['demoneeded'], example_dest, links=[], executable=py_path,
... index=None, include_site_packages=False,
... allowed_eggs_from_site_packages=['demoneeded'])
Traceback (most recent call last):
...
MissingDistribution: Couldn't find a distribution for 'demoneeded'.
"""
def subprocesses_have_same_environment_by_default():
"""
The scripts generated by sitepackage_safe_scripts set the PYTHONPATH so that,
if the environment is maintained (the default behavior), subprocesses get
the same Python packages.
First, we set up a script and an interpreter.
>>> interpreter_dir = tmpdir('interpreter')
>>> interpreter_parts_dir = os.path.join(
... interpreter_dir, 'parts', 'interpreter')
>>> interpreter_bin_dir = os.path.join(interpreter_dir, 'bin')
>>> mkdir(interpreter_bin_dir)
>>> mkdir(interpreter_dir, 'eggs')
>>> mkdir(interpreter_dir, 'parts')
>>> mkdir(interpreter_parts_dir)
>>> ws = zc.buildout.easy_install.install(
... ['demo'], join(interpreter_dir, 'eggs'), links=[link_server],
... index=link_server+'index/')
>>> test = (
... "import subprocess, sys; subprocess.call("
... "[sys.executable, '-c', "
... "'import eggrecipedemo; print eggrecipedemo.x'])")
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... reqs=['demo'], interpreter='py',
... script_initialization=test + '; sys.exit(0)')
This works for the script.
>>> print system(join(interpreter_bin_dir, 'demo'))
3
<BLANKLINE>
This also works for the generated interpreter.
>>> print call_py(join(interpreter_bin_dir, 'py'), test)
3
<BLANKLINE>
If you have a PYTHONPATH in your environment, it will be honored, after
the buildout-generated path.
>>> original_pythonpath = os.environ.get('PYTHONPATH')
>>> os.environ['PYTHONPATH'] = 'foo'
>>> test = (
... "import subprocess, sys; subprocess.call("
... "[sys.executable, '-c', "
... "'import sys, pprint; pprint.pprint(sys.path)'])")
>>> generated = zc.buildout.easy_install.sitepackage_safe_scripts(
... interpreter_bin_dir, ws, sys.executable, interpreter_parts_dir,
... reqs=['demo'], interpreter='py',
... script_initialization=test + '; sys.exit(0)')
This works for the script. As you can see, /sample_buildout/foo is included
right after the "parts" directory that contains site.py and sitecustomize.py.
You can also see, actually more easily than in the other example, that we
have the desired eggs available.
>>> print system(join(interpreter_bin_dir, 'demo')), # doctest: +ELLIPSIS
['',
'/interpreter/parts/interpreter',
'/sample-buildout/foo',
...
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
This also works for the generated interpreter, with identical results.
>>> print call_py(join(interpreter_bin_dir, 'py'), test),
... # doctest: +ELLIPSIS
['',
'/interpreter/parts/interpreter',
'/sample-buildout/foo',
...
'/interpreter/eggs/demo-0.3-pyN.N.egg',
'/interpreter/eggs/demoneeded-1.1-pyN.N.egg']
>>> # Cleanup
>>> if original_pythonpath:
... os.environ['PYTHONPATH'] = original_pythonpath
... else:
... del os.environ['PYTHONPATH']
...
"""
def bootstrap_makes_buildout_that_works_with_system_python():
r"""
In order to work smoothly with a system Python, bootstrapping creates
the buildout script with
zc.buildout.easy_install.sitepackage_safe_scripts. If it did not, a
variety of problems might happen. For instance, if another version of
buildout or setuptools is installed in the site-packages than is
desired, it may cause a problem.
A problem actually experienced in the field is when
a recipe wants a different version of a dependency that is installed in
site-packages. We will create a similar situation, and show that it is now
handled.
First let's write a dummy recipe.
>>> mkdir(sample_buildout, 'recipes')
>>> write(sample_buildout, 'recipes', 'dummy.py',
... '''
... import logging, os, zc.buildout
... class Dummy:
... def __init__(self, buildout, name, options):
... pass
... def install(self):
... return ()
... def update(self):
... pass
... ''')
>>> write(sample_buildout, 'recipes', 'setup.py',
... '''
... from setuptools import setup
...
... setup(
... name = "recipes",
... entry_points = {'zc.buildout': ['dummy = dummy:Dummy']},
... install_requires = 'demoneeded==1.2c1',
... )
... ''')
>>> write(sample_buildout, 'recipes', 'README.txt', " ")
Now we'll try to use it with a Python that has a different version of
demoneeded installed.
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> rmdir('develop-eggs')
>>> from zc.buildout.testing import make_buildout
>>> make_buildout(executable=py_path)
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... develop = recipes
... parts = dummy
... find-links = %(link_server)s
... executable = %(py_path)s
...
... [dummy]
... recipe = recipes:dummy
... ''' % globals())
Now we actually run the buildout. Before the change, we got the following
error:
Develop: '/sample-buildout/recipes'
While:
Installing.
Getting section dummy.
Initializing section dummy.
Installing recipe recipes.
Error: There is a version conflict.
We already have: demoneeded 1.1
but recipes 0.0.0 requires 'demoneeded==1.2c1'.
Now, it is handled smoothly.
>>> print system(buildout)
Develop: '/sample-buildout/recipes'
Getting distribution for 'demoneeded==1.2c1'.
Got demoneeded 1.2c1.
Installing dummy.
<BLANKLINE>
Here's the same story with a namespace package, which has some additional
complications behind the scenes. First, a recipe, in the "tellmy" namespace.
>>> mkdir(sample_buildout, 'ns')
>>> mkdir(sample_buildout, 'ns', 'tellmy')
>>> write(sample_buildout, 'ns', 'tellmy', '__init__.py',
... "__import__('pkg_resources').declare_namespace(__name__)\n")
>>> mkdir(sample_buildout, 'ns', 'tellmy', 'recipes')
>>> write(sample_buildout, 'ns', 'tellmy', 'recipes', '__init__.py', ' ')
>>> write(sample_buildout, 'ns', 'tellmy', 'recipes', 'dummy.py',
... '''
... import logging, os, zc.buildout
... class Dummy:
... def __init__(self, buildout, name, options):
... pass
... def install(self):
... return ()
... def update(self):
... pass
... ''')
>>> write(sample_buildout, 'ns', 'setup.py',
... '''
... from setuptools import setup
... setup(
... name="tellmy.recipes",
... packages=['tellmy', 'tellmy.recipes'],
... install_requires=['setuptools'],
... namespace_packages=['tellmy'],
... entry_points = {'zc.buildout':
... ['dummy = tellmy.recipes.dummy:Dummy']},
... )
... ''')
Now, a buildout that uses it.
>>> create_sample_namespace_eggs(sample_eggs, site_packages_path)
>>> rmdir('develop-eggs')
>>> from zc.buildout.testing import make_buildout
>>> make_buildout(executable=py_path)
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... develop = ns
... recipes
... parts = dummy
... find-links = %(link_server)s
... executable = %(py_path)s
...
... [dummy]
... recipe = tellmy.recipes:dummy
... ''' % globals())
Now we actually run the buildout.
>>> print system(buildout)
Develop: '/sample-buildout/ns'
Develop: '/sample-buildout/recipes'
Uninstalling dummy.
Installing dummy.
<BLANKLINE>
"""
if sys.version_info > (2, 4): if sys.version_info > (2, 4):
def test_exit_codes(): def test_exit_codes():
""" """
...@@ -2367,6 +3085,7 @@ Distribution setup scripts can import modules in the distribution directory: ...@@ -2367,6 +3085,7 @@ Distribution setup scripts can import modules in the distribution directory:
>>> ls('develop-eggs') >>> ls('develop-eggs')
- foo.egg-link - foo.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
""" """
...@@ -2385,6 +3104,13 @@ honoring our version specification. ...@@ -2385,6 +3104,13 @@ honoring our version specification.
>>> write('foo.py', '') >>> write('foo.py', '')
>>> _ = system(buildout+' setup . sdist') >>> _ = system(buildout+' setup . sdist')
>>> if zc.buildout.easy_install.is_distribute:
... distribute_version = 'distribute = %s' % (
... pkg_resources.working_set.find(
... pkg_resources.Requirement.parse('distribute')).version,)
... else:
... distribute_version = ''
...
>>> write('buildout.cfg', >>> write('buildout.cfg',
... ''' ... '''
... [buildout] ... [buildout]
...@@ -2396,12 +3122,14 @@ honoring our version specification. ...@@ -2396,12 +3122,14 @@ honoring our version specification.
... [versions] ... [versions]
... setuptools = %s ... setuptools = %s
... foo = 1 ... foo = 1
... %s
... ...
... [foo] ... [foo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
... eggs = foo ... eggs = foo
... ''' % pkg_resources.working_set.find( ... ''' % (pkg_resources.working_set.find(
... pkg_resources.Requirement.parse('setuptools')).version) ... pkg_resources.Requirement.parse('setuptools')).version,
... distribute_version))
>>> print system(buildout), >>> print system(buildout),
Installing foo. Installing foo.
...@@ -2654,44 +3382,70 @@ def increment_on_command_line(): ...@@ -2654,44 +3382,70 @@ def increment_on_command_line():
###################################################################### ######################################################################
def create_sample_eggs(test, executable=sys.executable): def make_py_with_system_install(make_py, sample_eggs):
write = test.globs['write'] py_path, site_packages_path = make_py()
dest = test.globs['sample_eggs'] create_sample_namespace_eggs(sample_eggs, site_packages_path)
return py_path
def create_sample_namespace_eggs(dest, site_packages_path=None):
from zc.buildout.testing import write, mkdir
for pkg, version in (('version', '1.0'), ('version', '1.1'),
('fortune', '1.0')):
tmp = tempfile.mkdtemp() tmp = tempfile.mkdtemp()
try: try:
write(tmp, 'README.txt', '') write(tmp, 'README.txt', '')
mkdir(tmp, 'src')
for i in (0, 1, 2): mkdir(tmp, 'src', 'tellmy')
write(tmp, 'eggrecipedemoneeded.py', 'y=%s\ndef f():\n pass' % i) write(tmp, 'src', 'tellmy', '__init__.py',
c1 = i==2 and 'c1' or '' "__import__("
"'pkg_resources').declare_namespace(__name__)\n")
mkdir(tmp, 'src', 'tellmy', pkg)
write(tmp, 'src', 'tellmy', pkg,
'__init__.py', '__version__=%r\n' % version)
write( write(
tmp, 'setup.py', tmp, 'setup.py',
"from setuptools import setup\n" "from setuptools import setup\n"
"setup(name='demoneeded', py_modules=['eggrecipedemoneeded']," "setup(\n"
" zip_safe=True, version='1.%s%s', author='bob', url='bob', " " name='tellmy.%(pkg)s',\n"
"author_email='bob')\n" " package_dir = {'': 'src'},\n"
% (i, c1) " packages = ['tellmy', 'tellmy.%(pkg)s'],\n"
" install_requires = ['setuptools'],\n"
" namespace_packages=['tellmy'],\n"
" zip_safe=True, version=%(version)r,\n"
" author='bob', url='bob', author_email='bob')\n"
% locals()
) )
zc.buildout.testing.sdist(tmp, dest) zc.buildout.testing.sdist(tmp, dest)
if (site_packages_path and pkg == 'version' and version == '1.1'):
# We install the 1.1 version in site packages the way a
# system packaging system (debs, rpms) would do it.
zc.buildout.testing.sys_install(tmp, site_packages_path)
finally:
shutil.rmtree(tmp)
def _write_eggrecipedemoneeded(tmp, minor_version, suffix=''):
from zc.buildout.testing import write
write(tmp, 'README.txt', '')
write(tmp, 'eggrecipedemoneeded.py',
'y=%s\ndef f():\n pass' % minor_version)
write( write(
tmp, 'setup.py', tmp, 'setup.py',
"from setuptools import setup\n" "from setuptools import setup\n"
"setup(name='other', zip_safe=False, version='1.0', " "setup(name='demoneeded', py_modules=['eggrecipedemoneeded'],"
"py_modules=['eggrecipedemoneeded'])\n" " zip_safe=True, version='1.%s%s', author='bob', url='bob', "
"author_email='bob')\n"
% (minor_version, suffix)
) )
zc.buildout.testing.bdist_egg(tmp, executable, dest)
os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py'))
for i in (1, 2, 3, 4): def _write_eggrecipedemo(tmp, minor_version, suffix=''):
from zc.buildout.testing import write
write(tmp, 'README.txt', '')
write( write(
tmp, 'eggrecipedemo.py', tmp, 'eggrecipedemo.py',
'import eggrecipedemoneeded\n' 'import eggrecipedemoneeded\n'
'x=%s\n' 'x=%s\n'
'def main(): print x, eggrecipedemoneeded.y\n' 'def main(): print x, eggrecipedemoneeded.y\n'
% i) % minor_version)
c1 = i==4 and 'c1' or ''
write( write(
tmp, 'setup.py', tmp, 'setup.py',
"from setuptools import setup\n" "from setuptools import setup\n"
...@@ -2699,10 +3453,46 @@ def create_sample_eggs(test, executable=sys.executable): ...@@ -2699,10 +3453,46 @@ def create_sample_eggs(test, executable=sys.executable):
" install_requires = 'demoneeded'," " install_requires = 'demoneeded',"
" entry_points={'console_scripts': " " entry_points={'console_scripts': "
"['demo = eggrecipedemo:main']}," "['demo = eggrecipedemo:main']},"
" zip_safe=True, version='0.%s%s')\n" % (i, c1) " zip_safe=True, version='0.%s%s')\n" % (minor_version, suffix)
)
def create_sample_sys_install(site_packages_path):
for creator, minor_version in (
(_write_eggrecipedemoneeded, 1),
(_write_eggrecipedemo, 3)):
# Write the files and install in site_packages_path.
tmp = tempfile.mkdtemp()
try:
creator(tmp, minor_version)
zc.buildout.testing.sys_install(tmp, site_packages_path)
finally:
shutil.rmtree(tmp)
def create_sample_eggs(test, executable=sys.executable):
from zc.buildout.testing import write
dest = test.globs['sample_eggs']
tmp = tempfile.mkdtemp()
try:
for i in (0, 1, 2):
suffix = i==2 and 'c1' or ''
_write_eggrecipedemoneeded(tmp, i, suffix)
zc.buildout.testing.sdist(tmp, dest)
write(
tmp, 'setup.py',
"from setuptools import setup\n"
"setup(name='other', zip_safe=False, version='1.0', "
"py_modules=['eggrecipedemoneeded'])\n"
) )
zc.buildout.testing.bdist_egg(tmp, executable, dest) zc.buildout.testing.bdist_egg(tmp, executable, dest)
os.remove(os.path.join(tmp, 'eggrecipedemoneeded.py'))
for i in (1, 2, 3, 4):
suffix = i==4 and 'c1' or ''
_write_eggrecipedemo(tmp, i, suffix)
zc.buildout.testing.bdist_egg(tmp, executable, dest)
write(tmp, 'eggrecipebigdemo.py', 'import eggrecipedemo') write(tmp, 'eggrecipebigdemo.py', 'import eggrecipedemo')
write( write(
tmp, 'setup.py', tmp, 'setup.py',
...@@ -2776,6 +3566,7 @@ def easy_install_SetUp(test): ...@@ -2776,6 +3566,7 @@ def easy_install_SetUp(test):
test.globs['sample_eggs']) test.globs['sample_eggs'])
test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5) test.globs['update_extdemo'] = lambda : add_source_dist(test, 1.5)
zc.buildout.testing.install_develop('zc.recipe.egg', test) zc.buildout.testing.install_develop('zc.recipe.egg', test)
zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$' egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'
).match ).match
...@@ -2818,7 +3609,8 @@ def updateSetup(test): ...@@ -2818,7 +3609,8 @@ def updateSetup(test):
here = os.getcwd() here = os.getcwd()
os.chdir(os.path.dirname(dist.location)) os.chdir(os.path.dirname(dist.location))
assert os.spawnle( assert os.spawnle(
os.P_WAIT, sys.executable, zc.buildout.easy_install._safe_arg (sys.executable), os.P_WAIT, sys.executable,
zc.buildout.easy_install._safe_arg(sys.executable),
os.path.join(os.path.dirname(dist.location), 'setup.py'), os.path.join(os.path.dirname(dist.location), 'setup.py'),
'-q', 'bdist_egg', '-d', eggs, '-q', 'bdist_egg', '-d', eggs,
dict(os.environ, dict(os.environ,
...@@ -2841,13 +3633,15 @@ def updateSetup(test): ...@@ -2841,13 +3633,15 @@ def updateSetup(test):
# now let's make the new releases # now let's make the new releases
makeNewRelease('zc.buildout', ws, new_releases) makeNewRelease('zc.buildout', ws, new_releases)
makeNewRelease('setuptools', ws, new_releases)
os.mkdir(os.path.join(new_releases, 'zc.buildout')) os.mkdir(os.path.join(new_releases, 'zc.buildout'))
if zc.buildout.easy_install.is_distribute:
makeNewRelease('distribute', ws, new_releases)
os.mkdir(os.path.join(new_releases, 'distribute'))
else:
makeNewRelease('setuptools', ws, new_releases)
os.mkdir(os.path.join(new_releases, 'setuptools')) os.mkdir(os.path.join(new_releases, 'setuptools'))
normalize_bang = ( normalize_bang = (
re.compile(re.escape('#!'+ re.compile(re.escape('#!'+
zc.buildout.easy_install._safe_arg(sys.executable))), zc.buildout.easy_install._safe_arg(sys.executable))),
...@@ -2869,7 +3663,8 @@ def test_suite(): ...@@ -2869,7 +3663,8 @@ def test_suite():
'__buildout_signature__ = recipes-SSSSSSSSSSS'), '__buildout_signature__ = recipes-SSSSSSSSSSS'),
(re.compile('executable = [\S ]+python\S*', re.I), (re.compile('executable = [\S ]+python\S*', re.I),
'executable = python'), 'executable = python'),
(re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile('[-d] (setuptools|distribute)-\S+[.]egg'),
'setuptools.egg'),
(re.compile('zc.buildout(-\S+)?[.]egg(-link)?'), (re.compile('zc.buildout(-\S+)?[.]egg(-link)?'),
'zc.buildout.egg'), 'zc.buildout.egg'),
(re.compile('creating \S*setup.cfg'), 'creating setup.cfg'), (re.compile('creating \S*setup.cfg'), 'creating setup.cfg'),
...@@ -2884,6 +3679,7 @@ def test_suite(): ...@@ -2884,6 +3679,7 @@ def test_suite():
r'when that file already exists: '), r'when that file already exists: '),
'[Errno 17] File exists: ' '[Errno 17] File exists: '
), ),
(re.compile('distribute'), 'setuptools'),
]) ])
), ),
doctest.DocFileSuite( doctest.DocFileSuite(
...@@ -2913,9 +3709,18 @@ def test_suite(): ...@@ -2913,9 +3709,18 @@ def test_suite():
(re.compile('(zc.buildout|setuptools)-\d+[.]\d+\S*' (re.compile('(zc.buildout|setuptools)-\d+[.]\d+\S*'
'-py\d.\d.egg'), '-py\d.\d.egg'),
'\\1.egg'), '\\1.egg'),
(re.compile('distribute-\d+[.]\d+\S*'
'-py\d.\d.egg'),
'setuptools.egg'),
(re.compile('(zc.buildout|setuptools)( version)? \d+[.]\d+\S*'), (re.compile('(zc.buildout|setuptools)( version)? \d+[.]\d+\S*'),
'\\1 V.V'), '\\1 V.V'),
(re.compile('[-d] setuptools'), '- setuptools'), (re.compile('distribute( version)? \d+[.]\d+\S*'),
'setuptools V.V'),
(re.compile('[-d] (setuptools|distribute)'), '- setuptools'),
(re.compile('distribute'), 'setuptools'),
(re.compile("\nUnused options for buildout: "
"'(distribute|setuptools)\-version'\."),
'')
]) ])
), ),
...@@ -2931,9 +3736,17 @@ def test_suite(): ...@@ -2931,9 +3736,17 @@ def test_suite():
zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize_egg_py,
normalize_bang, normalize_bang,
(re.compile('extdemo[.]pyd'), 'extdemo.so'), (re.compile('extdemo[.]pyd'), 'extdemo.so'),
(re.compile('[-d] setuptools-\S+[.]egg'), 'setuptools.egg'), (re.compile('[-d] (setuptools|distribute)-\S+[.]egg'),
'setuptools.egg'),
(re.compile(r'\\[\\]?'), '/'), (re.compile(r'\\[\\]?'), '/'),
(re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'), (re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),
# Normalize generate_script's Windows interpreter to UNIX:
(re.compile(r'\nimport subprocess\n'), '\n'),
(re.compile('subprocess\\.call\\(argv, env=environ\\)'),
'os.execve(sys.executable, argv, environ)'),
(re.compile('distribute'), 'setuptools'),
# Distribute unzips eggs by default.
(re.compile('\- demoneeded'), 'd demoneeded'),
]+(sys.version_info < (2, 5) and [ ]+(sys.version_info < (2, 5) and [
(re.compile('.*No module named runpy.*', re.S), ''), (re.compile('.*No module named runpy.*', re.S), ''),
(re.compile('.*usage: pdb.py scriptfile .*', re.S), ''), (re.compile('.*usage: pdb.py scriptfile .*', re.S), ''),
...@@ -2965,7 +3778,7 @@ def test_suite(): ...@@ -2965,7 +3778,7 @@ def test_suite():
zc.buildout.testing.normalize_egg_py, zc.buildout.testing.normalize_egg_py,
(re.compile("buildout: Running \S*setup.py"), (re.compile("buildout: Running \S*setup.py"),
'buildout: Running setup.py'), 'buildout: Running setup.py'),
(re.compile('setuptools-\S+-'), (re.compile('(setuptools|distribute)-\S+-'),
'setuptools.egg'), 'setuptools.egg'),
(re.compile('zc.buildout-\S+-'), (re.compile('zc.buildout-\S+-'),
'zc.buildout.egg'), 'zc.buildout.egg'),
...@@ -2973,7 +3786,7 @@ def test_suite(): ...@@ -2973,7 +3786,7 @@ def test_suite():
'File "one.py"'), 'File "one.py"'),
(re.compile(r'We have a develop egg: (\S+) (\S+)'), (re.compile(r'We have a develop egg: (\S+) (\S+)'),
r'We have a develop egg: \1 V'), r'We have a develop egg: \1 V'),
(re.compile('Picked: setuptools = \S+'), (re.compile('Picked: (setuptools|distribute) = \S+'),
'Picked: setuptools = V'), 'Picked: setuptools = V'),
(re.compile(r'\\[\\]?'), '/'), (re.compile(r'\\[\\]?'), '/'),
(re.compile( (re.compile(
...@@ -2981,6 +3794,12 @@ def test_suite(): ...@@ -2981,6 +3794,12 @@ def test_suite():
'-q develop -mxN -d /sample-buildout/develop-eggs' '-q develop -mxN -d /sample-buildout/develop-eggs'
), ),
(re.compile(r'^[*]...'), '...'), (re.compile(r'^[*]...'), '...'),
# for bug_92891_bootstrap_crashes_with_egg_recipe_in_buildout_section
(re.compile(r"Unused options for buildout: 'eggs' 'scripts'\."),
"Unused options for buildout: 'scripts' 'eggs'."),
(re.compile('distribute'), 'setuptools'),
# Distribute unzips eggs by default.
(re.compile('\- demoneeded'), 'd demoneeded'),
]), ]),
), ),
zc.buildout.testselectingpython.test_suite(), zc.buildout.testselectingpython.test_suite(),
...@@ -3040,6 +3859,8 @@ def test_suite(): ...@@ -3040,6 +3859,8 @@ def test_suite():
zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_script,
normalize_bang, normalize_bang,
(re.compile('Downloading.*setuptools.*egg\n'), ''), (re.compile('Downloading.*setuptools.*egg\n'), ''),
(re.compile('options:'), 'Options:'),
(re.compile('usage:'), 'Usage:'),
]), ]),
)) ))
......
...@@ -11,7 +11,7 @@ ...@@ -11,7 +11,7 @@
# FOR A PARTICULAR PURPOSE. # FOR A PARTICULAR PURPOSE.
# #
############################################################################## ##############################################################################
import os, re, sys, unittest import os, re, subprocess, sys, textwrap, unittest
from zope.testing import doctest, renormalizing from zope.testing import doctest, renormalizing
import zc.buildout.tests import zc.buildout.tests
import zc.buildout.testing import zc.buildout.testing
...@@ -42,6 +42,33 @@ We can specify a specific Python executable. ...@@ -42,6 +42,33 @@ We can specify a specific Python executable.
def multi_python(test): def multi_python(test):
other_executable = zc.buildout.testing.find_python(other_version) other_executable = zc.buildout.testing.find_python(other_version)
command = textwrap.dedent('''\
try:
import setuptools
except ImportError:
import sys
sys.exit(1)
''')
if subprocess.call([other_executable, '-c', command],
env=os.environ):
# the other executable does not have setuptools. Get setuptools.
# We will do this using the same tools we are testing, for better or
# worse. Alternatively, we could try using bootstrap.
executable_dir = test.globs['tmpdir']('executable_dir')
executable_parts = os.path.join(executable_dir, 'parts')
test.globs['mkdir'](executable_parts)
ws = zc.buildout.easy_install.install(
['setuptools'], executable_dir,
index='http://www.python.org/pypi/',
always_unzip=True, executable=other_executable)
zc.buildout.easy_install.sitepackage_safe_scripts(
executable_dir, ws, other_executable, executable_parts,
reqs=['setuptools'], interpreter='py')
original_executable = other_executable
other_executable = os.path.join(executable_dir, 'py')
assert not subprocess.call(
[other_executable, '-c', command], env=os.environ), (
'test set up failed')
sample_eggs = test.globs['tmpdir']('sample_eggs') sample_eggs = test.globs['tmpdir']('sample_eggs')
os.mkdir(os.path.join(sample_eggs, 'index')) os.mkdir(os.path.join(sample_eggs, 'index'))
test.globs['sample_eggs'] = sample_eggs test.globs['sample_eggs'] = sample_eggs
......
...@@ -78,25 +78,30 @@ new versions found in new releases: ...@@ -78,25 +78,30 @@ new versions found in new releases:
zc.buildout 99.99 zc.buildout 99.99
setuptools 99.99 setuptools 99.99
Our buildout script has been updated to use the new eggs: Our buildout script's site.py has been updated to use the new eggs:
>>> cat(sample_buildout, 'bin', 'buildout') >>> cat(sample_buildout, 'parts', 'buildout', 'site.py')
#!/usr/local/bin/python2.4 ... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
"...
def addsitepackages(known_paths):
"""Add site packages, as determined by zc.buildout.
<BLANKLINE> <BLANKLINE>
import sys See original_addsitepackages, below, for the original version."""
sys.path[0:0] = [ buildout_paths = [
'/sample-buildout/eggs/zc.buildout-99.99-py2.4.egg', '/sample-buildout/eggs/zc.buildout-99.99-pyN.N.egg',
'/sample-buildout/eggs/setuptools-99.99-py2.4.egg', '/sample-buildout/eggs/setuptools-99.99-pyN.N.egg'
] ]
<BLANKLINE> for path in buildout_paths:
import zc.buildout.buildout sitedir, sitedircase = makepath(path)
<BLANKLINE> if not sitedircase in known_paths and os.path.exists(sitedir):
if __name__ == '__main__': sys.path.append(sitedir)
zc.buildout.buildout.main() known_paths.add(sitedircase)
return known_paths
...
Now, let's recreate the sample buildout. If we specify constraints on Now, let's recreate the sample buildout. If we specify constraints on
the versions of zc.buildout and setuptools to use, running the the versions of zc.buildout and setuptools (or distribute) to use,
buildout will install earlier versions of these packages: running the buildout will install earlier versions of these packages:
>>> write(sample_buildout, 'buildout.cfg', >>> write(sample_buildout, 'buildout.cfg',
... """ ... """
...@@ -107,6 +112,7 @@ buildout will install earlier versions of these packages: ...@@ -107,6 +112,7 @@ buildout will install earlier versions of these packages:
... develop = showversions ... develop = showversions
... zc.buildout-version = < 99 ... zc.buildout-version = < 99
... setuptools-version = < 99 ... setuptools-version = < 99
... distribute-version = < 99
... ...
... [show-versions] ... [show-versions]
... recipe = showversions ... recipe = showversions
...@@ -119,7 +125,6 @@ Now we can see that we actually "upgrade" to an earlier version. ...@@ -119,7 +125,6 @@ Now we can see that we actually "upgrade" to an earlier version.
zc.buildout version 1.0.0, zc.buildout version 1.0.0,
setuptools version 0.6; setuptools version 0.6;
restarting. restarting.
Generated script '/sample-buildout/bin/buildout'.
Develop: '/sample-buildout/showversions' Develop: '/sample-buildout/showversions'
Updating show-versions. Updating show-versions.
zc.buildout 1.0.0 zc.buildout 1.0.0
......
Change History
**************
1.0.0
=====
Initial public version.
********************************
Buildout Script Recipe
********************************
.. contents::
The script recipe installs eggs into a buildout eggs directory, exactly
like zc.recipe.egg, and then generates scripts in a buildout bin
directory with egg paths baked into them.
##############################################################################
#
# Copyright (c) 2007 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Setup for z3c.recipe.scripts package
$Id: setup.py 106736 2009-12-18 02:33:08Z gary $
"""
version = '1.0.0dev'
import os
from setuptools import setup, find_packages
def read(*rnames):
return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
name = "z3c.recipe.scripts"
setup(
name = name,
version = version,
author = "Gary Poster",
author_email = "gary.poster@canonical.com",
description = "Recipe for installing Python scripts",
long_description = (
read('README.txt')
+ '\n' +
read('CHANGES.txt')
+ '\n' +
'Detailed Documentation\n'
'**********************\n'
+ '\n' +
read('src', 'z3c', 'recipe', 'scripts', 'README.txt')
+ '\n' +
'Download\n'
'*********\n'
),
keywords = "development build",
classifiers = [
'Development Status :: 5 - Production/Stable',
'Framework :: Buildout',
'Intended Audience :: Developers',
'License :: OSI Approved :: Zope Public License',
'Topic :: Software Development :: Build Tools',
'Topic :: Software Development :: Libraries :: Python Modules',
],
url='http://cheeseshop.python.org/pypi/z3c.recipe.scripts',
license = "ZPL 2.1",
packages = find_packages('src'),
package_dir = {'':'src'},
namespace_packages = ['z3c', 'z3c.recipe'],
install_requires = [
'zc.buildout >=1.5.0dev',
'zc.recipe.egg >=1.2.3dev',
'setuptools'],
tests_require = ['zope.testing'],
test_suite = name+'.tests.test_suite',
entry_points = {'zc.buildout': ['default = %s:Scripts' % name,
'script = %s:Scripts' % name,
'scripts = %s:Scripts' % name,
'interpreter = %s:Interpreter' % name,
]
},
include_package_data = True,
zip_safe=False,
)
__import__('pkg_resources').declare_namespace(__name__)
__import__('pkg_resources').declare_namespace(__name__)
Script and interpreter generation
=================================
This recipe is very similar to zc.recipe.egg, and if you are familiar with its
options, you will be able to use this one easily.
The script and interpreter generation in this recipe are improved from
those provided by zc.recipe.egg in two basic ways.
- The interpreter generated by the script supports all interpreter
options, as opposed to the subset provided by zc.recipe.egg.
- Both scripts and interpreters from this recipe can optionally choose
to include site-packages, and even sitecustomize.
The recipe takes several options. First, here's the list of the options
that overlap from the standard zc.recipe.eggs scripts recipe. After
this, we'll list the new options and describe them.
* eggs
* find-links
* index
* python
* extra-paths
* entry-points
* scripts
* dependent-scripts
* interpreter
* arguments
* initialization
* relative-paths
In addition to these, the recipe offers these new options. They are
introduced here, and described more in depth below.
include-site-packages
You can choose to have the site-packages of the underlying Python
available to your script or interpreter, in addition to the packages
from your eggs. See the section on this option for motivations and
warnings.
allowed-eggs-from-site-packages
Sometimes you need or want to control what eggs from site-packages are
used. The allowed-eggs-from-site-packages option allows you to specify a
whitelist of project names that may be included from site-packages. You
can use globs to specify the value. It defaults to a single value of '*',
indicating that any package may come from site-packages.
Here's a usage example::
[buildout]
...
allowed-eggs-from-site-packages =
demo
bigdemo
zope.*
This option interacts with the ``include-site-packages`` option in the
following ways.
If ``include-site-packages`` is true, then
``allowed-eggs-from-site-packages`` filters what eggs from site-packages
may be chosen. Therefore, if ``allowed-eggs-from-site-packages`` is an
empty list, then no eggs from site-packages are chosen, but site-packages
will still be included at the end of path lists.
If ``include-site-packages`` is false, the value of
``allowed-eggs-from-site-packages`` is irrelevant.
extends
You can extend another section using this value. It is intended to be
used by extending a section that uses this package's scripts recipe.
In this manner, you can avoid repeating yourself.
exec-sitecustomize
Normally the Python's real sitecustomize module is not processed.
If you want it to be processed, set this value to 'true'. This will
be honored irrespective of the setting for include-site-packages.
script-initialization
The standard initialization code affects both an interpreter and scripts.
The code in script-initialization is used only for the generated scripts.
Finally, the "interpreter" entry point ignores ``script-initialization``,
``scripts``, and ``arguments``, and provides yet another additional option.
name
While, by default, the interpreter recipe takes the name of the
section to be the desired interpreter name, you can specify the
interpreter name here instead.
Script generation
-----------------
Generating a basic script looks virtually identical to using zc.recipe.egg.
(Note that the find-links and index values are typically not needed; they
are included to help make this document run as a test successfully.)
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))
>>> print system(buildout),
Installing demo.
Getting distribution for 'demo<0.3'.
Got demo 0.2.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
Generated script '/sample-buildout/bin/demo'.
>>> print system(join(sample_buildout, 'bin', 'demo')),
2 2
Interpreter generation
----------------------
As with zc.recipe.egg, you can generate an interpreter with the default
script recipe shown above by supplying the "interpreter" option.
This example will create both an entry point script and an interpreter.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... interpreter = py
... """ % dict(server=link_server))
>>> print system(buildout),
Uninstalling demo.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.
You can also generate an interpreter alone with the ``interpreter`` recipe.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))
>>> print system(buildout),
Uninstalling demo.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.
In both cases, the bin/py script works by restarting Python after
specifying a special path in PYTHONPATH. This example shows the UNIX version;
the Windows version actually uses subprocess instead.
>>> cat(sample_buildout, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
#!/usr/bin/python2.4 -S
<BLANKLINE>
import os
import sys
<BLANKLINE>
argv = [sys.executable] + sys.argv[1:]
environ = os.environ.copy()
path = '/sample-buildout/parts/py'
if environ.get('PYTHONPATH'):
path = os.pathsep.join([path, environ['PYTHONPATH']])
environ['PYTHONPATH'] = path
os.execve(sys.executable, argv, environ)
The path is a directory that contains two files: our own site.py and
sitecustomize.py. The site.py is modified from the underlying Python's
site.py, and is responsible for setting up our paths. The
sitecustomize.py is responsible for running the initialization code
provided.
>>> ls(sample_buildout, 'parts', 'py')
- site.py
- sitecustomize.py
Here's an example of using the generated interpreter.
>>> print system(join(sample_buildout, 'bin', 'py') +
... ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
'/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
<BLANKLINE>
Including site-packages and sitecustomize
-----------------------------------------
As introduced above, this recipe supports including site packages. This has
some advantages and some serious dangers.
A typical reason to include site-packages is that it is easier to
install one or more dependencies in your Python than it is with
buildout. Some packages, such as lxml or Python PostgreSQL integration,
have dependencies that can be much easier to build and/or install using
other mechanisms, such as your operating system's package manager. By
installing some core packages into your Python's site-packages, this can
significantly simplify some application installations.
However, doing this has a significant danger. One of the primary goals
of buildout is to provide repeatability. Some packages (one of the
better known Python openid packages, for instance) change their behavior
depending on what packages are available. If Python curl bindings are
available, these may be preferred by the library. If a certain XML
package is installed, it may be preferred by the library. These hidden
choices may cause small or large behavior differences. The fact that
they can be rarely encountered can actually make it worse: you forget
that this might be a problem, and debugging the differences can be
difficult. If you allow site-packages to be included in your buildout,
and the Python you use is not managed precisely by your application (for
instance, it is a system Python), you open yourself up to these
possibilities. Don't be unaware of the dangers.
To show off these features, we need to use buildout with a Python
executable with some extra paths to show ``include-site-packages``; and one
guaranteed to have a sitecustomize module to show
``exec-sitecustomize``. We'll make one using a test fixture called
``make_py``. The os.environ change below will go into the sitecustomize,
and the site_packages_path will be in the Python's path.
>>> py_path, site_packages_path = make_py(initialization='''\
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... ''')
>>> print site_packages_path
/executable_buildout/site-packages
Now let's take a look at include-site-packages. The default is false,
so we will set it to true.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... include-site-packages = true
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server, py_path=py_path))
>>> print system(buildout),
Uninstalling py.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.
Now executable_buildout/site-packages is included in sys.path.
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "import sys, pprint; pprint.pprint(sys.path)"''')
... # doctest: +ELLIPSIS
['',
'/sample-buildout/parts/py',
...,
'/sample-buildout/eggs/demo-0.2-pyN.N.egg',
'/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg',
'/executable_buildout/eggs/setuptools-X-pyN.N.egg',
'/executable_buildout/site-packages']
<BLANKLINE>
As described above, the allowed-eggs-from-site-packages option lets us
control what site-packages eggs zc.buildout will allow to fulfill
dependencies. The behavior was described above with an example (and the
implementation is tested elsewhere), so we'll only look at some simple and
common use cases here.
Sometimes you may want to allow site-packages to be available but you don't
want your package to depend on it using setup.py. For instance, perhaps you
are writing an application, and you want to depend on your system's packaging
of the PostgreSQL code, but the system Python does not use eggs to
package it, so you need to manage the two separately. In this case, you
might not want to use any eggs from site-packages, but you want it available.
In this case, you can use allowed-eggs-from-site-packages with an empty value
to keep any egg from being used from site-packages.
Here's an example. Let's say we have a Python with demo and demoneeded
installed as eggs in the system Python. Normally, they will be used to
fulfill dependencies, because allowed-eggs-from-site-packages defaults to
the value "*" (allow any package). (We use an empty find-links value to say
that buildout may not look elsewhere for the package. We use a different
eggs-directory for isolation, so that eggs obtained other parts of the
document do not affect this example.)
>>> from zc.buildout.tests import create_sample_sys_install
>>> create_sample_sys_install(site_packages_path)
>>> import zc.buildout.easy_install
>>> zc.buildout.easy_install.clear_index_cache()
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... eggs-directory = tmpeggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... python = primed_python
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout)
Creating directory '/sample-buildout/tmpeggs'.
Uninstalling py.
Installing eggs.
<BLANKLINE>
That succeeds fine, getting demoneeded from the Python site-packages.
However, when allowed-eggs-from-site-packages is an empty value, demoneeded
is not allowed to come from site-packages, and the buildout fails.
>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir('tmpeggs')
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... eggs-directory = tmpeggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... allowed-eggs-from-site-packages =
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout)
Creating directory '/sample-buildout/tmpeggs'.
Uninstalling eggs.
Installing eggs.
Couldn't find index page for 'demoneeded' (maybe misspelled?)
Getting distribution for 'demoneeded'.
While:
Installing eggs.
Getting distribution for 'demoneeded'.
Error: Couldn't find a distribution for 'demoneeded'.
<BLANKLINE>
Remember that you can provide multiple lines to the
allowed-eggs-from-site-packages option, each specifying a whitelist of
allowed packages. Globs (* and ?) are allowed.
Next we will use the exec-sitecustomize option. It simply copies
Python's underlying sitecustomize module, if it exists, to the local
version. The os.environ change shown above in the make_py call will go
into the sitecustomize.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... exec-sitecustomize = true
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server, py_path=py_path))
>>> print system(buildout),
Installing py.
Generated interpreter '/sample-buildout/bin/py'.
>>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
<BLANKLINE>
# The following is from
# /executable_buildout/parts/py/sitecustomize.py
...
import os
os.environ['zc.buildout'] = 'foo bar baz shazam'
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "import os; print os.environ['zc.buildout']"''')
foo bar baz shazam
<BLANKLINE>
Options
-------
We'll focus now on the remaining options that are different than
zc.recipe.egg.
Let's look at the ``extends`` option first.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo python
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
...
... [python]
... recipe = z3c.recipe.scripts:interpreter
... extends = demo
... initialization =
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... """ % dict(server=link_server))
That makes it easier to specify some initialization for the interpreter
that is different than a script, while duplicating other configuration.
Now let's put it in action.
>>> print system(buildout),
Uninstalling py.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Installing python.
Generated interpreter '/sample-buildout/bin/python'.
>>> print system(join(sample_buildout, 'bin', 'python') +
... ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
'/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
<BLANKLINE>
>>> print system(join(sample_buildout, 'bin', 'python') +
... ''' -c "import os; print os.environ['zc.buildout']"'''),
foo bar baz shazam
Note that the parts/py directory has been cleaned up, and parts/python has
been created.
>>> ls(sample_buildout, 'parts')
d buildout
d demo
d python
If you want to have initialization that only affects scripts, not the
interpreter, you can use script-initialization. Here's a demonstration.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... interpreter = py
... script-initialization =
... print "Hi from the script"
... """ % dict(server=link_server))
>>> print system(buildout),
Uninstalling python.
Uninstalling demo.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "print 'Hi from the interpreter'"'''),
Hi from the interpreter
>>> print system(join(sample_buildout, 'bin', 'demo')),
Hi from the script
2 2
The last new option is ``name``. This simply changes the name of the
interpreter, so that you are not forced to use the name of the section.
>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = interpreter
...
... [interpreter]
... name = python2
... recipe = z3c.recipe.scripts:interpreter
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... """ % dict(server=link_server))
>>> print system(buildout),
Uninstalling demo.
Installing interpreter.
Generated interpreter '/sample-buildout/bin/python2'.
>>> print system(join(sample_buildout, 'bin', 'python2') +
... ' -c "print 42"')
42
<BLANKLINE>
The other options all identical to zc.recipe.egg.
from z3c.recipe.scripts.scripts import Scripts, Interpreter
##############################################################################
#
# Copyright (c) 2009-2010 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""Install scripts from eggs.
"""
import os
import zc.buildout
import zc.buildout.easy_install
from zc.recipe.egg.egg import ScriptBase
class Base(ScriptBase):
def __init__(self, buildout, name, options):
if 'extends' in options:
options.update(buildout[options['extends']])
super(Base, self).__init__(buildout, name, options)
self.default_eggs = '' # Disables feature from zc.recipe.egg.
b_options = buildout['buildout']
options['parts-directory'] = os.path.join(
b_options['parts-directory'], self.name)
value = options.setdefault(
'allowed-eggs-from-site-packages',
'*')
self.allowed_eggs = tuple(name.strip() for name in value.split('\n'))
value = options.setdefault(
'include-site-packages',
b_options.get('include-site-packages', 'false'))
if value not in ('true', 'false'):
raise zc.buildout.UserError(
"Invalid value for include-site-packages option: %s" %
(value,))
self.include_site_packages = (value == 'true')
value = options.setdefault(
'exec-sitecustomize',
b_options.get('exec-sitecustomize', 'false'))
if value not in ('true', 'false'):
raise zc.buildout.UserError(
"Invalid value for exec-sitecustomize option: %s" %
(value,))
self.exec_sitecustomize = (value == 'true')
class Interpreter(Base):
def __init__(self, buildout, name, options):
super(Interpreter, self).__init__(buildout, name, options)
options.setdefault('name', name)
def install(self):
reqs, ws = self.working_set()
options = self.options
generated = []
if not os.path.exists(options['parts-directory']):
os.mkdir(options['parts-directory'])
generated.append(options['parts-directory'])
generated.extend(zc.buildout.easy_install.sitepackage_safe_scripts(
options['bin-directory'], ws, options['executable'],
options['parts-directory'],
interpreter=options['name'],
extra_paths=self.extra_paths,
initialization=options.get('initialization', ''),
include_site_packages=self.include_site_packages,
exec_sitecustomize=self.exec_sitecustomize,
relative_paths=self._relative_paths,
))
return generated
update = install
class Scripts(Base):
def _install(self, reqs, ws, scripts):
options = self.options
generated = []
if not os.path.exists(options['parts-directory']):
os.mkdir(options['parts-directory'])
generated.append(options['parts-directory'])
generated.extend(zc.buildout.easy_install.sitepackage_safe_scripts(
options['bin-directory'], ws, options['executable'],
options['parts-directory'], reqs=reqs, scripts=scripts,
interpreter=options.get('interpreter'),
extra_paths=self.extra_paths,
initialization=options.get('initialization', ''),
include_site_packages=self.include_site_packages,
exec_sitecustomize=self.exec_sitecustomize,
relative_paths=self._relative_paths,
script_arguments=options.get('arguments', ''),
script_initialization=options.get('script-initialization', '')
))
return generated
##############################################################################
#
# Copyright (c) 2006 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
import os, re, shutil, sys
import zc.buildout.tests
import zc.buildout.testselectingpython
import zc.buildout.testing
import unittest
from zope.testing import doctest, renormalizing
# We do not explicitly test the recipe support for the ``eggs``,
# ``find-links``, and ``index`` options because they are used for most or
# all of the examples. The README tests ``extends``,
# ``include-site-customization`` and ``name``. That leaves ``python``,
# ``extra-paths``, ``initialization``, ``relative-paths``, and
# ``include-site-packages``.
def supports_python_option():
"""
This simply shows that the ``python`` option can specify another section to
find the ``executable``. (The ``python`` option defaults to looking in the
``buildout`` section.) We do this by creating a custom Python that will have
some initialization that we can look for.
>>> py_path, site_packages_path = make_py(initialization='''
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... ''')
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... parts = py
...
... [custom_python]
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... exec-sitecustomize = true
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... python = custom_python
... ''' % dict(server=link_server, py_path=py_path))
>>> print system(buildout),
Installing py.
Getting distribution for 'demo<0.3'.
Got demo 0.2.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
Generated interpreter '/sample-buildout/bin/py'.
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "import os; print os.environ['zc.buildout']"'''),
foo bar baz shazam
"""
def interpreter_recipe_supports_extra_paths_option():
"""
This shows that specifying extra-paths will affect sys.path.
This recipe will not add paths that do not exist, so we create them.
>>> mkdir(sample_buildout, 'foo')
>>> mkdir(sample_buildout, 'foo', 'bar')
>>> mkdir(sample_buildout, 'spam')
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... parts = py
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... find-links = %(server)s
... index = %(server)s/index
... extra-paths =
... ${buildout:directory}/foo/bar
... ${buildout:directory}/spam
... ''' % dict(server=link_server))
>>> print system(buildout),
Installing py.
Generated interpreter '/sample-buildout/bin/py'.
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "import sys;print 'path' + ' '.join(sys.path)"''')
... # doctest:+ELLIPSIS
path.../foo/bar /sample-buildout/spam...
"""
def interpreter_recipe_supports_initialization_option():
"""
This simply shows that the ``initialization`` option can specify code to
run on initialization.
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... parts = py
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... initialization =
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... ''' % dict(server=link_server))
>>> print system(buildout),
Installing py.
Getting distribution for 'demo<0.3'.
Got demo 0.2.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
Generated interpreter '/sample-buildout/bin/py'.
>>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
... # doctest: +NORMALIZE_WHITESPACE
<BLANKLINE>
import os
os.environ['zc.buildout'] = 'foo bar baz shazam'
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "import os; print os.environ['zc.buildout']"'''),
foo bar baz shazam
This also works with the exec-sitecustomize option, processing local
initialization, and then the Python's initialization. We show this with a
custom Python.
>>> py_path, site_packages_path = make_py(initialization='''
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... ''')
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... parts = py
...
... [custom_python]
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... initialization =
... import os
... os.environ['zc.recipe.egg'] = 'baLOOba'
... exec-sitecustomize = true
... eggs = demo<0.3
... find-links = %(server)s
... index = %(server)s/index
... python = custom_python
... ''' % dict(server=link_server, py_path=py_path))
>>> print system(buildout),
Uninstalling py.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.
>>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
<BLANKLINE>
import os
os.environ['zc.recipe.egg'] = 'baLOOba'
<BLANKLINE>
# The following is from
# /executable_buildout/parts/py/sitecustomize.py
...
import os
os.environ['zc.buildout'] = 'foo bar baz shazam'
>>> print system(join(sample_buildout, 'bin', 'py') + ' -c ' +
... '''"import os; print os.environ['zc.recipe.egg']"'''),
baLOOba
>>> print system(join(sample_buildout, 'bin', 'py') +
... ''' -c "import os; print os.environ['zc.buildout']"'''),
foo bar baz shazam
"""
def interpreter_recipe_supports_relative_paths_option():
"""
This shows that the relative-paths option affects the code for inserting
paths into sys.path.
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
... parts = py
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... find-links = %(server)s
... index = %(server)s/index
... relative-paths = true
... extra-paths =
... /foo/bar
... ${buildout:directory}/spam
... ''' % dict(server=link_server))
>>> print system(buildout),
Installing py.
Generated interpreter '/sample-buildout/bin/py'.
Let's look at the site.py that was generated:
>>> import sys
>>> sys.stdout.write('#'); cat(sample_buildout, 'parts', 'py', 'site.py')
... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
#...
def addsitepackages(known_paths):
"..."
join = os.path.join
base = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
base = os.path.dirname(base)
base = os.path.dirname(base)
buildout_paths = [
'/foo/bar',
join(base, 'spam')
]...
"""
def include_site_packages_option_reusing_eggs():
"""
The include-site-packages buildout option not only controls whether
site-packages are included in the path, but whether eggs in site-packages
can be used to fulfill direct and indirect dependencies of your package. If
it did not, it might fail to exclude site-packages because one of the
dependencies actually was supposed to be fulfilled with it.
The default is ``include-site-packages = false``. This makes it possible to
easily use a system Python. As a demonstration, we will start with a
Python executable that has the "demoneeded" and "demo" eggs installed.
The eggs are not found.
>>> from zc.buildout.tests import create_sample_sys_install
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... python = primed_python
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout)
Installing eggs.
Couldn't find index page for 'demoneeded' (maybe misspelled?)
Getting distribution for 'demoneeded'.
While:
Installing eggs.
Getting distribution for 'demoneeded'.
Error: Couldn't find a distribution for 'demoneeded'.
<BLANKLINE>
However, if we set include-site-packages to true, the package will be found.
Notice we do not set find-links, but the eggs are still found because
they are in the executable's path.
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... python = primed_python
... include-site-packages = true
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout)
Installing eggs.
<BLANKLINE>
We get an error if we specify anything but true or false:
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links = %(link_server)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = no
... eggs = other
... ''' % globals())
>>> print system(buildout)
While:
Installing.
Getting section eggs.
Initializing part eggs.
Error: Invalid value for include-site-packages option: no
<BLANKLINE>
"""
def allowed_eggs_from_site_packages_option():
"""
The allowed-eggs-from-site-packages option allows you to specify a
whitelist of project names that may be included from site-packages.
In the test below, our "py_path" has the "demoneeded" and "demo"
packages available. We'll simply be asking for "demoneeded" here. The
default value of '*' will allow it, as we've seen elsewhere. Here we
explicitly use a "*" for the same result. This also shows that we
correctly parse a single-line value.
>>> from zc.buildout.tests import create_sample_sys_install
>>> py_path, site_packages_path = make_py()
>>> create_sample_sys_install(site_packages_path)
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... allowed-eggs-from-site-packages = *
... python = primed_python
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout)
Installing eggs.
<BLANKLINE>
Specifying the egg exactly will work as well. This shows we correctly
parse a multi-line value.
>>> zc.buildout.easy_install.clear_index_cache()
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... find-links =
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... allowed-eggs-from-site-packages = other
... demoneeded
... python = primed_python
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout)
Uninstalling eggs.
Installing eggs.
<BLANKLINE>
It will also work if we use a glob ("*" or "?"). (We won't show that here
because we already tested it in
zc.buildout.tests.allowed_eggs_from_site_packages.)
However, if we do not include "demoneeded" in the
"allowed-eggs-from-site-packages" key, we get an error, because the
packages are not available in any links, and they are not allowed to
come from the executable's site packages. (We won't show that here
because we already tested it in the same test mentioned above.)
"""
def setUp(test):
zc.buildout.tests.easy_install_SetUp(test)
zc.buildout.testing.install_develop('zc.recipe.egg', test)
zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
def setUpSelecting(test):
zc.buildout.testselectingpython.setup(test)
zc.buildout.testing.install_develop('zc.recipe.egg', test)
zc.buildout.testing.install_develop('z3c.recipe.scripts', test)
def test_suite():
suite = unittest.TestSuite((
doctest.DocFileSuite(
'README.txt',
setUp=setUp, tearDown=zc.buildout.testing.buildoutTearDown,
checker=renormalizing.RENormalizing([
zc.buildout.testing.normalize_path,
zc.buildout.testing.normalize_endings,
zc.buildout.testing.normalize_script,
zc.buildout.testing.normalize_egg_py,
zc.buildout.tests.normalize_bang,
(re.compile(r'zc.buildout(-\S+)?[.]egg(-link)?'),
'zc.buildout.egg'),
(re.compile('[-d] (setuptools|distribute)-[^-]+-'), 'setuptools-X-'),
(re.compile(r'(setuptools|distribute)-[\w.]+-py'), 'setuptools-X-py'),
(re.compile(r'eggs\\\\demo'), 'eggs/demo'),
(re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'),
(re.compile(r'\#!\S+\bpython\S*'), '#!/usr/bin/python'),
# Normalize generate_script's Windows interpreter to UNIX:
(re.compile(r'\nimport subprocess\n'), '\n'),
(re.compile('subprocess\\.call\\(argv, env=environ\\)'),
'os.execve(sys.executable, argv, environ)'),
(re.compile('distribute'), 'setuptools'),
])
),
doctest.DocTestSuite(
setUp=setUp,
tearDown=zc.buildout.testing.buildoutTearDown,
checker=renormalizing.RENormalizing([
zc.buildout.testing.normalize_path,
zc.buildout.testing.normalize_endings,
zc.buildout.testing.normalize_egg_py,
(re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'),
]),
),
))
return suite
if __name__ == '__main__':
unittest.main(defaultTest='test_suite')
...@@ -16,7 +16,7 @@ ...@@ -16,7 +16,7 @@
$Id$ $Id$
""" """
version = '0' version = '1.2.3dev'
import os import os
from setuptools import setup, find_packages from setuptools import setup, find_packages
...@@ -66,7 +66,7 @@ setup( ...@@ -66,7 +66,7 @@ setup(
package_dir = {'':'src'}, package_dir = {'':'src'},
namespace_packages = ['zc', 'zc.recipe'], namespace_packages = ['zc', 'zc.recipe'],
install_requires = [ install_requires = [
'zc.buildout >=1.2.0', 'zc.buildout >=1.5.0dev',
'setuptools'], 'setuptools'],
tests_require = ['zope.testing'], tests_require = ['zope.testing'],
test_suite = name+'.tests.test_suite', test_suite = name+'.tests.test_suite',
......
...@@ -154,6 +154,8 @@ dependent-scripts ...@@ -154,6 +154,8 @@ dependent-scripts
interpreter interpreter
The name of a script to generate that allows access to a Python The name of a script to generate that allows access to a Python
interpreter that has the path set based on the eggs installed. interpreter that has the path set based on the eggs installed.
(See the ``z3c.recipe.scripts`` recipe for a more full-featured
interpreter.)
extra-paths extra-paths
Extra paths to include in a generated script. Extra paths to include in a generated script.
...@@ -577,7 +579,7 @@ declare entry points using the entry-points option: ...@@ -577,7 +579,7 @@ declare entry points using the entry-points option:
- demo - demo
- other - other
>>> cat(sample_buildout, 'bin', 'other') >>> cat(sample_buildout, 'bin', 'other') # doctest: +NORMALIZE_WHITESPACE
#!/usr/local/bin/python2.4 #!/usr/local/bin/python2.4
<BLANKLINE> <BLANKLINE>
import sys import sys
...@@ -640,3 +642,4 @@ be made to contact an index server: ...@@ -640,3 +642,4 @@ be made to contact an index server:
Uninstalling bigdemo. Uninstalling bigdemo.
Installing demo. Installing demo.
Generated script '/sample-buildout/bin/foo'. Generated script '/sample-buildout/bin/foo'.
...@@ -117,6 +117,7 @@ computed by the egg recipe by looking at .installed.cfg: ...@@ -117,6 +117,7 @@ computed by the egg recipe by looking at .installed.cfg:
extras = other extras = other
find-links = http://localhost:27071/ find-links = http://localhost:27071/
index = http://localhost:27071/index index = http://localhost:27071/index
python = buildout
recipe = sample recipe = sample
If we use the extra-paths option: If we use the extra-paths option:
......
...@@ -150,6 +150,7 @@ eggs directory can be shared across multiple buildouts. ...@@ -150,6 +150,7 @@ eggs directory can be shared across multiple buildouts.
>>> ls(sample_buildout, 'develop-eggs') >>> ls(sample_buildout, 'develop-eggs')
d extdemo-1.4-py2.4-unix-i686.egg d extdemo-1.4-py2.4-unix-i686.egg
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
Note that no scripts or dependencies are installed. To install Note that no scripts or dependencies are installed. To install
...@@ -231,6 +232,7 @@ We won't get an update. ...@@ -231,6 +232,7 @@ We won't get an update.
>>> ls(sample_buildout, 'develop-eggs') >>> ls(sample_buildout, 'develop-eggs')
- demo.egg-link - demo.egg-link
d extdemo-1.4-py2.4-unix-i686.egg d extdemo-1.4-py2.4-unix-i686.egg
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
But if we run the buildout in the default on-line and newest modes, we But if we run the buildout in the default on-line and newest modes, we
...@@ -248,6 +250,7 @@ version is imported: ...@@ -248,6 +250,7 @@ version is imported:
- demo.egg-link - demo.egg-link
d extdemo-1.4-py2.4-linux-i686.egg d extdemo-1.4-py2.4-linux-i686.egg
d extdemo-1.5-py2.4-linux-i686.egg d extdemo-1.5-py2.4-linux-i686.egg
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
Controlling the version used Controlling the version used
...@@ -287,6 +290,7 @@ We can specify a specific version using the egg option: ...@@ -287,6 +290,7 @@ We can specify a specific version using the egg option:
>>> ls(sample_buildout, 'develop-eggs') >>> ls(sample_buildout, 'develop-eggs')
- demo.egg-link - demo.egg-link
d extdemo-1.4-py2.4-linux-i686.egg d extdemo-1.4-py2.4-linux-i686.egg
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
...@@ -553,6 +557,7 @@ Our develop-eggs now includes an egg link for extdemo: ...@@ -553,6 +557,7 @@ Our develop-eggs now includes an egg link for extdemo:
>>> ls('develop-eggs') >>> ls('develop-eggs')
- demo.egg-link - demo.egg-link
- extdemo.egg-link - extdemo.egg-link
- z3c.recipe.scripts.egg-link
- zc.recipe.egg.egg-link - zc.recipe.egg.egg-link
and the extdemo now has a built extension: and the extdemo now has a built extension:
......
...@@ -19,14 +19,17 @@ $Id$ ...@@ -19,14 +19,17 @@ $Id$
import logging, os, re, zipfile import logging, os, re, zipfile
import zc.buildout.easy_install import zc.buildout.easy_install
class Eggs(object): class Eggs(object):
include_site_packages = allowed_eggs = None
def __init__(self, buildout, name, options): def __init__(self, buildout, name, options):
self.buildout = buildout self.buildout = buildout
self.name = name self.name = self.default_eggs = name
self.options = options self.options = options
links = options.get('find-links', b_options = buildout['buildout']
buildout['buildout'].get('find-links')) links = options.get('find-links', b_options['find-links'])
if links: if links:
links = links.split() links = links.split()
options['find-links'] = '\n'.join(links) options['find-links'] = '\n'.join(links)
...@@ -34,25 +37,25 @@ class Eggs(object): ...@@ -34,25 +37,25 @@ class Eggs(object):
links = () links = ()
self.links = links self.links = links
index = options.get('index', buildout['buildout'].get('index')) index = options.get('index', b_options.get('index'))
if index is not None: if index is not None:
options['index'] = index options['index'] = index
self.index = index self.index = index
allow_hosts = buildout['buildout'].get('allow-hosts', '*') allow_hosts = b_options['allow-hosts']
allow_hosts = tuple([host.strip() for host in allow_hosts.split('\n') allow_hosts = tuple([host.strip() for host in allow_hosts.split('\n')
if host.strip()!='']) if host.strip()!=''])
self.allow_hosts = allow_hosts self.allow_hosts = allow_hosts
options['eggs-directory'] = buildout['buildout']['eggs-directory'] options['eggs-directory'] = b_options['eggs-directory']
options['_e'] = options['eggs-directory'] # backward compat. options['_e'] = options['eggs-directory'] # backward compat.
options['develop-eggs-directory' options['develop-eggs-directory'] = b_options['develop-eggs-directory']
] = buildout['buildout']['develop-eggs-directory']
options['_d'] = options['develop-eggs-directory'] # backward compat. options['_d'] = options['develop-eggs-directory'] # backward compat.
assert options.get('unzip') in ('true', 'false', None) # verify that this is None, 'true' or 'false'
get_bool(options, 'unzip')
python = options.get('python', buildout['buildout']['python']) python = options.setdefault('python', b_options['python'])
options['executable'] = buildout[python]['executable'] options['executable'] = buildout[python]['executable']
def working_set(self, extra=()): def working_set(self, extra=()):
...@@ -61,31 +64,36 @@ class Eggs(object): ...@@ -61,31 +64,36 @@ class Eggs(object):
This is intended for reuse by similar recipes. This is intended for reuse by similar recipes.
""" """
options = self.options options = self.options
b_options = self.buildout['buildout']
distributions = [ distributions = [
r.strip() r.strip()
for r in options.get('eggs', self.name).split('\n') for r in options.get('eggs', self.default_eggs).split('\n')
if r.strip()] if r.strip()]
orig_distributions = distributions[:] orig_distributions = distributions[:]
distributions.extend(extra) distributions.extend(extra)
if self.buildout['buildout'].get('offline') == 'true': if b_options.get('offline') == 'true':
ws = zc.buildout.easy_install.working_set( ws = zc.buildout.easy_install.working_set(
distributions, options['executable'], distributions, options['executable'],
[options['develop-eggs-directory'], options['eggs-directory']] [options['develop-eggs-directory'],
options['eggs-directory']],
include_site_packages=self.include_site_packages,
allowed_eggs_from_site_packages=self.allowed_eggs,
) )
else: else:
kw = {} kw = {}
if options.get('unzip'): if 'unzip' in options:
kw['always_unzip'] = get_bool(options, 'unzip') kw['always_unzip'] = get_bool(options, 'unzip')
ws = zc.buildout.easy_install.install( ws = zc.buildout.easy_install.install(
distributions, options['eggs-directory'], distributions, options['eggs-directory'],
links=self.links, links=self.links,
index=self.index, index=self.index,
executable=options['executable'], executable=options['executable'],
path=[options['develop-eggs-directory']], path=[options['develop-eggs-directory']],
newest=self.buildout['buildout'].get('newest') == 'true', newest=b_options.get('newest') == 'true',
include_site_packages=self.include_site_packages,
allowed_eggs_from_site_packages=self.allowed_eggs,
allow_hosts=self.allow_hosts, allow_hosts=self.allow_hosts,
**kw) **kw)
...@@ -97,16 +105,19 @@ class Eggs(object): ...@@ -97,16 +105,19 @@ class Eggs(object):
update = install update = install
class Scripts(Eggs):
class ScriptBase(Eggs):
def __init__(self, buildout, name, options): def __init__(self, buildout, name, options):
super(Scripts, self).__init__(buildout, name, options) super(ScriptBase, self).__init__(buildout, name, options)
b_options = buildout['buildout']
options['bin-directory'] = buildout['buildout']['bin-directory'] options['bin-directory'] = b_options['bin-directory']
options['_b'] = options['bin-directory'] # backward compat. options['_b'] = options['bin-directory'] # backward compat.
self.extra_paths = [ self.extra_paths = [
os.path.join(buildout['buildout']['directory'], p.strip()) os.path.join(b_options['directory'], p.strip())
for p in options.get('extra-paths', '').split('\n') for p in options.get('extra-paths', '').split('\n')
if p.strip() if p.strip()
] ]
...@@ -115,11 +126,9 @@ class Scripts(Eggs): ...@@ -115,11 +126,9 @@ class Scripts(Eggs):
relative_paths = options.get( relative_paths = options.get(
'relative-paths', 'relative-paths', b_options.get('relative-paths', 'false'))
buildout['buildout'].get('relative-paths', 'false')
)
if relative_paths == 'true': if relative_paths == 'true':
options['buildout-directory'] = buildout['buildout']['directory'] options['buildout-directory'] = b_options['directory']
self._relative_paths = options['buildout-directory'] self._relative_paths = options['buildout-directory']
else: else:
self._relative_paths = '' self._relative_paths = ''
...@@ -128,12 +137,13 @@ class Scripts(Eggs): ...@@ -128,12 +137,13 @@ class Scripts(Eggs):
parse_entry_point = re.compile( parse_entry_point = re.compile(
'([^=]+)=(\w+(?:[.]\w+)*):(\w+(?:[.]\w+)*)$' '([^=]+)=(\w+(?:[.]\w+)*):(\w+(?:[.]\w+)*)$'
).match ).match
def install(self): def install(self):
reqs, ws = self.working_set() reqs, ws = self.working_set()
options = self.options options = self.options
scripts = options.get('scripts') scripts = options.get('scripts')
if scripts or scripts is None: if scripts or scripts is None or options.get('interpreter'):
if scripts is not None: if scripts is not None:
scripts = scripts.split() scripts = scripts.split()
scripts = dict([ scripts = dict([
...@@ -157,7 +167,20 @@ class Scripts(Eggs): ...@@ -157,7 +167,20 @@ class Scripts(Eggs):
name = dist.project_name name = dist.project_name
if name != 'setuptools' and name not in reqs: if name != 'setuptools' and name not in reqs:
reqs.append(name) reqs.append(name)
return self._install(reqs, ws, scripts)
return ()
update = install
def _install(self, reqs, ws, scripts):
# Subclasses implement this.
raise NotImplementedError()
class Scripts(ScriptBase):
def _install(self, reqs, ws, scripts):
options = self.options
return zc.buildout.easy_install.scripts( return zc.buildout.easy_install.scripts(
reqs, ws, options['executable'], reqs, ws, options['executable'],
options['bin-directory'], options['bin-directory'],
...@@ -166,12 +189,9 @@ class Scripts(Eggs): ...@@ -166,12 +189,9 @@ class Scripts(Eggs):
interpreter=options.get('interpreter'), interpreter=options.get('interpreter'),
initialization=options.get('initialization', ''), initialization=options.get('initialization', ''),
arguments=options.get('arguments', ''), arguments=options.get('arguments', ''),
relative_paths=self._relative_paths, relative_paths=self._relative_paths
) )
return ()
update = install
def get_bool(options, name, default=False): def get_bool(options, name, default=False):
value = options.get(name) value = options.get(name)
......
...@@ -35,7 +35,7 @@ install the demo package using Python 2.4. ...@@ -35,7 +35,7 @@ install the demo package using Python 2.4.
... index = http://www.python.org/pypi/ ... index = http://www.python.org/pypi/
... ...
... [python2.4] ... [python2.4]
... executable = %(python23)s ... executable = %(python24)s
... ...
... [demo] ... [demo]
... recipe = zc.recipe.egg ... recipe = zc.recipe.egg
...@@ -43,7 +43,7 @@ install the demo package using Python 2.4. ...@@ -43,7 +43,7 @@ install the demo package using Python 2.4.
... find-links = %(server)s ... find-links = %(server)s
... python = python2.4 ... python = python2.4
... interpreter = py-demo ... interpreter = py-demo
... """ % dict(server=link_server, python23=other_executable)) ... """ % dict(server=link_server, python24=other_executable))
Now, if we run the buildout: Now, if we run the buildout:
......
...@@ -50,9 +50,12 @@ def test_suite(): ...@@ -50,9 +50,12 @@ def test_suite():
zc.buildout.tests.normalize_bang, zc.buildout.tests.normalize_bang,
(re.compile('zc.buildout(-\S+)?[.]egg(-link)?'), (re.compile('zc.buildout(-\S+)?[.]egg(-link)?'),
'zc.buildout.egg'), 'zc.buildout.egg'),
(re.compile('[-d] setuptools-[^-]+-'), 'setuptools-X-'), (re.compile('[-d] (setuptools|distribute)-[^-]+-'),
'setuptools-X-'),
(re.compile(r'eggs\\\\demo'), 'eggs/demo'), (re.compile(r'eggs\\\\demo'), 'eggs/demo'),
(re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'), (re.compile(r'[a-zA-Z]:\\\\foo\\\\bar'), '/foo/bar'),
# Distribute unzips eggs by default.
(re.compile('\- demoneeded'), 'd demoneeded'),
]) ])
), ),
doctest.DocFileSuite( doctest.DocFileSuite(
...@@ -64,7 +67,7 @@ def test_suite(): ...@@ -64,7 +67,7 @@ def test_suite():
(re.compile('__buildout_signature__ = ' (re.compile('__buildout_signature__ = '
'sample-\S+\s+' 'sample-\S+\s+'
'zc.recipe.egg-\S+\s+' 'zc.recipe.egg-\S+\s+'
'setuptools-\S+\s+' '(setuptools|distribute)-\S+\s+'
'zc.buildout-\S+\s*' 'zc.buildout-\S+\s*'
), ),
'__buildout_signature__ = sample- zc.recipe.egg-\n'), '__buildout_signature__ = sample- zc.recipe.egg-\n'),
...@@ -104,14 +107,17 @@ def test_suite(): ...@@ -104,14 +107,17 @@ def test_suite():
zc.buildout.testing.normalize_path, zc.buildout.testing.normalize_path,
zc.buildout.testing.normalize_endings, zc.buildout.testing.normalize_endings,
zc.buildout.testing.normalize_script, zc.buildout.testing.normalize_script,
(re.compile('Got setuptools \S+'), 'Got setuptools V'), (re.compile('Got (setuptools|distribute) \S+'),
(re.compile('([d-] )?setuptools-\S+-py'), 'Got setuptools V'),
(re.compile('([d-] )?(setuptools|distribute)-\S+-py'),
'setuptools-V-py'), 'setuptools-V-py'),
(re.compile('-py2[.][0-35-9][.]'), 'py2.5.'), (re.compile('-py2[.][0-35-9][.]'), 'py2.5.'),
(re.compile('zc.buildout-\S+[.]egg'), (re.compile('zc.buildout-\S+[.]egg'),
'zc.buildout.egg'), 'zc.buildout.egg'),
(re.compile('zc.buildout[.]egg-link'), (re.compile('zc.buildout[.]egg-link'),
'zc.buildout.egg'), 'zc.buildout.egg'),
# Distribute unzips eggs by default.
(re.compile('\- demoneeded'), 'd demoneeded'),
]), ]),
), ),
) )
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment