Add script installation support. Use distutils' exceptions for option

errors.  Include Python version in setuptools' egg name for compatibility
w/installs via easy_install.  Add isdir/listdir facilities for metadata,
along with support for running scripts from eggs.

--HG--
branch : setuptools
extra : convert_revision : svn%3A6015fed2-1504-0410-9fe1-9d1591cc4771/sandbox/trunk/setuptools%4041053

EasyInstall.txt

@@ -65,10 +65,10 @@ version, and automatically downloading, building, and installing it::
 
     easy_install SQLObject
 
-**Example 2**. Install a package by name and version from a given
-"download page"::
+**Example 2**. Install or upgrade a package by name and version by finding
+links on a given "download page"::
 
-    easy_install -s http://peak.telecommunity.com/dist "setuptools>=0.4a1"
+    easy_install -f http://peak.telecommunity.com/dist "setuptools>=0.4a1"
 
 **Example 3**. Download a source distribution from a specified URL,
 automatically building and installing it::
@@ -90,17 +90,26 @@ distributions as well.
 
 By default, packages are installed to the running Python installation's
 ``site-packages`` directory, unless you provide the ``-d`` or ``--install-dir``
-option to specify an alternative directory.
+option to specify an alternative directory, or specify an alternate location
+using distutils configuration files.  (See `Configuration Files`_, below.)
+
+By default, any scripts included with the package are installed to the running
+Python installation's standard script installation location.  However, if you
+specify an installation directory via the command line or a config file, then
+the default directory for installing scripts will be the same as the package
+installation directory, to ensure that the script will have access to the
+installed package.  You can override this using the ``-s`` or ``--script-dir``
+option.
 
 Packages installed to ``site-packages`` are added to an ``easy-install.pth``
-file, so that Python will be able to import the package by default. If you do
-not want this to happen, you should use the ``-m`` or ``--multi`` option, which
-allows multiple versions of the same package to be selected at runtime.
+file, so that Python will always use the most-recently-installed version of
+the package.  If you would like to be able to select which version to use at
+runtime, you should use the ``-m`` or ``--multi-version`` option.
 
-Note that installing to a directory other than ``site-packages`` already
-implies the ``-m`` option, so if you cannot install to ``site-packages``,
-please see the `Command-Line Options`_ section below (under ``--multi``) to 
-find out how to select packages at runtime.
+Note, however, that installing to a directory other than ``site-packages``
+already implies the ``-m`` option, so if you cannot install to
+``site-packages``, please see the `Command-Line Options`_ section below (under
+``--multi-version``) to find out how to select packages at runtime.
 
 
 Upgrading a Package
@@ -133,6 +142,11 @@ package automatically replaces any previous version in the ``easy-install.pth``
 file, so that Python will import the most-recently installed version by
 default.
 
+If you haven't suppressed script installation (using ``--exclude-scripts`` or
+``-x``), then the upgraded version's scripts will be installed, and they will
+be automatically patched to ``require()`` the corresponding version of the
+package, so that you can use them even if not installing to ``site-packages``.
+
 ``easy_install`` never actually deletes packages (unless you're installing a
 package with the same name and version number as an existing package), so if
 you want to get rid of older versions of a package, please see `Uninstalling
@@ -148,15 +162,22 @@ version, you can do so like this::
     easy_install PackageName==1.2.3
 
 Where ``1.2.3`` is replaced by the exact version number you wish to switch to.
-Note that the named package and version must already have been installed to
-``site-packages``.
+If a package matching the requested name and version is not already installed
+in a directory on ``sys.path``, it will be located via PyPI and installed.
 
-If you'd like to switch to the latest version of ``PackageName``, you can do so
-like this::
+If you'd like to switch to the latest installed version of ``PackageName``, you
+can do so like this::
 
     easy_install PackageName
 
-This will activate the latest installed version.
+This will activate the latest installed version.  (Note: if you have set any
+``find_links`` via distutils configuration files, those download pages will be
+checked for the latest available version of the package, and it will be
+downloaded and installed if it is newer than your current version.)
+
+Note that changing the active version of a package will install the newly
+active version's scripts, unless the ``--exclude-scripts`` or ``-x`` option is
+specified.
 
 
 Uninstalling Packages
@@ -173,7 +194,45 @@ versions of a package), you should first run::
 
 This will ensure that Python doesn't continue to search for a package you're
 planning to remove. After you've done this, you can safely delete the .egg
-files or directories.
+files or directories, along with any scripts you wish to remove.
+
+
+Managing Scripts
+----------------
+
+Whenever you install, upgrade, or change versions of a package, EasyInstall
+automatically installs the scripts for the selected package version, unless
+you tell it not to with ``-x`` or ``--exclude-scripts``.  If any scripts in
+the script directory have the same name, they are overwritten.
+
+Thus, you do not normally need to manually delete scripts for older versions of
+a package, unless the newer version of the package does not include a script
+of the same name.  However, if you are completely uninstalling a package, you
+may wish to manually delete its scripts.
+
+EasyInstall's default behavior means that you can normally only run scripts
+from one version of a package at a time.  If you want to keep multiple versions
+of a script available, however, you can simply use the ``--multi-version`` or
+``-m`` option, and rename the scripts that EasyInstall creates.  This works
+because EasyInstall installs scripts as short code stubs that ``require()`` the
+matching version of the package the script came from, so renaming the script
+has no effect on what it executes.
+
+For example, suppose you want to use two versions of the ``rst2html`` tool
+provided by the `docutils <http://docutils.sf.net/>`_ package.  You might
+first install one version::
+
+    easy_install -m docutils==0.3.9
+
+then rename the ``rst2html.py`` to ``r2h_039``, and install another version::
+
+    easy_install -m docutils==0.3.10
+
+This will create another ``rst2html.py`` script, this one using docutils
+version 0.3.10 instead of 0.3.9.  You now have two scripts, each using a
+different version of the package.  (Notice that we used ``-m`` for both
+installations, so that Python won't lock us out of using anything but the most
+recently-installed version of the package.)
 
 
 Reference Manual
@@ -267,8 +326,22 @@ Command-Line Options
     or in a distutils configuration file, the distutils default installation
     location is used.  Normally, this would be the ``site-packages`` directory,
     but if you are using distutils configuration files, setting things like
-    ``--prefix`` or ``--install-lib``, then those settings are taken into
-    account when computing the default directory.
+    ``prefix`` or ``install_lib``, then those settings are taken into
+    account when computing the default installation directory.
+
+``--script-dir=DIR, -s DIR``
+    Set the script installation directory.  If you don't supply this option
+    (via the command line or a configuration file), but you *have* supplied
+    an ``--install-dir`` (via command line or config file), then this option
+    defaults to the same directory, so that the scripts will be able to find
+    their associated package installation.  Otherwise, this setting defaults
+    to the location where the distutils would normally install scripts, taking
+    any distutils configuration file settings into account.
+
+``--exclude-scripts, -x``
+    Don't install scripts.  This is useful if you need to install multiple
+    versions of a package, but do not want to reset the version that will be
+    run by scripts that are already installed.
 
 ``--find-links=URL, -f URL`` (Option renamed in 0.4a2)
     Scan the specified "download pages" for direct links to downloadable eggs
@@ -319,6 +392,8 @@ Known Issues
    time out or be missing a file.
 
 0.4a2
+ * Added support for installing scripts
+ 
  * Added support for setting options via distutils configuration files
 
  * Renamed ``--scan-url/-s`` to ``--find-links/-f`` to free up ``-s`` for the
@@ -405,9 +480,9 @@ Known Issues
 Future Plans
 ============
 
-* Support packages that include scripts
 * Log progress to a logger, with -v and -q options to control verbosity
 * Process the installed package's dependencies as well as the base package
 * Additional utilities to list/remove/verify packages
 * Signature checking?  SSL?  Ability to suppress PyPI search?
+* Support installation from bdist_wininst packages?
 

easy_install.py

@@ -17,7 +17,7 @@ import sys, os.path, zipimport, shutil, tempfile
 from setuptools import Command
 from setuptools.sandbox import run_setup
 from distutils.sysconfig import get_python_lib
-
+from distutils.errors import DistutilsArgError
 from setuptools.archive_util import unpack_archive
 from setuptools.package_index import PackageIndex
 from pkg_resources import *
@@ -43,31 +43,31 @@ class easy_install(Command):
     """Manage a download/build/install process"""
 
     description = "Find/get/install Python packages"
-
     command_consumes_arguments = True
-    
     user_options = [
         ("zip-ok", "z", "install package as a zipfile"),
         ("multi-version", "m", "make apps have to require() a version"),
         ("install-dir=", "d", "install package to DIR"),
+        ("script-dir=", "s", "install scripts to DIR"),
+        ("exclude-scripts", "x", "Don't install scripts"),
         ("index-url=", "i", "base URL of Python Package Index"),
         ("find-links=", "f", "additional URL(s) to search for packages"),
         ("build-directory=", "b",
             "download/extract/build in DIR; keep the results"),
     ]
 
-    boolean_options = [ 'zip-ok', 'multi-version' ]
+    boolean_options = [ 'zip-ok', 'multi-version', 'exclude-scripts' ]
     create_index = PackageIndex
     
     def initialize_options(self):
         self.zip_ok = None
         self.multi_version = None
-        self.install_dir = None
+        self.install_dir = self.script_dir = self.exclude_scripts = None
         self.index_url = None
         self.find_links = None
         self.build_directory = None
         self.args = None
-
+        
         # Options not specifiable via command line
         self.package_index = None
         self.pth_file = None
@@ -81,11 +81,22 @@ class easy_install(Command):
         return tmpdir
         
     def finalize_options(self):
-        # Let install_lib get set by install_lib command, which in turn
+        # If a non-default installation directory was specified, default the
+        # script directory to match it.
+        if self.script_dir is None:
+            self.script_dir = self.install_dir
+
+        # Let install_dir get set by install_lib command, which in turn
         # gets its info from the install command, and takes into account
         # --prefix and --home and all that other crud.
-        #
-        self.set_undefined_options('install_lib',('install_dir','install_dir'))
+        self.set_undefined_options('install_lib',
+            ('install_dir','install_dir')
+        )         
+        # Likewise, set default script_dir from 'install_scripts.install_dir'
+        self.set_undefined_options('install_scripts',
+            ('install_dir', 'script_dir')
+        )
+
         site_packages = get_python_lib()       
         instdir = self.install_dir
 
@@ -102,10 +113,10 @@ class easy_install(Command):
 
         elif not self.multi_version:
             # explicit false set from Python code; raise an error
-            raise RuntimeError(
+            raise DistutilsArgError(
                 "Can't do single-version installs outside site-packages"
             )
-            
+
         self.index_url = self.index_url or "http://www.python.org/pypi"
         if self.package_index is None:
             self.package_index = self.create_index(self.index_url)
@@ -117,10 +128,13 @@ class easy_install(Command):
                 self.package_index.scan_url(link)
 
         if not self.args:
-            parser.error("No urls, filenames, or requirements specified")
+            raise DistutilsArgError(
+                "No urls, filenames, or requirements specified (see --help)")
         elif len(self.args)>1 and self.build_directory is not None:
-            parser.error("Build directory can only be set when using one URL")
-            
+            raise DistutilsArgError(
+                "Build directory can only be set when using one URL"   
+            )
+
     def run(self):
         for spec in self.args:
             self.easy_install(spec)
@@ -139,6 +153,7 @@ class easy_install(Command):
             print "Installing", os.path.basename(download)
             for dist in self.install_eggs(download, self.zip_ok, tmpdir):
                 self.package_index.add(dist)
+                self.install_egg_scripts(dist)
                 print self.installation_report(dist)
 
         finally:
@@ -147,16 +162,42 @@ class easy_install(Command):
 
 
 
+    def install_egg_scripts(self, dist):
+        metadata = dist.metadata
+        if self.exclude_scripts or not metadata.metadata_isdir('scripts'):
+            return
 
+        from distutils.command.build_scripts import first_line_re
 
+        for script_name in metadata.metadata_listdir('scripts'):
+            target = os.path.join(self.script_dir, script_name)
 
+            print "Installing", script_name, "to", target
 
+            script_text = metadata.get_metadata('scripts/'+script_name)
+            script_text = script_text.replace('\r','\n')
+            first, rest = script_text.split('\n',1)
 
+            match = first_line_re.match(first)
+            options = ''
+            if match:
+                options = match.group(1) or ''
+                if options:
+                    options = ' '+options
 
+            spec = '%s==%s' % (dist.name,dist.version)
 
+            script_text = '\n'.join([
+                "#!%s%s" % (os.path.normpath(sys.executable),options),
+                "# EASY-INSTALL-SCRIPT: %r,%r" % (spec, script_name),
+                "import pkg_resources",
+                "pkg_resources.run_main(%r, %r)" % (spec, script_name)
+            ])
 
-
-
+            f = open(target,"w")
+            f.write(script_text)
+            f.close()
+            
 
 
 
@@ -329,7 +370,7 @@ class PthDistributions(AvailableDistributions):
 def main(argv, cmds={'easy_install':easy_install}):
     from setuptools import setup
     try:
-        setup(cmdclass = cmds, script_args = ['easy_install']+argv)
+        setup(cmdclass = cmds, script_args = ['-q','easy_install', '-v']+argv)
     except RuntimeError, v:
         print >>sys.stderr,"error:",v
         sys.exit(1)

pkg_resources.py

@@ -22,7 +22,7 @@ __all__ = [
     'InvalidOption', 'Distribution', 'Requirement', 'yield_lines',
     'get_importer', 'find_distributions', 'find_on_path', 'register_finder',
     'split_sections', 'declare_namespace', 'register_namespace_handler',
-    'safe_name', 'safe_version'
+    'safe_name', 'safe_version', 'run_main',
 ]
 
 import sys, os, zipimport, time, re, imp
@@ -102,12 +102,12 @@ def compatible_platforms(provided,required):
     # XXX all the tricky cases go here
     return False
 
-
-
-
-
-
-
+def run_main(dist_spec, script_name):
+    """Locate distribution `dist_spec` and run its `script_name` script"""
+    import __main__
+    __main__.__dict__.clear()
+    __main__.__dict__.update({'__name__':'__main__'})
+    require(dist_spec)[0].metadata.run_script(script_name, __main__.__dict__)
 
 
 
@@ -135,6 +135,33 @@ class IMetadataProvider:
        Leading and trailing whitespace is stripped from each line, and lines
        with ``#`` as the first non-blank character are omitted."""
 
+    def metadata_isdir(name):
+        """Is the named metadata a directory?  (like ``os.path.isdir()``)"""
+
+    def metadata_listdir(name):
+        """List of metadata names in the directory (like ``os.listdir()``)"""
+
+    def run_script(script_name, namespace):
+        """Execute the named script in the supplied namespace dictionary"""
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 class IResourceProvider(IMetadataProvider):
     """An object that provides access to package resources"""
 
@@ -162,6 +189,20 @@ class IResourceProvider(IMetadataProvider):
     def resource_listdir(resource_name):
         """List of resource names in the directory (like ``os.listdir()``)"""
 
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 class AvailableDistributions(object):
     """Searchable snapshot of distributions on a search path"""
 
@@ -460,9 +501,10 @@ def require(*requirements):
     """
 
     requirements = parse_requirements(requirements)
-
-    for dist in AvailableDistributions().resolve(requirements):
+    to_install = AvailableDistributions().resolve(requirements)
+    for dist in to_install:
         dist.install_on(sys.path)
+    return to_install
 
 
 def safe_name(name):
@@ -489,7 +531,6 @@ def safe_version(version):
 
 
 
-
 class NullProvider:
     """Try to implement resources and metadata for arbitrary PEP 302 loaders"""
 
@@ -502,40 +543,79 @@ class NullProvider:
         self.module_path = os.path.dirname(getattr(module, '__file__', ''))
 
     def get_resource_filename(self, manager, resource_name):
-        return self._fn(resource_name)
+        return self._fn(self.module_path, resource_name)
 
     def get_resource_stream(self, manager, resource_name):
-        return open(self._fn(resource_name), 'rb')
+        return open(self._fn(self.module_path, resource_name), 'rb')
 
     def get_resource_string(self, manager, resource_name):
-        return self._get(self._fn(resource_name))
+        return self._get(self._fn(self.module_path, resource_name))
 
     def has_resource(self, resource_name):
-        return self._has(self._fn(resource_name))
+        return self._has(self._fn(self.module_path, resource_name))
 
     def has_metadata(self, name):
-        if not self.egg_info:
-            raise NotImplementedError("Only .egg supports metadata")
-        return self._has(os.path.join(self.egg_info, *name.split('/')))
+        return self.egg_info and self._has(self._fn(self.egg_info,name))
 
     def get_metadata(self, name):
         if not self.egg_info:
-            raise NotImplementedError("Only .egg supports metadata")
-        return self._get(os.path.join(self.egg_info, *name.split('/')))
+            return ""
+        return self._get(self._fn(self.egg_info,name))
 
     def get_metadata_lines(self, name):
         return yield_lines(self.get_metadata(name))
 
-    def resource_isdir(self,name): return False
+    def resource_isdir(self,name):
+        return self._isdir(self._fn(self.module_path, resource_name))
+
+    def metadata_isdir(self,name):
+        return self.egg_info and self._isdir(self._fn(self.egg_info,name))
+
 
     def resource_listdir(self,name):
+        return self._listdir(self._fn(self.egg_info,name))
+
+    def metadata_listdir(self,name):
+        if self.egg_info:
+            return self._listdir(self._fn(self.egg_info,name))
         return []
 
+    def run_script(self,script_name,namespace):
+        script = 'scripts/'+script_name
+        if not self.has_metadata(script):
+            raise ResolutionError("No script named %r" % script_name)
+        script_text = self.get_metadata(script).replace('\r\n','\n')
+        script_text = script_text.replace('\r','\n')
+        script_filename = self._fn(self.egg_info,script)
+
+        if os.path.exists(script_filename):
+            execfile(script_filename, namespace, namespace)
+        else:
+            from linecache import cache
+            cache[script_filename] = (
+                len(script_text), 0, script_text.split('\n'), script_filename
+            )
+            script_code = compile(script_text,script_filename,'exec')
+            exec script_code in namespace, namespace
+
     def _has(self, path):
         raise NotImplementedError(
             "Can't perform this operation for unregistered loader type"
         )
 
+    def _isdir(self, path):
+        raise NotImplementedError(
+            "Can't perform this operation for unregistered loader type"
+        )
+
+    def _listdir(self, path):
+        raise NotImplementedError(
+            "Can't perform this operation for unregistered loader type"
+        )
+
+    def _fn(self, base, resource_name):
+        return os.path.join(base, *resource_name.split('/'))
+
     def _get(self, path):
         if hasattr(self.loader, 'get_data'):
             return self.loader.get_data(path)
@@ -543,9 +623,6 @@ class NullProvider:
             "Can't perform this operation for loaders without 'get_data()'"
         )
 
-    def _fn(self, resource_name):
-        return os.path.join(self.module_path, *resource_name.split('/'))
-
 register_loader_type(object, NullProvider)
 
 
@@ -566,6 +643,11 @@ register_loader_type(object, NullProvider)
 
 
 
+
+
+
+
+
 
 
 
@@ -578,7 +660,7 @@ class DefaultProvider(NullProvider):
     def __init__(self,module):
         NullProvider.__init__(self,module)
         self._setup_prefix()
-        
+
     def _setup_prefix(self):
         # we assume here that our metadata may be nested inside a "basket"
         # of multiple eggs; that's why we use module_path instead of .archive
@@ -597,11 +679,11 @@ class DefaultProvider(NullProvider):
     def _has(self, path):
         return os.path.exists(path)
 
-    def resource_isdir(self,name):
-        return os.path.isdir(self._fn(name))
+    def _isdir(self,path):
+        return os.path.isdir(path)
 
-    def resource_listdir(self,name):
-        return os.listdir(self._fn(name))
+    def _listdir(self,path):
+        return os.listdir(path)
 
     def _get(self, path):
         stream = open(path, 'rb')
@@ -628,10 +710,6 @@ class ZipProvider(DefaultProvider):
             return path[len(self.zip_pre):]
         return path
 
-    def _has(self, path): return self._short_name(path) in self.zipinfo or self.resource_isdir(path)
-
-    def _get(self, path): return self.loader.get_data(path)
-
     def get_resource_stream(self, manager, resource_name):
         return StringIO(self.get_resource_string(manager, resource_name))
 
@@ -649,27 +727,19 @@ class ZipProvider(DefaultProvider):
 
         return self._extract_resource(manager, resource_name)
 
-    def resource_isdir(self, resource_name):
-        if resource_name.endswith('/'):
-            resource_name = resource_name[:-1]
-        return resource_name in self._index()
-
-    def resource_listdir(self, resource_name):
-        if resource_name.endswith('/'):
-            resource_name = resource_name[:-1]
-        return list(self._index().get(resource_name, ()))
-
     def _extract_directory(self, manager, resource_name):
         if resource_name.endswith('/'):
             resource_name = resource_name[:-1]
         for resource in self.resource_listdir(resource_name):
             last = self._extract_resource(manager, resource_name+'/'+resource)
         return os.path.dirname(last)    # return the directory path
-        
+
+
+
     def _extract_resource(self, manager, resource_name):
         if self.resource_isdir(resource_name):
             return self._extract_dir(resource_name)
-            
+
         parts = resource_name.split('/')
         zip_path = os.path.join(self.module_path, *parts)
         zip_stat = self.zipinfo[os.path.join(*self.prefix+parts)]
@@ -704,6 +774,9 @@ class ZipProvider(DefaultProvider):
             self.eagers = eagers
         return self.eagers
 
+
+
+
     def _index(self):
         try:
             return self._dirindex
@@ -724,14 +797,23 @@ class ZipProvider(DefaultProvider):
             self._dirindex = ind
             return ind
 
+    def _has(self, path):
+        return self._short_name(path) in self.zipinfo or self._isdir(path)
 
-register_loader_type(zipimport.zipimporter, ZipProvider)
-
-
+    def _isdir(self,path):
+        path = self._short_name(path).replace(os.sep, '/')
+        if path.endswith('/'): path = path[:-1]
+        return path in self._index()
 
+    def _listdir(self,path):
+        path = self._short_name(path).replace(os.sep, '/')
+        if path.endswith('/'): path = path[:-1]
+        return list(self._index().get(path, ()))
 
+    _get = NullProvider._get
 
 
+register_loader_type(zipimport.zipimporter, ZipProvider)
 
 
 
@@ -886,7 +968,7 @@ def find_in_zip(importer,path_item):
             subpath = os.path.join(path_item, subitem)
             for dist in find_in_zip(zipimport.zipimporter(subpath), subpath):
                 yield dist
-        
+
 register_finder(zipimport.zipimporter,find_in_zip)
 
 
@@ -1148,7 +1230,7 @@ def parse_version(s):
 
 class Distribution(object):
     """Wrap an actual or potential sys.path entry w/metadata"""
-    
+
     def __init__(self,
         path_str, metadata=None, name=None, version=None,
         py_version=PY_MAJOR, platform=None, distro_type = EGG_DIST

setup.py

@@ -2,8 +2,9 @@
 """Distutils setup file, used to install or test 'setuptools'"""
 
 VERSION = "0.4a1"
+import sys
 from setuptools import setup, find_packages, Require
-from distutils.version import LooseVersion
+PYVER = sys.version[:3]
 
 setup(
     name="setuptools",
@@ -38,11 +39,10 @@ setup(
         Require('PyUnit', None, 'unittest', "http://pyunit.sf.net/"),
     ],
 
-    
     packages = find_packages(),
     py_modules = ['pkg_resources', 'easy_install'],
     scripts = ['easy_install.py'],
-    extra_path = ('setuptools', 'setuptools-%s.egg' % VERSION),
+    extra_path = ('setuptools', 'setuptools-%s-py%s.egg' % (VERSION,PYVER)),
 
     classifiers = [f.strip() for f in """
     Development Status :: 3 - Alpha