Implement ``namespace_packages`` keyword to ``setup()``. Added keyword

summary to setuptools doc.  Begin work on ``zip_safe`` flag.

--HG--
branch : setuptools
extra : convert_revision : svn%3A6015fed2-1504-0410-9fe1-9d1591cc4771/sandbox/trunk/setuptools%4041113

setuptools.txt

@@ -133,6 +133,56 @@ arguments do (except for the metadata ones), and the various ways you might use
 them in your own project(s).
 
 
+New and Changed ``setup()`` Keywords
+====================================
+
+The following keyword arguments to ``setup()`` are added or changed by
+``setuptools``.  All of them are optional; you do not have to supply them
+unless you need the associated ``setuptools`` feature.
+
+``package_data``
+    A dictionary mapping package names to lists of glob patterns.  For a
+    complete description and examples, see the section below on `Including
+    Data Files`_.
+
+``zip_safe``
+    A boolean (True or False) flag specifying whether the project can be
+    safely installed and run from a zip file.  If this argument is not
+    supplied, the ``bdist_egg`` command will have to analyze all of your
+    project's contents for possible problems each time it buids an egg.
+
+``install_requires``
+    A string or list of strings specifying what other distributions need to
+    be installed when this one is.  See the section below on `Declaring
+    Dependencies`_ for details and examples of the format of this argument.
+    
+``extras_require``
+    A dictionary mapping names of "extras" (optional features of your project)
+    to strings or lists of strings specifying what other distributions must be
+    installed to support those features.  See the section below on `Declaring
+    Dependencies`_ for details and examples of the format of this argument.
+
+``test_suite``
+    A string naming a ``unittest.TestCase`` subclass (or a module containing
+    one or more of them, or a method of such a subclass), or naming a function
+    that can be called with no arguments and returns a ``unittest.TestSuite``.
+    Specifying this argument enables use of the `test`_ command to run the
+    specified test suite, e.g. via ``setup.py test``.  See the section on the
+    `test`_ command below for more details.
+
+``namespace_packages``
+    A list of strings naming the project's "namespace packages".  A namespace
+    package is a package that may be split across multiple project
+    distributions.  For example, Zope 3's ``zope`` package is a namespace
+    package, because subpackages like ``zope.interface`` and ``zope.publisher``
+    may be distributed separately.  The egg runtime system can automatically
+    merge such subpackages into a single parent package at runtime, as long
+    as you declare them in each project that contains any subpackages of the
+    namespace package, and as long as the namespace package's ``__init__.py``
+    does not contain any code.  See the section below on `Namespace Packages`_
+    for more information.
+
+
 Declaring Dependencies
 ======================
 
@@ -406,7 +456,11 @@ a quick example of converting code that uses ``__file__`` to use
 .. _Resource Management API: http://peak.telecommunity.com/DevCenter/PythonEggs#resource-management
 .. _Accessing Package Resources: http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources
 
-.. XXX put doc about zip_safe flag here, once it's implemented
+
+Setting the ``zip_safe`` flag
+-----------------------------
+
+XXX put doc about zip_safe flag here, once it's implemented
 
 
 "Development Mode"
@@ -483,6 +537,62 @@ Building Extensions written with Pyrex
 XXX
 
 
+Namespace Packages
+==================
+
+Sometimes, a large package is more useful if distributed as a collection of
+smaller eggs.  However, Python does not normally allow the contents of a
+package to be retrieved from more than one location.  "Namespace packages"
+are a solution for this problem.  When you declare a package to be a namespace
+package, it means that the package has no meaningful contents in its
+``__init__.py``, and that it is merely a container for modules and subpackages.
+
+The ``pkg_resources`` runtime will automatically ensure that the contents of
+namespace packages that are spread over multiple eggs or directories are
+combined into a single virtual package.
+
+The ``namespace_packages`` argument to ``setup()`` lets you declare your
+project's namespace packages, so that they will be included in your project's
+metadata.  Then, the runtime will automatically detect this when it adds the
+distribution to ``sys.path``, and ensure that the packages are properly merged.
+
+The argument should list the namespace packages that the egg participates in.
+For example, the ZopeInterface project might do this::
+
+    setup(
+        # ...
+        namespace_packages = ['zope']
+    )
+
+because it contains a ``zope.interface`` package that lives in the ``zope``
+namespace package.  Similarly, a project for a standalone ``zope.publisher``
+would also declare the ``zope`` namespace package.
+
+Namespace packages don't have to be top-level packages.  For example, Zope 3's
+``zope.app`` package is a namespace package, and in the future PEAK's
+``peak.util`` package will be too.
+
+Note, by the way, that your project's source tree must include the namespace
+packages' ``__init__.py`` files (and the ``__init__.py`` of any parent
+packages), in a normal Python package layout.  These ``__init__.py`` files
+should not contain any code or data, because only *one* egg's ``__init__.py``
+files will be used to construct the parent packages in memory at runtime, and
+there is no guarantee which egg will be used.
+
+For example, if both ``zope.interface`` and ``zope.publisher`` have been
+installed from separate distributions, it is unspecified which of the two
+distributions' ``zope/__init__.py`` files will be used to create the ``zope``
+package in memory.  Therefore, it is better not to have any code or data in
+a namespace package's ``__init__`` module, so as to prevent any complications.
+
+(This is one reason the concept is called a "namespace package": it is a
+package that exists *only* to provide a namespace under which other modules or
+packages are gathered.  In Java, for example, namespace packages are often used
+just to avoid naming collisions between different projects, using package names
+like ``org.apache`` as a namespace for packages that are part of apache.org
+projects.)
+
+
 -----------------
 Command Reference
 -----------------
@@ -916,6 +1026,8 @@ options`_ (listed under the `saveopts`_ command, above) to determine which
 distutils configuration file the option will be added to (or removed from).
 
 
+.. _test:
+
 ``test`` - Build package and run a unittest suite
 =================================================
 
@@ -1079,6 +1191,8 @@ Release Notes/Change History
    This is used by the ``easy_install`` command to find possibly-conflicting
    "unmanaged" packages when installing the distribution.
 
+ * Added ``zip_safe`` and ``namespace_packages`` arguments to ``setup()``.
+
  * Fixed the swapped ``-d`` and ``-b`` options of ``bdist_egg``.
 
 0.5a8

setuptools/command/easy_install.py

@@ -21,7 +21,7 @@ from distutils.errors import DistutilsArgError, DistutilsOptionError, DistutilsE
 from setuptools.archive_util import unpack_archive
 from setuptools.package_index import PackageIndex, parse_bdist_wininst
 from setuptools.package_index import URL_SCHEME
-from setuptools.command import bdist_egg
+from setuptools.command import bdist_egg, egg_info
 from pkg_resources import *
 
 __all__ = [
@@ -697,6 +697,7 @@ PYTHONPATH, or by being added to sys.path by your code.)
 
     def build_and_install(self, setup_script, setup_base, zip_ok):
         sys.modules.setdefault('distutils.command.bdist_egg', bdist_egg)
+        sys.modules.setdefault('distutils.command.bdist_egg', egg_info)
 
         args = ['bdist_egg', '--dist-dir']
         if self.verbose>2:
@@ -735,7 +736,6 @@ PYTHONPATH, or by being added to sys.path by your code.)
             shutil.rmtree(dist_dir)
             log.set_verbosity(self.verbose) # restore our log verbosity
 
-
     def update_pth(self,dist):
         if self.pth_file is None:
             return

setuptools/command/egg_info.py

@@ -9,7 +9,7 @@ from distutils.errors import *
 from distutils import log
 from pkg_resources import parse_requirements, safe_name, \
     safe_version, yield_lines
-
+from setuptools.dist import iter_distribution_names
 
 class egg_info(Command):
 
@@ -83,8 +83,8 @@ class egg_info(Command):
     def run(self):
         # Make the .egg-info directory, then write PKG-INFO and requires.txt
         self.mkpath(self.egg_info)
-
         log.info("writing %s" % os.path.join(self.egg_info,'PKG-INFO'))
+
         if not self.dry_run:
             metadata = self.distribution.metadata
             metadata.version, oldver = self.egg_version, metadata.version
@@ -96,15 +96,16 @@ class egg_info(Command):
             finally:
                 metadata.name, metadata.version = oldname, oldver
 
+        self.write_namespace_packages()
         self.write_requirements()
         self.write_toplevel_names()
+
         if os.path.exists(os.path.join(self.egg_info,'depends.txt')):
             log.warn(
                 "WARNING: 'depends.txt' will not be used by setuptools 0.6!\n"
                 "Use the install_requires/extras_require setup() args instead."
             )
 
-
     def write_requirements(self):
         dist = self.distribution
         if not getattr(dist,'install_requires',None) and \
@@ -120,7 +121,6 @@ class egg_info(Command):
                 f.write('\n\n[%s]\n%s' % (extra, '\n'.join(yield_lines(reqs))))
             f.close()
 
-
     def tagged_version(self):
         version = self.distribution.get_version()
         if self.tag_build:
@@ -144,21 +144,62 @@ class egg_info(Command):
 
 
     def write_toplevel_names(self):
-        pkgs = dict.fromkeys(self.distribution.packages or ())
-        pkgs.update(dict.fromkeys(self.distribution.py_modules or ()))
-        for ext in self.distribution.ext_modules or ():
-            if isinstance(ext,tuple):
-                name,buildinfo = ext
-            else:
-                name = ext.name
-            pkgs[name]=1
-        pkgs = dict.fromkeys([k.split('.',1)[0] for k in pkgs])
-        toplevel = os.path.join(self.egg_info,"top_level.txt")
+        pkgs = dict.fromkeys(
+            [k.split('.',1)[0]
+                for k in iter_distribution_names(self.distribution)
+            ]
+        )
+        toplevel = os.path.join(self.egg_info, "top_level.txt")
         log.info("writing list of top-level names to %s" % toplevel)
         if not self.dry_run:
             f = open(toplevel, 'wt')
             f.write('\n'.join(pkgs))
             f.write('\n')
             f.close()
+
+
+
+
+
+
+    def write_namespace_packages(self):
+        nsp = getattr(self.distribution,'namespace_packages',None)
+        if nsp is None:
+            return
+
+        filename = os.path.join(self.egg_info,"namespace_packages.txt")
+
+        if nsp:
+            log.info("writing %s", filename)
+            if not self.dry_run:
+                f = open(filename, 'wt')
+                f.write('\n'.join(nsp))
+                f.write('\n')
+                f.close()
+
+        elif os.path.exists(filename):
+            log.info("deleting %s", filename)
+            if not self.dry_run:
+                os.unlink(filename)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 

setuptools/dist.py

@@ -90,6 +90,8 @@ class Distribution(_Distribution):
         self.install_requires = []
         self.extras_require = {}
         self.dist_files = []
+        self.zip_safe = None
+        self.namespace_packages = None
         _Distribution.__init__(self,attrs)
         if not have_package_data:
             from setuptools.command.build_py import build_py
@@ -100,19 +102,17 @@ class Distribution(_Distribution):
         self.cmdclass.setdefault('install_lib',install_lib)
         self.cmdclass.setdefault('sdist',sdist)
 
+    def parse_command_line(self):
+        """Process features after parsing command line options"""
+        result = _Distribution.parse_command_line(self)
+        if self.features:
+            self._finalize_features()
+        return result
 
 
-
-
-
-
-
-
-
-
-
-
-
+    def _feature_attrname(self,name):
+        """Convert feature name to corresponding option attribute name"""
+        return 'with_'+name.replace('-','_')
 
 
 
@@ -123,10 +123,8 @@ class Distribution(_Distribution):
 
     def finalize_options(self):
         _Distribution.finalize_options(self)
-
         if self.features:
             self._set_global_opts_from_features()
-
         if self.extra_path:
             raise DistutilsSetupError(
                 "The 'extra_path' parameter is not needed when using "
@@ -148,19 +146,21 @@ class Distribution(_Distribution):
                 "strings or lists of strings containing valid project/version "
                 "requirement specifiers."
             )
-
-    def parse_command_line(self):
-        """Process features after parsing command line options"""
-        result = _Distribution.parse_command_line(self)
-        if self.features:
-            self._finalize_features()
-        return result
-
-
-    def _feature_attrname(self,name):
-        """Convert feature name to corresponding option attribute name"""
-        return 'with_'+name.replace('-','_')
-
+        if self.namespace_packages is not None:
+            try:
+                assert ''.join(self.namespace_packages)!=self.namespace_packages
+            except (TypeError,ValueError,AttributeError,AssertionError):
+                raise DistutilsSetupError(
+                    "'namespace_packages' must be a sequence of strings"
+                )
+            for nsp in self.namespace_packages:
+                for name in iter_distribution_names(self):
+                    if name.startswith(nsp+'.'): break
+                else:
+                    raise DistutilsSetupError(
+                        "Distribution contains no modules or packages for " +
+                        "namespace package %r" % nsp
+                    )
 
     def _set_global_opts_from_features(self):
         """Add --with-X/--without-X options based on optional features"""
@@ -488,6 +488,47 @@ class Distribution(_Distribution):
                 d.setdefault(cmd,{})[opt] = val
 
         return d
+
+
+def iter_distribution_names(distribution):
+    """Yield all packages, modules, and extensions declared by distribution"""
+
+    for pkg in distribution.packages or ():
+        yield pkg
+
+    for module in distribution.py_modules or ():
+        yield module
+
+    for ext in distribution.ext_modules or ():
+        if isinstance(ext,tuple):
+            name,buildinfo = ext
+            yield name
+        else:
+            yield ext.name
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
 class Feature: