Significantly enhanced support and docs for "non-root" installation,

including both "virtual" and PYTHONPATH-based installs.  The activation
precedence of distributions has also changed so that PYTHONPATH-based
non-root installs can include eggs that override system-defined packages
(whether managed or unmanaged).  This version should eliminate most
common installation complaints from non-root Python users.
Note: this version includes a hacked 'site.py' to support processing
.pth files in directories that come *before* site-packages on sys.path.
However, because of its placement, it should only come into play when
a user puts the setuptools .egg file *directly* on PYTHONPATH, so it
doesn't affect "virtual" or "root" installations.  It's strictly to
provide support for luddites who refuse to give up their
existing non-root PYTHONPATH setup unless you pry it from their cold,
dead hands.  :)

--HG--
branch : setuptools
extra : convert_revision : svn%3A6015fed2-1504-0410-9fe1-9d1591cc4771/sandbox/trunk/setuptools%4041262

EasyInstall.txt

@@ -25,6 +25,8 @@ Using "Easy Install"
 ====================
 
 
+.. _installation instructions:
+
 Installing "Easy Install"
 -------------------------
 
@@ -718,12 +720,46 @@ configuration file will work correctly no matter what Python version you use,
 now or in the future.)
 
 If you are on a Linux, BSD, Cygwin, or other similar Unix-like operating
-system, you should create a ``~/lib/python2.x/site-packages`` directory
-instead.  (Note: Ian Bicking has created a script that can automate most of the
-process that follows; see http://svn.colorstudy.com/home/ianb/non_root_python.py
-for details.)
+system, you have a couple of different options.  You can create a "virtual"
+Python installation, which uses its own library directories and some symlinks
+to the site-wide Python.  Or, you can use a "traditional" ``PYTHONPATH``-based
+installation, which isn't as flexible, but which you may find more familiar,
+especially if you already have a custom ``PYTHONPATH`` set up.
+
+
+Creating a "Virtual" Python
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the simplest case, your virtual Python installation will live under the
+``~/lib/python2.x``, ``~/include/python2.x``, and ``~/bin`` directories.  Just
+download `virtual-python.py`_ and run it using the site-wide Python.  If you
+want to customize the location, you can use the ``--prefix`` option to specify
+an installation base directory in place of ``~``.  (Use ``--help`` to get the
+complete list of options.)
+
+.. _virtual-python.py: http://peak.telecommunity.com/dist/virtual-python.py
+
+When you're done, you'll have a ``~/bin/python`` executable that's linked to
+the local Python installation and inherits all its current libraries, but which
+allows you to add as many new libraries as you want.  Simply use this new
+Python in place of your system-defined one, and you can modify it as you like
+without breaking anything that relies on the system Python.  You'll also still
+need to follow the standard `installation instructions`_ to install setuptools
+and EasyInstall, using your new ``~/bin/python`` executable in place of the
+system Python.
+
+Note that if you were previously setting a ``PYTHONPATH`` and/or had other
+special configuration options in your ``~/.pydistutils.cfg``, you may need to
+remove these settings *before* running ``virtual-python.py``.  You should
+also make sure that the ``~/bin`` directory (or whatever directory you choose)
+is on your ``PATH``, because that is where EasyInstall will install new Python
+scripts.
+
+If you'd prefer to do the installation steps by hand, or just want to know what
+the script will do, here are the steps.  (If you don't care how it works, you
+can just skip the rest of this section.)
 
-You will need to know your Python version's ``sys.prefix`` and
+First, you will need to know your Python version's ``sys.prefix`` and
 ``sys.exec_prefix``, which you can find out by running::
 
     python -c "import sys; print sys.prefix; print sys.exec_prefix"
@@ -761,13 +797,43 @@ is on a different filesystem), you should use ``copy -p`` instead of ``ln``.
 Do NOT use a symlink!  The Python binary must be copied or hardlinked,
 otherwise it will use the system ``site-packages`` directory and not yours.
 
-Note that if you were previously setting a ``PYTHONPATH`` and/or had other
-special configuration options in your ``~/.pydistutils.cfg``, you may need to
-remove these settings and relocate any older installed modules to your
-new ``~/lib/python2.x/site-packages`` directory.  Also note that you must now
-make sure to use the ``~/bin/python`` executable instead of the system Python,
-and ideally you should put the ``~/bin`` directory first on your ``PATH`` as
-well, because that is where EasyInstall will install new Python scripts.
+You can now proceed with the standard `installation instructions`_.
+
+
+"Traditional" ``PYTHONPATH``-based Installation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This installation method is not as robust or as flexible as `creating a
+"virtual" python`_ installation, as it uses various tricks to fool Python into
+processing ``.pth`` files where it normally wouldn't.  We suggest you try the
+virtual Python approach first, as we are providing this method mainly for
+people who just can't get past their unshakeable belief that creating a virtual
+python is somehow "Not Right", or that putting stuff on ``PYTHONPATH`` "Should
+Just Work, Darnit."  So, if you're not one of those people, you don't
+need these instructions.  :-)
+
+Assuming that you want to install packages in a directory called ``~/py-lib``,
+and scripts in ``~/bin``, here's what you need to do:
+
+First, edit ``~/.pydistutils.cfg`` to include these settings::
+
+    [install]
+    install_lib = ~/py-lib
+    install_scripts = ~/bin
+    
+    [easy_install]
+    site_dirs = ~/py_lib
+
+Be sure to do this *before* you try to run the ``ez_setup.py`` installation
+script.  Then, follow the standard `installation instructions`_, but take
+careful note of the full pathname of the ``.egg`` file that gets installed, so
+that you can add it to your ``PYTHONPATH``, along with ``~/py_lib``.
+
+You *must* add the setuptools egg file to your ``PYTHONPATH`` manually, or it
+will not work, and neither will any other packages you install with
+EasyInstall.  You will not, however, have to manually add any other
+packages to the ``PYTHONPATH``; EasyInstall will take care of them for you, as
+long as the setuptools egg is explicitly listed in ``PYTHONPATH``.
 
 
 Release Notes/Change History
@@ -778,6 +844,12 @@ Known Issues
    time out or be missing a file.
 
 0.6a6
+ * Added support for "traditional" PYTHONPATH-based non-root installation, and
+   also the convenient ``virtual-python.py`` script, based on a contribution
+   by Ian Bicking.  The setuptools egg now contains a hacked ``site`` module
+   that makes the PYTHONPATH-based approach work with .pth files, so that you
+   can get the full EasyInstall feature set on such installations.
+
  * Added ``--no-deps`` option.
 
  * Improved Windows ``.exe`` script wrappers so that the script can have the

api_tests.txt

@@ -182,12 +182,12 @@ once)::
     ['...example.com...', '...pkg_resources...', '...pkg_resources...']
 
 And you can specify the path entry a distribution was found under, using the
-optional second parameter to ``add()``
+optional second parameter to ``add()``::
 
+    >>> ws = WorkingSet([])
     >>> ws.add(dist,"foo")
-    >>> ws.add(dist,"bar")
     >>> ws.entries
-    ['http://example.com/something', ..., 'foo', 'bar']
+    ['foo']
 
 But even if a distribution is found under multiple path entries, it still only
 shows up once when iterating the working set:
@@ -222,14 +222,14 @@ again for new distributions added thereafter::
     >>> def added(dist): print "Added", dist
     >>> ws.subscribe(added)
     Added Bar 0.9
-    >>> foo12 = Distribution(project_name="Foo", version="1.2") 
+    >>> foo12 = Distribution(project_name="Foo", version="1.2", location="f12") 
     >>> ws.add(foo12)
     Added Foo 1.2
 
 Note, however, that only the first distribution added for a given project name
 will trigger a callback, even during the initial ``subscribe()`` callback::
 
-    >>> foo14 = Distribution(project_name="Foo", version="1.4") 
+    >>> foo14 = Distribution(project_name="Foo", version="1.4", location="f14") 
     >>> ws.add(foo14)   # no callback, because Foo 1.2 is already active
 
     >>> ws = WorkingSet([])

pkg_resources.py

@@ -356,7 +356,7 @@ class WorkingSet(object):
         self.entry_keys.setdefault(entry, [])
         self.entries.append(entry)
         for dist in find_distributions(entry, True):
-            self.add(dist, entry)
+            self.add(dist, entry, False)
 
 
     def __contains__(self,dist):
@@ -421,7 +421,7 @@ class WorkingSet(object):
                     seen[key]=1
                     yield self.by_key[key]
 
-    def add(self, dist, entry=None):
+    def add(self, dist, entry=None, insert=True):
         """Add `dist` to working set, associated with `entry`
 
         If `entry` is unspecified, it defaults to the ``.location`` of `dist`.
@@ -432,23 +432,23 @@ class WorkingSet(object):
         doesn't already have a distribution in the set.  If it's added, any
         callbacks registered with the ``subscribe()`` method will be called.
         """
+        if insert:
+            dist.insert_on(self.entries, entry)
+
         if entry is None:
             entry = dist.location
-
-        if entry not in self.entry_keys:
-            self.entries.append(entry)
-            self.entry_keys[entry] = []
+        keys = self.entry_keys.setdefault(entry,[])
 
         if dist.key in self.by_key:
             return      # ignore hidden distros
 
         self.by_key[dist.key] = dist
-        keys = self.entry_keys[entry]
         if dist.key not in keys:
             keys.append(dist.key)
 
         self._added_new(dist)
 
+
     def resolve(self, requirements, env=None, installer=None):
         """List all distributions needed to (recursively) meet `requirements`
 
@@ -1837,12 +1837,12 @@ class Distribution(object):
     def activate(self,path=None):
         """Ensure distribution is importable on `path` (default=sys.path)"""
         if path is None: path = sys.path
-        if self.location not in path:
-            path.append(self.location)
+        self.insert_on(path)
         if path is sys.path:
             fixup_namespace_packages(self.location)
             map(declare_namespace, self._get_metadata('namespace_packages.txt'))
 
+
     def egg_name(self):
         """Return what this distribution's standard .egg filename should be"""
         filename = "%s-%s-py%s" % (
@@ -1907,6 +1907,47 @@ class Distribution(object):
         """Return the EntryPoint object for `group`+`name`, or ``None``"""
         return self.get_entry_map(group).get(name)
 
+    def insert_on(self, path, loc = None):
+        """Insert self.location in path before its nearest parent directory"""
+        loc = loc or self.location
+        if not loc: return
+        if path is sys.path:
+            self.check_version_conflict()
+        best, pos = 0, -1
+        for p,item in enumerate(path):
+            if loc.startswith(item) and len(item)>best and loc<>item:
+                best, pos = len(item), p
+        if pos==-1:
+            if loc not in path: path.append(loc)
+        elif loc not in path[:pos+1]:
+            while loc in path: path.remove(loc)
+            path.insert(pos,loc)
+
+
+
+    def check_version_conflict(self):
+        if self.key=='setuptools':
+            return      # ignore the inevitable setuptools self-conflicts  :(
+
+        nsp = dict.fromkeys(self._get_metadata('namespace_packages.txt'))
+
+        for modname in self._get_metadata('top_level.txt'):
+            if modname not in sys.modules or modname in nsp:
+                continue
+
+            fn = getattr(sys.modules[modname], '__file__', None)
+            if fn and fn.startswith(self.location):
+                continue
+
+            from warnings import warn
+            warn(
+                "Module %s was already imported from %s, but %s is being added"
+                " to sys.path" % (modname, fn, self.location)
+            )
+
+
+
+
 
 
 
@@ -2165,9 +2206,9 @@ iter_entry_points = working_set.iter_entry_points
 add_activation_listener = working_set.subscribe
 run_script = working_set.run_script
 run_main = run_script   # backward compatibility
-
 # Activate all distributions already on sys.path, and ensure that
 # all distributions added to the working set in the future (e.g. by
 # calling ``require()``) will get activated as well.
 add_activation_listener(lambda dist: dist.activate())
+working_set.entries=[]; map(working_set.add_entry,sys.path) # match order
 

pkg_resources.txt

@@ -1488,6 +1488,17 @@ File/Path Utilities
 Release Notes/Change History
 ----------------------------
 
+0.6a6
+ * Activated distributions are now inserted in ``sys.path`` (and the working
+   set) just before the directory that contains them, instead of at the end.
+   This allows e.g. eggs in ``site-packages`` to override unmanged modules in
+   the same location, and allows eggs found earlier on ``sys.path`` to override
+   ones found later.
+
+ * When a distribution is activated, it now checks whether any contained
+   non-namespace modules have already been imported and issues a warning if
+   a conflicting module has already been imported.
+
 0.6a4
  * Fix a bug in ``WorkingSet.resolve()`` that was introduced in 0.6a3.
 

setup.py

@@ -37,7 +37,7 @@ setup(
     test_suite = 'setuptools.tests.test_suite',
     packages = find_packages(),
     package_data = {'setuptools': ['*.exe']},
-    py_modules = ['pkg_resources', 'easy_install'],
+    py_modules = ['pkg_resources', 'easy_install', 'site'],
 
     zip_safe = False,   # We want 'python -m easy_install' to work, for now :(
     entry_points = {

virtual-python.py

+"""Create a "virtual" Python installation
+
+Based on a script created by Ian Bicking."""
+
+import sys, os, optparse, shutil
+join = os.path.join
+py_version = 'python%s.%s' % (sys.version_info[0], sys.version_info[1])
+
+def mkdir(path):
+    if not os.path.exists(path):
+        print 'Creating %s' % path
+        os.makedirs(path)
+    else:
+        if verbose:
+            print 'Directory %s already exists'
+
+def symlink(src, dest):
+    if not os.path.exists(dest):
+        if verbose:
+            print 'Creating symlink %s' % dest
+        os.symlink(src, dest)
+    else:
+        print 'Symlink %s already exists' % dest
+
+
+def rmtree(dir):
+    if os.path.exists(dir):
+        print 'Deleting tree %s' % dir
+        shutil.rmtree(dir)
+    else:
+        if verbose:
+            print 'Do not need to delete %s; already gone' % dir
+
+def make_exe(fn):
+    if os.name == 'posix':
+        oldmode = os.stat(fn).st_mode & 07777
+        newmode = (oldmode | 0555) & 07777
+        os.chmod(fn, newmode)
+        if verbose:
+            print 'Changed mode of %s to %s' % (fn, oct(newmode))
+
+def main():
+    if os.name != 'posix':
+        print "This script only works on Unix-like platforms, sorry."
+        return
+
+    parser = optparse.OptionParser()
+    
+    parser.add_option('-v', '--verbose', action='count', dest='verbose',
+        default=0, help="Increase verbosity")
+        
+    parser.add_option('--prefix', dest="prefix", default='~',
+        help="The base directory to install to (default ~)")
+        
+    parser.add_option('--clear', dest='clear', action='store_true',
+        help="Clear out the non-root install and start from scratch")
+        
+    parser.add_option('--no-site-packages', dest='no_site_packages',
+        action='store_true',
+        help="Don't copy the contents of the global site-packages dir to the "
+             "non-root site-packages")
+    
+    options, args = parser.parse_args()
+    global verbose
+
+    home_dir = os.path.expanduser(options.prefix)
+    lib_dir = join(home_dir, 'lib', py_version)
+    inc_dir = join(home_dir, 'include', py_version)
+    bin_dir = join(home_dir, 'bin')
+
+    if sys.executable.startswith(bin_dir):
+        print 'Please use the *system* python to run this script'
+        return
+
+    verbose = options.verbose
+    assert not args, "No arguments allowed"
+        
+    if options.clear:
+        rmtree(lib_dir)
+        rmtree(inc_dir)
+        print 'Not deleting', bin_dir
+
+    prefix = sys.prefix
+    mkdir(lib_dir)
+    stdlib_dir = join(prefix, 'lib', py_version)
+    for fn in os.listdir(stdlib_dir):
+        if fn != 'site-packages':
+            symlink(join(stdlib_dir, fn), join(lib_dir, fn))
+
+    mkdir(join(lib_dir, 'site-packages'))
+    if not options.no_site_packages:
+        for fn in os.listdir(join(stdlib_dir, 'site-packages')):
+            symlink(join(stdlib_dir, 'site-packages', fn),
+                    join(lib_dir, 'site-packages', fn))
+
+    mkdir(inc_dir)
+    stdinc_dir = join(prefix, 'include', py_version)
+    for fn in os.listdir(stdinc_dir):
+        symlink(join(stdinc_dir, fn), join(inc_dir, fn))
+
+    if sys.exec_prefix != sys.prefix:
+        exec_dir = join(sys.exec_prefix, 'lib', py_version)
+        for fn in os.listdir(exec_dir):
+            symlink(join(exec_dir, fn), join(lib_dir, fn))
+
+    mkdir(bin_dir)
+    print 'Copying %s to %s' % (sys.executable, bin_dir)
+    py_executable = join(bin_dir, 'python')
+    if sys.executable != py_executable:
+        shutil.copyfile(sys.executable, py_executable)
+        make_exe(py_executable)
+
+    pydistutils = os.path.expanduser('~/.pydistutils.cfg')
+    if os.path.exists(pydistutils):
+        print 'Please make sure you remove any previous custom paths from'
+        print "your", pydistutils, "file."
+
+    print "You're now ready to download ez_setup.py, and run"
+    print py_executable, "ez_setup.py"
+      
+if __name__ == '__main__':
+    main()
+