Merged revisions 59275-59303 via svnmerge from

svn+ssh://pythondev@svn.python.org/python/trunk

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
NOTE: The merge does NOT contain the modified file Python/import.c from
      r59288. I can't get it running. Nick, please check in the PEP 366
      manually.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

........
  r59279 | georg.brandl | 2007-12-02 19:17:50 +0100 (Sun, 02 Dec 2007) | 2 lines

  Fix a sentence I missed before. Do not merge to 3k.
........
  r59281 | georg.brandl | 2007-12-02 22:58:54 +0100 (Sun, 02 Dec 2007) | 3 lines

  Add documentation for PySys_* functions.
  Written by Charlie Shepherd for GHOP. Also fixes #1245.
........
  r59288 | nick.coghlan | 2007-12-03 13:55:17 +0100 (Mon, 03 Dec 2007) | 1 line

  Implement PEP 366
........
  r59290 | christian.heimes | 2007-12-03 14:47:29 +0100 (Mon, 03 Dec 2007) | 3 lines

  Applied my patch #1455 with some extra fixes for VS 2005
  The new msvc9compiler module supports VS 2005 and VS 2008. I've also fixed build_ext to support PCbuild8 and PCbuild9 and backported my fix for xxmodule.c from py3k. The old code msvccompiler is still in place in case somebody likes to build an extension with VS 2003 or earlier.
  I've also updated the cygwin compiler module for VS 2005 and VS 2008. It works with VS 2005 but I'm unable to test it with VS 2008. We have to wait for a new version of cygwin.
........
  r59291 | christian.heimes | 2007-12-03 14:55:16 +0100 (Mon, 03 Dec 2007) | 1 line

  Added comment to Misc/NEWS for r59290
........
  r59292 | christian.heimes | 2007-12-03 15:28:04 +0100 (Mon, 03 Dec 2007) | 1 line

  I followed MA Lemberg's suggestion and added comments to the late initialization of the type slots.
........
  r59293 | facundo.batista | 2007-12-03 17:29:52 +0100 (Mon, 03 Dec 2007) | 3 lines


  Speedup and cleaning of __str__.  Thanks Mark Dickinson.
........
  r59294 | facundo.batista | 2007-12-03 18:55:00 +0100 (Mon, 03 Dec 2007) | 4 lines


  Faster _fix function, and some reordering for a more elegant
  coding. Thanks Mark Dickinson.
........
  r59295 | martin.v.loewis | 2007-12-03 20:20:02 +0100 (Mon, 03 Dec 2007) | 5 lines

  Issue #1727780: Support loading pickles of random.Random objects created
  on 32-bit systems on 64-bit systems, and vice versa. As a consequence
  of the change, Random pickles created by Python 2.6 cannot be loaded
  in Python 2.5.
........
  r59297 | facundo.batista | 2007-12-03 20:49:54 +0100 (Mon, 03 Dec 2007) | 3 lines


  Two small fixes. Issue 1547.
........
  r59299 | georg.brandl | 2007-12-03 20:57:02 +0100 (Mon, 03 Dec 2007) | 2 lines

  #1548: fix apostroph placement.
........
  r59300 | christian.heimes | 2007-12-03 21:01:02 +0100 (Mon, 03 Dec 2007) | 3 lines

  Patch #1537 from Chad Austin
  Change GeneratorExit's base class from Exception to BaseException
  (This time I'm applying the patch to the correct sandbox.)
........
  r59302 | georg.brandl | 2007-12-03 21:03:46 +0100 (Mon, 03 Dec 2007) | 3 lines

  Add examples to the xmlrpclib docs.
  Written for GHOP by Josip Dzolonga.
........

Doc/ACKS.txt

@@ -161,6 +161,7 @@ docs@python.org), and we'll be glad to correct the problem.
 * Barry Scott
 * Joakim Sernbrant
 * Justin Sheehy
+* Charlie Shepherd
 * Michael Simcich
 * Ionel Simionescu
 * Michael Sloan

Doc/c-api/init.rst

@@ -360,8 +360,6 @@ Initialization, Finalization, and Threads
    .. % XXX impl. doesn't seem consistent in allowing 0/NULL for the params;
    .. % check w/ Guido.
 
-.. % XXX Other PySys thingies (doesn't really belong in this chapter)
-
 
 .. _threads:
 

Doc/c-api/utilities.rst

@@ -66,6 +66,66 @@ Operating System Utilities
    not call those functions directly!  :ctype:`PyOS_sighandler_t` is a typedef
    alias for :ctype:`void (\*)(int)`.
 
+.. _systemfunctions:
+
+System Functions
+================
+
+These are utility functions that make functionality from the :mod:`sys` module
+accessible to C code.  They all work with the current interpreter thread's
+:mod:`sys` module's dict, which is contained in the internal thread state structure.
+
+.. cfunction:: PyObject *PySys_GetObject(char *name)
+
+   Return the object *name* from the :mod:`sys` module or *NULL* if it does
+   not exist, without setting an exception.
+
+.. cfunction:: FILE *PySys_GetFile(char *name, FILE *def)
+
+   Return the :ctype:`FILE*` associated with the object *name* in the
+   :mod:`sys` module, or *def* if *name* is not in the module or is not associated
+   with a :ctype:`FILE*`.
+
+.. cfunction:: int PySys_SetObject(char *name, PyObject *v)
+
+   Set *name* in the :mod:`sys` module to *v* unless *v* is *NULL*, in which
+   case *name* is deleted from the sys module. Returns ``0`` on success, ``-1``
+   on error.
+
+.. cfunction:: void PySys_ResetWarnOptions(void)
+
+   Reset :data:`sys.warnoptions` to an empty list.
+
+.. cfunction:: void PySys_AddWarnOption(char *s)
+
+   Append *s* to :data:`sys.warnoptions`.
+
+.. cfunction:: void PySys_SetPath(char *path)
+
+   Set :data:`sys.path` to a list object of paths found in *path* which should
+   be a list of paths separated with the platform's search path delimiter
+   (``:`` on Unix, ``;`` on Windows).
+
+.. cfunction:: void PySys_WriteStdout(const char *format, ...)
+
+   Write the output string described by *format* to :data:`sys.stdout`.  No
+   exceptions are raised, even if truncation occurs (see below).
+
+   *format* should limit the total size of the formatted output string to
+   1000 bytes or less -- after 1000 bytes, the output string is truncated.
+   In particular, this means that no unrestricted "%s" formats should occur;
+   these should be limited using "%.<N>s" where <N> is a decimal number
+   calculated so that <N> plus the maximum size of other formatted text does not
+   exceed 1000 bytes.  Also watch out for "%f", which can print hundreds of
+   digits for very large numbers.
+
+   If a problem occurs, or :data:`sys.stdout` is unset, the formatted message
+   is written to the real (C level) *stdout*.
+
+.. cfunction:: void PySys_WriteStderr(const char *format, ...)
+
+   As above, but write to :data:`sys.stderr` or *stderr* instead.
+
 
 .. _processcontrol:
 

Doc/data/refcounts.dat

@@ -1251,10 +1251,32 @@ PyString_AsEncodedString:PyObject*:str::
 PyString_AsEncodedString:const char*:encoding::
 PyString_AsEncodedString:const char*:errors::
 
+PySys_AddWarnOption:void:::
+PySys_AddWarnOption:char*:s::
+
+PySys_GetFile:FILE*:::
+PySys_GetFile:char*:name::
+PySys_GetFile:FILE*:def::
+
+PySys_GetObject:PyObject*::0:
+PySys_GetObject:char*:name::
+
 PySys_SetArgv:int:::
 PySys_SetArgv:int:argc::
 PySys_SetArgv:char**:argv::
 
+PySys_SetObject:int:::
+PySys_SetObject:char*:name::
+PySys_SetObject:PyObject*:v:+1:
+
+PySys_ResetWarnOptions:void:::
+
+PySys_WriteStdout:void:::
+PySys_WriteStdout:char*:format::
+
+PySys_WriteStderr:void:::
+PySys_WriteStderr:char*:format::
+
 PyThreadState_Clear:void:::
 PyThreadState_Clear:PyThreadState*:tstate::
 

Doc/glossary.rst

@@ -400,7 +400,7 @@ Glossary
       :term:`immutable` keys rather than integers.
 
    slice
-      An object usually containing a portion of a :term:`sequence`.  A slice is
+      A list containing a portion of an indexed list-like object.  A slice is
       created using the subscript notation, ``[]`` with colons between numbers
       when several are given, such as in ``variable_name[1:3:5]``.  The bracket
       (subscript) notation uses :class:`slice` objects internally.

Doc/library/exceptions.rst

@@ -135,7 +135,9 @@ The following exceptions are the exceptions that are actually raised.
 
 .. exception:: GeneratorExit
 
-   Raise when a :term:`generator`\'s :meth:`close` method is called.
+   Raise when a :term:`generator`\'s :meth:`close` method is called.  It
+   directly inherits from :exc:`BaseException` instead of :exc:`Exception` since
+   it is technically not an error.
 
 
 .. exception:: IOError

Doc/library/random.rst

@@ -69,6 +69,8 @@ Bookkeeping functions:
    Return an object capturing the current internal state of the generator.  This
    object can be passed to :func:`setstate` to restore the state.
 
+   State values produced in Python 2.6 cannot be loaded into earlier versions.
+
 
 .. function:: setstate(state)
 

Doc/library/simplexmlrpcserver.rst

@@ -109,7 +109,11 @@ alone XML-RPC servers.
    considered valid. The default value is ``('/', '/RPC2')``.
 
 
-Example::
+.. _simplexmlrpcserver-example:
+
+SimpleXMLRPCServer Example
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Server code::
 
    from SimpleXMLRPCServer import SimpleXMLRPCServer
 
@@ -137,7 +141,7 @@ Example::
    # Run the server's main loop
    server.serve_forever()
 
-The following client code will call the methods made available by  the preceding
+The following client code will call the methods made available by the preceding
 server::
 
    import xmlrpclib

Doc/library/xmlrpclib.rst

@@ -192,6 +192,26 @@ unmarshalling code:
 
    Write the XML-RPC encoding of this Boolean item to the out stream object.
 
+A working example follows. The server code::
+
+   import xmlrpclib
+   from SimpleXMLRPCServer import SimpleXMLRPCServer
+
+   def is_even(n):
+       return n%2 == 0
+
+   server = SimpleXMLRPCServer(("localhost", 8000))
+   print "Listening on port 8000..."
+   server.register_function(is_even, "is_even")
+   server.serve_forever()
+
+The client code for the preceding server::
+
+   import xmlrpclib
+
+   proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
+   print "3 is even: %s" % str(proxy.is_even(3))
+   print "100 is even: %s" % str(proxy.is_even(100))
 
 .. _datetime-objects:
 
@@ -217,6 +237,32 @@ mainly for internal use by the marshalling/unmarshalling code:
 It also supports certain of Python's built-in operators through  :meth:`__cmp__`
 and :meth:`__repr__` methods.
 
+A working example follows. The server code::
+
+   import datetime
+   from SimpleXMLRPCServer import SimpleXMLRPCServer
+   import xmlrpclib
+
+   def today():
+       today = datetime.datetime.today()
+       return xmlrpclib.DateTime(today)
+
+   server = SimpleXMLRPCServer(("localhost", 8000))
+   print "Listening on port 8000..."
+   server.register_function(today, "today")
+   server.serve_forever()
+
+The client code for the preceding server::
+
+   import xmlrpclib
+   import datetime
+
+   proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
+
+   today = proxy.today()
+   # convert the ISO8601 string to a datetime object
+   converted = datetime.datetime.strptime(today.value, "%Y%m%dT%H:%M:%S")
+   print "Today: %s" % converted.strftime("%d.%m.%Y, %H:%M")
 
 .. _binary-objects:
 
@@ -249,6 +295,31 @@ internal use by the marshalling/unmarshalling code:
 It also supports certain of Python's built-in operators through a
 :meth:`__cmp__` method.
 
+Example usage of the binary objects.  We're going to transfer an image over
+XMLRPC::
+
+   from SimpleXMLRPCServer import SimpleXMLRPCServer
+   import xmlrpclib
+
+   def python_logo():
+        handle = open("python_logo.jpg")
+        return xmlrpclib.Binary(handle.read())
+        handle.close()
+
+   server = SimpleXMLRPCServer(("localhost", 8000))
+   print "Listening on port 8000..."
+   server.register_function(python_logo, 'python_logo')
+
+   server.serve_forever()
+
+The client gets the image and saves it to a file::
+
+   import xmlrpclib
+
+   proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
+   handle = open("fetched_python_logo.jpg", "w")
+   handle.write(proxy.python_logo().data)
+   handle.close()
 
 .. _fault-objects:
 
@@ -268,6 +339,35 @@ objects have the following members:
 
    A string containing a diagnostic message associated with the fault.
 
+In the following example we're going to intentionally cause a :exc:`Fault` by
+returning a complex type object.  The server code::
+
+   from SimpleXMLRPCServer import SimpleXMLRPCServer
+
+   # A marshalling error is going to occur because we're returning a
+   # complex number
+   def add(x,y):
+       return x+y+0j
+
+   server = SimpleXMLRPCServer(("localhost", 8000))
+   print "Listening on port 8000..."
+   server.register_function(add, 'add')
+
+   server.serve_forever()
+
+The client code for the preceding server::
+
+   import xmlrpclib
+
+   proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
+   try:
+       proxy.add(2, 5)
+   except xmlrpclib.Fault, err:
+       print "A fault occured"
+       print "Fault code: %d" % err.faultCode
+       print "Fault string: %s" % err.faultString
+
+
 
 .. _protocol-error-objects:
 
@@ -299,6 +399,22 @@ does not exist).  It has the following members:
    A dict containing the headers of the HTTP/HTTPS request that triggered the
    error.
 
+In the following example we're going to intentionally cause a :exc:`ProtocolError`
+by providing an invalid URI::
+
+   import xmlrpclib
+
+   # create a ServerProxy with an invalid URI
+   proxy = xmlrpclib.ServerProxy("http://invalidaddress/")
+
+   try:
+       proxy.some_method()
+   except xmlrpclib.ProtocolError, err:
+       print "A protocol error occured"
+       print "URL: %s" % err.url
+       print "HTTP/HTTPS headers: %s" % err.headers
+       print "Error code: %d" % err.errcode
+       print "Error message: %s" % err.errmsg
 
 MultiCall Objects
 -----------------
@@ -317,12 +433,45 @@ encapsulate multiple calls to a remote server into a single request.
    is a :term:`generator`; iterating over this generator yields the individual
    results.
 
-A usage example of this class is ::
+A usage example of this class follows.  The server code ::
+
+   from SimpleXMLRPCServer import SimpleXMLRPCServer
+
+   def add(x,y):
+       return x+y
 
-   multicall = MultiCall(server_proxy)
-   multicall.add(2,3)
-   multicall.get_address("Guido")
-   add_result, address = multicall()
+   def subtract(x, y):
+       return x-y
+
+   def multiply(x, y):
+       return x*y
+
+   def divide(x, y):
+       return x/y
+
+   # A simple server with simple arithmetic functions
+   server = SimpleXMLRPCServer(("localhost", 8000))
+   print "Listening on port 8000..."
+   server.register_multicall_functions()
+   server.register_function(add, 'add')
+   server.register_function(subtract, 'subtract')
+   server.register_function(multiply, 'multiply')
+   server.register_function(divide, 'divide')
+   server.serve_forever()
+
+The client code for the preceding server::
+
+   import xmlrpclib
+
+   proxy = xmlrpclib.ServerProxy("http://localhost:8000/")
+   multicall = xmlrpclib.MultiCall(proxy)
+   multicall.add(7,3)
+   multicall.subtract(7,3)
+   multicall.multiply(7,3)
+   multicall.divide(7,3)
+   result = multicall()
+
+   print "7+3=%d, 7-3=%d, 7*3=%d, 7/3=%d" % tuple(result)
 
 
 Convenience Functions
@@ -407,3 +556,10 @@ transport.  The following example,  written by NoboNobo, shows how:
    server = xmlrpclib.Server('http://time.xmlrpc.com/RPC2', transport=p)
    print(server.currentTime.getCurrentTime())
 
+
+Example of Client and Server Usage
+----------------------------------
+
+See :ref:`simplexmlrpcserver-example`.
+
+

Doc/reference/expressions.rst

@@ -413,9 +413,6 @@ generator functions::
    ...         while True:
    ...             try:
    ...                 value = (yield value)
-   ...             except GeneratorExit:
-   ...                 # never catch GeneratorExit
-   ...                 raise
    ...             except Exception, e:
    ...                 value = e
    ...     finally:

Doc/using/cmdline.rst

@@ -10,7 +10,7 @@ settings.
 
 .. note:: 
    
-   Other implementation's command line schemes may differ.  See
+   Other implementations' command line schemes may differ.  See
    :ref:`implementations` for further resources.
 
 

Doc/whatsnew/2.6.rst

@@ -108,7 +108,7 @@ PEP 343: The 'with' statement
 
 The previous version, Python 2.5, added the ':keyword:`with`'
 statement an optional feature, to be enabled by a ``from __future__
-import generators`` directive.  In 2.6 the statement no longer need to
+import with_statement`` directive.  In 2.6 the statement no longer need to
 be specially enabled; this means that :keyword:`with` is now always a
 keyword.  The rest of this section is a copy of the corresponding 
 section from "What's New in Python 2.5" document; if you read
@@ -481,7 +481,7 @@ Here are all of the changes that Python 2.6 makes to the core Python language.
   of strings containing the names of valid attributes for the object,
   and lets the object control the value that :func:`dir` produces.
   Objects that have :meth:`__getattr__` or :meth:`__getattribute__` 
-  methods.
+  methods can use this to advertise pseudo-attributes they will honor.
 
   .. % Patch 1591665
 

Lib/distutils/command/build_ext.py

@@ -14,6 +14,10 @@ from distutils.dep_util import newer_group
 from distutils.extension import Extension
 from distutils import log
 
+if os.name == 'nt':
+    from distutils.msvccompiler import get_build_version
+    MSVC_VERSION = int(get_build_version())
+
 # An extension name is just a dot-separated list of Python NAMEs (ie.
 # the same as a fully-qualified module name).
 extension_name_re = re.compile \
@@ -172,7 +176,15 @@ class build_ext(Command):
             # Append the source distribution include and library directories,
             # this allows distutils on windows to work in the source tree
             self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC'))
-            self.library_dirs.append(os.path.join(sys.exec_prefix, 'PCBuild'))
+            if MSVC_VERSION == 9:
+                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                                         'PCBuild9'))
+            elif MSVC_VERSION == 8:
+                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                                         'PCBuild8', 'win32release'))
+            else:
+                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                                         'PCBuild'))
 
         # OS/2 (EMX) doesn't support Debug vs Release builds, but has the
         # import libraries in its "Config" subdirectory

Lib/distutils/cygwinccompiler.py

@@ -54,6 +54,29 @@ from distutils.file_util import write_file
 from distutils.errors import DistutilsExecError, CompileError, UnknownFileError
 from distutils import log
 
+def get_msvcr():
+    """Include the appropriate MSVC runtime library if Python was built
+    with MSVC 7.0 or later.
+    """
+    msc_pos = sys.version.find('MSC v.')
+    if msc_pos != -1:
+        msc_ver = sys.version[msc_pos+6:msc_pos+10]
+        if msc_ver == '1300':
+            # MSVC 7.0
+            return ['msvcr70']
+        elif msc_ver == '1310':
+            # MSVC 7.1
+            return ['msvcr71']
+        elif msc_ver == '1400':
+            # VS2005 / MSVC 8.0
+            return ['msvcr80']
+        elif msc_ver == '1500':
+            # VS2008 / MSVC 9.0
+            return ['msvcr90']
+        else:
+            raise ValueError("Unknown MS Compiler version %i " % msc_Ver)
+
+
 class CygwinCCompiler (UnixCCompiler):
 
     compiler_type = 'cygwin'
@@ -119,18 +142,9 @@ class CygwinCCompiler (UnixCCompiler):
             self.warn(
                 "Consider upgrading to a newer version of gcc")
         else:
-            self.dll_libraries=[]
             # Include the appropriate MSVC runtime library if Python was built
-            # with MSVC 7.0 or 7.1.
-            msc_pos = sys.version.find('MSC v.')
-            if msc_pos != -1:
-                msc_ver = sys.version[msc_pos+6:msc_pos+10]
-                if msc_ver == '1300':
-                    # MSVC 7.0
-                    self.dll_libraries = ['msvcr70']
-                elif msc_ver == '1310':
-                    # MSVC 7.1
-                    self.dll_libraries = ['msvcr71']
+            # with MSVC 7.0 or later.
+            self.dll_libraries = get_msvcr()
 
     # __init__ ()
 
@@ -317,16 +331,8 @@ class Mingw32CCompiler (CygwinCCompiler):
         self.dll_libraries=[]
 
         # Include the appropriate MSVC runtime library if Python was built
-        # with MSVC 7.0 or 7.1.
-        msc_pos = sys.version.find('MSC v.')
-        if msc_pos != -1:
-            msc_ver = sys.version[msc_pos+6:msc_pos+10]
-            if msc_ver == '1300':
-                # MSVC 7.0
-                self.dll_libraries = ['msvcr70']
-            elif msc_ver == '1310':
-                # MSVC 7.1
-                self.dll_libraries = ['msvcr71']
+        # with MSVC 7.0 or later.
+        self.dll_libraries = get_msvcr()
 
     # __init__ ()
 

Lib/distutils/msvccompiler.py

@@ -632,3 +632,11 @@ class MSVCCompiler(CCompiler) :
             p = self.get_msvc_paths(name)
         if p:
             os.environ[name] = ';'.join(p)
+
+
+if get_build_version() >= 8.0:
+    log.debug("Importing new compiler from distutils.msvc9compiler")
+    OldMSVCCompiler = MSVCCompiler
+    from distutils.msvc9compiler import MSVCCompiler
+    from distutils.msvc9compiler import get_build_architecture
+    from distutils.msvc9compiler import MacroExpander

Lib/random.py

@@ -83,7 +83,7 @@ class Random(_random.Random):
 
     """
 
-    VERSION = 2     # used by getstate/setstate
+    VERSION = 3     # used by getstate/setstate
 
     def __init__(self, x=None):
         """Initialize an instance.
@@ -120,9 +120,20 @@ class Random(_random.Random):
     def setstate(self, state):
         """Restore internal state from object returned by getstate()."""
         version = state[0]
-        if version == 2:
+        if version == 3:
             version, internalstate, self.gauss_next = state
             super().setstate(internalstate)
+        elif version == 2:
+            version, internalstate, self.gauss_next = state
+            # In version 2, the state was saved as signed ints, which causes
+            #   inconsistencies between 32/64-bit systems. The state is
+            #   really unsigned 32-bit ints, so we convert negative ints from
+            #   version 2 to positive longs for version 3.
+            try:
+                internalstate = tuple( x % (2**32) for x in internalstate )
+            except ValueError as e:
+                raise TypeError from e
+            super(Random, self).setstate(internalstate)
         else:
             raise ValueError("state with version %s passed to "
                              "Random.setstate() of version %s" %

Lib/runpy.py

@@ -23,19 +23,20 @@ __all__ = [
 
 def _run_code(code, run_globals, init_globals=None,
               mod_name=None, mod_fname=None,
-              mod_loader=None):
+              mod_loader=None, pkg_name=None):
     """Helper for _run_module_code"""
     if init_globals is not None:
         run_globals.update(init_globals)
     run_globals.update(__name__ = mod_name,
                        __file__ = mod_fname,
-                       __loader__ = mod_loader)
+                       __loader__ = mod_loader,
+                       __package__ = pkg_name)
     exec(code, run_globals)
     return run_globals
 
 def _run_module_code(code, init_globals=None,
                     mod_name=None, mod_fname=None,
-                    mod_loader=None):
+                    mod_loader=None, pkg_name=None):
     """Helper for run_module"""
     # Set up the top level namespace dictionary
     temp_module = imp.new_module(mod_name)
@@ -49,7 +50,8 @@ def _run_module_code(code, init_globals=None,
     sys.modules[mod_name] = temp_module
     try:
         _run_code(code, mod_globals, init_globals,
-                    mod_name, mod_fname, mod_loader)
+                    mod_name, mod_fname,
+                    mod_loader, pkg_name)
     finally:
         sys.argv[0] = saved_argv0
         if restore_module:
@@ -95,11 +97,12 @@ def _run_module_as_main(mod_name, set_argv0=True):
            __loader__
     """
     loader, code, fname = _get_module_details(mod_name)
+    pkg_name = mod_name.rpartition('.')[0]
     main_globals = sys.modules["__main__"].__dict__
     if set_argv0:
         sys.argv[0] = fname
     return _run_code(code, main_globals, None,
-                     "__main__", fname, loader)
+                     "__main__", fname, loader, pkg_name)
 
 def run_module(mod_name, init_globals=None,
                run_name=None, alter_sys=False):
@@ -110,13 +113,14 @@ def run_module(mod_name, init_globals=None,
     loader, code, fname = _get_module_details(mod_name)
     if run_name is None:
         run_name = mod_name
+    pkg_name = mod_name.rpartition('.')[0]
     if alter_sys:
         return _run_module_code(code, init_globals, run_name,
-                                fname, loader)
+                                fname, loader, pkg_name)
     else:
         # Leave the sys module alone
-        return _run_code(code, {}, init_globals,
-                         run_name, fname, loader)
+        return _run_code(code, {}, init_globals, run_name,
+                         fname, loader, pkg_name)
 
 
 if __name__ == "__main__":

Lib/test/exception_hierarchy.txt

 BaseException
  +-- SystemExit
  +-- KeyboardInterrupt
+ +-- GeneratorExit
  +-- Exception
-      +-- GeneratorExit
       +-- StopIteration
       +-- ArithmeticError
       |    +-- FloatingPointError

Lib/test/test_cmd_line_script.py

@@ -35,15 +35,15 @@ def temp_dir():
     finally:
         shutil.rmtree(dirname)
 
-test_source = ("""\
+test_source = """\
 # Script may be run with optimisation enabled, so don't rely on assert
 # statements being executed
 def assertEqual(lhs, rhs):
     if lhs != rhs:
-        raise AssertionError("%r != %r" % (lhs, rhs))
+        raise AssertionError('%r != %r' % (lhs, rhs))
 def assertIdentical(lhs, rhs):
     if lhs is not rhs:
-        raise AssertionError("%r is not %r" % (lhs, rhs))
+        raise AssertionError('%r is not %r' % (lhs, rhs))
 # Check basic code execution
 result = ['Top level assignment']
 def f():
@@ -53,17 +53,18 @@ assertEqual(result, ['Top level assignment', 'Lower level reference'])
 # Check population of magic variables
 assertEqual(__name__, '__main__')
 print('__file__==%r' % __file__)
+print('__package__==%r' % __package__)
 # Check the sys module
 import sys
 assertIdentical(globals(), sys.modules[__name__].__dict__)
 print('sys.argv[0]==%r' % sys.argv[0])
-""")
+"""
 
-def _make_test_script(script_dir, script_basename):
-    script_filename = script_basename+os.path.extsep+"py"
+def _make_test_script(script_dir, script_basename, source=test_source):
+    script_filename = script_basename+os.path.extsep+'py'
     script_name = os.path.join(script_dir, script_filename)
-    script_file = open(script_name, "w")
-    script_file.write(test_source)
+    script_file = open(script_name, 'w')
+    script_file.write(source)
     script_file.close()
     return script_name
 
@@ -88,59 +89,96 @@ def _make_test_zip(zip_dir, zip_basename, script_name):
     #    zip_file.close()
     return zip_name
 
+def _make_test_pkg(pkg_dir):
+    os.mkdir(pkg_dir)
+    _make_test_script(pkg_dir, '__init__', '')
+
+# There's no easy way to pass the script directory in to get
+# -m to work (avoiding that is the whole point of making
+# directories and zipfiles executable!)
+# So we fake it for testing purposes with a custom launch script
+launch_source = """\
+import sys, os.path, runpy
+sys.path[0:0] = os.path.dirname(__file__)
+runpy._run_module_as_main(%r)
+"""
+
+def _make_launch_script(script_dir, script_basename, module_name):
+    return _make_test_script(script_dir, script_basename,
+                             launch_source % module_name)
+
 class CmdLineTest(unittest.TestCase):
-    def _check_script(self, script_name, expected_file, expected_argv0):
-        exit_code, data = _run_python(script_name)
+    def _check_script(self, script_name, expected_file,
+                            expected_argv0, expected_package,
+                            *cmd_line_switches):
+        run_args = cmd_line_switches + (script_name,)
+        exit_code, data = _run_python(*run_args)
         if verbose:
             print("Output from test script %r:" % script_name)
             print(data)
-        self.assertEqual(exit_code, 0, data)
+        self.assertEqual(exit_code, 0)
         printed_file = '__file__==%r' % expected_file
         printed_argv0 = 'sys.argv[0]==%r' % expected_argv0
-        self.assert_(printed_file in data, (printed_file, data))
-        self.assert_(printed_argv0 in data, (printed_argv0, data))
+        printed_package = '__package__==%r' % expected_package
+        if verbose:
+            print('Expected output:')
+            print(printed_file)
+            print(printed_package)
+            print(printed_argv0)
+        self.assert_(printed_file in data)
+        self.assert_(printed_package in data)
+        self.assert_(printed_argv0 in data)
 
     def test_basic_script(self):
         with temp_dir() as script_dir:
-            script_name = _make_test_script(script_dir, "script")
-            self._check_script(script_name, script_name, script_name)
+            script_name = _make_test_script(script_dir, 'script')
+            self._check_script(script_name, script_name, script_name, None)
 
     def test_script_compiled(self):
         with temp_dir() as script_dir:
-            script_name = _make_test_script(script_dir, "script")
+            script_name = _make_test_script(script_dir, 'script')
             compiled_name = _compile_test_script(script_name)
             os.remove(script_name)
-            self._check_script(compiled_name, compiled_name, compiled_name)
+            self._check_script(compiled_name, compiled_name, compiled_name, None)
 
     def test_directory(self):
         with temp_dir() as script_dir:
-            script_name = _make_test_script(script_dir, "__main__")
-            self._check_script(script_dir, script_name, script_dir)
+            script_name = _make_test_script(script_dir, '__main__')
+            self._check_script(script_dir, script_name, script_dir, '')
 
     def test_directory_compiled(self):
         with temp_dir() as script_dir:
-            script_name = _make_test_script(script_dir, "__main__")
+            script_name = _make_test_script(script_dir, '__main__')
             compiled_name = _compile_test_script(script_name)
             os.remove(script_name)
-            self._check_script(script_dir, compiled_name, script_dir)
+            self._check_script(script_dir, compiled_name, script_dir, '')
 
     def test_zipfile(self):
         with temp_dir() as script_dir:
-            script_name = _make_test_script(script_dir, "__main__")
-            zip_name = _make_test_zip(script_dir, "test_zip", script_name)
-            self._check_script(zip_name, None, zip_name)
+            script_name = _make_test_script(script_dir, '__main__')
+            zip_name = _make_test_zip(script_dir, 'test_zip', script_name)
+            self._check_script(zip_name, None, zip_name, '')
 
     def test_zipfile_compiled(self):
         with temp_dir() as script_dir:
-            script_name = _make_test_script(script_dir, "__main__")
+            script_name = _make_test_script(script_dir, '__main__')
             compiled_name = _compile_test_script(script_name)
-            zip_name = _make_test_zip(script_dir, "test_zip", compiled_name)
-            self._check_script(zip_name, None, zip_name)
+            zip_name = _make_test_zip(script_dir, 'test_zip', compiled_name)
+            self._check_script(zip_name, None, zip_name, '')
+
+    def test_module_in_package(self):
+        with temp_dir() as script_dir:
+            pkg_dir = os.path.join(script_dir, 'test_pkg')
+            _make_test_pkg(pkg_dir)
+            script_name = _make_test_script(pkg_dir, 'script')
+            launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg.script')
+            self._check_script(launch_name, script_name,
+                               script_name, 'test_pkg')
 
 
 def test_main():
     test.test_support.run_unittest(CmdLineTest)
     test.test_support.reap_children()
 
-if __name__ == "__main__":
+if __name__ == '__main__':
     test_main()

Lib/test/test_frozen.py

@@ -12,7 +12,7 @@ class FrozenTests(unittest.TestCase):
         except ImportError as x:
             self.fail("import __hello__ failed:" + str(x))
         self.assertEqual(__hello__.initialized, True)
-        self.assertEqual(len(dir(__hello__)), 5)
+        self.assertEqual(len(dir(__hello__)), 6, dir(__hello__))
 
         try:
             import __phello__
@@ -20,17 +20,17 @@ class FrozenTests(unittest.TestCase):
             self.fail("import __phello__ failed:" + str(x))
         self.assertEqual(__phello__.initialized, True)
         if not "__phello__.spam" in sys.modules:
-            self.assertEqual(len(dir(__phello__)), 6, dir(__phello__))
-        else:
             self.assertEqual(len(dir(__phello__)), 7, dir(__phello__))
+        else:
+            self.assertEqual(len(dir(__phello__)), 8, dir(__phello__))
 
         try:
             import __phello__.spam
         except ImportError as x:
             self.fail("import __phello__.spam failed:" + str(x))
         self.assertEqual(__phello__.spam.initialized, True)
-        self.assertEqual(len(dir(__phello__.spam)), 5)
-        self.assertEqual(len(dir(__phello__)), 7)
+        self.assertEqual(len(dir(__phello__.spam)), 6)
+        self.assertEqual(len(dir(__phello__)), 8)
 
         try:
             import __phello__.foo

Lib/test/test_generators.py

@@ -1668,6 +1668,21 @@ And finalization:
 exiting
 
 
+GeneratorExit is not caught by except Exception:
+
+>>> def f():
+...     try: yield
+...     except Exception:
+...         print('except')
+...     finally:
+...         print('finally')
+
+>>> g = f()
+>>> next(g)
+>>> del g
+finally
+
+
 Now let's try some ill-behaved generators:
 
 >>> def f():

Lib/test/test_pkg.py

@@ -192,11 +192,13 @@ class TestPkg(unittest.TestCase):
         import t5
         self.assertEqual(fixdir(dir(t5)),
                          ['__doc__', '__file__', '__name__',
-                          '__path__', 'foo', 'string', 't5'])
+                          '__package__', '__path__', 'foo', 'string', 't5'])
         self.assertEqual(fixdir(dir(t5.foo)),
-                         ['__doc__', '__file__', '__name__', 'string'])
+                         ['__doc__', '__file__', '__name__', '__package__',
+                          'string'])
         self.assertEqual(fixdir(dir(t5.string)),
-                         ['__doc__', '__file__', '__name__', 'spam'])
+                         ['__doc__', '__file__', '__name__','__package__',
+                          'spam'])
 
     def test_6(self):
         hier = [
@@ -212,14 +214,14 @@ class TestPkg(unittest.TestCase):
         import t6
         self.assertEqual(fixdir(dir(t6)),
                          ['__all__', '__doc__', '__file__',
-                          '__name__', '__path__'])
+                          '__name__', '__package__', '__path__'])
         s = """
             import t6
             from t6 import *
             self.assertEqual(fixdir(dir(t6)),
                              ['__all__', '__doc__', '__file__',
-                              '__name__', '__path__', 'eggs',
-                              'ham', 'spam'])
+                              '__name__', '__package__', '__path__',
+                              'eggs', 'ham', 'spam'])
             self.assertEqual(dir(), ['eggs', 'ham', 'self', 'spam', 't6'])
             """
         self.run_code(s)
@@ -245,17 +247,19 @@ class TestPkg(unittest.TestCase):
         t7, sub, subsub = None, None, None
         import t7 as tas
         self.assertEqual(fixdir(dir(tas)),
-                         ['__doc__', '__file__', '__name__', '__path__'])
+                         ['__doc__', '__file__', '__name__',
+                          '__package__', '__path__'])
         self.failIf(t7)
         from t7 import sub as subpar
         self.assertEqual(fixdir(dir(subpar)),
-                         ['__doc__', '__file__', '__name__', '__path__'])
+                         ['__doc__', '__file__', '__name__',
+                          '__package__', '__path__'])
         self.failIf(t7)
         self.failIf(sub)
         from t7.sub import subsub as subsubsub
         self.assertEqual(fixdir(dir(subsubsub)),
-                         ['__doc__', '__file__', '__name__', '__path__',
-                          'spam'])
+                         ['__doc__', '__file__', '__name__',
+                         '__package__', '__path__', 'spam'])
         self.failIf(t7)
         self.failIf(sub)
         self.failIf(subsub)

Lib/test/test_random.py

@@ -144,6 +144,19 @@ class TestBasicOps(unittest.TestCase):
         restoredseq = [newgen.random() for i in range(10)]
         self.assertEqual(origseq, restoredseq)
 
+    def test_bug_1727780(self):
+        # verify that version-2-pickles can be loaded
+        # fine, whether they are created on 32-bit or 64-bit
+        # platforms, and that version-3-pickles load fine.
+        files = [("randv2_32.pck", 780),
+                 ("randv2_64.pck", 866),
+                 ("randv3.pck", 343)]
+        for file, value in files:
+            f = open(test_support.findfile(file),"rb")
+            r = pickle.load(f)
+            f.close()
+            self.assertEqual(r.randrange(1000), value)
+
 class WichmannHill_TestBasicOps(TestBasicOps):
     gen = random.WichmannHill()
 

Lib/test/test_runpy.py

@@ -5,7 +5,12 @@ import os.path
 import sys
 import tempfile
 from test.test_support import verbose, run_unittest, forget
-from runpy import _run_code, _run_module_code, _run_module_as_main, run_module
+from runpy import _run_code, _run_module_code, run_module
+
+# Note: This module can't safely test _run_module_as_main as it
+# runs its tests in the current process, which would mess with the
+# real __main__ module (usually test.regrtest)
+# See test_cmd_line_script for a test that executes that code path
 
 # Set up the test code and expected results
 
@@ -36,6 +41,7 @@ class RunModuleCodeTest(unittest.TestCase):
         self.failUnless(d["__name__"] is None)
         self.failUnless(d["__file__"] is None)
         self.failUnless(d["__loader__"] is None)
+        self.failUnless(d["__package__"] is None)
         self.failUnless(d["run_argv0"] is saved_argv0)
         self.failUnless("run_name" not in d)
         self.failUnless(sys.argv[0] is saved_argv0)
@@ -45,13 +51,15 @@ class RunModuleCodeTest(unittest.TestCase):
         name = "<Nonsense>"
         file = "Some other nonsense"
         loader = "Now you're just being silly"
+        package = '' # Treat as a top level module
         d1 = dict(initial=initial)
         saved_argv0 = sys.argv[0]
         d2 = _run_module_code(self.test_source,
                               d1,
                               name,
                               file,
-                              loader)
+                              loader,
+                              package)
         self.failUnless("result" not in d1)
         self.failUnless(d2["initial"] is initial)
         self.assertEqual(d2["result"], self.expected_result)
@@ -62,6 +70,7 @@ class RunModuleCodeTest(unittest.TestCase):
         self.failUnless(d2["__file__"] is file)
         self.failUnless(d2["run_argv0"] is file)
         self.failUnless(d2["__loader__"] is loader)
+        self.failUnless(d2["__package__"] is package)
         self.failUnless(sys.argv[0] is saved_argv0)
         self.failUnless(name not in sys.modules)
 
@@ -164,7 +173,7 @@ class RunModuleTest(unittest.TestCase):
             self._del_pkg(pkg_dir, depth, mod_name)
         if verbose: print("Module executed successfully")
 
-    def _add_relative_modules(self, base_dir, depth):
+    def _add_relative_modules(self, base_dir, source, depth):
         if depth <= 1:
             raise ValueError("Relative module test needs depth > 1")
         pkg_name = "__runpy_pkg__"
@@ -190,7 +199,7 @@ class RunModuleTest(unittest.TestCase):
         if verbose: print("  Added nephew module:", nephew_fname)
 
     def _check_relative_imports(self, depth, run_name=None):
-        contents = """\
+        contents = r"""\
 from __future__ import absolute_import
 from . import sibling
 from ..uncle.cousin import nephew
@@ -198,16 +207,21 @@ from ..uncle.cousin import nephew
         pkg_dir, mod_fname, mod_name = (
                self._make_pkg(contents, depth))
         try:
-            self._add_relative_modules(pkg_dir, depth)
+            self._add_relative_modules(pkg_dir, contents, depth)
+            pkg_name = mod_name.rpartition('.')[0]
             if verbose: print("Running from source:", mod_name)
-            d1 = run_module(mod_name) # Read from source
+            d1 = run_module(mod_name, run_name=run_name) # Read from source
+            self.failUnless("__package__" in d1)
+            self.failUnless(d1["__package__"] == pkg_name)
             self.failUnless("sibling" in d1)
             self.failUnless("nephew" in d1)
             del d1 # Ensure __loader__ entry doesn't keep file open
             __import__(mod_name)
             os.remove(mod_fname)
             if verbose: print("Running from compiled:", mod_name)
-            d2 = run_module(mod_name) # Read from bytecode
+            d2 = run_module(mod_name, run_name=run_name) # Read from bytecode
+            self.failUnless("__package__" in d2)
+            self.failUnless(d2["__package__"] == pkg_name)
             self.failUnless("sibling" in d2)
             self.failUnless("nephew" in d2)
             del d2 # Ensure __loader__ entry doesn't keep file open
@@ -225,6 +239,11 @@ from ..uncle.cousin import nephew
             if verbose: print("Testing relative imports at depth:", depth)
             self._check_relative_imports(depth)
 
+    def test_main_relative_import(self):
+        for depth in range(2, 5):
+            if verbose: print("Testing main relative imports at depth:", depth)
+            self._check_relative_imports(depth, "__main__")
+
 
 def test_main():
     run_unittest(RunModuleCodeTest)

Misc/ACKS

@@ -398,6 +398,7 @@ Marc-Andre Lemburg
 Mark Levinson
 William Lewis
 Robert van Liere
+Shawn Ligocki
 Martin Ligr
 Christopher Lindblad
 Eric Lindvall

Modules/_randommodule.c

@@ -319,7 +319,7 @@ random_getstate(RandomObject *self)
 	if (state == NULL)
 		return NULL;
 	for (i=0; i<N ; i++) {
-		element = PyLong_FromLong((long)(self->state[i]));
+		element = PyLong_FromUnsignedLong(self->state[i]);
 		if (element == NULL)
 			goto Fail;
 		PyTuple_SET_ITEM(state, i, element);
@@ -339,7 +339,8 @@ static PyObject *
 random_setstate(RandomObject *self, PyObject *state)
 {
 	int i;
-	long element;
+	unsigned long element;
+	long index;
 
 	if (!PyTuple_Check(state)) {
 		PyErr_SetString(PyExc_TypeError,
@@ -353,16 +354,16 @@ random_setstate(RandomObject *self, PyObject *state)
 	}
 
 	for (i=0; i<N ; i++) {
-		element = PyLong_AsLong(PyTuple_GET_ITEM(state, i));
+		element = PyLong_AsUnsignedLong(PyTuple_GET_ITEM(state, i));
 		if (element == -1 && PyErr_Occurred())
 			return NULL;
-		self->state[i] = (unsigned long)element;
+		self->state[i] = element & 0xffffffffUL; /* Make sure we get sane state */
 	}
 
-	element = PyLong_AsLong(PyTuple_GET_ITEM(state, i));
-	if (element == -1 && PyErr_Occurred())
+	index = PyLong_AsLong(PyTuple_GET_ITEM(state, i));
+	if (index == -1 && PyErr_Occurred())
 		return NULL;
-	self->index = (int)element;
+	self->index = (int)index;
 
 	Py_INCREF(Py_None);
 	return Py_None;

Modules/xxmodule.c

@@ -246,7 +246,7 @@ static PyTypeObject Str_Type = {
 	0,			/*tp_methods*/
 	0,			/*tp_members*/
 	0,			/*tp_getset*/
-	0,			/*tp_base*/
+	0, /* see initxx */	/*tp_base*/
 	0,			/*tp_dict*/
 	0,			/*tp_descr_get*/
 	0,			/*tp_descr_set*/
@@ -301,14 +301,14 @@ static PyTypeObject Null_Type = {
 	0,			/*tp_methods*/
 	0,			/*tp_members*/
 	0,			/*tp_getset*/
-	0,			/*tp_base*/
+	0, /* see initxx */	/*tp_base*/
 	0,			/*tp_dict*/
 	0,			/*tp_descr_get*/
 	0,			/*tp_descr_set*/
 	0,			/*tp_dictoffset*/
 	0,			/*tp_init*/
 	0,			/*tp_alloc*/
-	PyType_GenericNew,	/*tp_new*/
+	0, /* see initxx */	/*tp_new*/
 	0,			/*tp_free*/
 	0,			/*tp_is_gc*/
 };
@@ -341,12 +341,15 @@ initxx(void)
 {
 	PyObject *m;
 
+	/* Due to cross platform compiler issues the slots must be filled
+	 * here. It's required for portability to Windows without requiring
+	 * C++. */
 	Null_Type.tp_base = &PyBaseObject_Type;
+	Null_Type.tp_new = PyType_GenericNew;
 	Str_Type.tp_base = &PyUnicode_Type;
 
 	/* Finalize the type object including setting type of the new type
-	 * object; doing it here is required for portability to Windows 
-	 * without requiring C++. */
+	 * object; doing it here is required for portability, too. /*
 	if (PyType_Ready(&Xxo_Type) < 0)
 		return;
 

Objects/exceptions.c

@@ -424,9 +424,9 @@ SimpleExtendsException(PyExc_Exception, StopIteration,
 
 
 /*
- *    GeneratorExit extends Exception
+ *    GeneratorExit extends BaseException
  */
-SimpleExtendsException(PyExc_Exception, GeneratorExit,
+SimpleExtendsException(PyExc_BaseException, GeneratorExit,
                        "Request that a generator exit.");
 
 

Objects/moduleobject.c

@@ -30,6 +30,8 @@ PyModule_New(const char *name)
 		goto fail;
 	if (PyDict_SetItemString(m->md_dict, "__doc__", Py_None) != 0)
 		goto fail;
+	if (PyDict_SetItemString(m->md_dict, "__package__", Py_None) != 0)
+		goto fail;
 	Py_DECREF(nameobj);
 	PyObject_GC_Track(m);
 	return (PyObject *)m;

Tools/msi/msi.py

@@ -975,6 +975,7 @@ def add_files(db):
             lib.add_file("empty.vbs")
             lib.glob("*.uue")
             lib.glob("*.pem")
+            lib.glob("*.pck")
             lib.add_file("readme.txt", src="README")
         if dir=='decimaltestdata':
             lib.glob("*.decTest")