Merged upstream changes.

Doc/howto/index.rst

@@ -28,4 +28,5 @@ Currently, the HOWTOs are:
    urllib2.rst
    webservers.rst
    argparse.rst
+   ipaddress.rst
 

Doc/howto/ipaddress.rst

+.. _ipaddress-howto:
+
+***************
+Ipaddress Howto
+***************
+
+:author: Peter Moody
+
+.. topic:: Abstract
+
+   This document is a gentle introduction to :mod:`ipaddress` module.
+
+
+Creating Address/Network/Interface objects
+==========================================
+
+Since :mod:`ipaddress` is a module for inspecting and manipulating IP address,
+the first thing you'll want to do is create some objects.  You can use
+:mod:`ipaddress` to create objects from strings and integers.
+
+
+A Note on IP Versions
+---------------------
+
+For readers that aren't particularly familiar with IP addressing, it's
+important to know that the Internet Protocol is currently in the process
+of moving from version 4 of the protocol to version 6. This transition is
+occurring largely because version 4 of the protocol doesn't provide enough
+addresses to handle the needs of the whole world, especially given the
+increasing number of devices with direct connections to the internet.
+
+Explaining the details of the differences between the two versions of the
+protocol is beyond the scope of this introduction, but readers need to at
+least be aware that these two versions exist, and it will sometimes be
+necessary to force the use of one version or the other.
+
+
+IP Host Addresses
+-----------------
+
+Addresses, often referred to as "host addresses" are the most basic unit
+when working with IP addressing. The simplest way to create addresses is
+to use the ``ip_address`` factory function, which automatically determines
+whether to create an IPv4 or IPv6 address based on the passed in value::
+
+   >>> ipaddress.ip_address('192.0.2.1')
+   IPv4Address('192.0.2.1')
+   >>> ipaddress.ip_address('2001:DB8::1')
+   IPv6Address('2001:db8::1')
+
+Addresses can also be created directly from integers. Values that will
+fit within 32 bits are assumed to be IPv4 addresses::
+
+   >>> ipaddress.ip_address(3221225985)
+   IPv4Address('192.0.2.1')
+   >>> ipaddress.ip_address(42540766411282592856903984951653826561)
+   IPv6Address('2001:db8::1')
+
+To force the use of IPv4 or IPv6 addresses, the relevant classes can be
+invoked directly. This is particularly useful to force creation of IPv6
+addresses for small integers::
+
+   >>> ipaddress.ip_address(1)
+   IPv4Address('0.0.0.1')
+   >>> ipaddress.IPv4Address(1)
+   IPv4Address('0.0.0.1')
+   >>> ipaddress.IPv6Address(1)
+   IPv6Address('::1')
+
+
+Defining Networks
+-----------------
+
+Host addresses are usually grouped together into IP networks, so
+:mod:`ipaddress` provides a way to create, inspect and manipulate network
+definitions. IP network objects are constructed from strings that define the
+range of host addresses that are part of that network. The simplest form
+for that information is a "network address/network prefix" pair, where the
+prefix defines the number of leading bits that are compared to determine
+whether or not an address is part of the network and the network address
+defines the expected value of those bits.
+
+As for addresses, a factory function is provided that determines the correct
+IP version automatically::
+
+   >>> ipaddress.ip_network('192.0.2.0/24')
+   IPv4Network('192.0.2.0/24')
+   >>> ipaddress.ip_network('2001:db8::0/96')
+   IPv6Network('2001:db8::/96')
+
+Network objects cannot have any host bits set.  The practical effect of this
+is that ``192.0.2.1/24`` does not describe a network.  Such definitions are
+referred to as interface objects since the ip-on-a-network notation is
+commonly used to describe network interfaces of a computer on a given network
+and are described further in the next section.
+
+By default, attempting to create a network object with host bits set will
+result in :exc:`ValueError` being raised. To request that the
+additional bits instead be coerced to zero, the flag ``strict=False`` can
+be passed to the constructor::
+
+   >>> ipaddress.ip_network('192.0.2.1/24')
+   Traceback (most recent call last):
+      ...
+   ValueError: 192.0.2.1/24 has host bits set
+   >>> ipaddress.ip_network('192.0.2.1/24', strict=False)
+   IPv4Network('192.0.2.0/24')
+
+While the string form offers significantly more flexibility, networks can
+also be defined with integers, just like host addresses. In this case, the
+network is considered to contain only the single address identified by the
+integer, so the network prefix includes the entire network address::
+
+   >>> ipaddress.ip_network(3221225984)
+   IPv4Network('192.0.2.0/32')
+   >>> ipaddress.ip_network(42540766411282592856903984951653826560L)
+   IPv6Network('2001:db8::/128')
+
+Creation of a particular kind of network can be forced by calling the
+class constructor directly instead of using the factory function.
+
+
+Host Interfaces
+---------------
+
+As mentioned just above, if you need to describe an address on a particular
+network, neither the address nor the network classes are sufficient.
+Notation like ``192.0.2.1/24`` is commonly used network engineers and the
+people who write tools for firewalls and routers as shorthand for "the host
+``192.0.2.1`` on the network ``192.0.2.0/24``", Accordingly, :mod:`ipaddress`
+provides a set of hybrid classes that associate an address with a particular
+network. The interface for creation is identical to that for defining network
+objects, except that the address portion isn't constrained to being a network
+address.
+
+   >>> ipaddress.ip_interface('192.0.2.1/24')
+   IPv4Interface('192.0.2.1/24')
+   >>> ipaddress.ip_network('2001:db8::1/96')
+   IPv6Interface('2001:db8::1/96')
+
+Integer inputs are accepted (as with networks), and use of a particular IP
+version can be forced by calling the relevant constructor directly.
+
+
+Inspecting Address/Network/Interface Objects
+============================================
+
+You've gone to the trouble of creating an IPv(4|6)(Address|Network|Interface)
+object, so you probably want to get information about it.  :mod:`ipaddress`
+tries to make doing this easy and intuitive.
+
+Extracting the IP version::
+
+   >>> addr4 = ipaddress.ip_address('192.0.2.1')
+   >>> addr6 = ipaddress.ip_address('2001:db8::1')
+   >>> addr6.version
+   6
+   >>> addr4.version
+   4
+
+Obtaining the network from an interface::
+
+   >>> host4 = ipaddress.ip_interface('192.0.2.1/24')
+   >>> host4.network
+   IPv4Network('192.0.2.0/24')
+   >>> host6 = ipaddress.ip_interface('2001:db8::1/96')
+   >>> host6.network
+   IPv6Network('2001:db8::/96')
+
+Finding out how many individual addresses are in a network::
+
+   >>> net4 = ipaddress.ip_network('192.0.2.0/24')
+   >>> net4.numhosts
+   256
+   >>> net6 = ipaddress.ip_network('2001:db8::0/96')
+   >>> net6.numhosts
+   4294967296
+
+Iterating through the 'usable' addresses on a network::
+
+   >>> net4 = ipaddress.ip_network('192.0.2.0/24')
+   >>> for x in net4.iterhosts():
+          print(x)
+   192.0.2.1
+   192.0.2.2
+   192.0.2.3
+   192.0.2.4
+   <snip>
+   192.0.2.252
+   192.0.2.253
+   192.0.2.254
+
+
+Obtaining the netmask (i.e. set bits corresponding to the network prefix) or
+the hostmask (any bits that are not part of the netmask):
+
+   >>> net4 = ipaddress.ip_network('192.0.2.0/24')
+   >>> net4.netmask
+   IPv4Address('255.255.255.0')
+   >>> net4.hostmask
+   IPv4Address('0.0.0.255')
+   >>> net6 = ipaddress.ip_network('2001:db8::0/96')
+   >>> net6.netmask
+   IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::')
+   >>> net6.hostmask
+   IPv6Address('::ffff:ffff')
+
+
+Exploding or compressing the address::
+
+   >>> net6.exploded
+   '2001:0000:0000:0000:0000:0000:0000:0000/96'
+   >>> addr6.exploded
+   '2001:0000:0000:0000:0000:0000:0000:0001'
+
+
+Networks as lists of Addresses
+==============================
+
+It's sometimes useful to treat networks as lists.  This means it is possible
+to index them like this::
+
+   >>> net4[1]
+   IPv4Address('192.0.2.1')
+   >>> net4[-1]
+   IPv4Address('192.0.2.255')
+   >>> net6[1]
+   IPv6Address('2001::1')
+   >>> net6[-1]
+   IPv6Address('2001::ffff:ffff')
+
+
+It also means that network objects lend themselves to using the list
+membership test syntax like this::
+
+   if address in network:
+       # do something
+
+Containment testing is done efficiently based on the network prefix::
+
+   >>> addr4 = ipaddress.ip_address('192.0.2.1')
+   >>> addr4 in ipaddress.ip_network('192.0.2.0/24')
+   True
+   >>> addr4 in ipaddress.ip_network('192.0.3.0/24')
+   False
+
+
+Comparisons
+===========
+
+:mod:`ipaddress` provides some simple, hopefully intuitive ways to compare
+objects, where it makes sense::
+
+   >>> ipaddress.ip_address('192.0.2.1') < ipaddress.ip_address('192.0.2.2')
+   True
+
+A :exc:`TypeError` exception is raised if you try to compare objects of
+different versions or different types.
+
+
+Using IP Addresses with other modules
+=====================================
+
+Other modules that use IP addresses (such as :mod:`socket`) usually won't
+accept objects from this module directly. Instead, they must be coerced to
+an integer or string that the other module will accept::
+
+   >>> addr4 = ipaddress.ip_address('192.0.2.1')
+   >>> str(addr4)
+   '192.0.2.1'
+   >>> int(addr4)
+   3221225985
+
+
+Exceptions raised by :mod:`ipaddress`
+=====================================
+
+If you try to create an address/network/interface object with an invalid value
+for either the address or netmask, :mod:`ipaddress` will raise an
+:exc:`AddressValueError` or :exc:`NetmaskValueError` respectively. However,
+this applies only when calling the class constructors directly. The factory
+functions and other module level functions will just raise :exc:`ValueError`.
+
+Both of the module specific exceptions have :exc:`ValueError` as their
+parent class, so if you're not concerned with the particular type of error,
+you can still do the following::
+
+   try:
+       ipaddress.IPv4Address(address)
+   except ValueError:
+       print 'address/netmask is invalid: %s' % address

Doc/library/smtpd.rst

@@ -20,17 +20,24 @@ specific mail-sending strategies.
 Additionally the SMTPChannel may be extended to implement very specific
 interaction behaviour with SMTP clients.
 
+The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension.
+
+
 SMTPServer Objects
 ------------------
 
 
-.. class:: SMTPServer(localaddr, remoteaddr)
+.. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432)
 
    Create a new :class:`SMTPServer` object, which binds to local address
    *localaddr*.  It will treat *remoteaddr* as an upstream SMTP relayer.  It
    inherits from :class:`asyncore.dispatcher`, and so will insert itself into
    :mod:`asyncore`'s event loop on instantiation.
 
+   *data_size_limit* specifies the maximum number of bytes that will be
+   accepted in a ``DATA`` command.  A value of ``None`` or ``0`` means no
+   limit.
+
    .. method:: process_message(peer, mailfrom, rcpttos, data)
 
       Raise :exc:`NotImplementedError` exception. Override this in subclasses to
@@ -155,11 +162,15 @@ SMTPChannel Objects
    Command  Action taken
    ======== ===================================================================
    HELO     Accepts the greeting from the client and stores it in
-            :attr:`seen_greeting`.
+            :attr:`seen_greeting`.  Sets server to base command mode.
+   EHLO     Accepts the greeting from the client and stores it in
+            :attr:`seen_greeting`.  Sets server to extended command mode.
    NOOP     Takes no action.
    QUIT     Closes the connection cleanly.
    MAIL     Accepts the "MAIL FROM:" syntax and stores the supplied address as
-            :attr:`mailfrom`.
+            :attr:`mailfrom`.  In extended command mode, accepts the
+            :rfc:`1870` SIZE attribute and responds appropriately based on the
+            value of ``data_size_limit``.
    RCPT     Accepts the "RCPT TO:" syntax and stores the supplied addresses in
             the :attr:`rcpttos` list.
    RSET     Resets the :attr:`mailfrom`, :attr:`rcpttos`, and
@@ -167,4 +178,7 @@ SMTPChannel Objects
    DATA     Sets the internal state to :attr:`DATA` and stores remaining lines
             from the client in :attr:`received_data` until the terminator
             "\r\n.\r\n" is received.
+   HELP     Returns minimal information on command syntax
+   VRFY     Returns code 252 (the server doesn't know if the address is valid)
+   EXPN     Reports that the command is not implemented.
    ======== ===================================================================

Lib/email/_header_value_parser.py

@@ -791,6 +791,8 @@ class AngleAddr(TokenList):
         for x in self:
             if x.token_type == 'addr-spec':
                 return x.addr_spec
+        else:
+            return '<>'
 
 
 class ObsRoute(TokenList):
@@ -1829,6 +1831,14 @@ def get_angle_addr(value):
             "expected angle-addr but found '{}'".format(value))
     angle_addr.append(ValueTerminal('<', 'angle-addr-start'))
     value = value[1:]
+    # Although it is not legal per RFC5322, SMTP uses '<>' in certain
+    # circumstances.
+    if value[0] == '>':
+        angle_addr.append(ValueTerminal('>', 'angle-addr-end'))
+        angle_addr.defects.append(errors.InvalidHeaderDefect(
+            "null addr-spec in angle-addr"))
+        value = value[1:]
+        return angle_addr, value
     try:
         token, value = get_addr_spec(value)
     except errors.HeaderParseError:
@@ -1838,7 +1848,7 @@ def get_angle_addr(value):
                 "obsolete route specification in angle-addr"))
         except errors.HeaderParseError:
             raise errors.HeaderParseError(
-                "expected addr-spec or but found '{}'".format(value))
+                "expected addr-spec or obs-route but found '{}'".format(value))
         angle_addr.append(token)
         token, value = get_addr_spec(value)
     angle_addr.append(token)

Lib/importlib/_bootstrap.py

@@ -949,8 +949,6 @@ class NamespaceLoader:
     def module_repr(cls, module):
         return "<module '{}' (namespace)>".format(module.__name__)
 
-    @set_package
-    @set_loader
     @module_for_loader
     def load_module(self, module):
         """Load a namespace module."""

Lib/test/test_email/test__header_value_parser.py

@@ -1429,6 +1429,19 @@ class TestParser(TestEmailBase):
         self.assertIsNone(angle_addr.route)
         self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com')
 
+    def test_get_angle_addr_empty(self):
+        angle_addr = self._test_get_x(parser.get_angle_addr,
+            '<>',
+            '<>',
+            '<>',
+            [errors.InvalidHeaderDefect],
+            '')
+        self.assertEqual(angle_addr.token_type, 'angle-addr')
+        self.assertIsNone(angle_addr.local_part)
+        self.assertIsNone(angle_addr.domain)
+        self.assertIsNone(angle_addr.route)
+        self.assertEqual(angle_addr.addr_spec, '<>')
+
     def test_get_angle_addr_with_cfws(self):
         angle_addr = self._test_get_x(parser.get_angle_addr,
             ' (foo) <dinsdale@example.com>(bar)',
@@ -2007,7 +2020,7 @@ class TestParser(TestEmailBase):
         self.assertEqual(group.mailboxes,
                          group.all_mailboxes)
 
-    def test_get_troup_null_addr_spec(self):
+    def test_get_group_null_addr_spec(self):
         group = self._test_get_x(parser.get_group,
             'foo: <>;',
             'foo: <>;',

Lib/test/test_ipaddress.py

-#!/usr/bin/python3
-#
 # Copyright 2007 Google Inc.
 #  Licensed to PSF under a Contributor Agreement.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Unittest for ipaddressmodule."""
+
+"""Unittest for ipaddress module."""
 
 
 import unittest
@@ -404,7 +390,7 @@ class IpaddrUnitTest(unittest.TestCase):
         self.assertRaises(ValueError, list,
                           self.ipv4_interface.network.subnets(-1))
         self.assertRaises(ValueError, list,
-                          self.ipv4_network.network.subnets(-1))
+                          self.ipv4_network.subnets(-1))
         self.assertRaises(ValueError, list,
                           self.ipv6_interface.network.subnets(-1))
         self.assertRaises(ValueError, list,
@@ -780,12 +766,6 @@ class IpaddrUnitTest(unittest.TestCase):
         self.assertEqual(self.ipv4_address.version, 4)
         self.assertEqual(self.ipv6_address.version, 6)
 
-        with self.assertRaises(ValueError):
-            ipaddress.ip_address('1', version=[])
-
-        with self.assertRaises(ValueError):
-            ipaddress.ip_address('1', version=5)
-
     def testMaxPrefixLength(self):
         self.assertEqual(self.ipv4_interface.max_prefixlen, 32)
         self.assertEqual(self.ipv6_interface.max_prefixlen, 128)
@@ -1052,12 +1032,7 @@ class IpaddrUnitTest(unittest.TestCase):
 
     def testForceVersion(self):
         self.assertEqual(ipaddress.ip_network(1).version, 4)
-        self.assertEqual(ipaddress.ip_network(1, version=6).version, 6)
-
-        with self.assertRaises(ValueError):
-            ipaddress.ip_network(1, version='l')
-        with self.assertRaises(ValueError):
-            ipaddress.ip_network(1, version=3)
+        self.assertEqual(ipaddress.IPv6Network(1).version, 6)
 
     def testWithStar(self):
         self.assertEqual(str(self.ipv4_interface.with_prefixlen), "1.2.3.4/24")
@@ -1148,13 +1123,6 @@ class IpaddrUnitTest(unittest.TestCase):
                          sixtofouraddr.sixtofour)
         self.assertFalse(bad_addr.sixtofour)
 
-    def testIpInterfaceVersion(self):
-        with self.assertRaises(ValueError):
-            ipaddress.ip_interface(1, version=123)
-
-        with self.assertRaises(ValueError):
-            ipaddress.ip_interface(1, version='')
-
 
 if __name__ == '__main__':
     unittest.main()

Lib/test/test_logging.py

@@ -663,6 +663,7 @@ if threading:
             self.smtp_server = server
             self.conn = conn
             self.addr = addr
+            self.data_size_limit = None
             self.received_lines = []
             self.smtp_state = self.COMMAND
             self.seen_greeting = ''
@@ -682,6 +683,7 @@ if threading:
                 return
             self.push('220 %s %s' % (self.fqdn, smtpd.__version__))
             self.set_terminator(b'\r\n')
+            self.extended_smtp = False
 
 
     class TestSMTPServer(smtpd.SMTPServer):
@@ -709,6 +711,7 @@ if threading:
         def __init__(self, addr, handler, poll_interval, sockmap):
             self._localaddr = addr
             self._remoteaddr = None
+            self.data_size_limit = None
             self.sockmap = sockmap
             asyncore.dispatcher.__init__(self, map=sockmap)
             try:

Lib/test/test_smtplib.py

@@ -229,13 +229,13 @@ class DebuggingServerTests(unittest.TestCase):
 
     def testNOOP(self):
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        expected = (250, b'Ok')
+        expected = (250, b'OK')
         self.assertEqual(smtp.noop(), expected)
         smtp.quit()
 
     def testRSET(self):
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        expected = (250, b'Ok')
+        expected = (250, b'OK')
         self.assertEqual(smtp.rset(), expected)
         smtp.quit()
 
@@ -246,10 +246,18 @@ class DebuggingServerTests(unittest.TestCase):
         self.assertEqual(smtp.ehlo(), expected)
         smtp.quit()
 
+    def testNotImplemented(self):
+        # EXPN isn't implemented in DebuggingServer
+        smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
+        expected = (502, b'EXPN not implemented')
+        smtp.putcmd('EXPN')
+        self.assertEqual(smtp.getreply(), expected)
+        smtp.quit()
+
     def testVRFY(self):
-        # VRFY isn't implemented in DebuggingServer
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        expected = (502, b'Error: command "VRFY" not implemented')
+        expected = (252, b'Cannot VRFY user, but will accept message ' + \
+                         b'and attempt delivery')
         self.assertEqual(smtp.vrfy('nobody@nowhere.com'), expected)
         self.assertEqual(smtp.verify('nobody@nowhere.com'), expected)
         smtp.quit()
@@ -265,7 +273,8 @@ class DebuggingServerTests(unittest.TestCase):
 
     def testHELP(self):
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        self.assertEqual(smtp.help(), b'Error: command "HELP" not implemented')
+        self.assertEqual(smtp.help(), b'Supported commands: EHLO HELO MAIL ' + \
+                                      b'RCPT DATA RSET NOOP QUIT VRFY')
         smtp.quit()
 
     def testSend(self):

Misc/ACKS

@@ -112,6 +112,7 @@ Gregory Bond
 Matias Bordese
 Jurjen Bos
 Peter Bosch
+Dan Boswell
 Eric Bouck
 Thierry Bousch
 Sebastian Boving
@@ -494,6 +495,7 @@ Geert Jansen
 Jack Jansen
 Bill Janssen
 Thomas Jarosch
+Juhana Jauhiainen
 Zbigniew Jędrzejewski-Szmek
 Julien Jehannet
 Drew Jenkins
@@ -1039,6 +1041,7 @@ Sandro Tosi
 Richard Townsend
 David Townshend
 Laurence Tratt
+Alberto Trevino
 Matthias Troffaes
 John Tromp
 Jason Trowbridge

Misc/NEWS

@@ -46,6 +46,9 @@ Core and Builtins
 Library
 -------
 
+- Issue #8739: Updated smtpd to support RFC 5321, and added support for the
+  RFC 1870 SIZE extension.
+
 - Issue #665194: Added a localtime function to email.utils to provide an
   aware local datetime for use in setting Date headers.