Merged upstream changes.

Doc/howto/index.rst

@@ -28,4 +28,5 @@ Currently, the HOWTOs are:
    urllib2.rst
    webservers.rst
    argparse.rst
+   ipaddress.rst
 

Doc/howto/ipaddress.rst

+.. _ipaddress-howto:
+
+***************
+Ipaddress Howto
+***************
+
+:author: Peter Moody
+
+.. topic:: Abstract
+
+   This document is a gentle introduction to :mod:`ipaddress` module.
+
+
+Creating Address/Network/Interface objects
+==========================================
+
+Since :mod:`ipaddress` is a module for inspecting and manipulating IP address,
+the first thing you'll want to do is create some objects.  You can use
+:mod:`ipaddress` to create objects from strings and integers.
+
+
+A Note on IP Versions
+---------------------
+
+For readers that aren't particularly familiar with IP addressing, it's
+important to know that the Internet Protocol is currently in the process
+of moving from version 4 of the protocol to version 6. This transition is
+occurring largely because version 4 of the protocol doesn't provide enough
+addresses to handle the needs of the whole world, especially given the
+increasing number of devices with direct connections to the internet.
+
+Explaining the details of the differences between the two versions of the
+protocol is beyond the scope of this introduction, but readers need to at
+least be aware that these two versions exist, and it will sometimes be
+necessary to force the use of one version or the other.
+
+
+IP Host Addresses
+-----------------
+
+Addresses, often referred to as "host addresses" are the most basic unit
+when working with IP addressing. The simplest way to create addresses is
+to use the ``ip_address`` factory function, which automatically determines
+whether to create an IPv4 or IPv6 address based on the passed in value::
+
+   >>> ipaddress.ip_address('192.0.2.1')
+   IPv4Address('192.0.2.1')
+   >>> ipaddress.ip_address('2001:DB8::1')
+   IPv6Address('2001:db8::1')
+
+Addresses can also be created directly from integers. Values that will
+fit within 32 bits are assumed to be IPv4 addresses::
+
+   >>> ipaddress.ip_address(3221225985)
+   IPv4Address('192.0.2.1')
+   >>> ipaddress.ip_address(42540766411282592856903984951653826561)
+   IPv6Address('2001:db8::1')
+
+To force the use of IPv4 or IPv6 addresses, the relevant classes can be
+invoked directly. This is particularly useful to force creation of IPv6
+addresses for small integers::
+
+   >>> ipaddress.ip_address(1)
+   IPv4Address('0.0.0.1')
+   >>> ipaddress.IPv4Address(1)
+   IPv4Address('0.0.0.1')
+   >>> ipaddress.IPv6Address(1)
+   IPv6Address('::1')
+
+
+Defining Networks
+-----------------
+
+Host addresses are usually grouped together into IP networks, so
+:mod:`ipaddress` provides a way to create, inspect and manipulate network
+definitions. IP network objects are constructed from strings that define the
+range of host addresses that are part of that network. The simplest form
+for that information is a "network address/network prefix" pair, where the
+prefix defines the number of leading bits that are compared to determine
+whether or not an address is part of the network and the network address
+defines the expected value of those bits.
+
+As for addresses, a factory function is provided that determines the correct
+IP version automatically::
+
+   >>> ipaddress.ip_network('192.0.2.0/24')
+   IPv4Network('192.0.2.0/24')
+   >>> ipaddress.ip_network('2001:db8::0/96')
+   IPv6Network('2001:db8::/96')
+
+Network objects cannot have any host bits set.  The practical effect of this
+is that ``192.0.2.1/24`` does not describe a network.  Such definitions are
+referred to as interface objects since the ip-on-a-network notation is
+commonly used to describe network interfaces of a computer on a given network
+and are described further in the next section.
+
+By default, attempting to create a network object with host bits set will
+result in :exc:`ValueError` being raised. To request that the
+additional bits instead be coerced to zero, the flag ``strict=False`` can
+be passed to the constructor::
+
+   >>> ipaddress.ip_network('192.0.2.1/24')
+   Traceback (most recent call last):
+      ...
+   ValueError: 192.0.2.1/24 has host bits set
+   >>> ipaddress.ip_network('192.0.2.1/24', strict=False)
+   IPv4Network('192.0.2.0/24')
+
+While the string form offers significantly more flexibility, networks can
+also be defined with integers, just like host addresses. In this case, the
+network is considered to contain only the single address identified by the
+integer, so the network prefix includes the entire network address::
+
+   >>> ipaddress.ip_network(3221225984)
+   IPv4Network('192.0.2.0/32')
+   >>> ipaddress.ip_network(42540766411282592856903984951653826560L)
+   IPv6Network('2001:db8::/128')
+
+Creation of a particular kind of network can be forced by calling the
+class constructor directly instead of using the factory function.
+
+
+Host Interfaces
+---------------
+
+As mentioned just above, if you need to describe an address on a particular
+network, neither the address nor the network classes are sufficient.
+Notation like ``192.0.2.1/24`` is commonly used network engineers and the
+people who write tools for firewalls and routers as shorthand for "the host
+``192.0.2.1`` on the network ``192.0.2.0/24``", Accordingly, :mod:`ipaddress`
+provides a set of hybrid classes that associate an address with a particular
+network. The interface for creation is identical to that for defining network
+objects, except that the address portion isn't constrained to being a network
+address.
+
+   >>> ipaddress.ip_interface('192.0.2.1/24')
+   IPv4Interface('192.0.2.1/24')
+   >>> ipaddress.ip_network('2001:db8::1/96')
+   IPv6Interface('2001:db8::1/96')
+
+Integer inputs are accepted (as with networks), and use of a particular IP
+version can be forced by calling the relevant constructor directly.
+
+
+Inspecting Address/Network/Interface Objects
+============================================
+
+You've gone to the trouble of creating an IPv(4|6)(Address|Network|Interface)
+object, so you probably want to get information about it.  :mod:`ipaddress`
+tries to make doing this easy and intuitive.
+
+Extracting the IP version::
+
+   >>> addr4 = ipaddress.ip_address('192.0.2.1')
+   >>> addr6 = ipaddress.ip_address('2001:db8::1')
+   >>> addr6.version
+   6
+   >>> addr4.version
+   4
+
+Obtaining the network from an interface::
+
+   >>> host4 = ipaddress.ip_interface('192.0.2.1/24')
+   >>> host4.network
+   IPv4Network('192.0.2.0/24')
+   >>> host6 = ipaddress.ip_interface('2001:db8::1/96')
+   >>> host6.network
+   IPv6Network('2001:db8::/96')
+
+Finding out how many individual addresses are in a network::
+
+   >>> net4 = ipaddress.ip_network('192.0.2.0/24')
+   >>> net4.numhosts
+   256
+   >>> net6 = ipaddress.ip_network('2001:db8::0/96')
+   >>> net6.numhosts
+   4294967296
+
+Iterating through the 'usable' addresses on a network::
+
+   >>> net4 = ipaddress.ip_network('192.0.2.0/24')
+   >>> for x in net4.iterhosts():
+          print(x)
+   192.0.2.1
+   192.0.2.2
+   192.0.2.3
+   192.0.2.4
+   <snip>
+   192.0.2.252
+   192.0.2.253
+   192.0.2.254
+
+
+Obtaining the netmask (i.e. set bits corresponding to the network prefix) or
+the hostmask (any bits that are not part of the netmask):
+
+   >>> net4 = ipaddress.ip_network('192.0.2.0/24')
+   >>> net4.netmask
+   IPv4Address('255.255.255.0')
+   >>> net4.hostmask
+   IPv4Address('0.0.0.255')
+   >>> net6 = ipaddress.ip_network('2001:db8::0/96')
+   >>> net6.netmask
+   IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::')
+   >>> net6.hostmask
+   IPv6Address('::ffff:ffff')
+
+
+Exploding or compressing the address::
+
+   >>> net6.exploded
+   '2001:0000:0000:0000:0000:0000:0000:0000/96'
+   >>> addr6.exploded
+   '2001:0000:0000:0000:0000:0000:0000:0001'
+
+
+Networks as lists of Addresses
+==============================
+
+It's sometimes useful to treat networks as lists.  This means it is possible
+to index them like this::
+
+   >>> net4[1]
+   IPv4Address('192.0.2.1')
+   >>> net4[-1]
+   IPv4Address('192.0.2.255')
+   >>> net6[1]
+   IPv6Address('2001::1')
+   >>> net6[-1]
+   IPv6Address('2001::ffff:ffff')
+
+
+It also means that network objects lend themselves to using the list
+membership test syntax like this::
+
+   if address in network:
+       # do something
+
+Containment testing is done efficiently based on the network prefix::
+
+   >>> addr4 = ipaddress.ip_address('192.0.2.1')
+   >>> addr4 in ipaddress.ip_network('192.0.2.0/24')
+   True
+   >>> addr4 in ipaddress.ip_network('192.0.3.0/24')
+   False
+
+
+Comparisons
+===========
+
+:mod:`ipaddress` provides some simple, hopefully intuitive ways to compare
+objects, where it makes sense::
+
+   >>> ipaddress.ip_address('192.0.2.1') < ipaddress.ip_address('192.0.2.2')
+   True
+
+A :exc:`TypeError` exception is raised if you try to compare objects of
+different versions or different types.
+
+
+Using IP Addresses with other modules
+=====================================
+
+Other modules that use IP addresses (such as :mod:`socket`) usually won't
+accept objects from this module directly. Instead, they must be coerced to
+an integer or string that the other module will accept::
+
+   >>> addr4 = ipaddress.ip_address('192.0.2.1')
+   >>> str(addr4)
+   '192.0.2.1'
+   >>> int(addr4)
+   3221225985
+
+
+Exceptions raised by :mod:`ipaddress`
+=====================================
+
+If you try to create an address/network/interface object with an invalid value
+for either the address or netmask, :mod:`ipaddress` will raise an
+:exc:`AddressValueError` or :exc:`NetmaskValueError` respectively. However,
+this applies only when calling the class constructors directly. The factory
+functions and other module level functions will just raise :exc:`ValueError`.
+
+Both of the module specific exceptions have :exc:`ValueError` as their
+parent class, so if you're not concerned with the particular type of error,
+you can still do the following::
+
+   try:
+       ipaddress.IPv4Address(address)
+   except ValueError:
+       print 'address/netmask is invalid: %s' % address

Doc/library/smtpd.rst

@@ -20,17 +20,24 @@ specific mail-sending strategies.
 Additionally the SMTPChannel may be extended to implement very specific
 interaction behaviour with SMTP clients.
 
+The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension.
+
+
 SMTPServer Objects
 ------------------
 
 
-.. class:: SMTPServer(localaddr, remoteaddr)
+.. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432)
 
    Create a new :class:`SMTPServer` object, which binds to local address
    *localaddr*.  It will treat *remoteaddr* as an upstream SMTP relayer.  It
    inherits from :class:`asyncore.dispatcher`, and so will insert itself into
    :mod:`asyncore`'s event loop on instantiation.
 
+   *data_size_limit* specifies the maximum number of bytes that will be
+   accepted in a ``DATA`` command.  A value of ``None`` or ``0`` means no
+   limit.
+
    .. method:: process_message(peer, mailfrom, rcpttos, data)
 
       Raise :exc:`NotImplementedError` exception. Override this in subclasses to
@@ -155,11 +162,15 @@ SMTPChannel Objects
    Command  Action taken
    ======== ===================================================================
    HELO     Accepts the greeting from the client and stores it in
-            :attr:`seen_greeting`.
+            :attr:`seen_greeting`.  Sets server to base command mode.
+   EHLO     Accepts the greeting from the client and stores it in
+            :attr:`seen_greeting`.  Sets server to extended command mode.
    NOOP     Takes no action.
    QUIT     Closes the connection cleanly.
    MAIL     Accepts the "MAIL FROM:" syntax and stores the supplied address as
-            :attr:`mailfrom`.
+            :attr:`mailfrom`.  In extended command mode, accepts the
+            :rfc:`1870` SIZE attribute and responds appropriately based on the
+            value of ``data_size_limit``.
    RCPT     Accepts the "RCPT TO:" syntax and stores the supplied addresses in
             the :attr:`rcpttos` list.
    RSET     Resets the :attr:`mailfrom`, :attr:`rcpttos`, and
@@ -167,4 +178,7 @@ SMTPChannel Objects
    DATA     Sets the internal state to :attr:`DATA` and stores remaining lines
             from the client in :attr:`received_data` until the terminator
             "\r\n.\r\n" is received.
+   HELP     Returns minimal information on command syntax
+   VRFY     Returns code 252 (the server doesn't know if the address is valid)
+   EXPN     Reports that the command is not implemented.
    ======== ===================================================================

Lib/email/_header_value_parser.py

@@ -791,6 +791,8 @@ class AngleAddr(TokenList):
         for x in self:
             if x.token_type == 'addr-spec':
                 return x.addr_spec
+        else:
+            return '<>'
 
 
 class ObsRoute(TokenList):
@@ -1829,6 +1831,14 @@ def get_angle_addr(value):
             "expected angle-addr but found '{}'".format(value))
     angle_addr.append(ValueTerminal('<', 'angle-addr-start'))
     value = value[1:]
+    # Although it is not legal per RFC5322, SMTP uses '<>' in certain
+    # circumstances.
+    if value[0] == '>':
+        angle_addr.append(ValueTerminal('>', 'angle-addr-end'))
+        angle_addr.defects.append(errors.InvalidHeaderDefect(
+            "null addr-spec in angle-addr"))
+        value = value[1:]
+        return angle_addr, value
     try:
         token, value = get_addr_spec(value)
     except errors.HeaderParseError:
@@ -1838,7 +1848,7 @@ def get_angle_addr(value):
                 "obsolete route specification in angle-addr"))
         except errors.HeaderParseError:
             raise errors.HeaderParseError(
-                "expected addr-spec or but found '{}'".format(value))
+                "expected addr-spec or obs-route but found '{}'".format(value))
         angle_addr.append(token)
         token, value = get_addr_spec(value)
     angle_addr.append(token)

Lib/importlib/_bootstrap.py

@@ -949,8 +949,6 @@ class NamespaceLoader:
     def module_repr(cls, module):
         return "<module '{}' (namespace)>".format(module.__name__)
 
-    @set_package
-    @set_loader
     @module_for_loader
     def load_module(self, module):
         """Load a namespace module."""

Lib/ipaddress.py

 # Copyright 2007 Google Inc.
 #  Licensed to PSF under a Contributor Agreement.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-# implied. See the License for the specific language governing
-# permissions and limitations under the License.
 
 """A fast, lightweight IPv4/IPv6 manipulation library in Python.
 
@@ -36,34 +24,22 @@ class NetmaskValueError(ValueError):
     """A Value Error related to the netmask."""
 
 
-def ip_address(address, version=None):
+def ip_address(address):
     """Take an IP string/int and return an object of the correct type.
 
     Args:
         address: A string or integer, the IP address.  Either IPv4 or
           IPv6 addresses may be supplied; integers less than 2**32 will
           be considered to be IPv4 by default.
-        version: An integer, 4 or 6.  If set, don't try to automatically
-          determine what the IP address type is.  Important for things
-          like ip_address(1), which could be IPv4, '192.0.2.1',  or IPv6,
-          '2001:db8::1'.
 
     Returns:
         An IPv4Address or IPv6Address object.
 
     Raises:
         ValueError: if the *address* passed isn't either a v4 or a v6
-          address, or if the version is not None, 4, or 6.
+          address
 
     """
-    if version is not None:
-        if version == 4:
-            return IPv4Address(address)
-        elif version == 6:
-            return IPv6Address(address)
-        else:
-            raise ValueError()
-
     try:
         return IPv4Address(address)
     except (AddressValueError, NetmaskValueError):
@@ -78,35 +54,22 @@ def ip_address(address, version=None):
                      address)
 
 
-def ip_network(address, version=None, strict=True):
+def ip_network(address, strict=True):
     """Take an IP string/int and return an object of the correct type.
 
     Args:
         address: A string or integer, the IP network.  Either IPv4 or
           IPv6 networks may be supplied; integers less than 2**32 will
           be considered to be IPv4 by default.
-        version: An integer, 4 or 6.  If set, don't try to automatically
-          determine what the IP address type is.  Important for things
-          like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6,
-          '2001:db8::1/128'.
 
     Returns:
         An IPv4Network or IPv6Network object.
 
     Raises:
         ValueError: if the string passed isn't either a v4 or a v6
-          address. Or if the network has host bits set.  Or if the version
-          is not None, 4, or 6.
+          address. Or if the network has host bits set.
 
     """
-    if version is not None:
-        if version == 4:
-            return IPv4Network(address, strict)
-        elif version == 6:
-            return IPv6Network(address, strict)
-        else:
-            raise ValueError()
-
     try:
         return IPv4Network(address, strict)
     except (AddressValueError, NetmaskValueError):
@@ -121,24 +84,20 @@ def ip_network(address, version=None, strict=True):
                      address)
 
 
-def ip_interface(address, version=None):
+def ip_interface(address):
     """Take an IP string/int and return an object of the correct type.
 
     Args:
         address: A string or integer, the IP address.  Either IPv4 or
           IPv6 addresses may be supplied; integers less than 2**32 will
           be considered to be IPv4 by default.
-        version: An integer, 4 or 6.  If set, don't try to automatically
-          determine what the IP address type is.  Important for things
-          like ip_interface(1), which could be IPv4, '192.0.2.1/32', or IPv6,
-          '2001:db8::1/128'.
 
     Returns:
         An IPv4Interface or IPv6Interface object.
 
     Raises:
         ValueError: if the string passed isn't either a v4 or a v6
-          address.  Or if the version is not None, 4, or 6.
+          address.
 
     Notes:
         The IPv?Interface classes describe an Address on a particular
@@ -146,14 +105,6 @@ def ip_interface(address, version=None):
         and Network classes.
 
     """
-    if version is not None:
-        if version == 4:
-            return IPv4Interface(address)
-        elif version == 6:
-            return IPv6Interface(address)
-        else:
-            raise ValueError()
-
     try:
         return IPv4Interface(address)
     except (AddressValueError, NetmaskValueError):
@@ -281,7 +232,7 @@ def summarize_address_range(first, last):
             If the first and last objects are not the same version.
         ValueError:
             If the last object is not greater than the first.
-            If the version is not 4 or 6.
+            If the version of the first address is not 4 or 6.
 
     """
     if (not (isinstance(first, _BaseAddress) and
@@ -318,7 +269,7 @@ def summarize_address_range(first, last):
         if current == ip._ALL_ONES:
             break
         first_int = current + 1
-        first = ip_address(first_int, version=first._version)
+        first = first.__class__(first_int)
 
 
 def _collapse_addresses_recursive(addresses):
@@ -586,12 +537,12 @@ class _BaseAddress(_IPAddressBase):
     def __add__(self, other):
         if not isinstance(other, int):
             return NotImplemented
-        return ip_address(int(self) + other, version=self._version)
+        return self.__class__(int(self) + other)
 
     def __sub__(self, other):
         if not isinstance(other, int):
             return NotImplemented
-        return ip_address(int(self) - other, version=self._version)
+        return self.__class__(int(self) - other)
 
     def __repr__(self):
         return '%s(%r)' % (self.__class__.__name__, str(self))
@@ -612,13 +563,12 @@ class _BaseAddress(_IPAddressBase):
 
 class _BaseNetwork(_IPAddressBase):
 
-    """A generic IP object.
+    """A generic IP network object.
 
     This IP class contains the version independent methods which are
     used by networks.
 
     """
-
     def __init__(self, address):
         self._cache = {}
 
@@ -642,14 +592,14 @@ class _BaseNetwork(_IPAddressBase):
         bcast = int(self.broadcast_address) - 1
         while cur <= bcast:
             cur += 1
-            yield ip_address(cur - 1, version=self._version)
+            yield self._address_class(cur - 1)
 
     def __iter__(self):
         cur = int(self.network_address)
         bcast = int(self.broadcast_address)
         while cur <= bcast:
             cur += 1
-            yield ip_address(cur - 1, version=self._version)
+            yield self._address_class(cur - 1)
 
     def __getitem__(self, n):
         network = int(self.network_address)
@@ -657,12 +607,12 @@ class _BaseNetwork(_IPAddressBase):
         if n >= 0:
             if network + n > broadcast:
                 raise IndexError
-            return ip_address(network + n, version=self._version)
+            return self._address_class(network + n)
         else:
             n += 1
             if broadcast + n < network:
                 raise IndexError
-            return ip_address(broadcast + n, version=self._version)
+            return self._address_class(broadcast + n)
 
     def __lt__(self, other):
         if self._version != other._version:
@@ -746,8 +696,8 @@ class _BaseNetwork(_IPAddressBase):
     def broadcast_address(self):
         x = self._cache.get('broadcast_address')
         if x is None:
-            x = ip_address(int(self.network_address) | int(self.hostmask),
-                           version=self._version)
+            x = self._address_class(int(self.network_address) |
+                                    int(self.hostmask))
             self._cache['broadcast_address'] = x
         return x
 
@@ -755,16 +705,10 @@ class _BaseNetwork(_IPAddressBase):
     def hostmask(self):
         x = self._cache.get('hostmask')
         if x is None:
-            x = ip_address(int(self.netmask) ^ self._ALL_ONES,
-                          version=self._version)
+            x = self._address_class(int(self.netmask) ^ self._ALL_ONES)
             self._cache['hostmask'] = x
         return x
 
-    @property
-    def network(self):
-        return ip_network('%s/%d' % (str(self.network_address),
-                                     self.prefixlen))
-
     @property
     def with_prefixlen(self):
         return '%s/%d' % (str(self.ip), self._prefixlen)
@@ -786,6 +730,10 @@ class _BaseNetwork(_IPAddressBase):
     def version(self):
         raise NotImplementedError('BaseNet has no version')
 
+    @property
+    def _address_class(self):
+        raise NotImplementedError('BaseNet has no associated address class')
+
     @property
     def prefixlen(self):
         return self._prefixlen
@@ -840,9 +788,8 @@ class _BaseNetwork(_IPAddressBase):
             raise StopIteration
 
         # Make sure we're comparing the network of other.
-        other = ip_network('%s/%s' % (str(other.network_address),
-                                      str(other.prefixlen)),
-                           version=other._version)
+        other = other.__class__('%s/%s' % (str(other.network_address),
+                                           str(other.prefixlen)))
 
         s1, s2 = self.subnets()
         while s1 != other and s2 != other:
@@ -973,9 +920,9 @@ class _BaseNetwork(_IPAddressBase):
                 'prefix length diff %d is invalid for netblock %s' % (
                     new_prefixlen, str(self)))
 
-        first = ip_network('%s/%s' % (str(self.network_address),
-                                     str(self._prefixlen + prefixlen_diff)),
-                         version=self._version)
+        first = self.__class__('%s/%s' %
+                                 (str(self.network_address),
+                                  str(self._prefixlen + prefixlen_diff)))
 
         yield first
         current = first
@@ -983,17 +930,12 @@ class _BaseNetwork(_IPAddressBase):
             broadcast = current.broadcast_address
             if broadcast == self.broadcast_address:
                 return
-            new_addr = ip_address(int(broadcast) + 1, version=self._version)
-            current = ip_network('%s/%s' % (str(new_addr), str(new_prefixlen)),
-                                version=self._version)
+            new_addr = self._address_class(int(broadcast) + 1)
+            current = self.__class__('%s/%s' % (str(new_addr),
+                                                str(new_prefixlen)))
 
             yield current
 
-    def masked(self):
-        """Return the network object with the host bits masked out."""
-        return ip_network('%s/%d' % (self.network_address, self._prefixlen),
-                         version=self._version)
-
     def supernet(self, prefixlen_diff=1, new_prefix=None):
         """The supernet containing the current network.
 
@@ -1030,11 +972,10 @@ class _BaseNetwork(_IPAddressBase):
                 'current prefixlen is %d, cannot have a prefixlen_diff of %d' %
                 (self.prefixlen, prefixlen_diff))
         # TODO (pmoody): optimize this.
-        t = ip_network('%s/%d' % (str(self.network_address),
+        t = self.__class__('%s/%d' % (str(self.network_address),
                                       self.prefixlen - prefixlen_diff),
-                         version=self._version, strict=False)
-        return ip_network('%s/%d' % (str(t.network_address), t.prefixlen),
-                          version=t._version)
+                                     strict=False)
+        return t.__class__('%s/%d' % (str(t.network_address), t.prefixlen))
 
 
 class _BaseV4(object):
@@ -1391,6 +1332,9 @@ class IPv4Network(_BaseV4, _BaseNetwork):
         .prefixlen: 27
 
     """
+    # Class to use when creating address objects
+    # TODO (ncoghlan): Investigate using IPv4Interface instead
+    _address_class = IPv4Address
 
     # the valid octets for host and netmasks. only useful for IPv4.
     _valid_mask_octets = set((255, 254, 252, 248, 240, 224, 192, 128, 0))
@@ -1952,7 +1896,7 @@ class _BaseV6(object):
 
         """
         if isinstance(self, IPv6Network):
-            return int(self.network) == 1 and getattr(
+            return int(self) == 1 and getattr(
                 self, '_prefixlen', 128) == 128
         elif isinstance(self, IPv6Interface):
             return int(self.network.network_address) == 1 and getattr(
@@ -2071,6 +2015,10 @@ class IPv6Network(_BaseV6, _BaseNetwork):
 
     """
 
+    # Class to use when creating address objects
+    # TODO (ncoghlan): Investigate using IPv6Interface instead
+    _address_class = IPv6Address
+
     def __init__(self, address, strict=True):
         """Instantiate a new IPv6 Network object.
 

Lib/smtpd.py

 #! /usr/bin/env python3
-"""An RFC 2821 smtp proxy.
+"""An RFC 5321 smtp proxy.
 
 Usage: %(program)s [options] [localhost:localport [remotehost:remoteport]]
 
@@ -20,6 +20,11 @@ Options:
         Use `classname' as the concrete SMTP proxy class.  Uses `PureProxy' by
         default.
 
+    --size limit
+    -s limit
+        Restrict the total size of the incoming message to "limit" number of
+        bytes via the RFC 1870 SIZE extension.  Defaults to 33554432 bytes.
+
     --debug
     -d
         Turn on debugging prints.
@@ -35,10 +40,9 @@ given then 8025 is used.  If remotehost is not given then `localhost' is used,
 and if remoteport is not given, then 25 is used.
 """
 
-
 # Overview:
 #
-# This file implements the minimal SMTP protocol as defined in RFC 821.  It
+# This file implements the minimal SMTP protocol as defined in RFC 5321.  It
 # has a hierarchy of classes which implement the backend functionality for the
 # smtpd.  A number of classes are provided:
 #
@@ -66,7 +70,7 @@ and if remoteport is not given, then 25 is used.
 #
 # - support mailbox delivery
 # - alias files
-# - ESMTP
+# - Handle more ESMTP extensions
 # - handle error codes from the backend smtpd
 
 import sys
@@ -77,12 +81,14 @@ import time
 import socket
 import asyncore
 import asynchat
+import collections
 from warnings import warn
+from email._header_value_parser import get_addr_spec, get_angle_addr
 
 __all__ = ["SMTPServer","DebuggingServer","PureProxy","MailmanProxy"]
 
 program = sys.argv[0]
-__version__ = 'Python SMTP proxy version 0.2'
+__version__ = 'Python SMTP proxy version 0.3'
 
 
 class Devnull:
@@ -94,9 +100,9 @@ DEBUGSTREAM = Devnull()
 NEWLINE = '\n'
 EMPTYSTRING = ''
 COMMASPACE = ', '
+DATA_SIZE_DEFAULT = 33554432
 
 
-
 def usage(code, msg=''):
     print(__doc__ % globals(), file=sys.stderr)
     if msg:
@@ -104,19 +110,23 @@ def usage(code, msg=''):
     sys.exit(code)
 
 
-
 class SMTPChannel(asynchat.async_chat):
     COMMAND = 0
     DATA = 1
 
-    data_size_limit = 33554432
     command_size_limit = 512
+    command_size_limits = collections.defaultdict(lambda x=command_size_limit: x)
+    command_size_limits.update({
+        'MAIL': command_size_limit + 26,
+        })
+    max_command_size_limit = max(command_size_limits.values())
 
-    def __init__(self, server, conn, addr):
+    def __init__(self, server, conn, addr, data_size_limit=DATA_SIZE_DEFAULT):
         asynchat.async_chat.__init__(self, conn)
         self.smtp_server = server
         self.conn = conn
         self.addr = addr
+        self.data_size_limit = data_size_limit
         self.received_lines = []
         self.smtp_state = self.COMMAND
         self.seen_greeting = ''
@@ -137,6 +147,7 @@ class SMTPChannel(asynchat.async_chat):
         print('Peer:', repr(self.peer), file=DEBUGSTREAM)
         self.push('220 %s %s' % (self.fqdn, __version__))
         self.set_terminator(b'\r\n')
+        self.extended_smtp = False
 
     # properties for backwards-compatibility
     @property
@@ -268,7 +279,7 @@ class SMTPChannel(asynchat.async_chat):
     def collect_incoming_data(self, data):
         limit = None
         if self.smtp_state == self.COMMAND:
-            limit = self.command_size_limit
+            limit = self.max_command_size_limit
         elif self.smtp_state == self.DATA:
             limit = self.data_size_limit
         if limit and self.num_bytes > limit:
@@ -283,11 +294,7 @@ class SMTPChannel(asynchat.async_chat):
         print('Data:', repr(line), file=DEBUGSTREAM)
         self.received_lines = []
         if self.smtp_state == self.COMMAND:
-            if self.num_bytes > self.command_size_limit:
-                self.push('500 Error: line too long')
-                self.num_bytes = 0
-                return
-            self.num_bytes = 0
+            sz, self.num_bytes = self.num_bytes, 0
             if not line:
                 self.push('500 Error: bad syntax')
                 return
@@ -299,9 +306,14 @@ class SMTPChannel(asynchat.async_chat):
             else:
                 command = line[:i].upper()
                 arg = line[i+1:].strip()
+            max_sz = (self.command_size_limits[command]
+                        if self.extended_smtp else self.command_size_limit)
+            if sz > max_sz:
+                self.push('500 Error: line too long')
+                return
             method = getattr(self, 'smtp_' + command, None)
             if not method:
-                self.push('502 Error: command "%s" not implemented' % command)
+                self.push('500 Error: command "%s" not recognized' % command)
                 return
             method(arg)
             return
@@ -310,12 +322,12 @@ class SMTPChannel(asynchat.async_chat):
                 self.push('451 Internal confusion')
                 self.num_bytes = 0
                 return
-            if self.num_bytes > self.data_size_limit:
+            if self.data_size_limit and self.num_bytes > self.data_size_limit:
                 self.push('552 Error: Too much mail data')
                 self.num_bytes = 0
                 return
             # Remove extraneous carriage returns and de-transparency according
-            # to RFC 821, Section 4.5.2.
+            # to RFC 5321, Section 4.5.2.
             data = []
             for text in line.split('\r\n'):
                 if text and text[0] == '.':
@@ -333,7 +345,7 @@ class SMTPChannel(asynchat.async_chat):
             self.num_bytes = 0
             self.set_terminator(b'\r\n')
             if not status:
-                self.push('250 Ok')
+                self.push('250 OK')
             else:
                 self.push(status)
 
@@ -346,66 +358,188 @@ class SMTPChannel(asynchat.async_chat):
             self.push('503 Duplicate HELO/EHLO')
         else:
             self.seen_greeting = arg
+            self.extended_smtp = False
             self.push('250 %s' % self.fqdn)
 
+    def smtp_EHLO(self, arg):
+        if not arg:
+            self.push('501 Syntax: EHLO hostname')
+            return
+        if self.seen_greeting:
+            self.push('503 Duplicate HELO/EHLO')
+        else:
+            self.seen_greeting = arg
+            self.extended_smtp = True
+            self.push('250-%s' % self.fqdn)
+            if self.data_size_limit:
+                self.push('250-SIZE %s' % self.data_size_limit)
+            self.push('250 HELP')
+
     def smtp_NOOP(self, arg):
         if arg:
             self.push('501 Syntax: NOOP')
         else:
-            self.push('250 Ok')
+            self.push('250 OK')
 
     def smtp_QUIT(self, arg):
         # args is ignored
         self.push('221 Bye')
         self.close_when_done()
 
-    # factored
-    def __getaddr(self, keyword, arg):
-        address = None
+    def _strip_command_keyword(self, keyword, arg):
         keylen = len(keyword)
         if arg[:keylen].upper() == keyword:
-            address = arg[keylen:].strip()
+            return arg[keylen:].strip()
+        return ''
+
+    def _getaddr(self, arg):
+        if not arg:
+            return '', ''
+        if arg.lstrip().startswith('<'):
+            address, rest = get_angle_addr(arg)
+        else:
+            address, rest = get_addr_spec(arg)
         if not address:
-                pass
-            elif address[0] == '<' and address[-1] == '>' and address != '<>':
-                # Addresses can be in the form <person@dom.com> but watch out
-                # for null address, e.g. <>
-                address = address[1:-1]
-        return address
+            return address, rest
+        return address.addr_spec, rest
+
+    def _getparams(self, params):
+        # Return any parameters that appear to be syntactically valid according
+        # to RFC 1869, ignore all others.  (Postel rule: accept what we can.)
+        params = [param.split('=', 1) for param in params.split()
+                                      if '=' in param]
+        return {k: v for k, v in params if k.isalnum()}
+
+    def smtp_HELP(self, arg):
+        if arg:
+            extended = ' [SP <mail parameters]'
+            lc_arg = arg.upper()
+            if lc_arg == 'EHLO':
+                self.push('250 Syntax: EHLO hostname')
+            elif lc_arg == 'HELO':
+                self.push('250 Syntax: HELO hostname')
+            elif lc_arg == 'MAIL':
+                msg = '250 Syntax: MAIL FROM: <address>'
+                if self.extended_smtp:
+                    msg += extended
+                self.push(msg)
+            elif lc_arg == 'RCPT':
+                msg = '250 Syntax: RCPT TO: <address>'
+                if self.extended_smtp:
+                    msg += extended
+                self.push(msg)
+            elif lc_arg == 'DATA':
+                self.push('250 Syntax: DATA')
+            elif lc_arg == 'RSET':
+                self.push('250 Syntax: RSET')
+            elif lc_arg == 'NOOP':
+                self.push('250 Syntax: NOOP')
+            elif lc_arg == 'QUIT':
+                self.push('250 Syntax: QUIT')
+            elif lc_arg == 'VRFY':
+                self.push('250 Syntax: VRFY <address>')
+            else:
+                self.push('501 Supported commands: EHLO HELO MAIL RCPT '
+                          'DATA RSET NOOP QUIT VRFY')
+        else:
+            self.push('250 Supported commands: EHLO HELO MAIL RCPT DATA '
+                      'RSET NOOP QUIT VRFY')
+
+    def smtp_VRFY(self, arg):
+        if arg:
+            address, params = self._getaddr(arg)
+            if address:
+                self.push('252 Cannot VRFY user, but will accept message '
+                          'and attempt delivery')
+            else:
+                self.push('502 Could not VRFY %s' % arg)
+        else:
+            self.push('501 Syntax: VRFY <address>')
 
     def smtp_MAIL(self, arg):
         if not self.seen_greeting:
             self.push('503 Error: send HELO first');
             return
-
         print('===> MAIL', arg, file=DEBUGSTREAM)
-        address = self.__getaddr('FROM:', arg) if arg else None
+        syntaxerr = '501 Syntax: MAIL FROM: <address>'
+        if self.extended_smtp:
+            syntaxerr += ' [SP <mail-parameters>]'
+        if arg is None:
+            self.push(syntaxerr)
+            return
+        arg = self._strip_command_keyword('FROM:', arg)
+        address, params = self._getaddr(arg)
         if not address:
-            self.push('501 Syntax: MAIL FROM:<address>')
+            self.push(syntaxerr)
+            return
+        if not self.extended_smtp and params:
+            self.push(syntaxerr)
+            return
+        if not address:
+            self.push(syntaxerr)
             return
         if self.mailfrom:
             self.push('503 Error: nested MAIL command')
             return
+        params = self._getparams(params.upper())
+        if params is None:
+            self.push(syntaxerr)
+            return
+        size = params.pop('SIZE', None)
+        if size:
+            if not size.isdigit():
+                self.push(syntaxerr)
+                return
+            elif self.data_size_limit and int(size) > self.data_size_limit:
+                self.push('552 Error: message size exceeds fixed maximum message size')
+                return
+        if len(params.keys()) > 0:
+            self.push('555 MAIL FROM parameters not recognized or not implemented')
+            return
         self.mailfrom = address
         print('sender:', self.mailfrom, file=DEBUGSTREAM)
-        self.push('250 Ok')
+        self.push('250 OK')
 
     def smtp_RCPT(self, arg):
         if not self.seen_greeting:
             self.push('503 Error: send HELO first');
             return
-
         print('===> RCPT', arg, file=DEBUGSTREAM)
         if not self.mailfrom:
             self.push('503 Error: need MAIL command')
             return
-        address = self.__getaddr('TO:', arg) if arg else None
+        syntaxerr = '501 Syntax: RCPT TO: <address>'
+        if self.extended_smtp:
+            syntaxerr += ' [SP <mail-parameters>]'
+        if arg is None:
+            self.push(syntaxerr)
+            return
+        arg = self._strip_command_keyword('TO:', arg)
+        address, params = self._getaddr(arg)
+        if not address:
+            self.push(syntaxerr)
+            return
+        if params:
+            if self.extended_smtp:
+                params = self._getparams(params.upper())
+                if params is None:
+                    self.push(syntaxerr)
+                    return
+            else:
+                self.push(syntaxerr)
+                return
+        if not address:
+            self.push(syntaxerr)
+            return
+        if params and len(params.keys()) > 0:
+            self.push('555 RCPT TO parameters not recognized or not implemented')
+            return
         if not address:
             self.push('501 Syntax: RCPT TO: <address>')
             return
         self.rcpttos.append(address)
         print('recips:', self.rcpttos, file=DEBUGSTREAM)
-        self.push('250 Ok')
+        self.push('250 OK')
 
     def smtp_RSET(self, arg):
         if arg:
@@ -416,13 +550,12 @@ class SMTPChannel(asynchat.async_chat):
         self.rcpttos = []
         self.received_data = ''
         self.smtp_state = self.COMMAND
-        self.push('250 Ok')
+        self.push('250 OK')
 
     def smtp_DATA(self, arg):
         if not self.seen_greeting:
             self.push('503 Error: send HELO first');
             return
-
         if not self.rcpttos:
             self.push('503 Error: need RCPT command')
             return
@@ -433,15 +566,20 @@ class SMTPChannel(asynchat.async_chat):
         self.set_terminator(b'\r\n.\r\n')
         self.push('354 End data with <CR><LF>.<CR><LF>')
 
+    # Commands that have not been implemented
+    def smtp_EXPN(self, arg):
+        self.push('502 EXPN not implemented')
+
 
-
 class SMTPServer(asyncore.dispatcher):
     # SMTPChannel class to use for managing client connections
     channel_class = SMTPChannel
 
-    def __init__(self, localaddr, remoteaddr):
+    def __init__(self, localaddr, remoteaddr,
+                 data_size_limit=DATA_SIZE_DEFAULT):
         self._localaddr = localaddr
         self._remoteaddr = remoteaddr
+        self.data_size_limit = data_size_limit
         asyncore.dispatcher.__init__(self)
         try:
             self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
@@ -459,7 +597,7 @@ class SMTPServer(asyncore.dispatcher):
 
     def handle_accepted(self, conn, addr):
         print('Incoming connection from %s' % repr(addr), file=DEBUGSTREAM)
-        channel = self.channel_class(self, conn, addr)
+        channel = self.channel_class(self, conn, addr, self.data_size_limit)
 
     # API for "doing something useful with the message"
     def process_message(self, peer, mailfrom, rcpttos, data):
@@ -487,7 +625,6 @@ class SMTPServer(asyncore.dispatcher):
         raise NotImplementedError
 
 
-
 class DebuggingServer(SMTPServer):
     # Do something with the gathered message
     def process_message(self, peer, mailfrom, rcpttos, data):
@@ -503,7 +640,6 @@ class DebuggingServer(SMTPServer):
         print('------------ END MESSAGE ------------')
 
 
-
 class PureProxy(SMTPServer):
     def process_message(self, peer, mailfrom, rcpttos, data):
         lines = data.split('\n')
@@ -544,7 +680,6 @@ class PureProxy(SMTPServer):
         return refused
 
 
-
 class MailmanProxy(PureProxy):
     def process_message(self, peer, mailfrom, rcpttos, data):
         from io import StringIO
@@ -623,19 +758,18 @@ class MailmanProxy(PureProxy):
                 msg.Enqueue(mlist, torequest=1)
 
 
-
 class Options:
     setuid = 1
     classname = 'PureProxy'
+    size_limit = None
 
 
-
 def parseargs():
     global DEBUGSTREAM
     try:
         opts, args = getopt.getopt(
-            sys.argv[1:], 'nVhc:d',
-            ['class=', 'nosetuid', 'version', 'help', 'debug'])
+            sys.argv[1:], 'nVhc:s:d',
+            ['class=', 'nosetuid', 'version', 'help', 'size=', 'debug'])
     except getopt.error as e:
         usage(1, e)
 
@@ -652,6 +786,13 @@ def parseargs():
             options.classname = arg
         elif opt in ('-d', '--debug'):
             DEBUGSTREAM = sys.stderr
+        elif opt in ('-s', '--size'):
+            try:
+                int_size = int(arg)
+                options.size_limit = int_size
+            except:
+                print('Invalid size: ' + arg, file=sys.stderr)
+                sys.exit(1)
 
     # parse the rest of the arguments
     if len(args) < 1:
@@ -686,7 +827,6 @@ def parseargs():
     return options
 
 
-
 if __name__ == '__main__':
     options = parseargs()
     # Become nobody
@@ -699,7 +839,8 @@ if __name__ == '__main__':
         import __main__ as mod
     class_ = getattr(mod, classname)
     proxy = class_((options.localhost, options.localport),
-                   (options.remotehost, options.remoteport))
+                   (options.remotehost, options.remoteport),
+                   options.size_limit)
     if options.setuid:
         try:
             import pwd

Lib/test/test_email/test__header_value_parser.py

@@ -1429,6 +1429,19 @@ class TestParser(TestEmailBase):
         self.assertIsNone(angle_addr.route)
         self.assertEqual(angle_addr.addr_spec, 'dinsdale@example.com')
 
+    def test_get_angle_addr_empty(self):
+        angle_addr = self._test_get_x(parser.get_angle_addr,
+            '<>',
+            '<>',
+            '<>',
+            [errors.InvalidHeaderDefect],
+            '')
+        self.assertEqual(angle_addr.token_type, 'angle-addr')
+        self.assertIsNone(angle_addr.local_part)
+        self.assertIsNone(angle_addr.domain)
+        self.assertIsNone(angle_addr.route)
+        self.assertEqual(angle_addr.addr_spec, '<>')
+
     def test_get_angle_addr_with_cfws(self):
         angle_addr = self._test_get_x(parser.get_angle_addr,
             ' (foo) <dinsdale@example.com>(bar)',
@@ -2007,7 +2020,7 @@ class TestParser(TestEmailBase):
         self.assertEqual(group.mailboxes,
                          group.all_mailboxes)
 
-    def test_get_troup_null_addr_spec(self):
+    def test_get_group_null_addr_spec(self):
         group = self._test_get_x(parser.get_group,
             'foo: <>;',
             'foo: <>;',

Lib/test/test_ipaddress.py

-#!/usr/bin/python3
-#
 # Copyright 2007 Google Inc.
 #  Licensed to PSF under a Contributor Agreement.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Unittest for ipaddressmodule."""
+
+"""Unittest for ipaddress module."""
 
 
 import unittest
@@ -404,7 +390,7 @@ class IpaddrUnitTest(unittest.TestCase):
         self.assertRaises(ValueError, list,
                           self.ipv4_interface.network.subnets(-1))
         self.assertRaises(ValueError, list,
-                          self.ipv4_network.network.subnets(-1))
+                          self.ipv4_network.subnets(-1))
         self.assertRaises(ValueError, list,
                           self.ipv6_interface.network.subnets(-1))
         self.assertRaises(ValueError, list,
@@ -780,12 +766,6 @@ class IpaddrUnitTest(unittest.TestCase):
         self.assertEqual(self.ipv4_address.version, 4)
         self.assertEqual(self.ipv6_address.version, 6)
 
-        with self.assertRaises(ValueError):
-            ipaddress.ip_address('1', version=[])
-
-        with self.assertRaises(ValueError):
-            ipaddress.ip_address('1', version=5)
-
     def testMaxPrefixLength(self):
         self.assertEqual(self.ipv4_interface.max_prefixlen, 32)
         self.assertEqual(self.ipv6_interface.max_prefixlen, 128)
@@ -1052,12 +1032,7 @@ class IpaddrUnitTest(unittest.TestCase):
 
     def testForceVersion(self):
         self.assertEqual(ipaddress.ip_network(1).version, 4)
-        self.assertEqual(ipaddress.ip_network(1, version=6).version, 6)
-
-        with self.assertRaises(ValueError):
-            ipaddress.ip_network(1, version='l')
-        with self.assertRaises(ValueError):
-            ipaddress.ip_network(1, version=3)
+        self.assertEqual(ipaddress.IPv6Network(1).version, 6)
 
     def testWithStar(self):
         self.assertEqual(str(self.ipv4_interface.with_prefixlen), "1.2.3.4/24")
@@ -1148,13 +1123,6 @@ class IpaddrUnitTest(unittest.TestCase):
                          sixtofouraddr.sixtofour)
         self.assertFalse(bad_addr.sixtofour)
 
-    def testIpInterfaceVersion(self):
-        with self.assertRaises(ValueError):
-            ipaddress.ip_interface(1, version=123)
-
-        with self.assertRaises(ValueError):
-            ipaddress.ip_interface(1, version='')
-
 
 if __name__ == '__main__':
     unittest.main()

Lib/test/test_logging.py

@@ -663,6 +663,7 @@ if threading:
             self.smtp_server = server
             self.conn = conn
             self.addr = addr
+            self.data_size_limit = None
             self.received_lines = []
             self.smtp_state = self.COMMAND
             self.seen_greeting = ''
@@ -682,6 +683,7 @@ if threading:
                 return
             self.push('220 %s %s' % (self.fqdn, smtpd.__version__))
             self.set_terminator(b'\r\n')
+            self.extended_smtp = False
 
 
     class TestSMTPServer(smtpd.SMTPServer):
@@ -709,6 +711,7 @@ if threading:
         def __init__(self, addr, handler, poll_interval, sockmap):
             self._localaddr = addr
             self._remoteaddr = None
+            self.data_size_limit = None
             self.sockmap = sockmap
             asyncore.dispatcher.__init__(self, map=sockmap)
             try:

Lib/test/test_smtpd.py

-from unittest import TestCase
+import unittest
 from test import support, mock_socket
 import socket
 import io
@@ -26,7 +26,7 @@ class BrokenDummyServer(DummyServer):
         raise DummyDispatcherBroken()
 
 
-class SMTPDServerTest(TestCase):
+class SMTPDServerTest(unittest.TestCase):
     def setUp(self):
         smtpd.socket = asyncore.socket = mock_socket
 
@@ -39,7 +39,7 @@ class SMTPDServerTest(TestCase):
             channel.socket.queue_recv(line)
             channel.handle_read()
 
-        write_line(b'HELO test.example')
+        write_line(b'HELO example')
         write_line(b'MAIL From:eggs@example')
         write_line(b'RCPT To:spam@example')
         write_line(b'DATA')
@@ -50,7 +50,7 @@ class SMTPDServerTest(TestCase):
         asyncore.socket = smtpd.socket = socket
 
 
-class SMTPDChannelTest(TestCase):
+class SMTPDChannelTest(unittest.TestCase):
     def setUp(self):
         smtpd.socket = asyncore.socket = mock_socket
         self.old_debugstream = smtpd.DEBUGSTREAM
@@ -79,36 +79,94 @@ class SMTPDChannelTest(TestCase):
         self.assertEqual(self.channel.socket.last,
                          b'500 Error: bad syntax\r\n')
 
-    def test_EHLO_not_implemented(self):
-        self.write_line(b'EHLO test.example')
+    def test_EHLO(self):
+        self.write_line(b'EHLO example')
+        self.assertEqual(self.channel.socket.last, b'250 HELP\r\n')
+
+    def test_EHLO_bad_syntax(self):
+        self.write_line(b'EHLO')
+        self.assertEqual(self.channel.socket.last,
+                         b'501 Syntax: EHLO hostname\r\n')
+
+    def test_EHLO_duplicate(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'EHLO example')
+        self.assertEqual(self.channel.socket.last,
+                         b'503 Duplicate HELO/EHLO\r\n')
+
+    def test_EHLO_HELO_duplicate(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'HELO example')
         self.assertEqual(self.channel.socket.last,
-                         b'502 Error: command "EHLO" not implemented\r\n')
+                         b'503 Duplicate HELO/EHLO\r\n')
 
     def test_HELO(self):
         name = smtpd.socket.getfqdn()
-        self.write_line(b'HELO test.example')
+        self.write_line(b'HELO example')
         self.assertEqual(self.channel.socket.last,
                          '250 {}\r\n'.format(name).encode('ascii'))
 
+    def test_HELO_EHLO_duplicate(self):
+        self.write_line(b'HELO example')
+        self.write_line(b'EHLO example')
+        self.assertEqual(self.channel.socket.last,
+                         b'503 Duplicate HELO/EHLO\r\n')
+
+    def test_HELP(self):
+        self.write_line(b'HELP')
+        self.assertEqual(self.channel.socket.last,
+                         b'250 Supported commands: EHLO HELO MAIL RCPT ' + \
+                         b'DATA RSET NOOP QUIT VRFY\r\n')
+
+    def test_HELP_command(self):
+        self.write_line(b'HELP MAIL')
+        self.assertEqual(self.channel.socket.last,
+                         b'250 Syntax: MAIL FROM: <address>\r\n')
+
+    def test_HELP_command_unknown(self):
+        self.write_line(b'HELP SPAM')
+        self.assertEqual(self.channel.socket.last,
+                         b'501 Supported commands: EHLO HELO MAIL RCPT ' + \
+                         b'DATA RSET NOOP QUIT VRFY\r\n')
+
     def test_HELO_bad_syntax(self):
         self.write_line(b'HELO')
         self.assertEqual(self.channel.socket.last,
                          b'501 Syntax: HELO hostname\r\n')
 
     def test_HELO_duplicate(self):
-        self.write_line(b'HELO test.example')
-        self.write_line(b'HELO test.example')
+        self.write_line(b'HELO example')
+        self.write_line(b'HELO example')
         self.assertEqual(self.channel.socket.last,
                          b'503 Duplicate HELO/EHLO\r\n')
 
+    def test_HELO_parameter_rejected_when_extensions_not_enabled(self):
+        self.extended_smtp = False
+        self.write_line(b'HELO example')
+        self.write_line(b'MAIL from:<foo@example.com> SIZE=1234')
+        self.assertEqual(self.channel.socket.last,
+                         b'501 Syntax: MAIL FROM: <address>\r\n')
+
+    def test_MAIL_allows_space_after_colon(self):
+        self.write_line(b'HELO example')
+        self.write_line(b'MAIL from:   <foo@example.com>')
+        self.assertEqual(self.channel.socket.last,
+                         b'250 OK\r\n')
+
+    def test_extended_MAIL_allows_space_after_colon(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from:   <foo@example.com> size=20')
+        self.assertEqual(self.channel.socket.last,
+                         b'250 OK\r\n')
+
     def test_NOOP(self):
         self.write_line(b'NOOP')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
 
     def test_HELO_NOOP(self):
         self.write_line(b'HELO example')
         self.write_line(b'NOOP')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
 
     def test_NOOP_bad_syntax(self):
         self.write_line(b'NOOP hi')
@@ -136,15 +194,29 @@ class SMTPDChannelTest(TestCase):
 
     def test_command_too_long(self):
         self.write_line(b'HELO example')
-        self.write_line(b'MAIL from ' +
+        self.write_line(b'MAIL from: ' +
                         b'a' * self.channel.command_size_limit +
                         b'@example')
         self.assertEqual(self.channel.socket.last,
                          b'500 Error: line too long\r\n')
 
-    def test_data_too_long(self):
-        # Small hack. Setting limit to 2K octets here will save us some time.
-        self.channel.data_size_limit = 2048
+    def test_MAIL_command_limit_extended_with_SIZE(self):
+        self.write_line(b'EHLO example')
+        fill_len = self.channel.command_size_limit - len('MAIL from:<@example>')
+        self.write_line(b'MAIL from:<' +
+                        b'a' * fill_len +
+                        b'@example> SIZE=1234')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+
+        self.write_line(b'MAIL from:<' +
+                        b'a' * (fill_len + 26) +
+                        b'@example> SIZE=1234')
+        self.assertEqual(self.channel.socket.last,
+                         b'500 Error: line too long\r\n')
+
+    def test_data_longer_than_default_data_size_limit(self):
+        # Hack the default so we don't have to generate so much data.
+        self.channel.data_size_limit = 1048
         self.write_line(b'HELO example')
         self.write_line(b'MAIL From:eggs@example')
         self.write_line(b'RCPT To:spam@example')
@@ -154,28 +226,93 @@ class SMTPDChannelTest(TestCase):
         self.assertEqual(self.channel.socket.last,
                          b'552 Error: Too much mail data\r\n')
 
+    def test_MAIL_size_parameter(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL FROM:<eggs@example> SIZE=512')
+        self.assertEqual(self.channel.socket.last,
+                         b'250 OK\r\n')
+
+    def test_MAIL_invalid_size_parameter(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL FROM:<eggs@example> SIZE=invalid')
+        self.assertEqual(self.channel.socket.last,
+            b'501 Syntax: MAIL FROM: <address> [SP <mail-parameters>]\r\n')
+
+    def test_MAIL_RCPT_unknown_parameters(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL FROM:<eggs@example> ham=green')
+        self.assertEqual(self.channel.socket.last,
+            b'555 MAIL FROM parameters not recognized or not implemented\r\n')
+
+        self.write_line(b'MAIL FROM:<eggs@example>')
+        self.write_line(b'RCPT TO:<eggs@example> ham=green')
+        self.assertEqual(self.channel.socket.last,
+            b'555 RCPT TO parameters not recognized or not implemented\r\n')
+
+    def test_MAIL_size_parameter_larger_than_default_data_size_limit(self):
+        self.channel.data_size_limit = 1048
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL FROM:<eggs@example> SIZE=2096')
+        self.assertEqual(self.channel.socket.last,
+            b'552 Error: message size exceeds fixed maximum message size\r\n')
+
     def test_need_MAIL(self):
         self.write_line(b'HELO example')
         self.write_line(b'RCPT to:spam@example')
         self.assertEqual(self.channel.socket.last,
             b'503 Error: need MAIL command\r\n')
 
-    def test_MAIL_syntax(self):
+    def test_MAIL_syntax_HELO(self):
         self.write_line(b'HELO example')
         self.write_line(b'MAIL from eggs@example')
         self.assertEqual(self.channel.socket.last,
-            b'501 Syntax: MAIL FROM:<address>\r\n')
+            b'501 Syntax: MAIL FROM: <address>\r\n')
 
-    def test_MAIL_missing_from(self):
+    def test_MAIL_syntax_EHLO(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from eggs@example')
+        self.assertEqual(self.channel.socket.last,
+            b'501 Syntax: MAIL FROM: <address> [SP <mail-parameters>]\r\n')
+
+    def test_MAIL_missing_address(self):
         self.write_line(b'HELO example')
         self.write_line(b'MAIL from:')
         self.assertEqual(self.channel.socket.last,
-            b'501 Syntax: MAIL FROM:<address>\r\n')
+            b'501 Syntax: MAIL FROM: <address>\r\n')
 
     def test_MAIL_chevrons(self):
         self.write_line(b'HELO example')
         self.write_line(b'MAIL from:<eggs@example>')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+
+    def test_MAIL_empty_chevrons(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from:<>')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+
+    def test_MAIL_quoted_localpart(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from: <"Fred Blogs"@example.com>')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com')
+
+    def test_MAIL_quoted_localpart_no_angles(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from: "Fred Blogs"@example.com')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com')
+
+    def test_MAIL_quoted_localpart_with_size(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from: <"Fred Blogs"@example.com> SIZE=1000')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com')
+
+    def test_MAIL_quoted_localpart_with_size_no_angles(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL from: "Fred Blogs"@example.com SIZE=1000')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com')
 
     def test_nested_MAIL(self):
         self.write_line(b'HELO example')
@@ -184,6 +321,22 @@ class SMTPDChannelTest(TestCase):
         self.assertEqual(self.channel.socket.last,
             b'503 Error: nested MAIL command\r\n')
 
+    def test_VRFY(self):
+        self.write_line(b'VRFY eggs@example')
+        self.assertEqual(self.channel.socket.last,
+            b'252 Cannot VRFY user, but will accept message and attempt ' + \
+            b'delivery\r\n')
+
+    def test_VRFY_syntax(self):
+        self.write_line(b'VRFY')
+        self.assertEqual(self.channel.socket.last,
+            b'501 Syntax: VRFY <address>\r\n')
+
+    def test_EXPN_not_implemented(self):
+        self.write_line(b'EXPN')
+        self.assertEqual(self.channel.socket.last,
+            b'502 EXPN not implemented\r\n')
+
     def test_no_HELO_MAIL(self):
         self.write_line(b'MAIL from:<foo@example.com>')
         self.assertEqual(self.channel.socket.last,
@@ -196,13 +349,26 @@ class SMTPDChannelTest(TestCase):
         self.assertEqual(self.channel.socket.last,
             b'503 Error: need RCPT command\r\n')
 
-    def test_RCPT_syntax(self):
+    def test_RCPT_syntax_HELO(self):
         self.write_line(b'HELO example')
-        self.write_line(b'MAIL From:eggs@example')
+        self.write_line(b'MAIL From: eggs@example')
         self.write_line(b'RCPT to eggs@example')
         self.assertEqual(self.channel.socket.last,
             b'501 Syntax: RCPT TO: <address>\r\n')
 
+    def test_RCPT_syntax_EHLO(self):
+        self.write_line(b'EHLO example')
+        self.write_line(b'MAIL From: eggs@example')
+        self.write_line(b'RCPT to eggs@example')
+        self.assertEqual(self.channel.socket.last,
+            b'501 Syntax: RCPT TO: <address> [SP <mail-parameters>]\r\n')
+
+    def test_RCPT_lowercase_to_OK(self):
+        self.write_line(b'HELO example')
+        self.write_line(b'MAIL From: eggs@example')
+        self.write_line(b'RCPT to: <eggs@example>')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+
     def test_no_HELO_RCPT(self):
         self.write_line(b'RCPT to eggs@example')
         self.assertEqual(self.channel.socket.last,
@@ -211,15 +377,15 @@ class SMTPDChannelTest(TestCase):
     def test_data_dialog(self):
         self.write_line(b'HELO example')
         self.write_line(b'MAIL From:eggs@example')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
         self.write_line(b'RCPT To:spam@example')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
 
         self.write_line(b'DATA')
         self.assertEqual(self.channel.socket.last,
             b'354 End data with <CR><LF>.<CR><LF>\r\n')
         self.write_line(b'data\r\nmore\r\n.')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
         self.assertEqual(self.server.messages,
             [('peer', 'eggs@example', ['spam@example'], 'data\nmore')])
 
@@ -267,7 +433,7 @@ class SMTPDChannelTest(TestCase):
         self.write_line(b'MAIL From:eggs@example')
         self.write_line(b'RCPT To:spam@example')
         self.write_line(b'RSET')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
         self.write_line(b'MAIL From:foo@example')
         self.write_line(b'RCPT To:eggs@example')
         self.write_line(b'DATA')
@@ -278,12 +444,18 @@ class SMTPDChannelTest(TestCase):
     def test_HELO_RSET(self):
         self.write_line(b'HELO example')
         self.write_line(b'RSET')
-        self.assertEqual(self.channel.socket.last, b'250 Ok\r\n')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
 
     def test_RSET_syntax(self):
         self.write_line(b'RSET hi')
         self.assertEqual(self.channel.socket.last, b'501 Syntax: RSET\r\n')
 
+    def test_unknown_command(self):
+        self.write_line(b'UNKNOWN_CMD')
+        self.assertEqual(self.channel.socket.last,
+                         b'500 Error: command "UNKNOWN_CMD" not ' + \
+                         b'recognized\r\n')
+
     def test_attribute_deprecations(self):
         with support.check_warnings(('', DeprecationWarning)):
             spam = self.channel._SMTPChannel__server
@@ -330,8 +502,54 @@ class SMTPDChannelTest(TestCase):
         with support.check_warnings(('', DeprecationWarning)):
             self.channel._SMTPChannel__addr = 'spam'
 
-def test_main():
-    support.run_unittest(SMTPDServerTest, SMTPDChannelTest)
+
+class SMTPDChannelWithDataSizeLimitTest(unittest.TestCase):
+
+    def setUp(self):
+        smtpd.socket = asyncore.socket = mock_socket
+        self.debug = smtpd.DEBUGSTREAM = io.StringIO()
+        self.server = DummyServer('a', 'b')
+        conn, addr = self.server.accept()
+        # Set DATA size limit to 32 bytes for easy testing
+        self.channel = smtpd.SMTPChannel(self.server, conn, addr, 32)
+
+    def tearDown(self):
+        asyncore.close_all()
+        asyncore.socket = smtpd.socket = socket
+
+    def write_line(self, line):
+        self.channel.socket.queue_recv(line)
+        self.channel.handle_read()
+
+    def test_data_limit_dialog(self):
+        self.write_line(b'HELO example')
+        self.write_line(b'MAIL From:eggs@example')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.write_line(b'RCPT To:spam@example')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+
+        self.write_line(b'DATA')
+        self.assertEqual(self.channel.socket.last,
+            b'354 End data with <CR><LF>.<CR><LF>\r\n')
+        self.write_line(b'data\r\nmore\r\n.')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.assertEqual(self.server.messages,
+            [('peer', 'eggs@example', ['spam@example'], 'data\nmore')])
+
+    def test_data_limit_dialog_too_much_data(self):
+        self.write_line(b'HELO example')
+        self.write_line(b'MAIL From:eggs@example')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+        self.write_line(b'RCPT To:spam@example')
+        self.assertEqual(self.channel.socket.last, b'250 OK\r\n')
+
+        self.write_line(b'DATA')
+        self.assertEqual(self.channel.socket.last,
+            b'354 End data with <CR><LF>.<CR><LF>\r\n')
+        self.write_line(b'This message is longer than 32 bytes\r\n.')
+        self.assertEqual(self.channel.socket.last,
+                         b'552 Error: Too much mail data\r\n')
+
 
 if __name__ == "__main__":
-    test_main()
+    unittest.main()

Lib/test/test_smtplib.py

@@ -229,13 +229,13 @@ class DebuggingServerTests(unittest.TestCase):
 
     def testNOOP(self):
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        expected = (250, b'Ok')
+        expected = (250, b'OK')
         self.assertEqual(smtp.noop(), expected)
         smtp.quit()
 
     def testRSET(self):
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        expected = (250, b'Ok')
+        expected = (250, b'OK')
         self.assertEqual(smtp.rset(), expected)
         smtp.quit()
 
@@ -246,10 +246,18 @@ class DebuggingServerTests(unittest.TestCase):
         self.assertEqual(smtp.ehlo(), expected)
         smtp.quit()
 
+    def testNotImplemented(self):
+        # EXPN isn't implemented in DebuggingServer
+        smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
+        expected = (502, b'EXPN not implemented')
+        smtp.putcmd('EXPN')
+        self.assertEqual(smtp.getreply(), expected)
+        smtp.quit()
+
     def testVRFY(self):
-        # VRFY isn't implemented in DebuggingServer
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        expected = (502, b'Error: command "VRFY" not implemented')
+        expected = (252, b'Cannot VRFY user, but will accept message ' + \
+                         b'and attempt delivery')
         self.assertEqual(smtp.vrfy('nobody@nowhere.com'), expected)
         self.assertEqual(smtp.verify('nobody@nowhere.com'), expected)
         smtp.quit()
@@ -265,7 +273,8 @@ class DebuggingServerTests(unittest.TestCase):
 
     def testHELP(self):
         smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3)
-        self.assertEqual(smtp.help(), b'Error: command "HELP" not implemented')
+        self.assertEqual(smtp.help(), b'Supported commands: EHLO HELO MAIL ' + \
+                                      b'RCPT DATA RSET NOOP QUIT VRFY')
         smtp.quit()
 
     def testSend(self):

Misc/ACKS

@@ -112,6 +112,7 @@ Gregory Bond
 Matias Bordese
 Jurjen Bos
 Peter Bosch
+Dan Boswell
 Eric Bouck
 Thierry Bousch
 Sebastian Boving
@@ -494,6 +495,7 @@ Geert Jansen
 Jack Jansen
 Bill Janssen
 Thomas Jarosch
+Juhana Jauhiainen
 Zbigniew Jędrzejewski-Szmek
 Julien Jehannet
 Drew Jenkins
@@ -1039,6 +1041,7 @@ Sandro Tosi
 Richard Townsend
 David Townshend
 Laurence Tratt
+Alberto Trevino
 Matthias Troffaes
 John Tromp
 Jason Trowbridge

Misc/NEWS

@@ -46,6 +46,9 @@ Core and Builtins
 Library
 -------
 
+- Issue #8739: Updated smtpd to support RFC 5321, and added support for the
+  RFC 1870 SIZE extension.
+
 - Issue #665194: Added a localtime function to email.utils to provide an
   aware local datetime for use in setting Date headers.