Commit b1fdf47f authored by Antoine Pitrou's avatar Antoine Pitrou

Issue #21965: Add support for in-memory SSL to the ssl module.

Patch by Geert Jansen.
parent 414e15a8
......@@ -803,6 +803,29 @@ the specification of normal, OS-level sockets. See especially the
SSL sockets also have the following additional methods and attributes:
.. method:: SSLSocket.read(len=0, buffer=None)
Read up to *len* bytes of data from the SSL socket and return the result as
a ``bytes`` instance. If *buffer* is specified, then read into the buffer
instead, and return the number of bytes read.
.. method:: SSLSocket.write(buf)
Write *buf* to the SSL socket and return the number of bytes written. The
*buf* argument must be an object supporting the buffer interface.
.. note::
The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the
low-level methods that read and write unencrypted, application-level data
and and decrypt/encrypt it to encrypted, wire-level data. These methods
require an active SSL connection, i.e. the handshake was completed and
:meth:`SSLSocket.unwrap` was not called.
Normally you should use the socket API methods like
:meth:`~socket.socket.recv` and :meth:`~socket.socket.send` instead of these
methods.
.. method:: SSLSocket.do_handshake()
Perform the SSL setup handshake.
......@@ -935,6 +958,11 @@ SSL sockets also have the following additional methods and attributes:
.. versionadded:: 3.5
.. method:: SSLSocket.pending()
Returns the number of already decrypted bytes available for read, pending on
the connection.
.. attribute:: SSLSocket.context
The :class:`SSLContext` object this SSL socket is tied to. If the SSL
......@@ -944,6 +972,22 @@ SSL sockets also have the following additional methods and attributes:
.. versionadded:: 3.2
.. attribute:: SSLSocket.server_side
A boolean which is ``True`` for server-side sockets and ``False`` for
client-side sockets.
.. versionadded:: 3.5
.. attribute:: SSLSocket.server_hostname
A ``bytes`` instance containing the ``'idna'`` encoded version of the
hostname specified in the *server_hostname* argument in
:meth:`SSLContext.wrap_socket`. If no *server_hostname* was specified, this
attribute will be ``None``.
.. versionadded:: 3.5
SSL Contexts
------------
......@@ -1670,6 +1714,130 @@ thus several things you need to be aware of:
select.select([], [sock], [])
Memory BIO Support
------------------
.. versionadded:: 3.5
Ever since the SSL module was introduced in Python 2.6, the :class:`SSLSocket`
class has provided two related but distinct areas of functionality:
- SSL protocol handling
- Network IO
The network IO API is identical to that provided by :class:`socket.socket`,
from which :class:`SSLSocket` also inherits. This allows an SSL socket to be
used as a drop-in replacement for a regular socket, making it very easy to add
SSL support to an existing application.
Combining SSL protocol handling and network IO usually works well, but there
are some cases where it doesn't. An example is async IO frameworks that want to
use a different IO multiplexing model than the "select/poll on a file
descriptor" (readiness based) model that is assumed by :class:`socket.socket`
and by the internal OpenSSL socket IO routines. This is mostly relevant for
platforms like Windows where this model is not efficient. For this purpose, a
reduced scope variant of :class:`SSLSocket` called :class:`SSLObject` is
provided.
.. class:: SSLObject
A reduced-scope variant of :class:`SSLSocket` representing an SSL protocol
instance that does not contain any network IO methods.
The following methods are available from :class:`SSLSocket`:
- :attr:`~SSLSocket.context`
- :attr:`~SSLSocket.server_side`
- :attr:`~SSLSocket.server_hostname`
- :meth:`~SSLSocket.read`
- :meth:`~SSLSocket.write`
- :meth:`~SSLSocket.getpeercert`
- :meth:`~SSLSocket.selected_npn_protocol`
- :meth:`~SSLSocket.cipher`
- :meth:`~SSLSocket.compression`
- :meth:`~SSLSocket.pending`
- :meth:`~SSLSocket.do_handshake`
- :meth:`~SSLSocket.unwrap`
- :meth:`~SSLSocket.get_channel_binding`
An SSLObject communicates with the outside world using memory buffers. The
class :class:`MemoryBIO` provides a memory buffer that can be used for this
purpose. It wraps an OpenSSL memory BIO (Basic IO) object:
.. class:: MemoryBIO
A memory buffer that can be used to pass data between Python and an SSL
protocol instance.
.. attribute:: MemoryBIO.pending
Return the number of bytes currently in the memory buffer.
.. attribute:: MemoryBIO.eof
A boolean indicating whether the memory BIO is current at the end-of-file
position.
.. method:: MemoryBIO.read(n=-1)
Read up to *n* bytes from the memory buffer. If *n* is not specified or
negative, all bytes are returned.
.. method:: MemoryBIO.write(buf)
Write the bytes from *buf* to the memory BIO. The *buf* argument must be an
object supporting the buffer protocol.
The return value is the number of bytes written, which is always equal to
the length of *buf*.
.. method:: MemoryBIO.write_eof()
Write an EOF marker to the memory BIO. After this method has been called, it
is illegal to call :meth:`~MemoryBIO.write`. The attribute :attr:`eof` will
become true after all data currently in the buffer has been read.
An :class:`SSLObject` instance can be created using the
:meth:`~SSLContext.wrap_bio` method. This method will create the
:class:`SSLObject` instance and bind it to a pair of BIOs. The *incoming* BIO
is used to pass data from Python to the SSL protocol instance, while the
*outgoing* BIO is used to pass data the other way around.
.. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \
server_hostname=None)
Create a new :class:`SSLObject` instance by wrapping the BIO objects
*incoming* and *outgoing*. The SSL routines will read input data from the
incoming BIO and write data to the outgoing BIO.
The *server_side* and *server_hostname* parameters have the same meaning as
in :meth:`SSLContext.wrap_socket`.
Some notes related to the use of :class:`SSLObject`:
- All IO on an :class:`SSLObject` is non-blocking. This means that for example
:meth:`~SSLSocket.read` will raise an :exc:`SSLWantReadError` if it needs
more data than the incoming BIO has available.
- There is no module-level ``wrap_bio`` call like there is for
:meth:`~SSLContext.wrap_socket`. An :class:`SSLObject` is always created via
an :class:`SSLContext`.
- There is no *do_handshake_on_connect* machinery. You must always manually
call :meth:`~SSLSocket.do_handshake` to start the handshake.
- There is no handling of *suppress_ragged_eofs*. All end-of-file conditions
that are in violation of the protocol are reported via the :exc:`SSLEOFError`
exception.
- The method :meth:`~SSLSocket.unwrap` call does not return anything, unlike
for an SSL socket where it returns the underlying socket.
- The *server_name_callback* callback passed to
:meth:`SSLContext.set_servername_callback` will get an :class:`SSLObject`
instance instead of a :class:`SSLSocket` instance as its first parameter.
.. _ssl-security:
Security considerations
......
......@@ -97,7 +97,7 @@ from enum import Enum as _Enum, IntEnum as _IntEnum
import _ssl # if we can't import it, let the error propagate
from _ssl import OPENSSL_VERSION_NUMBER, OPENSSL_VERSION_INFO, OPENSSL_VERSION
from _ssl import _SSLContext
from _ssl import _SSLContext, MemoryBIO
from _ssl import (
SSLError, SSLZeroReturnError, SSLWantReadError, SSLWantWriteError,
SSLSyscallError, SSLEOFError,
......@@ -352,6 +352,12 @@ class SSLContext(_SSLContext):
server_hostname=server_hostname,
_context=self)
def wrap_bio(self, incoming, outgoing, server_side=False,
server_hostname=None):
sslobj = self._wrap_bio(incoming, outgoing, server_side=server_side,
server_hostname=server_hostname)
return SSLObject(sslobj)
def set_npn_protocols(self, npn_protocols):
protos = bytearray()
for protocol in npn_protocols:
......@@ -469,6 +475,129 @@ def _create_stdlib_context(protocol=PROTOCOL_SSLv23, *, cert_reqs=None,
return context
class SSLObject:
"""This class implements an interface on top of a low-level SSL object as
implemented by OpenSSL. This object captures the state of an SSL connection
but does not provide any network IO itself. IO needs to be performed
through separate "BIO" objects which are OpenSSL's IO abstraction layer.
This class does not have a public constructor. Instances are returned by
``SSLContext.wrap_bio``. This class is typically used by framework authors
that want to implement asynchronous IO for SSL through memory buffers.
When compared to ``SSLSocket``, this object lacks the following features:
* Any form of network IO incluging methods such as ``recv`` and ``send``.
* The ``do_handshake_on_connect`` and ``suppress_ragged_eofs`` machinery.
"""
def __init__(self, sslobj, owner=None):
self._sslobj = sslobj
# Note: _sslobj takes a weak reference to owner
self._sslobj.owner = owner or self
@property
def context(self):
"""The SSLContext that is currently in use."""
return self._sslobj.context
@context.setter
def context(self, ctx):
self._sslobj.context = ctx
@property
def server_side(self):
"""Whether this is a server-side socket."""
return self._sslobj.server_side
@property
def server_hostname(self):
"""The currently set server hostname (for SNI), or ``None`` if no
server hostame is set."""
return self._sslobj.server_hostname
def read(self, len=0, buffer=None):
"""Read up to 'len' bytes from the SSL object and return them.
If 'buffer' is provided, read into this buffer and return the number of
bytes read.
"""
if buffer is not None:
v = self._sslobj.read(len, buffer)
else:
v = self._sslobj.read(len or 1024)
return v
def write(self, data):
"""Write 'data' to the SSL object and return the number of bytes
written.
The 'data' argument must support the buffer interface.
"""
return self._sslobj.write(data)
def getpeercert(self, binary_form=False):
"""Returns a formatted version of the data in the certificate provided
by the other end of the SSL channel.
Return None if no certificate was provided, {} if a certificate was
provided, but not validated.
"""
return self._sslobj.peer_certificate(binary_form)
def selected_npn_protocol(self):
"""Return the currently selected NPN protocol as a string, or ``None``
if a next protocol was not negotiated or if NPN is not supported by one
of the peers."""
if _ssl.HAS_NPN:
return self._sslobj.selected_npn_protocol()
def cipher(self):
"""Return the currently selected cipher as a 3-tuple ``(name,
ssl_version, secret_bits)``."""
return self._sslobj.cipher()
def compression(self):
"""Return the current compression algorithm in use, or ``None`` if
compression was not negotiated or not supported by one of the peers."""
return self._sslobj.compression()
def pending(self):
"""Return the number of bytes that can be read immediately."""
return self._sslobj.pending()
def do_handshake(self, block=False):
"""Start the SSL/TLS handshake."""
self._sslobj.do_handshake()
if self.context.check_hostname:
if not self.server_hostname:
raise ValueError("check_hostname needs server_hostname "
"argument")
match_hostname(self.getpeercert(), self.server_hostname)
def unwrap(self):
"""Start the SSL shutdown handshake."""
return self._sslobj.shutdown()
def get_channel_binding(self, cb_type="tls-unique"):
"""Get channel binding data for current connection. Raise ValueError
if the requested `cb_type` is not supported. Return bytes of the data
or None if the data is not available (e.g. before the handshake)."""
if cb_type not in CHANNEL_BINDING_TYPES:
raise ValueError("Unsupported channel binding type")
if cb_type != "tls-unique":
raise NotImplementedError(
"{0} channel binding type not implemented"
.format(cb_type))
return self._sslobj.tls_unique_cb()
def version(self):
"""Return a string identifying the protocol version used by the
current SSL channel. """
return self._sslobj.version()
class SSLSocket(socket):
"""This class implements a subtype of socket.socket that wraps
the underlying OS socket in an SSL context when necessary, and
......@@ -556,8 +685,9 @@ class SSLSocket(socket):
if connected:
# create the SSL object
try:
self._sslobj = self._context._wrap_socket(self, server_side,
server_hostname)
sslobj = self._context._wrap_socket(self, server_side,
server_hostname)
self._sslobj = SSLObject(sslobj, owner=self)
if do_handshake_on_connect:
timeout = self.gettimeout()
if timeout == 0.0:
......@@ -602,11 +732,7 @@ class SSLSocket(socket):
if not self._sslobj:
raise ValueError("Read on closed or unwrapped SSL socket.")
try:
if buffer is not None:
v = self._sslobj.read(len, buffer)
else:
v = self._sslobj.read(len or 1024)
return v
return self._sslobj.read(len, buffer)
except SSLError as x:
if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
if buffer is not None:
......@@ -633,7 +759,7 @@ class SSLSocket(socket):
self._checkClosed()
self._check_connected()
return self._sslobj.peer_certificate(binary_form)
return self._sslobj.getpeercert(binary_form)
def selected_npn_protocol(self):
self._checkClosed()
......@@ -773,7 +899,7 @@ class SSLSocket(socket):
def unwrap(self):
if self._sslobj:
s = self._sslobj.shutdown()
s = self._sslobj.unwrap()
self._sslobj = None
return s
else:
......@@ -794,12 +920,6 @@ class SSLSocket(socket):
finally:
self.settimeout(timeout)
if self.context.check_hostname:
if not self.server_hostname:
raise ValueError("check_hostname needs server_hostname "
"argument")
match_hostname(self.getpeercert(), self.server_hostname)
def _real_connect(self, addr, connect_ex):
if self.server_side:
raise ValueError("can't connect in server-side mode")
......@@ -807,7 +927,8 @@ class SSLSocket(socket):
# connected at the time of the call. We connect it, then wrap it.
if self._connected:
raise ValueError("attempt to connect already-connected SSLSocket!")
self._sslobj = self.context._wrap_socket(self, False, self.server_hostname)
sslobj = self.context._wrap_socket(self, False, self.server_hostname)
self._sslobj = SSLObject(sslobj, owner=self)
try:
if connect_ex:
rc = socket.connect_ex(self, addr)
......@@ -850,15 +971,9 @@ class SSLSocket(socket):
if the requested `cb_type` is not supported. Return bytes of the data
or None if the data is not available (e.g. before the handshake).
"""
if cb_type not in CHANNEL_BINDING_TYPES:
raise ValueError("Unsupported channel binding type")
if cb_type != "tls-unique":
raise NotImplementedError(
"{0} channel binding type not implemented"
.format(cb_type))
if self._sslobj is None:
return None
return self._sslobj.tls_unique_cb()
return self._sslobj.get_channel_binding(cb_type)
def version(self):
"""
......
......@@ -518,9 +518,14 @@ class BasicSocketTests(unittest.TestCase):
def test_unknown_channel_binding(self):
# should raise ValueError for unknown type
s = socket.socket(socket.AF_INET)
with ssl.wrap_socket(s) as ss:
s.bind(('127.0.0.1', 0))
s.listen()
c = socket.socket(socket.AF_INET)
c.connect(s.getsockname())
with ssl.wrap_socket(c, do_handshake_on_connect=False) as ss:
with self.assertRaises(ValueError):
ss.get_channel_binding("unknown-type")
s.close()
@unittest.skipUnless("tls-unique" in ssl.CHANNEL_BINDING_TYPES,
"'tls-unique' channel binding not available")
......@@ -1247,6 +1252,69 @@ class SSLErrorTests(unittest.TestCase):
self.assertEqual(cm.exception.errno, ssl.SSL_ERROR_WANT_READ)
class MemoryBIOTests(unittest.TestCase):
def test_read_write(self):
bio = ssl.MemoryBIO()
bio.write(b'foo')
self.assertEqual(bio.read(), b'foo')
self.assertEqual(bio.read(), b'')
bio.write(b'foo')
bio.write(b'bar')
self.assertEqual(bio.read(), b'foobar')
self.assertEqual(bio.read(), b'')
bio.write(b'baz')
self.assertEqual(bio.read(2), b'ba')
self.assertEqual(bio.read(1), b'z')
self.assertEqual(bio.read(1), b'')
def test_eof(self):
bio = ssl.MemoryBIO()
self.assertFalse(bio.eof)
self.assertEqual(bio.read(), b'')
self.assertFalse(bio.eof)
bio.write(b'foo')
self.assertFalse(bio.eof)
bio.write_eof()
self.assertFalse(bio.eof)
self.assertEqual(bio.read(2), b'fo')
self.assertFalse(bio.eof)
self.assertEqual(bio.read(1), b'o')
self.assertTrue(bio.eof)
self.assertEqual(bio.read(), b'')
self.assertTrue(bio.eof)
def test_pending(self):
bio = ssl.MemoryBIO()
self.assertEqual(bio.pending, 0)
bio.write(b'foo')
self.assertEqual(bio.pending, 3)
for i in range(3):
bio.read(1)
self.assertEqual(bio.pending, 3-i-1)
for i in range(3):
bio.write(b'x')
self.assertEqual(bio.pending, i+1)
bio.read()
self.assertEqual(bio.pending, 0)
def test_buffer_types(self):
bio = ssl.MemoryBIO()
bio.write(b'foo')
self.assertEqual(bio.read(), b'foo')
bio.write(bytearray(b'bar'))
self.assertEqual(bio.read(), b'bar')
bio.write(memoryview(b'baz'))
self.assertEqual(bio.read(), b'baz')
def test_error_types(self):
bio = ssl.MemoryBIO()
self.assertRaises(TypeError, bio.write, 'foo')
self.assertRaises(TypeError, bio.write, None)
self.assertRaises(TypeError, bio.write, True)
self.assertRaises(TypeError, bio.write, 1)
class NetworkedTests(unittest.TestCase):
def test_connect(self):
......@@ -1577,6 +1645,95 @@ class NetworkedTests(unittest.TestCase):
self.assertIs(ss.context, ctx2)
self.assertIs(ss._sslobj.context, ctx2)
class NetworkedBIOTests(unittest.TestCase):
def ssl_io_loop(self, sock, incoming, outgoing, func, *args, **kwargs):
# A simple IO loop. Call func(*args) depending on the error we get
# (WANT_READ or WANT_WRITE) move data between the socket and the BIOs.
timeout = kwargs.get('timeout', 10)
count = 0
while True:
errno = None
count += 1
try:
ret = func(*args)
except ssl.SSLError as e:
# Note that we get a spurious -1/SSL_ERROR_SYSCALL for
# non-blocking IO. The SSL_shutdown manpage hints at this.
# It *should* be safe to just ignore SYS_ERROR_SYSCALL because
# with a Memory BIO there's no syscalls (for IO at least).
if e.errno not in (ssl.SSL_ERROR_WANT_READ,
ssl.SSL_ERROR_WANT_WRITE,
ssl.SSL_ERROR_SYSCALL):
raise
errno = e.errno
# Get any data from the outgoing BIO irrespective of any error, and
# send it to the socket.
buf = outgoing.read()
sock.sendall(buf)
# If there's no error, we're done. For WANT_READ, we need to get
# data from the socket and put it in the incoming BIO.
if errno is None:
break
elif errno == ssl.SSL_ERROR_WANT_READ:
buf = sock.recv(32768)
if buf:
incoming.write(buf)
else:
incoming.write_eof()
if support.verbose:
sys.stdout.write("Needed %d calls to complete %s().\n"
% (count, func.__name__))
return ret
def test_handshake(self):
with support.transient_internet("svn.python.org"):
sock = socket.socket(socket.AF_INET)
sock.connect(("svn.python.org", 443))
incoming = ssl.MemoryBIO()
outgoing = ssl.MemoryBIO()
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
ctx.verify_mode = ssl.CERT_REQUIRED
ctx.load_verify_locations(SVN_PYTHON_ORG_ROOT_CERT)
if ssl.HAS_SNI:
ctx.check_hostname = True
sslobj = ctx.wrap_bio(incoming, outgoing, False, 'svn.python.org')
else:
ctx.check_hostname = False
sslobj = ctx.wrap_bio(incoming, outgoing, False)
self.assertIs(sslobj._sslobj.owner, sslobj)
self.assertIsNone(sslobj.cipher())
self.assertRaises(ValueError, sslobj.getpeercert)
if 'tls-unique' in ssl.CHANNEL_BINDING_TYPES:
self.assertIsNone(sslobj.get_channel_binding('tls-unique'))
self.ssl_io_loop(sock, incoming, outgoing, sslobj.do_handshake)
self.assertTrue(sslobj.cipher())
self.assertTrue(sslobj.getpeercert())
if 'tls-unique' in ssl.CHANNEL_BINDING_TYPES:
self.assertTrue(sslobj.get_channel_binding('tls-unique'))
self.ssl_io_loop(sock, incoming, outgoing, sslobj.unwrap)
self.assertRaises(ssl.SSLError, sslobj.write, b'foo')
sock.close()
def test_read_write_data(self):
with support.transient_internet("svn.python.org"):
sock = socket.socket(socket.AF_INET)
sock.connect(("svn.python.org", 443))
incoming = ssl.MemoryBIO()
outgoing = ssl.MemoryBIO()
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
ctx.verify_mode = ssl.CERT_NONE
sslobj = ctx.wrap_bio(incoming, outgoing, False)
self.ssl_io_loop(sock, incoming, outgoing, sslobj.do_handshake)
req = b'GET / HTTP/1.0\r\n\r\n'
self.ssl_io_loop(sock, incoming, outgoing, sslobj.write, req)
buf = self.ssl_io_loop(sock, incoming, outgoing, sslobj.read, 1024)
self.assertEqual(buf[:5], b'HTTP/')
self.ssl_io_loop(sock, incoming, outgoing, sslobj.unwrap)
sock.close()
try:
import threading
except ImportError:
......@@ -3061,10 +3218,11 @@ def test_main(verbose=False):
if not os.path.exists(filename):
raise support.TestFailed("Can't read certificate file %r" % filename)
tests = [ContextTests, BasicSocketTests, SSLErrorTests]
tests = [ContextTests, BasicSocketTests, SSLErrorTests, MemoryBIOTests]
if support.is_resource_enabled('network'):
tests.append(NetworkedTests)
tests.append(NetworkedBIOTests)
if _have_threads:
thread_info = support.threading_setup()
......
......@@ -166,6 +166,9 @@ Core and Builtins
Library
-------
- Issue #21965: Add support for in-memory SSL to the ssl module. Patch
by Geert Jansen.
- Issue #21173: Fix len() on a WeakKeyDictionary when .clear() was called
with an iterator alive.
......
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment