Issue #14780: urllib.request.urlopen() now has a `cadefault` argument to use...

Issue #14780: urllib.request.urlopen() now has a `cadefault` argument to use the default certificate store.
Initial patch by James Oakley.

Doc/library/urllib.request.rst

@@ -16,7 +16,7 @@ authentication, redirections, cookies and more.
 The :mod:`urllib.request` module defines the following functions:
 
 
-.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None)
+.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None, cadefault=True)
 
    Open the URL *url*, which can be either a string or a
    :class:`Request` object.
@@ -53,9 +53,15 @@ The :mod:`urllib.request` module defines the following functions:
    point to a directory of hashed certificate files.  More information can
    be found in :meth:`ssl.SSLContext.load_verify_locations`.
 
+   The *cadefault* parameter specifies whether to fall back to loading a
+   default certificate store defined by the underlying OpenSSL library if the
+   *cafile* and *capath* parameters are omitted.  This will only work on
+   some non-Windows platforms.
+
    .. warning::
-      If neither *cafile* nor *capath* is specified, an HTTPS request
-      will not do any verification of the server's certificate.
+      If neither *cafile* nor *capath* is specified, and *cadefault* is False,
+      an HTTPS request will not do any verification of the server's
+      certificate.
 
    This function returns a file-like object that works as a :term:`context manager`,
    with two additional methods from the :mod:`urllib.response` module
@@ -92,6 +98,9 @@ The :mod:`urllib.request` module defines the following functions:
    .. versionadded:: 3.2
       *data* can be an iterable object.
 
+   .. versionchanged:: 3.3
+      *cadefault* was added.
+
 .. function:: install_opener(opener)
 
    Install an :class:`OpenerDirector` instance as the default global opener.

Lib/test/test_urllib2_localnet.py

@@ -474,6 +474,13 @@ class TestUrlopen(unittest.TestCase):
             self.urlopen("https://localhost:%s/bizarre" % handler.port,
                          cafile=CERT_fakehostname)
 
+    def test_https_with_cadefault(self):
+        handler = self.start_https_server(certfile=CERT_localhost)
+        # Self-signed cert should fail verification with system certificate store
+        with self.assertRaises(urllib.error.URLError) as cm:
+            self.urlopen("https://localhost:%s/bizarre" % handler.port,
+                         cadefault=True)
+
     def test_sending_headers(self):
         handler = self.start_server()
         req = urllib.request.Request("http://localhost:%s/" % handler.port,

Lib/urllib/request.py

@@ -135,16 +135,19 @@ __version__ = sys.version[:3]
 
 _opener = None
 def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
-            *, cafile=None, capath=None):
+            *, cafile=None, capath=None, cadefault=False):
     global _opener
-    if cafile or capath:
+    if cafile or capath or cadefault:
         if not _have_ssl:
             raise ValueError('SSL support not available')
         context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
         context.options |= ssl.OP_NO_SSLv2
-        if cafile or capath:
+        if cafile or capath or cadefault:
             context.verify_mode = ssl.CERT_REQUIRED
-            context.load_verify_locations(cafile, capath)
+            if cafile or capath:
+                context.load_verify_locations(cafile, capath)
+            else:
+                context.set_default_verify_paths()
             check_hostname = True
         else:
             check_hostname = False

Misc/ACKS

@@ -746,6 +746,7 @@ Nigel O'Brian
 John O'Connor
 Kevin O'Connor
 Tim O'Malley
+James Oakley
 Jon Oberheide
 Pascal Oberndoerfer
 Jeffrey Ollie

Misc/NEWS

@@ -34,6 +34,9 @@ Core and Builtins
 Library
 -------
 
+- Issue #14780: urllib.request.urlopen() now has a ``cadefault`` argument
+  to use the default certificate store.  Initial patch by James Oakley.
+
 - Issue #14829: Fix bisect and range() indexing with large indices
   (>= 2 ** 32) under 64-bit Windows.