PEP 489: Multi-phase extension module initialization

Known limitations of the current implementation:

- documentation changes are incomplete
- there's a reference leak I haven't tracked down yet

The leak is most visible by running:

  ./python -m test -R3:3 test_importlib

However, you can also see it by running:

  ./python -X showrefcount

Importing the array or _testmultiphase modules, and
then deleting them from both sys.modules and the local
namespace shows significant increases in the total
number of active references each cycle. By contrast,
with _testcapi (which continues to use single-phase
initialisation) the global refcounts stabilise after
a couple of cycles.

Doc/c-api/module.rst

@@ -7,8 +7,6 @@ Module Objects
 
 .. index:: object: module
 
-There are only a few functions special to module objects.
-
 
 .. c:var:: PyTypeObject PyModule_Type
 
@@ -109,6 +107,14 @@ There are only a few functions special to module objects.
       unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead.
 
 
+Per-interpreter module state
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Single-phase initialization creates singleton modules that can store additional
+information as part of the interpreter, allow that state to be retrieved later
+with only a reference to the module definition, rather than to the module
+itself.
+
 .. c:function:: void* PyModule_GetState(PyObject *module)
 
    Return the "state" of the module, that is, a pointer to the block of memory
@@ -146,27 +152,6 @@ There are only a few functions special to module objects.
 Initializing C modules
 ^^^^^^^^^^^^^^^^^^^^^^
 
-These functions are usually used in the module initialization function.
-
-.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
-
-   Create a new module object, given the definition in *module*.  This behaves
-   like :c:func:`PyModule_Create2` with *module_api_version* set to
-   :const:`PYTHON_API_VERSION`.
-
-
-.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
-
-   Create a new module object, given the definition in *module*, assuming the
-   API version *module_api_version*.  If that version does not match the version
-   of the running interpreter, a :exc:`RuntimeWarning` is emitted.
-
-   .. note::
-
-      Most uses of this function should be using :c:func:`PyModule_Create`
-      instead; only use this if you are sure you need it.
-
-
 .. c:type:: PyModuleDef
 
    This struct holds all information that is needed to create a module object.
@@ -210,9 +195,10 @@ These functions are usually used in the module initialization function.
       A pointer to a table of module-level functions, described by
       :c:type:`PyMethodDef` values.  Can be *NULL* if no functions are present.
 
-   .. c:member:: inquiry m_reload
+   .. c:member:: PyModuleDef_Slot* m_slots
 
-      Currently unused, should be *NULL*.
+      An array of slot definitions for multi-phase initialization, terminated by
+      a *NULL* entry.
 
    .. c:member:: traverseproc m_traverse
 
@@ -229,6 +215,61 @@ These functions are usually used in the module initialization function.
       A function to call during deallocation of the module object, or *NULL* if
       not needed.
 
+The module initialization function may create and return the module object
+directly. This is referred to as "single-phase initialization", and uses one
+of the following two module creation functions:
+
+.. c:function:: PyObject* PyModule_Create(PyModuleDef *module)
+
+   Create a new module object, given the definition in *module*.  This behaves
+   like :c:func:`PyModule_Create2` with *module_api_version* set to
+   :const:`PYTHON_API_VERSION`.
+
+
+.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version)
+
+   Create a new module object, given the definition in *module*, assuming the
+   API version *module_api_version*.  If that version does not match the version
+   of the running interpreter, a :exc:`RuntimeWarning` is emitted.
+
+   .. note::
+
+      Most uses of this function should be using :c:func:`PyModule_Create`
+      instead; only use this if you are sure you need it.
+
+
+Alternatively, the module initialization function may instead return a
+:c:type:`PyModuleDef` instance with a non-empty ``m_slots`` array. This is
+referred to as "multi-phase initialization", and ``PyModuleDef`` instance
+should be initialized with the following function:
+
+.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *module)
+
+   Ensures a module definition is a properly initialized Python object that
+   correctly reports its type and reference count.
+
+.. XXX (ncoghlan): It's not clear if it makes sense to document PyModule_ExecDef
+   PyModule_FromDefAndSpec or PyModule_FromDefAndSpec2 here, as end user code
+   generally shouldn't be calling those.
+
+The module initialization function (if using single phase initialization) or
+a function called from a module execution slot (if using multiphase
+initialization), can use the following functions to help initialize the module
+state:
+
+.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring)
+
+   Set the docstring for *module* to *docstring*. Return ``-1`` on error, ``0``
+   on success.
+
+.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
+
+   Add the functions from the ``NULL`` terminated *functions* array to *module*.
+   Refer to the :c:type:`PyMethodDef` documentation for details on individual
+   entries (due to the lack of a shared module namespace, module level
+   "functions" implemented in C typically receive the module as their first
+   parameter, making them similar to instance methods on Python classes).
+
 
 .. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value)
 
@@ -236,7 +277,6 @@ These functions are usually used in the module initialization function.
    be used from the module's initialization function.  This steals a reference to
    *value*.  Return ``-1`` on error, ``0`` on success.
 
-
 .. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value)
 
    Add an integer constant to *module* as *name*.  This convenience function can be

Doc/library/importlib.rst

@@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process.
     :pep:`451`
         A ModuleSpec Type for the Import System
 
+    :pep:`488`
+        Elimination of PYO files
+
+    :pep:`489`
+        Multi-phase extension module initialization
+
     :pep:`3120`
         Using UTF-8 as the Default Source Encoding
 
@@ -756,9 +762,9 @@ find and load modules.
     Only class methods are defined by this class to alleviate the need for
     instantiation.
 
-    .. note::
-       Due to limitations in the extension module C-API, for now
-       BuiltinImporter does not implement :meth:`Loader.exec_module`.
+    .. versionchanged:: 3.5
+       As part of :pep:`489`, the builtin importer now implements
+       :meth:`Loader.create_module` and :meth:`Loader.exec_module`
 
 
 .. class:: FrozenImporter
@@ -973,14 +979,18 @@ find and load modules.
 
       Path to the extension module.
 
-   .. method:: load_module(name=None)
+   .. method:: create_module(spec)
+
+      Creates the module object from the given specification in accordance
+      with :pep:`489`.
+
+      .. versionadded:: 3.5
+
+   .. method:: exec_module(module)
 
-      Loads the extension module if and only if *fullname* is the same as
-      :attr:`name` or is ``None``.
+      Initializes the given module object in accordance with :pep:`489`.
 
-      .. note::
-         Due to limitations in the extension module C-API, for now
-         ExtensionFileLoader does not implement :meth:`Loader.exec_module`.
+      .. versionadded:: 3.5
 
    .. method:: is_package(fullname)
 

Doc/whatsnew/3.5.rst

@@ -93,6 +93,7 @@ Implementation improvements:
   (:issue:`19977`).
 
 * :pep:`488`, the elimination of ``.pyo`` files.
+* :pep:`489`, multi-phase initialization of extension modules.
 
 Significantly Improved Library Modules:
 
@@ -265,6 +266,21 @@ updated API to help with this change.
    :pep:`488` -- Elimination of PYO files
 
 
+PEP 489: Multi-phase extension module initialization
+----------------------------------------------------
+
+:pep:`489` updates extension module initialization to take advantage of the
+two step module loading mechanism introduced by :pep:`451` in Python 3.4.
+
+This change brings the import semantics of extension modules that opt-in to
+using the new mechanism much closer to those of Python source and bytecode
+modules, including the ability to any valid identifier as a module name,
+rather than being restricted to ASCII.
+
+.. seealso::
+
+   :pep:`488` -- Multi-phase extension module initialization
+
 Other Language Changes
 ======================
 
@@ -667,7 +683,7 @@ time
 tkinter
 -------
 
-* The :module:`tkinter._fix` module used for setting up the Tcl/Tk environment
+* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment
   on Windows has been replaced by a private function in the :module:`_tkinter`
   module which makes no permanent changes to environment variables.
   (Contributed by Zachary Ware in :issue:`20035`.)
@@ -1012,7 +1028,6 @@ Changes in the Python API
   program depends on patching the module level variable to capture the debug
   output, you will need to update it to capture sys.stderr instead.
 
-
 Changes in the C API
 --------------------
 

Include/modsupport.h

@@ -12,13 +12,13 @@ extern "C" {
 /* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier
    to mean Py_ssize_t */
 #ifdef PY_SSIZE_T_CLEAN
-#define PyArg_Parse			_PyArg_Parse_SizeT
-#define PyArg_ParseTuple		_PyArg_ParseTuple_SizeT
-#define PyArg_ParseTupleAndKeywords	_PyArg_ParseTupleAndKeywords_SizeT
-#define PyArg_VaParse			_PyArg_VaParse_SizeT
-#define PyArg_VaParseTupleAndKeywords	_PyArg_VaParseTupleAndKeywords_SizeT
-#define Py_BuildValue			_Py_BuildValue_SizeT
-#define Py_VaBuildValue			_Py_VaBuildValue_SizeT
+#define PyArg_Parse                     _PyArg_Parse_SizeT
+#define PyArg_ParseTuple                _PyArg_ParseTuple_SizeT
+#define PyArg_ParseTupleAndKeywords     _PyArg_ParseTupleAndKeywords_SizeT
+#define PyArg_VaParse                   _PyArg_VaParse_SizeT
+#define PyArg_VaParseTupleAndKeywords   _PyArg_VaParseTupleAndKeywords_SizeT
+#define Py_BuildValue                   _Py_BuildValue_SizeT
+#define Py_VaBuildValue                 _Py_VaBuildValue_SizeT
 #else
 PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list);
 #endif
@@ -49,6 +49,9 @@ PyAPI_FUNC(int) PyModule_AddIntConstant(PyObject *, const char *, long);
 PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char *);
 #define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c)
 #define PyModule_AddStringMacro(m, c) PyModule_AddStringConstant(m, #c, c)
+PyAPI_FUNC(int) PyModule_SetDocString(PyObject *, const char *);
+PyAPI_FUNC(int) PyModule_AddFunctions(PyObject *, PyMethodDef *);
+PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);
 
 #define Py_CLEANUP_SUPPORTED 0x20000
 
@@ -67,35 +70,35 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
    Please add a line or two to the top of this log for each API
    version change:
 
-   22-Feb-2006  MvL	1013	PEP 353 - long indices for sequence lengths
+   22-Feb-2006  MvL     1013    PEP 353 - long indices for sequence lengths
 
-   19-Aug-2002  GvR	1012	Changes to string object struct for
-   				interning changes, saving 3 bytes.
+   19-Aug-2002  GvR     1012    Changes to string object struct for
+                                interning changes, saving 3 bytes.
 
-   17-Jul-2001	GvR	1011	Descr-branch, just to be on the safe side
+   17-Jul-2001  GvR     1011    Descr-branch, just to be on the safe side
 
    25-Jan-2001  FLD     1010    Parameters added to PyCode_New() and
                                 PyFrame_New(); Python 2.1a2
 
    14-Mar-2000  GvR     1009    Unicode API added
 
-   3-Jan-1999	GvR	1007	Decided to change back!  (Don't reuse 1008!)
+   3-Jan-1999   GvR     1007    Decided to change back!  (Don't reuse 1008!)
 
-   3-Dec-1998	GvR	1008	Python 1.5.2b1
+   3-Dec-1998   GvR     1008    Python 1.5.2b1
 
-   18-Jan-1997	GvR	1007	string interning and other speedups
+   18-Jan-1997  GvR     1007    string interning and other speedups
 
-   11-Oct-1996	GvR	renamed Py_Ellipses to Py_Ellipsis :-(
+   11-Oct-1996  GvR     renamed Py_Ellipses to Py_Ellipsis :-(
 
-   30-Jul-1996	GvR	Slice and ellipses syntax added
+   30-Jul-1996  GvR     Slice and ellipses syntax added
 
-   23-Jul-1996	GvR	For 1.4 -- better safe than sorry this time :-)
+   23-Jul-1996  GvR     For 1.4 -- better safe than sorry this time :-)
 
-   7-Nov-1995	GvR	Keyword arguments (should've been done at 1.3 :-( )
+   7-Nov-1995   GvR     Keyword arguments (should've been done at 1.3 :-( )
 
-   10-Jan-1995	GvR	Renamed globals to new naming scheme
+   10-Jan-1995  GvR     Renamed globals to new naming scheme
 
-   9-Jan-1995	GvR	Initial version (incompatible with older API)
+   9-Jan-1995   GvR     Initial version (incompatible with older API)
 */
 
 /* The PYTHON_ABI_VERSION is introduced in PEP 384. For the lifetime of
@@ -105,10 +108,11 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
 #define PYTHON_ABI_STRING "3"
 
 #ifdef Py_TRACE_REFS
- /* When we are tracing reference counts, rename PyModule_Create2 so
+ /* When we are tracing reference counts, rename module creation functions so
     modules compiled with incompatible settings will generate a
     link-time error. */
  #define PyModule_Create2 PyModule_Create2TraceRefs
+ #define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
 #endif
 
 PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
@@ -116,10 +120,22 @@ PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
 
 #ifdef Py_LIMITED_API
 #define PyModule_Create(module) \
-	PyModule_Create2(module, PYTHON_ABI_VERSION)
+        PyModule_Create2(module, PYTHON_ABI_VERSION)
 #else
 #define PyModule_Create(module) \
-	PyModule_Create2(module, PYTHON_API_VERSION)
+        PyModule_Create2(module, PYTHON_API_VERSION)
+#endif
+
+PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def,
+                                                PyObject *spec,
+                                                int module_api_version);
+
+#ifdef Py_LIMITED_API
+#define PyModule_FromDefAndSpec(module, spec) \
+    PyModule_FromDefAndSpec2(module, spec, PYTHON_ABI_VERSION)
+#else
+#define PyModule_FromDefAndSpec(module, spec) \
+    PyModule_FromDefAndSpec2(module, spec, PYTHON_API_VERSION)
 #endif
 
 #ifndef Py_LIMITED_API

Include/moduleobject.h

@@ -30,6 +30,9 @@ PyAPI_FUNC(void) _PyModule_ClearDict(PyObject *);
 PyAPI_FUNC(struct PyModuleDef*) PyModule_GetDef(PyObject*);
 PyAPI_FUNC(void*) PyModule_GetState(PyObject*);
 
+PyAPI_FUNC(PyObject *) PyModuleDef_Init(struct PyModuleDef*);
+PyTypeObject PyModuleDef_Type;
+
 typedef struct PyModuleDef_Base {
   PyObject_HEAD
   PyObject* (*m_init)(void);
@@ -44,18 +47,29 @@ typedef struct PyModuleDef_Base {
     NULL, /* m_copy */          \
   }
 
+typedef struct PyModuleDef_Slot{
+    int slot;
+    void *value;
+} PyModuleDef_Slot;
+
 typedef struct PyModuleDef{
   PyModuleDef_Base m_base;
   const char* m_name;
   const char* m_doc;
   Py_ssize_t m_size;
   PyMethodDef *m_methods;
-  inquiry m_reload;
+  PyModuleDef_Slot* m_slots;
   traverseproc m_traverse;
   inquiry m_clear;
   freefunc m_free;
 }PyModuleDef;
 
+#define Py_mod_create 1
+#define Py_mod_exec 2
+
+#ifndef Py_LIMITED_API
+#define _Py_mod_LAST_SLOT 2
+#endif
 
 #ifdef __cplusplus
 }

Lib/imp.py

@@ -8,15 +8,15 @@ functionality over this module.
 # (Probably) need to stay in _imp
 from _imp import (lock_held, acquire_lock, release_lock,
                   get_frozen_object, is_frozen_package,
-                  init_builtin, init_frozen, is_builtin, is_frozen,
+                  init_frozen, is_builtin, is_frozen,
                   _fix_co_filename)
 try:
-    from _imp import load_dynamic
+    from _imp import create_dynamic
 except ImportError:
     # Platform doesn't support dynamic loading.
-    load_dynamic = None
+    create_dynamic = None
 
-from importlib._bootstrap import _ERR_MSG, _exec, _load
+from importlib._bootstrap import _ERR_MSG, _exec, _load, _builtin_from_name
 from importlib._bootstrap_external import SourcelessFileLoader
 
 from importlib import machinery
@@ -312,3 +312,28 @@ def reload(module):
 
     """
     return importlib.reload(module)
+
+
+def init_builtin(name):
+    """**DEPRECATED**
+
+    Load and return a built-in module by name, or None is such module doesn't
+    exist
+    """
+    try:
+        return _builtin_from_name(name)
+    except ImportError:
+        return None
+
+
+if create_dynamic:
+    def load_dynamic(name, path, file=None):
+        """**DEPRECATED**
+
+        Load an extension module.
+        """
+        import importlib.machinery
+        loader = importlib.machinery.ExtensionFileLoader(name, path)
+        return loader.load_module()
+else:
+    load_dynamic = None

Lib/importlib/_bootstrap.py

@@ -735,16 +735,17 @@ class BuiltinImporter:
         return spec.loader if spec is not None else None
 
     @classmethod
-    @_requires_builtin
-    def load_module(cls, fullname):
-        """Load a built-in module."""
-        # Once an exec_module() implementation is added we can also
-        # add a deprecation warning here.
-        with _ManageReload(fullname):
-            module = _call_with_frames_removed(_imp.init_builtin, fullname)
-        module.__loader__ = cls
-        module.__package__ = ''
-        return module
+    def create_module(self, spec):
+        """Create a built-in module"""
+        if spec.name not in sys.builtin_module_names:
+            raise ImportError('{!r} is not a built-in module'.format(spec.name),
+                              name=spec.name)
+        return _call_with_frames_removed(_imp.create_builtin, spec)
+
+    @classmethod
+    def exec_module(self, module):
+        """Exec a built-in module"""
+        _call_with_frames_removed(_imp.exec_dynamic, module)
 
     @classmethod
     @_requires_builtin
@@ -764,6 +765,8 @@ class BuiltinImporter:
         """Return False as built-in modules are never packages."""
         return False
 
+    load_module = classmethod(_load_module_shim)
+
 
 class FrozenImporter:
 

Lib/importlib/_bootstrap_external.py

@@ -378,7 +378,8 @@ def _check_name(method):
         if name is None:
             name = self.name
         elif self.name != name:
-            raise ImportError('loader cannot handle %s' % name, name=name)
+            raise ImportError('loader for %s cannot handle %s' %
+                                (self.name, name), name=name)
         return method(self, name, *args, **kwargs)
     try:
         _wrap = _bootstrap._wrap
@@ -875,7 +876,7 @@ class SourcelessFileLoader(FileLoader, _LoaderBasics):
 EXTENSION_SUFFIXES = []
 
 
-class ExtensionFileLoader:
+class ExtensionFileLoader(FileLoader, _LoaderBasics):
 
     """Loader for extension modules.
 
@@ -894,24 +895,20 @@ class ExtensionFileLoader:
     def __hash__(self):
         return hash(self.name) ^ hash(self.path)
 
-    @_check_name
-    def load_module(self, fullname):
-        """Load an extension module."""
-        # Once an exec_module() implementation is added we can also
-        # add a deprecation warning here.
-        with _bootstrap._ManageReload(fullname):
-            module = _bootstrap._call_with_frames_removed(_imp.load_dynamic,
-                                                          fullname, self.path)
-        _verbose_message('extension module loaded from {!r}', self.path)
-        is_package = self.is_package(fullname)
-        if is_package and not hasattr(module, '__path__'):
-            module.__path__ = [_path_split(self.path)[0]]
-        module.__loader__ = self
-        module.__package__ = module.__name__
-        if not is_package:
-            module.__package__ = module.__package__.rpartition('.')[0]
+    def create_module(self, spec):
+        """Create an unitialized extension module"""
+        module = _bootstrap._call_with_frames_removed(
+            _imp.create_dynamic, spec)
+        _verbose_message('extension module {!r} loaded from {!r}',
+                         spec.name, self.path)
         return module
 
+    def exec_module(self, module):
+        """Initialize an extension module"""
+        _bootstrap._call_with_frames_removed(_imp.exec_dynamic, module)
+        _verbose_message('extension module {!r} executed from {!r}',
+                         self.name, self.path)
+
     def is_package(self, fullname):
         """Return True if the extension module is a package."""
         file_name = _path_split(self.path)[1]

Lib/test/test_importlib/extension/test_loader.py

@@ -7,6 +7,8 @@ import os.path
 import sys
 import types
 import unittest
+import importlib.util
+import importlib
 
 
 class LoaderTests(abc.LoaderTests):
@@ -80,6 +82,171 @@ class LoaderTests(abc.LoaderTests):
  Source_LoaderTests
  ) = util.test_both(LoaderTests, machinery=machinery)
 
+class MultiPhaseExtensionModuleTests(abc.LoaderTests):
+    """Test loading extension modules with multi-phase initialization (PEP 489)
+    """
+
+    def setUp(self):
+        self.name = '_testmultiphase'
+        finder = self.machinery.FileFinder(None)
+        self.spec = importlib.util.find_spec(self.name)
+        assert self.spec
+        self.loader = self.machinery.ExtensionFileLoader(
+            self.name, self.spec.origin)
+
+    # No extension module as __init__ available for testing.
+    test_package = None
+
+    # No extension module in a package available for testing.
+    test_lacking_parent = None
+
+    # Handling failure on reload is the up to the module.
+    test_state_after_failure = None
+
+    def test_module(self):
+        '''Test loading an extension module'''
+        with util.uncache(self.name):
+            module = self.load_module()
+            for attr, value in [('__name__', self.name),
+                                ('__file__', self.spec.origin),
+                                ('__package__', '')]:
+                self.assertEqual(getattr(module, attr), value)
+            with self.assertRaises(AttributeError):
+                module.__path__
+            self.assertIs(module, sys.modules[self.name])
+            self.assertIsInstance(module.__loader__,
+                                  self.machinery.ExtensionFileLoader)
+
+    def test_functionality(self):
+        '''Test basic functionality of stuff defined in an extension module'''
+        with util.uncache(self.name):
+            module = self.load_module()
+            self.assertIsInstance(module, types.ModuleType)
+            ex = module.Example()
+            self.assertEqual(ex.demo('abcd'), 'abcd')
+            self.assertEqual(ex.demo(), None)
+            with self.assertRaises(AttributeError):
+                ex.abc
+            ex.abc = 0
+            self.assertEqual(ex.abc, 0)
+            self.assertEqual(module.foo(9, 9), 18)
+            self.assertIsInstance(module.Str(), str)
+            self.assertEqual(module.Str(1) + '23', '123')
+            with self.assertRaises(module.error):
+                raise module.error()
+            self.assertEqual(module.int_const, 1969)
+            self.assertEqual(module.str_const, 'something different')
+
+    def test_reload(self):
+        '''Test that reload didn't re-set the module's attributes'''
+        with util.uncache(self.name):
+            module = self.load_module()
+            ex_class = module.Example
+            importlib.reload(module)
+            self.assertIs(ex_class, module.Example)
+
+    def test_try_registration(self):
+        '''Assert that the PyState_{Find,Add,Remove}Module C API doesn't work'''
+        module = self.load_module()
+        with self.subTest('PyState_FindModule'):
+            self.assertEqual(module.call_state_registration_func(0), None)
+        with self.subTest('PyState_AddModule'):
+            with self.assertRaises(SystemError):
+                module.call_state_registration_func(1)
+        with self.subTest('PyState_RemoveModule'):
+            with self.assertRaises(SystemError):
+                module.call_state_registration_func(2)
+
+    def load_module(self):
+        '''Load the module from the test extension'''
+        return self.loader.load_module(self.name)
+
+    def load_module_by_name(self, fullname):
+        '''Load a module from the test extension by name'''
+        origin = self.spec.origin
+        loader = self.machinery.ExtensionFileLoader(fullname, origin)
+        spec = importlib.util.spec_from_loader(fullname, loader)
+        module = importlib.util.module_from_spec(spec)
+        loader.exec_module(module)
+        return module
+
+    def test_load_twice(self):
+        '''Test that 2 loads result in 2 module objects'''
+        module1 = self.load_module_by_name(self.name)
+        module2 = self.load_module_by_name(self.name)
+        self.assertIsNot(module1, module2)
+
+    def test_unloadable(self):
+        '''Test nonexistent module'''
+        name = 'asdfjkl;'
+        with self.assertRaises(ImportError) as cm:
+            self.load_module_by_name(name)
+        self.assertEqual(cm.exception.name, name)
+
+    def test_unloadable_nonascii(self):
+        '''Test behavior with nonexistent module with non-ASCII name'''
+        name = 'fo\xf3'
+        with self.assertRaises(ImportError) as cm:
+            self.load_module_by_name(name)
+        self.assertEqual(cm.exception.name, name)
+
+    def test_nonmodule(self):
+        '''Test returning a non-module object from create works'''
+        name = self.name + '_nonmodule'
+        mod = self.load_module_by_name(name)
+        self.assertNotEqual(type(mod), type(unittest))
+        self.assertEqual(mod.three, 3)
+
+    def test_null_slots(self):
+        '''Test that NULL slots aren't a problem'''
+        name = self.name + '_null_slots'
+        module = self.load_module_by_name(name)
+        self.assertIsInstance(module, types.ModuleType)
+        assert module.__name__ == name
+
+    def test_bad_modules(self):
+        '''Test SystemError is raised for misbehaving extensions'''
+        for name_base in [
+                'bad_slot_large',
+                'bad_slot_negative',
+                'create_int_with_state',
+                'negative_size',
+                'export_null',
+                'export_uninitialized',
+                'export_raise',
+                'export_unreported_exception',
+                'create_null',
+                'create_raise',
+                'create_unreported_exception',
+                'nonmodule_with_exec_slots',
+                'exec_err',
+                'exec_raise',
+                'exec_unreported_exception',
+                ]:
+            with self.subTest(name_base):
+                name = self.name + '_' + name_base
+                with self.assertRaises(SystemError):
+                    self.load_module_by_name(name)
+
+    def test_nonascii(self):
+        '''Test that modules with non-ASCII names can be loaded'''
+        # punycode behaves slightly differently in some-ASCII and no-ASCII
+        # cases, so test both
+        cases = [
+            (self.name + '_zkou\u0161ka_na\u010dten\xed', 'Czech'),
+            ('\uff3f\u30a4\u30f3\u30dd\u30fc\u30c8\u30c6\u30b9\u30c8',
+             'Japanese'),
+            ]
+        for name, lang in cases:
+            with self.subTest(name):
+                module = self.load_module_by_name(name)
+                self.assertEqual(module.__name__, name)
+                self.assertEqual(module.__doc__, "Module named in %s" % lang)
+
+
+(Frozen_MultiPhaseExtensionModuleTests,
+ Source_MultiPhaseExtensionModuleTests
+ ) = util.test_both(MultiPhaseExtensionModuleTests, machinery=machinery)
 
 
 if __name__ == '__main__':

Misc/ACKS

@@ -1458,6 +1458,7 @@ Mike Verdone
 Jaap Vermeulen
 Nikita Vetoshkin
 Al Vezza
+Petr Victorin
 Jacques A. Vidrine
 John Viega
 Dino Viehland

Misc/NEWS

@@ -10,6 +10,8 @@ Release date: 2015-05-24
 Core and Builtins
 -----------------
 
+- Issue #24268: PEP 489: Multi-phase extension module initialization
+
 - Issue #23955: Add pyvenv.cfg option to suppress registry/environment
   lookup for generating sys.path on Windows.
 

Modules/_testcapimodule.c

@@ -4048,6 +4048,9 @@ static struct PyModuleDef _testcapimodule = {
     NULL
 };
 
+/* Per PEP 489, this module will not be converted to multi-phase initialization
+ */
+
 PyMODINIT_FUNC
 PyInit__testcapi(void)
 {

Modules/arraymodule.c

@@ -2981,34 +2981,17 @@ static PyMethodDef a_methods[] = {
     {NULL, NULL, 0, NULL}        /* Sentinel */
 };
 
-static struct PyModuleDef arraymodule = {
-    PyModuleDef_HEAD_INIT,
-    "array",
-    module_doc,
-    -1,
-    a_methods,
-    NULL,
-    NULL,
-    NULL,
-    NULL
-};
-
-
-PyMODINIT_FUNC
-PyInit_array(void)
+static int
+array_modexec(PyObject *m)
 {
-    PyObject *m;
     char buffer[Py_ARRAY_LENGTH(descriptors)], *p;
     PyObject *typecodes;
     Py_ssize_t size = 0;
     struct arraydescr *descr;
 
     if (PyType_Ready(&Arraytype) < 0)
-        return NULL;
+        return -1;
     Py_TYPE(&PyArrayIter_Type) = &PyType_Type;
-    m = PyModule_Create(&arraymodule);
-    if (m == NULL)
-        return NULL;
 
     Py_INCREF((PyObject *)&Arraytype);
     PyModule_AddObject(m, "ArrayType", (PyObject *)&Arraytype);
@@ -3031,5 +3014,30 @@ PyInit_array(void)
         Py_DECREF(m);
         m = NULL;
     }
-    return m;
+    return 0;
+}
+
+static PyModuleDef_Slot arrayslots[] = {
+    {Py_mod_exec, array_modexec},
+    {0, NULL}
+};
+
+
+static struct PyModuleDef arraymodule = {
+    PyModuleDef_HEAD_INIT,
+    "array",
+    module_doc,
+    0,
+    a_methods,
+    arrayslots,
+    NULL,
+    NULL,
+    NULL
+};
+
+
+PyMODINIT_FUNC
+PyInit_array(void)
+{
+    return PyModuleDef_Init(&arraymodule);
 }

Modules/config.c.in

@@ -13,7 +13,7 @@ redistribution of this file, and for a DISCLAIMER OF ALL WARRANTIES.
 /* !!! !!! !!! This file is edited by the makesetup script !!! !!! !!! */
 
 /* This file contains the table of built-in modules.
-   See init_builtin() in import.c. */
+   See create_builtin() in import.c. */
 
 #include "Python.h"
 

Modules/xxlimited.c

@@ -222,25 +222,9 @@ static PyMethodDef xx_methods[] = {
 PyDoc_STRVAR(module_doc,
 "This is a template module just for instruction.");
 
-/* Initialization function for the module (*must* be called PyInit_xx) */
-
-
-static struct PyModuleDef xxmodule = {
-    PyModuleDef_HEAD_INIT,
-    "xxlimited",
-    module_doc,
-    -1,
-    xx_methods,
-    NULL,
-    NULL,
-    NULL,
-    NULL
-};
-
-PyMODINIT_FUNC
-PyInit_xxlimited(void)
+static int
+xx_modexec(PyObject *m)
 {
-    PyObject *m = NULL;
     PyObject *o;
 
     /* Due to cross platform compiler issues the slots must be filled
@@ -254,11 +238,6 @@ PyInit_xxlimited(void)
     if (Xxo_Type == NULL)
         goto fail;
 
-    /* Create the module and add the functions */
-    m = PyModule_Create(&xxmodule);
-    if (m == NULL)
-        goto fail;
-
     /* Add some symbolic constants to the module */
     if (ErrorObject == NULL) {
         ErrorObject = PyErr_NewException("xxlimited.error", NULL, NULL);
@@ -279,8 +258,34 @@ PyInit_xxlimited(void)
     if (o == NULL)
         goto fail;
     PyModule_AddObject(m, "Null", o);
-    return m;
+    return 0;
  fail:
     Py_XDECREF(m);
-    return NULL;
+    return -1;
+}
+
+
+static PyModuleDef_Slot xx_slots[] = {
+    {Py_mod_exec, xx_modexec},
+    {0, NULL}
+};
+
+static struct PyModuleDef xxmodule = {
+    PyModuleDef_HEAD_INIT,
+    "xxlimited",
+    module_doc,
+    0,
+    xx_methods,
+    xx_slots,
+    NULL,
+    NULL,
+    NULL
+};
+
+/* Export function for the module (*must* be called PyInit_xx) */
+
+PyMODINIT_FUNC
+PyInit_xxlimited(void)
+{
+    return PyModuleDef_Init(&xxmodule);
 }

Modules/xxmodule.c

@@ -334,26 +334,10 @@ static PyMethodDef xx_methods[] = {
 PyDoc_STRVAR(module_doc,
 "This is a template module just for instruction.");
 
-/* Initialization function for the module (*must* be called PyInit_xx) */
 
-
-static struct PyModuleDef xxmodule = {
-    PyModuleDef_HEAD_INIT,
-    "xx",
-    module_doc,
-    -1,
-    xx_methods,
-    NULL,
-    NULL,
-    NULL,
-    NULL
-};
-
-PyMODINIT_FUNC
-PyInit_xx(void)
+static int
+xx_exec(PyObject *m)
 {
-    PyObject *m = NULL;
-
     /* Due to cross platform compiler issues the slots must be filled
      * here. It's required for portability to Windows without requiring
      * C++. */
@@ -366,11 +350,6 @@ PyInit_xx(void)
     if (PyType_Ready(&Xxo_Type) < 0)
         goto fail;
 
-    /* Create the module and add the functions */
-    m = PyModule_Create(&xxmodule);
-    if (m == NULL)
-        goto fail;
-
     /* Add some symbolic constants to the module */
     if (ErrorObject == NULL) {
         ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
@@ -389,8 +368,33 @@ PyInit_xx(void)
     if (PyType_Ready(&Null_Type) < 0)
         goto fail;
     PyModule_AddObject(m, "Null", (PyObject *)&Null_Type);
-    return m;
+    return 0;
  fail:
     Py_XDECREF(m);
-    return NULL;
+    return -1;
+}
+
+static struct PyModuleDef_Slot xx_slots[] = {
+    {Py_mod_exec, xx_exec},
+    {0, NULL},
+};
+
+static struct PyModuleDef xxmodule = {
+    PyModuleDef_HEAD_INIT,
+    "xx",
+    module_doc,
+    0,
+    xx_methods,
+    xx_slots,
+    NULL,
+    NULL,
+    NULL
+};
+
+/* Export function for the module (*must* be called PyInit_xx) */
+
+PyMODINIT_FUNC
+PyInit_xx(void)
+{
+    return PyModuleDef_Init(&xxmodule);
 }

Modules/xxsubtype.c

@@ -257,53 +257,58 @@ static PyMethodDef xxsubtype_functions[] = {
     {NULL,              NULL}           /* sentinel */
 };
 
-static struct PyModuleDef xxsubtypemodule = {
-    PyModuleDef_HEAD_INIT,
-    "xxsubtype",
-    xxsubtype__doc__,
-    -1,
-    xxsubtype_functions,
-    NULL,
-    NULL,
-    NULL,
-    NULL
-};
-
-
-PyMODINIT_FUNC
-PyInit_xxsubtype(void)
+static int
+xxsubtype_exec(PyObject* m)
 {
-    PyObject *m;
-
     /* Fill in deferred data addresses.  This must be done before
        PyType_Ready() is called.  Note that PyType_Ready() automatically
        initializes the ob.ob_type field to &PyType_Type if it's NULL,
        so it's not necessary to fill in ob_type first. */
     spamdict_type.tp_base = &PyDict_Type;
     if (PyType_Ready(&spamdict_type) < 0)
-        return NULL;
+        return -1;
 
     spamlist_type.tp_base = &PyList_Type;
     if (PyType_Ready(&spamlist_type) < 0)
-        return NULL;
-
-    m = PyModule_Create(&xxsubtypemodule);
-    if (m == NULL)
-        return NULL;
+        return -1;
 
     if (PyType_Ready(&spamlist_type) < 0)
-        return NULL;
+        return -1;
     if (PyType_Ready(&spamdict_type) < 0)
-        return NULL;
+        return -1;
 
     Py_INCREF(&spamlist_type);
     if (PyModule_AddObject(m, "spamlist",
                            (PyObject *) &spamlist_type) < 0)
-        return NULL;
+        return -1;
 
     Py_INCREF(&spamdict_type);
     if (PyModule_AddObject(m, "spamdict",
                            (PyObject *) &spamdict_type) < 0)
-        return NULL;
-    return m;
+        return -1;
+    return 0;
+}
+
+static struct PyModuleDef_Slot xxsubtype_slots[] = {
+    {Py_mod_exec, xxsubtype_exec},
+    {0, NULL},
+};
+
+static struct PyModuleDef xxsubtypemodule = {
+    PyModuleDef_HEAD_INIT,
+    "xxsubtype",
+    xxsubtype__doc__,
+    0,
+    xxsubtype_functions,
+    xxsubtype_slots,
+    NULL,
+    NULL,
+    NULL
+};
+
+
+PyMODINIT_FUNC
+PyInit_xxsubtype(void)
+{
+    return PyModuleDef_Init(&xxsubtypemodule);
 }

PC/config.c

 /* Module configuration */
 
 /* This file contains the table of built-in modules.
-   See init_builtin() in import.c. */
+   See create_builtin() in import.c. */
 
 #include "Python.h"
 

Python/clinic/import.c.h

@@ -97,6 +97,15 @@ exit:
     return return_value;
 }
 
+PyDoc_STRVAR(_imp_create_builtin__doc__,
+"create_builtin($module, spec, /)\n"
+"--\n"
+"\n"
+"Create an extension module.");
+
+#define _IMP_CREATE_BUILTIN_METHODDEF    \
+    {"create_builtin", (PyCFunction)_imp_create_builtin, METH_O, _imp_create_builtin__doc__},
+
 PyDoc_STRVAR(_imp_extension_suffixes__doc__,
 "extension_suffixes($module, /)\n"
 "--\n"
@@ -115,32 +124,6 @@ _imp_extension_suffixes(PyModuleDef *module, PyObject *Py_UNUSED(ignored))
     return _imp_extension_suffixes_impl(module);
 }
 
-PyDoc_STRVAR(_imp_init_builtin__doc__,
-"init_builtin($module, name, /)\n"
-"--\n"
-"\n"
-"Initializes a built-in module.");
-
-#define _IMP_INIT_BUILTIN_METHODDEF    \
-    {"init_builtin", (PyCFunction)_imp_init_builtin, METH_O, _imp_init_builtin__doc__},
-
-static PyObject *
-_imp_init_builtin_impl(PyModuleDef *module, PyObject *name);
-
-static PyObject *
-_imp_init_builtin(PyModuleDef *module, PyObject *arg)
-{
-    PyObject *return_value = NULL;
-    PyObject *name;
-
-    if (!PyArg_Parse(arg, "U:init_builtin", &name))
-        goto exit;
-    return_value = _imp_init_builtin_impl(module, name);
-
-exit:
-    return return_value;
-}
-
 PyDoc_STRVAR(_imp_init_frozen__doc__,
 "init_frozen($module, name, /)\n"
 "--\n"
@@ -273,31 +256,30 @@ exit:
 
 #if defined(HAVE_DYNAMIC_LOADING)
 
-PyDoc_STRVAR(_imp_load_dynamic__doc__,
-"load_dynamic($module, name, path, file=None, /)\n"
+PyDoc_STRVAR(_imp_create_dynamic__doc__,
+"create_dynamic($module, spec, file=None, /)\n"
 "--\n"
 "\n"
-"Loads an extension module.");
+"Create an extension module.");
 
-#define _IMP_LOAD_DYNAMIC_METHODDEF    \
-    {"load_dynamic", (PyCFunction)_imp_load_dynamic, METH_VARARGS, _imp_load_dynamic__doc__},
+#define _IMP_CREATE_DYNAMIC_METHODDEF    \
+    {"create_dynamic", (PyCFunction)_imp_create_dynamic, METH_VARARGS, _imp_create_dynamic__doc__},
 
 static PyObject *
-_imp_load_dynamic_impl(PyModuleDef *module, PyObject *name, PyObject *path,
-                       PyObject *file);
+_imp_create_dynamic_impl(PyModuleDef *module, PyObject *spec, PyObject *file);
 
 static PyObject *
-_imp_load_dynamic(PyModuleDef *module, PyObject *args)
+_imp_create_dynamic(PyModuleDef *module, PyObject *args)
 {
     PyObject *return_value = NULL;
-    PyObject *name;
-    PyObject *path;
+    PyObject *spec;
     PyObject *file = NULL;
 
-    if (!PyArg_ParseTuple(args, "UO&|O:load_dynamic",
-        &name, PyUnicode_FSDecoder, &path, &file))
+    if (!PyArg_UnpackTuple(args, "create_dynamic",
+        1, 2,
+        &spec, &file))
         goto exit;
-    return_value = _imp_load_dynamic_impl(module, name, path, file);
+    return_value = _imp_create_dynamic_impl(module, spec, file);
 
 exit:
     return return_value;
@@ -305,7 +287,42 @@ exit:
 
 #endif /* defined(HAVE_DYNAMIC_LOADING) */
 
-#ifndef _IMP_LOAD_DYNAMIC_METHODDEF
-    #define _IMP_LOAD_DYNAMIC_METHODDEF
-#endif /* !defined(_IMP_LOAD_DYNAMIC_METHODDEF) */
-/*[clinic end generated code: output=6d75cece35863874 input=a9049054013a1b77]*/
+#if defined(HAVE_DYNAMIC_LOADING)
+
+PyDoc_STRVAR(_imp_exec_dynamic__doc__,
+"exec_dynamic($module, mod, /)\n"
+"--\n"
+"\n"
+"Initialize an extension module.");
+
+#define _IMP_EXEC_DYNAMIC_METHODDEF    \
+    {"exec_dynamic", (PyCFunction)_imp_exec_dynamic, METH_O, _imp_exec_dynamic__doc__},
+
+static int
+_imp_exec_dynamic_impl(PyModuleDef *module, PyObject *mod);
+
+static PyObject *
+_imp_exec_dynamic(PyModuleDef *module, PyObject *mod)
+{
+    PyObject *return_value = NULL;
+    int _return_value;
+
+    _return_value = _imp_exec_dynamic_impl(module, mod);
+    if ((_return_value == -1) && PyErr_Occurred())
+        goto exit;
+    return_value = PyLong_FromLong((long)_return_value);
+
+exit:
+    return return_value;
+}
+
+#endif /* defined(HAVE_DYNAMIC_LOADING) */
+
+#ifndef _IMP_CREATE_DYNAMIC_METHODDEF
+    #define _IMP_CREATE_DYNAMIC_METHODDEF
+#endif /* !defined(_IMP_CREATE_DYNAMIC_METHODDEF) */
+
+#ifndef _IMP_EXEC_DYNAMIC_METHODDEF
+    #define _IMP_EXEC_DYNAMIC_METHODDEF
+#endif /* !defined(_IMP_EXEC_DYNAMIC_METHODDEF) */
+/*[clinic end generated code: output=0f1059766dd58f88 input=a9049054013a1b77]*/

Python/dynload_aix.c

@@ -154,8 +154,9 @@ aix_loaderror(const char *pathname)
 }
 
 
-dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
-                                    const char *pathname, FILE *fp)
+dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix,
+                                       const char *shortname,
+                                       const char *pathname, FILE *fp)
 {
     dl_funcptr p;
 

Python/dynload_dl.c

@@ -12,11 +12,12 @@ extern char *Py_GetProgramName(void);
 const char *_PyImport_DynLoadFiletab[] = {".o", NULL};
 
 
-dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
-                                    const char *pathname, FILE *fp)
+dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix,
+                                       const char *shortname,
+                                       const char *pathname, FILE *fp)
 {
     char funcname[258];
 
-    PyOS_snprintf(funcname, sizeof(funcname), "PyInit_%.200s", shortname);
+    PyOS_snprintf(funcname, sizeof(funcname), "%20s_%.200s", prefix, shortname);
     return dl_loadmod(Py_GetProgramName(), pathname, funcname);
 }

Python/dynload_hpux.c

@@ -8,15 +8,16 @@
 #include "importdl.h"
 
 #if defined(__hp9000s300)
-#define FUNCNAME_PATTERN "_PyInit_%.200s"
+#define FUNCNAME_PATTERN "_%20s_%.200s"
 #else
-#define FUNCNAME_PATTERN "PyInit_%.200s"
+#define FUNCNAME_PATTERN "%20s_%.200s"
 #endif
 
 const char *_PyImport_DynLoadFiletab[] = {SHLIB_EXT, NULL};
 
-dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
-                                    const char *pathname, FILE *fp)
+dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix,
+                                       const char *shortname,
+                                       const char *pathname, FILE *fp)
 {
     dl_funcptr p;
     shl_t lib;
@@ -50,7 +51,8 @@ dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
         Py_DECREF(pathname_ob);
         return NULL;
     }
-    PyOS_snprintf(funcname, sizeof(funcname), FUNCNAME_PATTERN, shortname);
+    PyOS_snprintf(funcname, sizeof(funcname), FUNCNAME_PATTERN,
+                  prefix, shortname);
     if (Py_VerboseFlag)
         printf("shl_findsym %s\n", funcname);
     if (shl_findsym(&lib, funcname, TYPE_UNDEFINED, (void *) &p) == -1) {

Python/dynload_next.c

@@ -27,8 +27,9 @@ const char *_PyImport_DynLoadFiletab[] = {".so", NULL};
 #define LINKOPTIONS NSLINKMODULE_OPTION_BINDNOW| \
     NSLINKMODULE_OPTION_RETURN_ON_ERROR|NSLINKMODULE_OPTION_PRIVATE
 #endif
-dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
-                                    const char *pathname, FILE *fp)
+dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix,
+                                       const char *shortname,
+                                       const char *pathname, FILE *fp)
 {
     dl_funcptr p = NULL;
     char funcname[258];
@@ -39,7 +40,7 @@ dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
     const char *errString;
     char errBuf[512];
 
-    PyOS_snprintf(funcname, sizeof(funcname), "_PyInit_%.200s", shortname);
+    PyOS_snprintf(funcname, sizeof(funcname), "_%20s_%.200s", prefix, shortname);
 
 #ifdef USE_DYLD_GLOBAL_NAMESPACE
     if (NSIsSymbolNameDefined(funcname)) {

Python/dynload_shlib.c

@@ -51,8 +51,10 @@ static struct {
 static int nhandles = 0;
 
 
-dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
-                                    const char *pathname, FILE *fp)
+dl_funcptr
+_PyImport_FindSharedFuncptr(const char *prefix,
+                            const char *shortname,
+                            const char *pathname, FILE *fp)
 {
     dl_funcptr p;
     void *handle;
@@ -67,7 +69,7 @@ dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
     }
 
     PyOS_snprintf(funcname, sizeof(funcname),
-                  LEAD_UNDERSCORE "PyInit_%.200s", shortname);
+                  LEAD_UNDERSCORE "%.20s_%.200s", prefix, shortname);
 
     if (fp != NULL) {
         int i;

Python/dynload_win.c

@@ -186,8 +186,9 @@ static char *GetPythonImport (HINSTANCE hModule)
     return NULL;
 }
 
-dl_funcptr _PyImport_GetDynLoadWindows(const char *shortname,
-                                       PyObject *pathname, FILE *fp)
+dl_funcptr _PyImport_FindSharedFuncptrWindows(const char *prefix,
+                                              const char *shortname,
+                                              PyObject *pathname, FILE *fp)
 {
     dl_funcptr p;
     char funcname[258], *import_python;
@@ -201,7 +202,7 @@ dl_funcptr _PyImport_GetDynLoadWindows(const char *shortname,
     if (wpathname == NULL)
         return NULL;
 
-    PyOS_snprintf(funcname, sizeof(funcname), "PyInit_%.200s", shortname);
+    PyOS_snprintf(funcname, sizeof(funcname), "%20_%.200s", prefix, shortname);
 
     {
         HINSTANCE hDLL = NULL;

Python/import.c

@@ -1026,50 +1026,74 @@ PyImport_GetImporter(PyObject *path) {
     return importer;
 }
 
+/*[clinic input]
+_imp.create_builtin
 
-static int init_builtin(PyObject *); /* Forward */
+    spec: object
+    /
 
-/* Initialize a built-in module.
-   Return 1 for success, 0 if the module is not found, and -1 with
-   an exception set if the initialization failed. */
+Create an extension module.
+[clinic start generated code]*/
 
-static int
-init_builtin(PyObject *name)
+static PyObject *
+_imp_create_builtin(PyModuleDef *module, PyObject *spec)
+/*[clinic end generated code: output=5038f467617226bd input=37f966f890384e47]*/
 {
     struct _inittab *p;
+    PyObject *name;
+    char *namestr;
     PyObject *mod;
 
+    name = PyObject_GetAttrString(spec, "name");
+    if (name == NULL) {
+        return NULL;
+    }
+
     mod = _PyImport_FindExtensionObject(name, name);
-    if (PyErr_Occurred())
-        return -1;
-    if (mod != NULL)
-        return 1;
+    if (mod || PyErr_Occurred()) {
+        Py_DECREF(name);
+        Py_INCREF(mod);
+        return mod;
+    }
+
+    namestr = PyUnicode_AsUTF8(name);
+    if (namestr == NULL) {
+        Py_DECREF(name);
+        return NULL;
+    }
 
     for (p = PyImport_Inittab; p->name != NULL; p++) {
-        PyObject *mod;
         PyModuleDef *def;
         if (PyUnicode_CompareWithASCIIString(name, p->name) == 0) {
             if (p->initfunc == NULL) {
-                PyErr_Format(PyExc_ImportError,
-                    "Cannot re-init internal module %R",
-                    name);
-                return -1;
+                /* Cannot re-init internal module ("sys" or "builtins") */
+                mod = PyImport_AddModule(namestr);
+                Py_DECREF(name);
+                return mod;
             }
             mod = (*p->initfunc)();
-            if (mod == 0)
-                return -1;
-            /* Remember pointer to module init function. */
-            def = PyModule_GetDef(mod);
-            def->m_base.m_init = p->initfunc;
-            if (_PyImport_FixupExtensionObject(mod, name, name) < 0)
-                return -1;
-            /* FixupExtension has put the module into sys.modules,
-               so we can release our own reference. */
-            Py_DECREF(mod);
-            return 1;
+            if (mod == NULL) {
+                Py_DECREF(name);
+                return NULL;
+            }
+            if (PyObject_TypeCheck(mod, &PyModuleDef_Type)) {
+                Py_DECREF(name);
+                return PyModule_FromDefAndSpec((PyModuleDef*)mod, spec);
+            } else {
+                /* Remember pointer to module init function. */
+                def = PyModule_GetDef(mod);
+                def->m_base.m_init = p->initfunc;
+                if (_PyImport_FixupExtensionObject(mod, name, name) < 0) {
+                    Py_DECREF(name);
+                    return NULL;
+                }
+                Py_DECREF(name);
+                return mod;
+            }
         }
     }
-    return 0;
+    Py_DECREF(name);
+    Py_RETURN_NONE;
 }
 
 
@@ -1820,34 +1844,6 @@ _imp_extension_suffixes_impl(PyModuleDef *module)
     return list;
 }
 
-/*[clinic input]
-_imp.init_builtin
-
-    name: unicode
-    /
-
-Initializes a built-in module.
-[clinic start generated code]*/
-
-static PyObject *
-_imp_init_builtin_impl(PyModuleDef *module, PyObject *name)
-/*[clinic end generated code: output=1868f473685f6d67 input=f934d2231ec52a2e]*/
-{
-    int ret;
-    PyObject *m;
-
-    ret = init_builtin(name);
-    if (ret < 0)
-        return NULL;
-    if (ret == 0) {
-        Py_INCREF(Py_None);
-        return Py_None;
-    }
-    m = PyImport_AddModuleObject(name);
-    Py_XINCREF(m);
-    return m;
-}
-
 /*[clinic input]
 _imp.init_frozen
 
@@ -1946,40 +1942,100 @@ _imp_is_frozen_impl(PyModuleDef *module, PyObject *name)
 #ifdef HAVE_DYNAMIC_LOADING
 
 /*[clinic input]
-_imp.load_dynamic
+_imp.create_dynamic
 
-    name: unicode
-    path: fs_unicode
+    spec: object
     file: object = NULL
     /
 
-Loads an extension module.
+Create an extension module.
 [clinic start generated code]*/
 
 static PyObject *
-_imp_load_dynamic_impl(PyModuleDef *module, PyObject *name, PyObject *path,
-                       PyObject *file)
-/*[clinic end generated code: output=e84e5f7f0f39bc54 input=af64f06e4bad3526]*/
+_imp_create_dynamic_impl(PyModuleDef *module, PyObject *spec, PyObject *file)
+/*[clinic end generated code: output=935cde5b3872d56d input=c31b954f4cf4e09d]*/
 {
-    PyObject *mod;
+    PyObject *mod, *name, *path;
     FILE *fp;
 
+    name = PyObject_GetAttrString(spec, "name");
+    if (name == NULL) {
+        return NULL;
+    }
+
+    path = PyObject_GetAttrString(spec, "origin");
+    if (path == NULL) {
+        Py_DECREF(name);
+        return NULL;
+    }
+
+    mod = _PyImport_FindExtensionObject(name, path);
+    if (mod != NULL) {
+        Py_DECREF(name);
+        Py_DECREF(path);
+        Py_INCREF(mod);
+        return mod;
+    }
+
     if (file != NULL) {
         fp = _Py_fopen_obj(path, "r");
         if (fp == NULL) {
+            Py_DECREF(name);
             Py_DECREF(path);
             return NULL;
         }
     }
     else
         fp = NULL;
-    mod = _PyImport_LoadDynamicModule(name, path, fp);
+
+    mod = _PyImport_LoadDynamicModuleWithSpec(spec, fp);
+
+    Py_DECREF(name);
     Py_DECREF(path);
     if (fp)
         fclose(fp);
     return mod;
 }
 
+/*[clinic input]
+_imp.exec_dynamic -> int
+
+    mod: object
+    /
+
+Initialize an extension module.
+[clinic start generated code]*/
+
+static int
+_imp_exec_dynamic_impl(PyModuleDef *module, PyObject *mod)
+/*[clinic end generated code: output=4b84f1301b22d4bd input=9fdbfcb250280d3a]*/
+{
+    PyModuleDef *def;
+    void *state;
+
+    if (!PyModule_Check(mod)) {
+        return 0;
+    }
+
+    def = PyModule_GetDef(mod);
+    if (def == NULL) {
+        if (PyErr_Occurred()) {
+            return -1;
+        }
+        return 0;
+    }
+    state = PyModule_GetState(mod);
+    if (PyErr_Occurred()) {
+        return -1;
+    }
+    if (state) {
+        /* Already initialized; skip reload */
+        return 0;
+    }
+    return PyModule_ExecDef(mod, def);
+}
+
+
 #endif /* HAVE_DYNAMIC_LOADING */
 
 /*[clinic input]
@@ -1998,11 +2054,12 @@ static PyMethodDef imp_methods[] = {
     _IMP_RELEASE_LOCK_METHODDEF
     _IMP_GET_FROZEN_OBJECT_METHODDEF
     _IMP_IS_FROZEN_PACKAGE_METHODDEF
-    _IMP_INIT_BUILTIN_METHODDEF
+    _IMP_CREATE_BUILTIN_METHODDEF
     _IMP_INIT_FROZEN_METHODDEF
     _IMP_IS_BUILTIN_METHODDEF
     _IMP_IS_FROZEN_METHODDEF
-    _IMP_LOAD_DYNAMIC_METHODDEF
+    _IMP_CREATE_DYNAMIC_METHODDEF
+    _IMP_EXEC_DYNAMIC_METHODDEF
     _IMP__FIX_CO_FILENAME_METHODDEF
     {NULL, NULL}  /* sentinel */
 };

Python/importdl.c

@@ -13,87 +13,186 @@
 #include "importdl.h"
 
 #ifdef MS_WINDOWS
-extern dl_funcptr _PyImport_GetDynLoadWindows(const char *shortname,
-                                              PyObject *pathname, FILE *fp);
+extern dl_funcptr _PyImport_FindSharedFuncptrWindows(const char *prefix,
+                                                     const char *shortname,
+                                                     PyObject *pathname,
+                                                     FILE *fp);
 #else
-extern dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname,
-                                           const char *pathname, FILE *fp);
+extern dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix,
+                                              const char *shortname,
+                                              const char *pathname, FILE *fp);
 #endif
 
+static const char *ascii_only_prefix = "PyInit";
+static const char *nonascii_prefix = "PyInitU";
+
+/* Get the variable part of a module's export symbol name.
+ * Returns a bytes instance. For non-ASCII-named modules, the name is
+ * encoded as per PEP 489.
+ * The hook_prefix pointer is set to either ascii_only_prefix or
+ * nonascii_prefix, as appropriate.
+ */
+static PyObject *
+get_encoded_name(PyObject *name, const char **hook_prefix) {
+    char *buf;
+    PyObject *tmp;
+    PyObject *encoded = NULL;
+    Py_ssize_t name_len, lastdot, i;
+
+    /* Get the short name (substring after last dot) */
+    name_len = PyUnicode_GetLength(name);
+    lastdot = PyUnicode_FindChar(name, '.', 0, name_len, -1);
+    if (lastdot < -1) {
+        return NULL;
+    } else if (lastdot >= 0) {
+        tmp = PyUnicode_Substring(name, lastdot, name_len);
+        if (tmp == NULL)
+            return NULL;
+        name = tmp;
+        /* "name" now holds a new reference to the substring */
+    } else {
+        Py_INCREF(name);
+    }
+
+    /* Encode to ASCII or Punycode, as needed */
+    encoded = PyUnicode_AsEncodedString(name, "ascii", NULL);
+    if (encoded != NULL) {
+        *hook_prefix = ascii_only_prefix;
+    } else {
+        if (PyErr_ExceptionMatches(PyExc_UnicodeEncodeError)) {
+            PyErr_Clear();
+            encoded = PyUnicode_AsEncodedString(name, "punycode", NULL);
+            if (encoded == NULL) {
+                goto error;
+            }
+            *hook_prefix = nonascii_prefix;
+        } else {
+            goto error;
+        }
+    }
+
+    buf = PyBytes_AS_STRING(encoded);
+    assert(Py_REFCNT(encoded) == 1);
+    for (i = 0; i < PyBytes_GET_SIZE(encoded) + 1; i++) {
+        if (buf[i] == '-') {
+            buf[i] = '_';
+        }
+    }
+
+    Py_DECREF(name);
+    return encoded;
+error:
+    Py_DECREF(name);
+    Py_XDECREF(encoded);
+    return NULL;
+}
+
 PyObject *
-_PyImport_LoadDynamicModule(PyObject *name, PyObject *path, FILE *fp)
+_PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp)
 {
-    PyObject *m = NULL;
 #ifndef MS_WINDOWS
-    PyObject *pathbytes;
+    PyObject *pathbytes = NULL;
 #endif
-    PyObject *nameascii;
-    char *namestr, *lastdot, *shortname, *packagecontext, *oldcontext;
-    dl_funcptr p0;
-    PyObject* (*p)(void);
-    struct PyModuleDef *def;
-
-    m = _PyImport_FindExtensionObject(name, path);
-    if (m != NULL) {
-        Py_INCREF(m);
-        return m;
-    }
-
-    /* name must be encodable to ASCII because dynamic module must have a
-       function called "PyInit_NAME", they are written in C, and the C language
-       doesn't accept non-ASCII identifiers. */
-    nameascii = PyUnicode_AsEncodedString(name, "ascii", NULL);
-    if (nameascii == NULL)
+    PyObject *name_unicode = NULL, *name = NULL, *path = NULL, *m = NULL;
+    const char *name_buf, *hook_prefix;
+    char *oldcontext;
+    dl_funcptr exportfunc;
+    PyModuleDef *def;
+    PyObject *(*p0)(void);
+
+    name_unicode = PyObject_GetAttrString(spec, "name");
+    if (name_unicode == NULL) {
         return NULL;
+    }
 
-    namestr = PyBytes_AS_STRING(nameascii);
-    if (namestr == NULL)
+    name = get_encoded_name(name_unicode, &hook_prefix);
+    if (name == NULL) {
         goto error;
-
-    lastdot = strrchr(namestr, '.');
-    if (lastdot == NULL) {
-        packagecontext = NULL;
-        shortname = namestr;
-    }
-    else {
-        packagecontext = namestr;
-        shortname = lastdot+1;
     }
+    name_buf = PyBytes_AS_STRING(name);
+
+    path = PyObject_GetAttrString(spec, "origin");
+    if (path == NULL)
+        goto error;
 
 #ifdef MS_WINDOWS
-    p0 = _PyImport_GetDynLoadWindows(shortname, path, fp);
+    exportfunc = _PyImport_FindSharedFuncptrWindows(hook_prefix, name_buf,
+                                                    path, fp);
 #else
     pathbytes = PyUnicode_EncodeFSDefault(path);
     if (pathbytes == NULL)
         goto error;
-    p0 = _PyImport_GetDynLoadFunc(shortname,
-                                  PyBytes_AS_STRING(pathbytes), fp);
+    exportfunc = _PyImport_FindSharedFuncptr(hook_prefix, name_buf,
+                                             PyBytes_AS_STRING(pathbytes),
+                                             fp);
     Py_DECREF(pathbytes);
 #endif
-    p = (PyObject*(*)(void))p0;
-    if (PyErr_Occurred())
-        goto error;
-    if (p == NULL) {
-        PyObject *msg = PyUnicode_FromFormat("dynamic module does not define "
-                                             "init function (PyInit_%s)",
-                                             shortname);
-        if (msg == NULL)
-            goto error;
-        PyErr_SetImportError(msg, name, path);
-        Py_DECREF(msg);
+
+    if (exportfunc == NULL) {
+        if (!PyErr_Occurred()) {
+            PyObject *msg;
+            msg = PyUnicode_FromFormat(
+                "dynamic module does not define "
+                "module export function (%s_%s)",
+                hook_prefix, name_buf);
+            if (msg == NULL)
+                goto error;
+            PyErr_SetImportError(msg, name_unicode, path);
+            Py_DECREF(msg);
+        }
         goto error;
     }
+
+    p0 = (PyObject *(*)(void))exportfunc;
+
+    /* Package context is needed for single-phase init */
     oldcontext = _Py_PackageContext;
-    _Py_PackageContext = packagecontext;
-    m = (*p)();
+    _Py_PackageContext = PyUnicode_AsUTF8(name_unicode);
+    m = p0();
     _Py_PackageContext = oldcontext;
-    if (m == NULL)
-        goto error;
 
-    if (PyErr_Occurred()) {
+    if (m == NULL) {
+        if (!PyErr_Occurred()) {
+            PyErr_Format(
+                PyExc_SystemError,
+                "initialization of %s failed without raising an exception",
+                name_buf);
+        }
+        goto error;
+    } else if (PyErr_Occurred()) {
+        PyErr_Clear();
+        PyErr_Format(
+            PyExc_SystemError,
+            "initialization of %s raised unreported exception",
+            name_buf);
+        m = NULL;
+        goto error;
+    }
+    if (Py_TYPE(m) == NULL) {
+        /* This can happen when a PyModuleDef is returned without calling
+         * PyModuleDef_Init on it
+         */
         PyErr_Format(PyExc_SystemError,
-                     "initialization of %s raised unreported exception",
-                     shortname);
+                     "init function of %s returned uninitialized object",
+                     name_buf);
+        m = NULL; /* prevent segfault in DECREF */
+        goto error;
+    }
+    if (PyObject_TypeCheck(m, &PyModuleDef_Type)) {
+        Py_DECREF(name_unicode);
+        Py_DECREF(name);
+        Py_DECREF(path);
+        return PyModule_FromDefAndSpec((PyModuleDef*)m, spec);
+    }
+
+    /* Fall back to single-phase init mechanism */
+
+    if (hook_prefix == nonascii_prefix) {
+        /* don't allow legacy init for non-ASCII module names */
+        PyErr_Format(
+            PyExc_SystemError,
+            "initialization of * did not return PyModuleDef",
+            name_buf);
         goto error;
     }
 
@@ -102,10 +201,10 @@ _PyImport_LoadDynamicModule(PyObject *name, PyObject *path, FILE *fp)
     if (def == NULL) {
         PyErr_Format(PyExc_SystemError,
                      "initialization of %s did not return an extension "
-                     "module", shortname);
+                     "module", name_buf);
         goto error;
     }
-    def->m_base.m_init = p;
+    def->m_base.m_init = p0;
 
     /* Remember the filename as the __file__ attribute */
     if (PyModule_AddObject(m, "__file__", path) < 0)
@@ -113,13 +212,19 @@ _PyImport_LoadDynamicModule(PyObject *name, PyObject *path, FILE *fp)
     else
         Py_INCREF(path);
 
-    if (_PyImport_FixupExtensionObject(m, name, path) < 0)
+    if (_PyImport_FixupExtensionObject(m, name_unicode, path) < 0)
         goto error;
-    Py_DECREF(nameascii);
+
+    Py_DECREF(name_unicode);
+    Py_DECREF(name);
+    Py_DECREF(path);
+
     return m;
 
 error:
-    Py_DECREF(nameascii);
+    Py_DECREF(name_unicode);
+    Py_XDECREF(name);
+    Py_XDECREF(path);
     Py_XDECREF(m);
     return NULL;
 }

Python/importdl.h

@@ -8,8 +8,7 @@ extern "C" {
 
 extern const char *_PyImport_DynLoadFiletab[];
 
-extern PyObject *_PyImport_LoadDynamicModule(PyObject *name, PyObject *pathname,
-                                             FILE *);
+extern PyObject *_PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *);
 
 /* Max length of module suffix searched for -- accommodates "module.slb" */
 #define MAXSUFFIXSIZE 12

Python/pystate.c

@@ -255,6 +255,9 @@ PyState_FindModule(struct PyModuleDef* module)
     Py_ssize_t index = module->m_base.m_index;
     PyInterpreterState *state = PyThreadState_GET()->interp;
     PyObject *res;
+    if (module->m_slots) {
+        return NULL;
+    }
     if (index == 0)
         return NULL;
     if (state->modules_by_index == NULL)
@@ -268,7 +271,13 @@ PyState_FindModule(struct PyModuleDef* module)
 int
 _PyState_AddModule(PyObject* module, struct PyModuleDef* def)
 {
-    PyInterpreterState *state = PyThreadState_GET()->interp;
+    PyInterpreterState *state;
+    if (def->m_slots) {
+        PyErr_SetString(PyExc_SystemError,
+                        "PyState_AddModule called on module with slots");
+        return -1;
+    }
+    state = PyThreadState_GET()->interp;
     if (!def)
         return -1;
     if (!state->modules_by_index) {
@@ -308,8 +317,14 @@ PyState_AddModule(PyObject* module, struct PyModuleDef* def)
 int
 PyState_RemoveModule(struct PyModuleDef* def)
 {
+    PyInterpreterState *state;
     Py_ssize_t index = def->m_base.m_index;
-    PyInterpreterState *state = PyThreadState_GET()->interp;
+    if (def->m_slots) {
+        PyErr_SetString(PyExc_SystemError,
+                        "PyState_RemoveModule called on module with slots");
+        return -1;
+    }
+    state = PyThreadState_GET()->interp;
     if (index == 0) {
         Py_FatalError("PyState_RemoveModule: Module index invalid.");
         return -1;

setup.py

@@ -620,6 +620,8 @@ class PyBuildExt(build_ext):
         exts.append( Extension('_testbuffer', ['_testbuffer.c']) )
         # Test loading multiple modules from one compiled file (http://bugs.python.org/issue16421)
         exts.append( Extension('_testimportmultiple', ['_testimportmultiple.c']) )
+        # Test multi-phase extension module init (PEP 489)
+        exts.append( Extension('_testmultiphase', ['_testmultiphase.c']) )
         # profiler (_lsprof is for cProfile.py)
         exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) )
         # static Unicode character database