Issue #24400: Introduce a distinct type for 'async def' coroutines.

Summary of changes:

1. Coroutines now have a distinct, separate from generators
   type at the C level: PyGen_Type, and a new typedef PyCoroObject.
   PyCoroObject shares the initial segment of struct layout with
   PyGenObject, making it possible to reuse existing generators
   machinery.  The new type is exposed as 'types.CoroutineType'.

   As a consequence of having a new type, CO_GENERATOR flag is
   no longer applied to coroutines.

2. Having a separate type for coroutines made it possible to add
   an __await__ method to the type.  Although it is not used by the
   interpreter (see details on that below), it makes coroutines
   naturally (without using __instancecheck__) conform to
   collections.abc.Coroutine and collections.abc.Awaitable ABCs.

   [The __instancecheck__ is still used for generator-based
   coroutines, as we don't want to add __await__ for generators.]

3. Add new opcode: GET_YIELD_FROM_ITER.  The opcode is needed to
   allow passing native coroutines to the YIELD_FROM opcode.

   Before this change, 'yield from o' expression was compiled to:

      (o)
      GET_ITER
      LOAD_CONST
      YIELD_FROM

   Now, we use GET_YIELD_FROM_ITER instead of GET_ITER.

   The reason for adding a new opcode is that GET_ITER is used
   in some contexts (such as 'for .. in' loops) where passing
   a coroutine object is invalid.

4. Add two new introspection functions to the inspec module:
   getcoroutinestate(c) and getcoroutinelocals(c).

5. inspect.iscoroutine(o) is updated to test if 'o' is a native
   coroutine object.  Before this commit it used abc.Coroutine,
   and it was requested to update inspect.isgenerator(o) to use
   abc.Generator; it was decided, however, that inspect functions
   should really be tailored for checking for native types.

6. sys.set_coroutine_wrapper(w) API is updated to work with only
   native coroutines.  Since types.coroutine decorator supports
   any type of callables now, it would be confusing that it does
   not work for all types of coroutines.

7. Exceptions logic in generators C implementation was updated
   to raise clearer messages for coroutines:

   Before: TypeError("generator raised StopIteration")
   After: TypeError("coroutine raised StopIteration")

Doc/c-api/concrete.rst

@@ -112,5 +112,6 @@ Other Objects
    weakref.rst
    capsule.rst
    gen.rst
+   coro.rst
    datetime.rst
 

Doc/c-api/coro.rst

+.. highlightlang:: c
+
+.. _coro-objects:
+
+Coroutine Objects
+-----------------
+
+.. versionadded:: 3.5
+
+Coroutine objects are what functions declared with an ``async`` keyword
+return.
+
+
+.. c:type:: PyCoroObject
+
+   The C structure used for coroutine objects.
+
+
+.. c:var:: PyTypeObject PyCoro_Type
+
+   The type object corresponding to coroutine objects.
+
+
+.. c:function:: int PyCoro_CheckExact(PyObject *ob)
+
+   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*.
+
+
+.. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
+
+   Create and return a new coroutine object based on the *frame* object,
+   with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
+   A reference to *frame* is stolen by this function.  The *frame* argument
+   must not be *NULL*.

Doc/c-api/gen.rst

@@ -7,7 +7,7 @@ Generator Objects
 
 Generator objects are what Python uses to implement generator iterators. They
 are normally created by iterating over a function that yields values, rather
-than explicitly calling :c:func:`PyGen_New`.
+than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 
 .. c:type:: PyGenObject
@@ -20,19 +20,25 @@ than explicitly calling :c:func:`PyGen_New`.
    The type object corresponding to generator objects
 
 
-.. c:function:: int PyGen_Check(ob)
+.. c:function:: int PyGen_Check(PyObject *ob)
 
    Return true if *ob* is a generator object; *ob* must not be *NULL*.
 
 
-.. c:function:: int PyGen_CheckExact(ob)
+.. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type* is a generator object; *ob* must not
-   be *NULL*.
+   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
 
-   Create and return a new generator object based on the *frame* object. A
-   reference to *frame* is stolen by this function. The parameter must not be
+   Create and return a new generator object based on the *frame* object.
+   A reference to *frame* is stolen by this function. The argument must not be
    *NULL*.
+
+.. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname)
+
+   Create and return a new generator object based on the *frame* object,
+   with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
+   A reference to *frame* is stolen by this function.  The *frame* argument
+   must not be *NULL*.

Doc/data/refcounts.dat

@@ -491,6 +491,12 @@ PyFunction_SetDefaults:PyObject*:defaults:+1:
 PyGen_New:PyObject*::+1:
 PyGen_New:PyFrameObject*:frame:0:
 
+PyGen_NewWithQualName:PyObject*::+1:
+PyGen_NewWithQualName:PyFrameObject*:frame:0:
+
+PyCoro_New:PyObject*::+1:
+PyCoro_New:PyFrameObject*:frame:0:
+
 Py_InitModule:PyObject*::0:
 Py_InitModule:const char*:name::
 Py_InitModule:PyMethodDef[]:methods::

Doc/glossary.rst

@@ -333,14 +333,23 @@ Glossary
       .. index:: single: generator
 
    generator
-      A function which returns an iterator.  It looks like a normal function
-      except that it contains :keyword:`yield` statements for producing a series
-      of values usable in a for-loop or that can be retrieved one at a time with
-      the :func:`next` function. Each :keyword:`yield` temporarily suspends
-      processing, remembering the location execution state (including local
-      variables and pending try-statements).  When the generator resumes, it
-      picks-up where it left-off (in contrast to functions which start fresh on
-      every invocation).
+      A function which returns a :term:`generator iterator`.  It looks like a
+      normal function except that it contains :keyword:`yield` expressions
+      for producing a series of values usable in a for-loop or that can be
+      retrieved one at a time with the :func:`next` function.
+
+      Usually refers to a generator function, but may refer to a
+      *generator iterator* in some contexts.  In cases where the intended
+      meaning isn't clear, using the full terms avoids ambiguity.
+
+   generator iterator
+      An object created by a :term:`generator` function.
+
+      Each :keyword:`yield` temporarily suspends processing, remembering the
+      location execution state (including local variables and pending
+      try-statements).  When the *generator iterator* resumes, it picks-up where
+      it left-off (in contrast to functions which start fresh on every
+      invocation).
 
       .. index:: single: generator expression
 

Doc/library/dis.rst

@@ -346,6 +346,14 @@ result back on the stack.
    Implements ``TOS = iter(TOS)``.
 
 
+.. opcode:: GET_YIELD_FROM_ITER
+
+   If ``TOS`` is a :term:`generator iterator` or :term:`coroutine` object
+   it is left as is.  Otherwise, implements ``TOS = iter(TOS)``.
+
+   .. versionadded:: 3.5
+
+
 **Binary operations**
 
 Binary operations remove the top of the stack (TOS) and the second top-most

Doc/library/inspect.rst

@@ -178,6 +178,16 @@ attributes:
 +-----------+-----------------+---------------------------+
 |           | gi_code         | code                      |
 +-----------+-----------------+---------------------------+
+| coroutine | __name__        | name                      |
++-----------+-----------------+---------------------------+
+|           | __qualname__    | qualified name            |
++-----------+-----------------+---------------------------+
+|           | cr_frame        | frame                     |
++-----------+-----------------+---------------------------+
+|           | cr_running      | is the coroutine running? |
++-----------+-----------------+---------------------------+
+|           | cr_code         | code                      |
++-----------+-----------------+---------------------------+
 | builtin   | __doc__         | documentation string      |
 +-----------+-----------------+---------------------------+
 |           | __name__        | original name of this     |
@@ -279,29 +289,16 @@ attributes:
 
 .. function:: iscoroutinefunction(object)
 
-   Return true if the object is a :term:`coroutine function`.
-
-   Coroutine functions are defined with an ``async def`` syntax,
-   or are generators decorated with :func:`types.coroutine`
-   or :func:`asyncio.coroutine`.
-
-   The function will return false for plain Python generator
-   functions.
+   Return true if the object is a :term:`coroutine function`
+   (a function defined with an :keyword:`async def` syntax).
 
    .. versionadded:: 3.5
 
 
 .. function:: iscoroutine(object)
 
-   Return true if the object is a :term:`coroutine`.
-
-   Coroutines are results of calls of coroutine functions or
-   generator functions decorated with :func:`types.coroutine`
-   or :func:`asyncio.coroutine`.
-
-   The function will return false for plain python generators.
-
-   See also :class:`collections.abc.Coroutine`.
+   Return true if the object is a :term:`coroutine` created by an
+   :keyword:`async def` function.
 
    .. versionadded:: 3.5
 
@@ -1116,8 +1113,8 @@ code execution::
            pass
 
 
-Current State of a Generator
-----------------------------
+Current State of Generators and Coroutines
+------------------------------------------
 
 When implementing coroutine schedulers and for other advanced uses of
 generators, it is useful to determine whether a generator is currently
@@ -1137,6 +1134,21 @@ generator to be determined easily.
 
    .. versionadded:: 3.2
 
+.. function:: getcoroutinestate(coroutine)
+
+   Get current state of a coroutine object.  The function is intended to be
+   used with coroutine objects created by :keyword:`async def` functions, but
+   will accept any coroutine-like object that has ``cr_running`` and
+   ``cr_frame`` attributes.
+
+   Possible states are:
+    * CORO_CREATED: Waiting to start execution.
+    * CORO_RUNNING: Currently being executed by the interpreter.
+    * CORO_SUSPENDED: Currently suspended at an await expression.
+    * CORO_CLOSED: Execution has completed.
+
+   .. versionadded:: 3.5
+
 The current internal state of the generator can also be queried. This is
 mostly useful for testing purposes, to ensure that internal state is being
 updated as expected:
@@ -1161,6 +1173,13 @@ updated as expected:
 
    .. versionadded:: 3.3
 
+.. function:: getcoroutinelocals(coroutine)
+
+   This function is analogous to :func:`~inspect.getgeneratorlocals`, but
+   works for coroutine objects created by :keyword:`async def` functions.
+
+   .. versionadded:: 3.5
+
 
 .. _inspect-module-cli:
 

Doc/library/sys.rst

@@ -1075,7 +1075,10 @@ always available.
 
 .. function:: set_coroutine_wrapper(wrapper)
 
-   Allows to intercept creation of :term:`coroutine` objects.
+   Allows intercepting creation of :term:`coroutine` objects (only ones that
+   are created by an :keyword:`async def` function; generators decorated with
+   :func:`types.coroutine` or :func:`asyncio.coroutine` will not be
+   intercepted).
 
    *wrapper* must be either:
 

Doc/library/types.rst

@@ -90,6 +90,14 @@ Standard names are defined for the following types:
    generator function.
 
 
+.. data:: CoroutineType
+
+   The type of :term:`coroutine` objects, produced by calling a
+   function defined with an :keyword:`async def` statement.
+
+   .. versionadded:: 3.5
+
+
 .. data:: CodeType
 
    .. index:: builtin: compile

Doc/reference/compound_stmts.rst

@@ -666,6 +666,15 @@ can be used to create instance variables with different implementation details.
 Coroutines
 ==========
 
+.. index::
+   statement: async def
+   statement: async for
+   statement: async with
+   keyword: async
+   keyword: await
+
+.. versionadded:: 3.5
+
 .. _`async def`:
 
 Coroutine function definition
@@ -683,7 +692,11 @@ even if they do not contain ``await`` or ``async`` keywords.
 
 It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in coroutines.
 
-.. versionadded:: 3.5
+An example of a coroutine function::
+
+    async def func(param1, param2):
+        do_stuff()
+        await some_coroutine()
 
 
 .. _`async for`:
@@ -725,7 +738,8 @@ Is semantically equivalent to::
 
 See also :meth:`__aiter__` and :meth:`__anext__` for details.
 
-.. versionadded:: 3.5
+It is a :exc:`SyntaxError` to use ``async for`` statement outside of an
+:keyword:`async def` function.
 
 
 .. _`async with`:
@@ -762,7 +776,8 @@ Is semantically equivalent to::
 
 See also :meth:`__aenter__` and :meth:`__aexit__` for details.
 
-.. versionadded:: 3.5
+It is a :exc:`SyntaxError` to use ``async with`` statement outside of an
+:keyword:`async def` function.
 
 .. seealso::
 

Doc/whatsnew/3.5.rst

@@ -531,6 +531,9 @@ inspect
   and :func:`~inspect.isawaitable` functions.  (Contributed by Yury Selivanov
   in :issue:`24017`.)
 
+* New :func:`~inspect.getcoroutinelocals` and :func:`~inspect.getcoroutinestate`
+  functions.  (Contributed by Yury Selivanov in :issue:`24400`.)
+
 ipaddress
 ---------
 
@@ -734,6 +737,9 @@ types
 * New :func:`~types.coroutine` function.  (Contributed by Yury Selivanov
   in :issue:`24017`.)
 
+* New :class:`~types.CoroutineType`.  (Contributed by Yury Selivanov
+  in :issue:`24400`.)
+
 urllib
 ------
 
@@ -1101,6 +1107,7 @@ Changes in the C API
   the :attr:`__module__` attribute.  Would be an AttributeError in future.
   (:issue:`20204`)
 
-* As part of PEP 492 implementation, ``tp_reserved`` slot of
+* As part of :pep:`492` implementation, ``tp_reserved`` slot of
   :c:type:`PyTypeObject` was replaced with a
-  :c:member:`PyTypeObject.tp_as_async` slot.
+  :c:member:`PyTypeObject.tp_as_async` slot.  Refer to :ref:`coro-objects` for
+  new types, structures and functions.

Include/genobject.h

@@ -10,27 +10,26 @@ extern "C" {
 
 struct _frame; /* Avoid including frameobject.h */
 
+/* _PyGenObject_HEAD defines the initial segment of generator
+   and coroutine objects. */
+#define _PyGenObject_HEAD(prefix)                                           \
+    PyObject_HEAD                                                           \
+    /* Note: gi_frame can be NULL if the generator is "finished" */         \
+    struct _frame *prefix##_frame;                                          \
+    /* True if generator is being executed. */                              \
+    char prefix##_running;                                                  \
+    /* The code object backing the generator */                             \
+    PyObject *prefix##_code;                                                \
+    /* List of weak reference. */                                           \
+    PyObject *prefix##_weakreflist;                                         \
+    /* Name of the generator. */                                            \
+    PyObject *prefix##_name;                                                \
+    /* Qualified name of the generator. */                                  \
+    PyObject *prefix##_qualname;
+
 typedef struct {
-    PyObject_HEAD
     /* The gi_ prefix is intended to remind of generator-iterator. */
-
-    /* Note: gi_frame can be NULL if the generator is "finished" */
-    struct _frame *gi_frame;
-
-    /* True if generator is being executed. */
-    char gi_running;
-
-    /* The code object backing the generator */
-    PyObject *gi_code;
-
-    /* List of weak reference. */
-    PyObject *gi_weakreflist;
-
-    /* Name of the generator. */
-    PyObject *gi_name;
-
-    /* Qualified name of the generator. */
-    PyObject *gi_qualname;
+    _PyGenObject_HEAD(gi)
 } PyGenObject;
 
 PyAPI_DATA(PyTypeObject) PyGen_Type;
@@ -38,12 +37,6 @@ PyAPI_DATA(PyTypeObject) PyGen_Type;
 #define PyGen_Check(op) PyObject_TypeCheck(op, &PyGen_Type)
 #define PyGen_CheckExact(op) (Py_TYPE(op) == &PyGen_Type)
 
-#define PyGen_CheckCoroutineExact(op) (PyGen_CheckExact(op) && \
-                                       (((PyCodeObject*) \
-                                           ((PyGenObject*)op)->gi_code) \
-                                         ->co_flags & (CO_ITERABLE_COROUTINE | \
-                                                       CO_COROUTINE)))
-
 PyAPI_FUNC(PyObject *) PyGen_New(struct _frame *);
 PyAPI_FUNC(PyObject *) PyGen_NewWithQualName(struct _frame *,
     PyObject *name, PyObject *qualname);
@@ -52,7 +45,21 @@ PyAPI_FUNC(int) _PyGen_FetchStopIterationValue(PyObject **);
 PyObject *_PyGen_Send(PyGenObject *, PyObject *);
 PyAPI_FUNC(void) _PyGen_Finalize(PyObject *self);
 
-PyObject *_PyGen_GetAwaitableIter(PyObject *o);
+#ifndef Py_LIMITED_API
+typedef struct {
+    _PyGenObject_HEAD(cr)
+} PyCoroObject;
+
+PyAPI_DATA(PyTypeObject) PyCoro_Type;
+PyAPI_DATA(PyTypeObject) _PyCoroWrapper_Type;
+
+#define PyCoro_CheckExact(op) (Py_TYPE(op) == &PyCoro_Type)
+PyObject *_PyCoro_GetAwaitableIter(PyObject *o);
+PyAPI_FUNC(PyObject *) PyCoro_New(struct _frame *,
+    PyObject *name, PyObject *qualname);
+#endif
+
+#undef _PyGenObject_HEAD
 
 #ifdef __cplusplus
 }

Include/opcode.h

@@ -45,6 +45,7 @@ extern "C" {
 #define BINARY_OR                66
 #define INPLACE_POWER            67
 #define GET_ITER                 68
+#define GET_YIELD_FROM_ITER      69
 #define PRINT_EXPR               70
 #define LOAD_BUILD_CLASS         71
 #define YIELD_FROM               72

Lib/_collections_abc.py

@@ -52,6 +52,12 @@ dict_items = type({}.items())
 ## misc ##
 mappingproxy = type(type.__dict__)
 generator = type((lambda: (yield))())
+## coroutine ##
+async def _coro(): pass
+_coro = _coro()
+coroutine = type(_coro)
+_coro.close()  # Prevent ResourceWarning
+del _coro
 
 
 ### ONE-TRICK PONIES ###
@@ -78,17 +84,15 @@ class Hashable(metaclass=ABCMeta):
 class _AwaitableMeta(ABCMeta):
 
     def __instancecheck__(cls, instance):
-        # 0x80 = CO_COROUTINE
-        # 0x100 = CO_ITERABLE_COROUTINE
-        # We don't want to import 'inspect' module, as
-        # a dependency for 'collections.abc'.
-        CO_COROUTINES = 0x80 | 0x100
-
-        if (isinstance(instance, generator) and
-            instance.gi_code.co_flags & CO_COROUTINES):
-
+        # This hook is needed because we can't add
+        # '__await__' method to generator objects, and
+        # we can't register GeneratorType on Awaitable.
+        # NB: 0x100 = CO_ITERABLE_COROUTINE
+        # (We don't want to import 'inspect' module, as
+        # a dependency for 'collections.abc')
+        if (instance.__class__ is generator and
+            instance.gi_code.co_flags & 0x100):
             return True
-
         return super().__instancecheck__(instance)
 
 
@@ -159,6 +163,9 @@ class Coroutine(Awaitable):
         return NotImplemented
 
 
+Coroutine.register(coroutine)
+
+
 class AsyncIterable(metaclass=ABCMeta):
 
     __slots__ = ()

Lib/asyncio/coroutines.py

@@ -34,30 +34,20 @@ _DEBUG = (not sys.flags.ignore_environment
 
 
 try:
-    types.coroutine
+    _types_coroutine = types.coroutine
 except AttributeError:
-    native_coroutine_support = False
-else:
-    native_coroutine_support = True
+    _types_coroutine = None
 
 try:
-    _iscoroutinefunction = inspect.iscoroutinefunction
+    _inspect_iscoroutinefunction = inspect.iscoroutinefunction
 except AttributeError:
-    _iscoroutinefunction = lambda func: False
+    _inspect_iscoroutinefunction = lambda func: False
 
 try:
-    inspect.CO_COROUTINE
-except AttributeError:
-    _is_native_coro_code = lambda code: False
-else:
-    _is_native_coro_code = lambda code: (code.co_flags &
-                                         inspect.CO_COROUTINE)
-
-try:
-    from collections.abc import Coroutine as CoroutineABC, \
-                                Awaitable as AwaitableABC
+    from collections.abc import Coroutine as _CoroutineABC, \
+                                Awaitable as _AwaitableABC
 except ImportError:
-    CoroutineABC = AwaitableABC = None
+    _CoroutineABC = _AwaitableABC = None
 
 
 # Check for CPython issue #21209
@@ -89,10 +79,7 @@ def debug_wrapper(gen):
     # We only wrap here coroutines defined via 'async def' syntax.
     # Generator-based coroutines are wrapped in @coroutine
     # decorator.
-    if _is_native_coro_code(gen.gi_code):
-        return CoroWrapper(gen, None)
-    else:
-        return gen
+    return CoroWrapper(gen, None)
 
 
 class CoroWrapper:
@@ -177,8 +164,7 @@ def coroutine(func):
     If the coroutine is not yielded from before it is destroyed,
     an error message is logged.
     """
-    is_coroutine = _iscoroutinefunction(func)
-    if is_coroutine and _is_native_coro_code(func.__code__):
+    if _inspect_iscoroutinefunction(func):
         # In Python 3.5 that's all we need to do for coroutines
         # defiend with "async def".
         # Wrapping in CoroWrapper will happen via
@@ -193,7 +179,7 @@ def coroutine(func):
             res = func(*args, **kw)
             if isinstance(res, futures.Future) or inspect.isgenerator(res):
                 res = yield from res
-            elif AwaitableABC is not None:
+            elif _AwaitableABC is not None:
                 # If 'func' returns an Awaitable (new in 3.5) we
                 # want to run it.
                 try:
@@ -201,15 +187,15 @@ def coroutine(func):
                 except AttributeError:
                     pass
                 else:
-                    if isinstance(res, AwaitableABC):
+                    if isinstance(res, _AwaitableABC):
                         res = yield from await_meth()
             return res
 
     if not _DEBUG:
-        if native_coroutine_support:
-            wrapper = types.coroutine(coro)
-        else:
+        if _types_coroutine is None:
             wrapper = coro
+        else:
+            wrapper = _types_coroutine(coro)
     else:
         @functools.wraps(func)
         def wrapper(*args, **kwds):
@@ -231,12 +217,12 @@ def coroutine(func):
 def iscoroutinefunction(func):
     """Return True if func is a decorated coroutine function."""
     return (getattr(func, '_is_coroutine', False) or
-            _iscoroutinefunction(func))
+            _inspect_iscoroutinefunction(func))
 
 
 _COROUTINE_TYPES = (types.GeneratorType, CoroWrapper)
-if CoroutineABC is not None:
-    _COROUTINE_TYPES += (CoroutineABC,)
+if _CoroutineABC is not None:
+    _COROUTINE_TYPES += (_CoroutineABC,)
 
 
 def iscoroutine(obj):

Lib/importlib/_bootstrap_external.py

@@ -222,12 +222,13 @@ _code_type = type(_write_atomic.__code__)
 #     Python 3.5a0  3320 (matrix multiplication operator)
 #     Python 3.5b1  3330 (PEP 448: Additional Unpacking Generalizations)
 #     Python 3.5b2  3340 (fix dictionary display evaluation order #11205)
+#     Python 3.5b2  3350 (add GET_YIELD_FROM_ITER opcode #24400)
 #
 # MAGIC must change whenever the bytecode emitted by the compiler may no
 # longer be understood by older implementations of the eval loop (usually
 # due to the addition of new opcodes).
 
-MAGIC_NUMBER = (3340).to_bytes(2, 'little') + b'\r\n'
+MAGIC_NUMBER = (3350).to_bytes(2, 'little') + b'\r\n'
 _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little')  # For import.c
 
 _PYCACHE = '__pycache__'

Lib/inspect.py

@@ -175,8 +175,7 @@ def isgeneratorfunction(object):
 
     See help(isfunction) for attributes listing."""
     return bool((isfunction(object) or ismethod(object)) and
-                object.__code__.co_flags & CO_GENERATOR and
-                not object.__code__.co_flags & CO_COROUTINE)
+                object.__code__.co_flags & CO_GENERATOR)
 
 def iscoroutinefunction(object):
     """Return true if the object is a coroutine function.
@@ -185,8 +184,7 @@ def iscoroutinefunction(object):
     or generators decorated with "types.coroutine".
     """
     return bool((isfunction(object) or ismethod(object)) and
-                object.__code__.co_flags & (CO_ITERABLE_COROUTINE |
-                                            CO_COROUTINE))
+                object.__code__.co_flags & CO_COROUTINE)
 
 def isawaitable(object):
     """Return true if the object can be used in "await" expression."""
@@ -207,12 +205,11 @@ def isgenerator(object):
         send            resumes the generator and "sends" a value that becomes
                         the result of the current yield-expression
         throw           used to raise an exception inside the generator"""
-    return (isinstance(object, types.GeneratorType) and
-            not object.gi_code.co_flags & CO_COROUTINE)
+    return isinstance(object, types.GeneratorType)
 
 def iscoroutine(object):
     """Return true if the object is a coroutine."""
-    return isinstance(object, collections.abc.Coroutine)
+    return isinstance(object, types.CoroutineType)
 
 def istraceback(object):
     """Return true if the object is a traceback.
@@ -1598,6 +1595,45 @@ def getgeneratorlocals(generator):
     else:
         return {}
 
+
+# ------------------------------------------------ coroutine introspection
+
+CORO_CREATED = 'CORO_CREATED'
+CORO_RUNNING = 'CORO_RUNNING'
+CORO_SUSPENDED = 'CORO_SUSPENDED'
+CORO_CLOSED = 'CORO_CLOSED'
+
+def getcoroutinestate(coroutine):
+    """Get current state of a coroutine object.
+
+    Possible states are:
+      CORO_CREATED: Waiting to start execution.
+      CORO_RUNNING: Currently being executed by the interpreter.
+      CORO_SUSPENDED: Currently suspended at an await expression.
+      CORO_CLOSED: Execution has completed.
+    """
+    if coroutine.cr_running:
+        return CORO_RUNNING
+    if coroutine.cr_frame is None:
+        return CORO_CLOSED
+    if coroutine.cr_frame.f_lasti == -1:
+        return CORO_CREATED
+    return CORO_SUSPENDED
+
+
+def getcoroutinelocals(coroutine):
+    """
+    Get the mapping of coroutine local variables to their current values.
+
+    A dict is returned, with the keys the local variable names and values the
+    bound values."""
+    frame = getattr(coroutine, "cr_frame", None)
+    if frame is not None:
+        return frame.f_locals
+    else:
+        return {}
+
+
 ###############################################################################
 ### Function Signature Object (PEP 362)
 ###############################################################################

Lib/opcode.py

@@ -103,6 +103,7 @@ def_op('BINARY_XOR', 65)
 def_op('BINARY_OR', 66)
 def_op('INPLACE_POWER', 67)
 def_op('GET_ITER', 68)
+def_op('GET_YIELD_FROM_ITER', 69)
 
 def_op('PRINT_EXPR', 70)
 def_op('LOAD_BUILD_CLASS', 71)

Lib/test/test_coroutines.py

@@ -24,7 +24,7 @@ class AsyncYield:
 
 
 def run_async(coro):
-    assert coro.__class__ is types.GeneratorType
+    assert coro.__class__ in {types.GeneratorType, types.CoroutineType}
 
     buffer = []
     result = None
@@ -37,6 +37,25 @@ def run_async(coro):
     return buffer, result
 
 
+def run_async__await__(coro):
+    assert coro.__class__ is types.CoroutineType
+    aw = coro.__await__()
+    buffer = []
+    result = None
+    i = 0
+    while True:
+        try:
+            if i % 2:
+                buffer.append(next(aw))
+            else:
+                buffer.append(aw.send(None))
+            i += 1
+        except StopIteration as ex:
+            result = ex.args[0] if ex.args else None
+            break
+    return buffer, result
+
+
 @contextlib.contextmanager
 def silence_coro_gc():
     with warnings.catch_warnings():
@@ -121,22 +140,24 @@ class CoroutineTest(unittest.TestCase):
             return 10
 
         f = foo()
-        self.assertIsInstance(f, types.GeneratorType)
-        self.assertTrue(bool(foo.__code__.co_flags & 0x80))
-        self.assertTrue(bool(foo.__code__.co_flags & 0x20))
-        self.assertTrue(bool(f.gi_code.co_flags & 0x80))
-        self.assertTrue(bool(f.gi_code.co_flags & 0x20))
+        self.assertIsInstance(f, types.CoroutineType)
+        self.assertTrue(bool(foo.__code__.co_flags & inspect.CO_COROUTINE))
+        self.assertFalse(bool(foo.__code__.co_flags & inspect.CO_GENERATOR))
+        self.assertTrue(bool(f.cr_code.co_flags & inspect.CO_COROUTINE))
+        self.assertFalse(bool(f.cr_code.co_flags & inspect.CO_GENERATOR))
         self.assertEqual(run_async(f), ([], 10))
 
+        self.assertEqual(run_async__await__(foo()), ([], 10))
+
         def bar(): pass
-        self.assertFalse(bool(bar.__code__.co_flags & 0x80))
+        self.assertFalse(bool(bar.__code__.co_flags & inspect.CO_COROUTINE))
 
     def test_func_2(self):
         async def foo():
             raise StopIteration
 
         with self.assertRaisesRegex(
-                RuntimeError, "generator raised StopIteration"):
+                RuntimeError, "coroutine raised StopIteration"):
 
             run_async(foo())
 
@@ -152,7 +173,7 @@ class CoroutineTest(unittest.TestCase):
             raise StopIteration
 
         check = lambda: self.assertRaisesRegex(
-            TypeError, "coroutine-objects do not support iteration")
+            TypeError, "'coroutine' object is not iterable")
 
         with check():
             list(foo())
@@ -166,9 +187,6 @@ class CoroutineTest(unittest.TestCase):
         with check():
             iter(foo())
 
-        with check():
-            next(foo())
-
         with silence_coro_gc(), check():
             for i in foo():
                 pass
@@ -185,7 +203,7 @@ class CoroutineTest(unittest.TestCase):
             await bar()
 
         check = lambda: self.assertRaisesRegex(
-            TypeError, "coroutine-objects do not support iteration")
+            TypeError, "'coroutine' object is not iterable")
 
         with check():
             for el in foo(): pass
@@ -221,7 +239,7 @@ class CoroutineTest(unittest.TestCase):
 
         with silence_coro_gc(), self.assertRaisesRegex(
             TypeError,
-            "cannot 'yield from' a coroutine object from a generator"):
+            "cannot 'yield from' a coroutine object in a non-coroutine generator"):
 
             list(foo())
 
@@ -244,6 +262,98 @@ class CoroutineTest(unittest.TestCase):
             foo()
             support.gc_collect()
 
+    def test_func_10(self):
+        N = 0
+
+        @types.coroutine
+        def gen():
+            nonlocal N
+            try:
+                a = yield
+                yield (a ** 2)
+            except ZeroDivisionError:
+                N += 100
+                raise
+            finally:
+                N += 1
+
+        async def foo():
+            await gen()
+
+        coro = foo()
+        aw = coro.__await__()
+        self.assertIs(aw, iter(aw))
+        next(aw)
+        self.assertEqual(aw.send(10), 100)
+
+        self.assertEqual(N, 0)
+        aw.close()
+        self.assertEqual(N, 1)
+
+        coro = foo()
+        aw = coro.__await__()
+        next(aw)
+        with self.assertRaises(ZeroDivisionError):
+            aw.throw(ZeroDivisionError, None, None)
+        self.assertEqual(N, 102)
+
+    def test_func_11(self):
+        async def func(): pass
+        coro = func()
+        # Test that PyCoro_Type and _PyCoroWrapper_Type types were properly
+        # initialized
+        self.assertIn('__await__', dir(coro))
+        self.assertIn('__iter__', dir(coro.__await__()))
+        self.assertIn('coroutine_wrapper', repr(coro.__await__()))
+        coro.close() # avoid RuntimeWarning
+
+    def test_func_12(self):
+        async def g():
+            i = me.send(None)
+            await foo
+        me = g()
+        with self.assertRaisesRegex(ValueError,
+                                    "coroutine already executing"):
+            me.send(None)
+
+    def test_func_13(self):
+        async def g():
+            pass
+        with self.assertRaisesRegex(
+            TypeError,
+            "can't send non-None value to a just-started coroutine"):
+
+            g().send('spam')
+
+    def test_func_14(self):
+        @types.coroutine
+        def gen():
+            yield
+        async def coro():
+            try:
+                await gen()
+            except GeneratorExit:
+                await gen()
+        c = coro()
+        c.send(None)
+        with self.assertRaisesRegex(RuntimeError,
+                                    "coroutine ignored GeneratorExit"):
+            c.close()
+
+    def test_corotype_1(self):
+        ct = types.CoroutineType
+        self.assertIn('into coroutine', ct.send.__doc__)
+        self.assertIn('inside coroutine', ct.close.__doc__)
+        self.assertIn('in coroutine', ct.throw.__doc__)
+        self.assertIn('of the coroutine', ct.__dict__['__name__'].__doc__)
+        self.assertIn('of the coroutine', ct.__dict__['__qualname__'].__doc__)
+        self.assertEqual(ct.__name__, 'coroutine')
+
+        async def f(): pass
+        c = f()
+        self.assertIn('coroutine object', repr(c))
+        c.close()
+
     def test_await_1(self):
 
         async def foo():
@@ -262,6 +372,7 @@ class CoroutineTest(unittest.TestCase):
             await AsyncYieldFrom([1, 2, 3])
 
         self.assertEqual(run_async(foo()), ([1, 2, 3], None))
+        self.assertEqual(run_async__await__(foo()), ([1, 2, 3], None))
 
     def test_await_4(self):
         async def bar():
@@ -1015,6 +1126,27 @@ class SysSetCoroWrapperTest(unittest.TestCase):
         finally:
             sys.set_coroutine_wrapper(None)
 
+    def test_set_wrapper_4(self):
+        @types.coroutine
+        def foo():
+            return 'spam'
+
+        wrapped = None
+        def wrap(gen):
+            nonlocal wrapped
+            wrapped = gen
+            return gen
+
+        sys.set_coroutine_wrapper(wrap)
+        try:
+            foo()
+            self.assertIs(
+                wrapped, None,
+                "generator-based coroutine was wrapped via "
+                "sys.set_coroutine_wrapper")
+        finally:
+            sys.set_coroutine_wrapper(None)
+
 
 class CAPITest(unittest.TestCase):
 

Lib/test/test_inspect.py

@@ -141,9 +141,9 @@ class TestPredicates(IsTestBase):
         gen_coro = gen_coroutine_function_example(1)
         coro = coroutine_function_example(1)
 
-        self.assertTrue(
+        self.assertFalse(
             inspect.iscoroutinefunction(gen_coroutine_function_example))
-        self.assertTrue(inspect.iscoroutine(gen_coro))
+        self.assertFalse(inspect.iscoroutine(gen_coro))
 
         self.assertTrue(
             inspect.isgeneratorfunction(gen_coroutine_function_example))
@@ -1737,6 +1737,70 @@ class TestGetGeneratorState(unittest.TestCase):
         self.assertRaises(TypeError, inspect.getgeneratorlocals, (2,3))
 
 
+class TestGetCoroutineState(unittest.TestCase):
+
+    def setUp(self):
+        @types.coroutine
+        def number_coroutine():
+            for number in range(5):
+                yield number
+        async def coroutine():
+            await number_coroutine()
+        self.coroutine = coroutine()
+
+    def tearDown(self):
+        self.coroutine.close()
+
+    def _coroutinestate(self):
+        return inspect.getcoroutinestate(self.coroutine)
+
+    def test_created(self):
+        self.assertEqual(self._coroutinestate(), inspect.CORO_CREATED)
+
+    def test_suspended(self):
+        self.coroutine.send(None)
+        self.assertEqual(self._coroutinestate(), inspect.CORO_SUSPENDED)
+
+    def test_closed_after_exhaustion(self):
+        while True:
+            try:
+                self.coroutine.send(None)
+            except StopIteration:
+                break
+
+        self.assertEqual(self._coroutinestate(), inspect.CORO_CLOSED)
+
+    def test_closed_after_immediate_exception(self):
+        with self.assertRaises(RuntimeError):
+            self.coroutine.throw(RuntimeError)
+        self.assertEqual(self._coroutinestate(), inspect.CORO_CLOSED)
+
+    def test_easy_debugging(self):
+        # repr() and str() of a coroutine state should contain the state name
+        names = 'CORO_CREATED CORO_RUNNING CORO_SUSPENDED CORO_CLOSED'.split()
+        for name in names:
+            state = getattr(inspect, name)
+            self.assertIn(name, repr(state))
+            self.assertIn(name, str(state))
+
+    def test_getcoroutinelocals(self):
+        @types.coroutine
+        def gencoro():
+            yield
+
+        gencoro = gencoro()
+        async def func(a=None):
+            b = 'spam'
+            await gencoro
+
+        coro = func()
+        self.assertEqual(inspect.getcoroutinelocals(coro),
+                         {'a': None, 'gencoro': gencoro})
+        coro.send(None)
+        self.assertEqual(inspect.getcoroutinelocals(coro),
+                         {'a': None, 'gencoro': gencoro, 'b': 'spam'})
+
+
 class MySignature(inspect.Signature):
     # Top-level to make it picklable;
     # used in test_signature_object_pickle
@@ -3494,7 +3558,8 @@ def test_main():
         TestNoEOL, TestSignatureObject, TestSignatureBind, TestParameterObject,
         TestBoundArguments, TestSignaturePrivateHelpers,
         TestSignatureDefinitions,
-        TestGetClosureVars, TestUnwrap, TestMain, TestReload
+        TestGetClosureVars, TestUnwrap, TestMain, TestReload,
+        TestGetCoroutineState
     )
 
 if __name__ == "__main__":

Lib/test/test_types.py

@@ -1205,10 +1205,28 @@ class CoroutineTests(unittest.TestCase):
     def test_wrong_func(self):
         @types.coroutine
         def foo():
-            pass
-        with self.assertRaisesRegex(TypeError,
-                                    'callable wrapped .* non-coroutine'):
-            foo()
+            return 'spam'
+        self.assertEqual(foo(), 'spam')
+
+    def test_async_def(self):
+        # Test that types.coroutine passes 'async def' coroutines
+        # without modification
+
+        async def foo(): pass
+        foo_code = foo.__code__
+        foo_flags = foo.__code__.co_flags
+        decorated_foo = types.coroutine(foo)
+        self.assertIs(foo, decorated_foo)
+        self.assertEqual(foo.__code__.co_flags, foo_flags)
+        self.assertIs(decorated_foo.__code__, foo_code)
+
+        foo_coro = foo()
+        @types.coroutine
+        def bar(): return foo_coro
+        coro = bar()
+        self.assertIs(foo_coro, coro)
+        self.assertEqual(coro.cr_code.co_flags, foo_flags)
+        coro.close()
 
     def test_duck_coro(self):
         class CoroLike:
@@ -1221,6 +1239,23 @@ class CoroutineTests(unittest.TestCase):
         @types.coroutine
         def foo():
             return coro
+        self.assertIs(foo(), coro)
+        self.assertIs(foo().__await__(), coro)
+
+    def test_duck_corogen(self):
+        class CoroGenLike:
+            def send(self): pass
+            def throw(self): pass
+            def close(self): pass
+            def __await__(self): return self
+            def __iter__(self): return self
+            def __next__(self): pass
+
+        coro = CoroGenLike()
+        @types.coroutine
+        def foo():
+            return coro
+        self.assertIs(foo(), coro)
         self.assertIs(foo().__await__(), coro)
 
     def test_duck_gen(self):
@@ -1236,7 +1271,7 @@ class CoroutineTests(unittest.TestCase):
         def foo():
             return gen
         self.assertIs(foo().__await__(), gen)
-
+        self.assertTrue(isinstance(foo(), collections.abc.Coroutine))
         with self.assertRaises(AttributeError):
             foo().gi_code
 
@@ -1251,6 +1286,7 @@ class CoroutineTests(unittest.TestCase):
                      'gi_running', 'gi_frame'):
             self.assertIs(getattr(foo(), name),
                           getattr(gen, name))
+        self.assertIs(foo().cr_code, gen.gi_code)
 
     def test_genfunc(self):
         def gen():
@@ -1259,7 +1295,13 @@ class CoroutineTests(unittest.TestCase):
         self.assertFalse(isinstance(gen(), collections.abc.Coroutine))
         self.assertFalse(isinstance(gen(), collections.abc.Awaitable))
 
-        self.assertIs(types.coroutine(gen), gen)
+        gen_code = gen.__code__
+        decorated_gen = types.coroutine(gen)
+        self.assertIs(decorated_gen, gen)
+        self.assertIsNot(decorated_gen.__code__, gen_code)
+
+        decorated_gen2 = types.coroutine(decorated_gen)
+        self.assertIs(decorated_gen2.__code__, decorated_gen.__code__)
 
         self.assertTrue(gen.__code__.co_flags & inspect.CO_ITERABLE_COROUTINE)
         self.assertFalse(gen.__code__.co_flags & inspect.CO_COROUTINE)

Lib/types.py

@@ -19,6 +19,11 @@ def _g():
     yield 1
 GeneratorType = type(_g())
 
+async def _c(): pass
+_c = _c()
+CoroutineType = type(_c)
+_c.close()  # Prevent ResourceWarning
+
 class _C:
     def _m(self): pass
 MethodType = type(_C()._m)
@@ -40,7 +45,7 @@ except TypeError:
 GetSetDescriptorType = type(FunctionType.__code__)
 MemberDescriptorType = type(FunctionType.__globals__)
 
-del sys, _f, _g, _C,                              # Not for export
+del sys, _f, _g, _C, _c,                           # Not for export
 
 
 # Provide a PEP 3115 compliant mechanism for class creation
@@ -164,29 +169,33 @@ import collections.abc as _collections_abc
 def coroutine(func):
     """Convert regular generator function to a coroutine."""
 
-    # We don't want to import 'dis' or 'inspect' just for
-    # these constants.
-    CO_GENERATOR = 0x20
-    CO_ITERABLE_COROUTINE = 0x100
-
     if not callable(func):
         raise TypeError('types.coroutine() expects a callable')
 
-    if (isinstance(func, FunctionType) and
-        isinstance(getattr(func, '__code__', None), CodeType) and
-        (func.__code__.co_flags & CO_GENERATOR)):
-
-        # TODO: Implement this in C.
-        co = func.__code__
-        func.__code__ = CodeType(
-            co.co_argcount, co.co_kwonlyargcount, co.co_nlocals,
-            co.co_stacksize,
-            co.co_flags | CO_ITERABLE_COROUTINE,
-            co.co_code,
-            co.co_consts, co.co_names, co.co_varnames, co.co_filename,
-            co.co_name, co.co_firstlineno, co.co_lnotab, co.co_freevars,
-            co.co_cellvars)
-        return func
+    if (func.__class__ is FunctionType and
+        getattr(func, '__code__', None).__class__ is CodeType):
+
+        co_flags = func.__code__.co_flags
+
+        # Check if 'func' is a coroutine function.
+        # (0x180 == CO_COROUTINE | CO_ITERABLE_COROUTINE)
+        if co_flags & 0x180:
+            return func
+
+        # Check if 'func' is a generator function.
+        # (0x20 == CO_GENERATOR)
+        if co_flags & 0x20:
+            # TODO: Implement this in C.
+            co = func.__code__
+            func.__code__ = CodeType(
+                co.co_argcount, co.co_kwonlyargcount, co.co_nlocals,
+                co.co_stacksize,
+                co.co_flags | 0x100,  # 0x100 == CO_ITERABLE_COROUTINE
+                co.co_code,
+                co.co_consts, co.co_names, co.co_varnames, co.co_filename,
+                co.co_name, co.co_firstlineno, co.co_lnotab, co.co_freevars,
+                co.co_cellvars)
+            return func
 
     # The following code is primarily to support functions that
     # return generator-like objects (for instance generators
@@ -195,11 +204,14 @@ def coroutine(func):
     class GeneratorWrapper:
         def __init__(self, gen):
             self.__wrapped__ = gen
-            self.send = gen.send
-            self.throw = gen.throw
-            self.close = gen.close
             self.__name__ = getattr(gen, '__name__', None)
             self.__qualname__ = getattr(gen, '__qualname__', None)
+        def send(self, val):
+            return self.__wrapped__.send(val)
+        def throw(self, *args):
+            return self.__wrapped__.throw(*args)
+        def close(self):
+            return self.__wrapped__.close()
         @property
         def gi_code(self):
             return self.__wrapped__.gi_code
@@ -209,24 +221,31 @@ def coroutine(func):
         @property
         def gi_running(self):
             return self.__wrapped__.gi_running
+        cr_code = gi_code
+        cr_frame = gi_frame
+        cr_running = gi_running
         def __next__(self):
             return next(self.__wrapped__)
         def __iter__(self):
             return self.__wrapped__
-        __await__ = __iter__
+        def __await__(self):
+            return self.__wrapped__
 
     @_functools.wraps(func)
     def wrapped(*args, **kwargs):
         coro = func(*args, **kwargs)
-        if coro.__class__ is GeneratorType:
+        if coro.__class__ is CoroutineType:
+            # 'coro' is a native coroutine object.
+            return coro
+        if (coro.__class__ is GeneratorType or
+                (isinstance(coro, _collections_abc.Generator) and
+                 not isinstance(coro, _collections_abc.Coroutine))):
+            # 'coro' is either a pure Python generator iterator, or it
+            # implements collections.abc.Generator (and does not implement
+            # collections.abc.Coroutine).
             return GeneratorWrapper(coro)
-        # slow checks
-        if not isinstance(coro, _collections_abc.Coroutine):
-            if isinstance(coro, _collections_abc.Generator):
-                return GeneratorWrapper(coro)
-            raise TypeError(
-                'callable wrapped with types.coroutine() returned '
-                'non-coroutine: {!r}'.format(coro))
+        # 'coro' is either an instance of collections.abc.Coroutine or
+        # some other object -- pass it through.
         return coro
 
     return wrapped

Misc/NEWS

@@ -12,6 +12,15 @@ Core and Builtins
 
 - Issue #24345: Add Py_tp_finalize slot for the stable ABI.
 
+- Issue #24400: Introduce a distinct type for PEP 492 coroutines; add
+  types.CoroutineType, inspect.getcoroutinestate, inspect.getcoroutinelocals;
+  coroutines no longer use CO_GENERATOR flag; sys.set_coroutine_wrapper
+  works only for 'async def' coroutines; inspect.iscoroutine no longer
+  uses collections.abc.Coroutine, it's intended to test for pure 'async def'
+  coroutines only; add new opcode: GET_YIELD_FROM_ITER; fix generators wrapper
+  used in types.coroutine to be instance of collections.abc.Generator.
+
+
 Library
 -------
 

Objects/object.c

@@ -1726,6 +1726,12 @@ _Py_ReadyTypes(void)
 
     if (PyType_Ready(&PySeqIter_Type) < 0)
         Py_FatalError("Can't initialize sequence iterator type");
+
+    if (PyType_Ready(&PyCoro_Type) < 0)
+        Py_FatalError("Can't initialize coroutine type");
+
+    if (PyType_Ready(&_PyCoroWrapper_Type) < 0)
+        Py_FatalError("Can't initialize coroutine wrapper type");
 }
 
 

Python/ceval.c

@@ -1191,7 +1191,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
     f->f_stacktop = NULL;       /* remains NULL unless yield suspends frame */
     f->f_executing = 1;
 
-    if (co->co_flags & CO_GENERATOR) {
+    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE)) {
         if (!throwflag && f->f_exc_type != NULL && f->f_exc_type != Py_None) {
             /* We were in an except handler when we left,
                restore the exception state which was put aside
@@ -1955,7 +1955,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
                 goto error;
             }
 
-            awaitable = _PyGen_GetAwaitableIter(iter);
+            awaitable = _PyCoro_GetAwaitableIter(iter);
             if (awaitable == NULL) {
                 SET_TOP(NULL);
                 PyErr_Format(
@@ -1998,7 +1998,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
                 goto error;
             }
 
-            awaitable = _PyGen_GetAwaitableIter(next_iter);
+            awaitable = _PyCoro_GetAwaitableIter(next_iter);
             if (awaitable == NULL) {
                 PyErr_Format(
                     PyExc_TypeError,
@@ -2017,7 +2017,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
 
         TARGET(GET_AWAITABLE) {
             PyObject *iterable = TOP();
-            PyObject *iter = _PyGen_GetAwaitableIter(iterable);
+            PyObject *iter = _PyCoro_GetAwaitableIter(iterable);
 
             Py_DECREF(iterable);
 
@@ -2034,25 +2034,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
             PyObject *v = POP();
             PyObject *reciever = TOP();
             int err;
-            if (PyGen_CheckExact(reciever)) {
-                if (
-                    (((PyCodeObject*) \
-                        ((PyGenObject*)reciever)->gi_code)->co_flags &
-                                                        CO_COROUTINE)
-                    && !(co->co_flags & (CO_COROUTINE | CO_ITERABLE_COROUTINE)))
-                {
-                    /* If we're yielding-from a coroutine object from a regular
-                       generator object - raise an error. */
-
-                    Py_CLEAR(v);
-                    Py_CLEAR(reciever);
-                    SET_TOP(NULL);
-
-                    PyErr_SetString(PyExc_TypeError,
-                                    "cannot 'yield from' a coroutine object "
-                                    "from a generator");
-                    goto error;
-                }
+            if (PyGen_CheckExact(reciever) || PyCoro_CheckExact(reciever)) {
                 retval = _PyGen_Send((PyGenObject *)reciever, v);
             } else {
                 _Py_IDENTIFIER(send);
@@ -2929,19 +2911,33 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
         TARGET(GET_ITER) {
             /* before: [obj]; after [getiter(obj)] */
             PyObject *iterable = TOP();
-            PyObject *iter;
-            /* If we have a generator object on top -- keep it there,
-               it's already an iterator.
-
-               This is needed to allow use of 'async def' coroutines
-               in 'yield from' expression from generator-based coroutines
-               (decorated with types.coroutine()).
+            PyObject *iter = PyObject_GetIter(iterable);
+            Py_DECREF(iterable);
+            SET_TOP(iter);
+            if (iter == NULL)
+                goto error;
+            PREDICT(FOR_ITER);
+            DISPATCH();
+        }
 
-               'yield from' is compiled to GET_ITER..YIELD_FROM combination,
-               but since coroutines raise TypeError in their 'tp_iter' we
-               need a way for them to "pass through" the GET_ITER.
-            */
-            if (!PyGen_CheckExact(iterable)) {
+        TARGET(GET_YIELD_FROM_ITER) {
+            /* before: [obj]; after [getiter(obj)] */
+            PyObject *iterable = TOP();
+            PyObject *iter;
+            if (PyCoro_CheckExact(iterable)) {
+                /* `iterable` is a coroutine */
+                if (!(co->co_flags & (CO_COROUTINE | CO_ITERABLE_COROUTINE))) {
+                    /* and it is used in a 'yield from' expression of a
+                       regular generator. */
+                    Py_DECREF(iterable);
+                    SET_TOP(NULL);
+                    PyErr_SetString(PyExc_TypeError,
+                                    "cannot 'yield from' a coroutine object "
+                                    "in a non-coroutine generator");
+                    goto error;
+                }
+            }
+            else if (!PyGen_CheckExact(iterable)) {
                 /* `iterable` is not a generator. */
                 iter = PyObject_GetIter(iterable);
                 Py_DECREF(iterable);
@@ -2949,7 +2945,6 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag)
                 if (iter == NULL)
                     goto error;
             }
-            PREDICT(FOR_ITER);
             DISPATCH();
         }
 
@@ -3517,7 +3512,7 @@ fast_block_end:
     assert((retval != NULL) ^ (PyErr_Occurred() != NULL));
 
 fast_yield:
-    if (co->co_flags & CO_GENERATOR) {
+    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE)) {
 
         /* The purpose of this block is to put aside the generator's exception
            state and restore that of the calling frame. If the current
@@ -3919,10 +3914,10 @@ _PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
         freevars[PyTuple_GET_SIZE(co->co_cellvars) + i] = o;
     }
 
-    if (co->co_flags & CO_GENERATOR) {
+    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE)) {
         PyObject *gen;
         PyObject *coro_wrapper = tstate->coroutine_wrapper;
-        int is_coro = co->co_flags & (CO_COROUTINE | CO_ITERABLE_COROUTINE);
+        int is_coro = co->co_flags & CO_COROUTINE;
 
         if (is_coro && tstate->in_coroutine_wrapper) {
             assert(coro_wrapper != NULL);
@@ -3942,7 +3937,11 @@ _PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
 
         /* Create a new generator that owns the ready to run frame
          * and return that as the value. */
-        gen = PyGen_NewWithQualName(f, name, qualname);
+        if (is_coro) {
+            gen = PyCoro_New(f, name, qualname);
+        } else {
+            gen = PyGen_NewWithQualName(f, name, qualname);
+        }
         if (gen == NULL)
             return NULL;
 

Python/compile.c

@@ -1064,6 +1064,8 @@ PyCompile_OpcodeStackEffect(int opcode, int oparg)
             return 0;
         case GET_ANEXT:
             return 1;
+        case GET_YIELD_FROM_ITER:
+            return 0;
         default:
             return PY_INVALID_STACK_EFFECT;
     }
@@ -1751,12 +1753,8 @@ compiler_function(struct compiler *c, stmt_ty s, int is_async)
     Py_DECREF(qualname);
     Py_DECREF(co);
 
-    if (is_async) {
+    if (is_async)
         co->co_flags |= CO_COROUTINE;
-        /* An async function is always a generator, even
-           if there is no 'yield' expressions in it. */
-        co->co_flags |= CO_GENERATOR;
-    }
 
     /* decorators */
     for (i = 0; i < asdl_seq_LEN(decos); i++) {
@@ -3850,7 +3848,7 @@ compiler_visit_expr(struct compiler *c, expr_ty e)
             return compiler_error(c, "'yield from' inside async function");
 
         VISIT(c, expr, e->v.YieldFrom.value);
-        ADDOP(c, GET_ITER);
+        ADDOP(c, GET_YIELD_FROM_ITER);
         ADDOP_O(c, LOAD_CONST, Py_None, consts);
         ADDOP(c, YIELD_FROM);
         break;

Python/opcode_targets.h

@@ -68,7 +68,7 @@ static void *opcode_targets[256] = {
     &&TARGET_BINARY_OR,
     &&TARGET_INPLACE_POWER,
     &&TARGET_GET_ITER,
-    &&_unknown_opcode,
+    &&TARGET_GET_YIELD_FROM_ITER,
     &&TARGET_PRINT_EXPR,
     &&TARGET_LOAD_BUILD_CLASS,
     &&TARGET_YIELD_FROM,