Implement PEP 380 - 'yield from' (closes #11682)

Doc/library/dis.rst

@@ -431,6 +431,13 @@ the stack so that it is available for further iterations of the loop.
    Pops ``TOS`` and yields it from a :term:`generator`.
 
 
+.. opcode:: YIELD_FROM
+
+   Pops ``TOS`` and delegates to it as a subiterator from a :term:`generator`.
+
+   .. versionadded:: 3.3
+
+
 .. opcode:: IMPORT_STAR
 
    Loads all symbols not starting with ``'_'`` directly from the module TOS to the

Doc/library/exceptions.rst

@@ -250,7 +250,16 @@ The following exceptions are the exceptions that are usually raised.
 .. exception:: StopIteration
 
    Raised by built-in function :func:`next` and an :term:`iterator`\'s
-   :meth:`__next__` method to signal that there are no further values.
+   :meth:`__next__` method to signal that there are no further items to be
+   produced by the iterator.
+
+   The exception object has a single attribute :attr:`value`, which is
+   given as an argument when constructing the exception, and defaults
+   to :const:`None`.
+
+   When a generator function returns, a new :exc:`StopIteration` instance is
+   raised, and the value returned by the function is used as the
+   :attr:`value` parameter to the constructor of the exception.
 
 
 .. exception:: SyntaxError

Doc/reference/expressions.rst

@@ -318,7 +318,7 @@ Yield expressions
 
 .. productionlist::
    yield_atom: "(" `yield_expression` ")"
-   yield_expression: "yield" [`expression_list`]
+   yield_expression: "yield" [`expression_list` | "from" `expression`]
 
 The :keyword:`yield` expression is only used when defining a generator function,
 and can only be used in the body of a function definition.  Using a
@@ -336,7 +336,10 @@ the internal evaluation stack.  When the execution is resumed by calling one of
 the generator's methods, the function can proceed exactly as if the
 :keyword:`yield` expression was just another external call.  The value of the
 :keyword:`yield` expression after resuming depends on the method which resumed
-the execution.
+the execution. If :meth:`__next__` is used (typically via either a
+:keyword:`for` or the :func:`next` builtin) then the result is :const:`None`,
+otherwise, if :meth:`send` is used, then the result will be the value passed
+in to that method.
 
 .. index:: single: coroutine
 
@@ -346,12 +349,29 @@ suspended.  The only difference is that a generator function cannot control
 where should the execution continue after it yields; the control is always
 transferred to the generator's caller.
 
-The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a
+:keyword:`yield` expressions are allowed in the :keyword:`try` clause of a
 :keyword:`try` ...  :keyword:`finally` construct.  If the generator is not
 resumed before it is finalized (by reaching a zero reference count or by being
 garbage collected), the generator-iterator's :meth:`close` method will be
 called, allowing any pending :keyword:`finally` clauses to execute.
 
+When ``yield from expression`` is used, it treats the supplied expression as
+a subiterator. All values produced by that subiterator are passed directly
+to the caller of the current generator's methods. Any values passed in with
+:meth:`send` and any exceptions passed in with :meth:`throw` are passed to
+the underlying iterator if it has the appropriate methods. If this is not the
+case, then :meth:`send` will raise :exc:`AttributeError` or :exc:`TypeError`,
+while :meth:`throw` will just raise the passed in exception immediately.
+
+When the underlying iterator is complete, the :attr:`~StopIteration.value`
+attribute of the raised :exc:`StopIteration` instance becomes the value of
+the yield expression. It can be either set explicitly when raising
+:exc:`StopIteration`, or automatically when the sub-iterator is a generator
+(by returning a value from the sub-generator).
+
+The parentheses can be omitted when the :keyword:`yield` expression is the
+sole expression on the right hand side of an assignment statement.
+
 .. index:: object: generator
 
 The following generator's methods can be used to control the execution of a
@@ -444,6 +464,10 @@ generator functions::
       The proposal to enhance the API and syntax of generators, making them
       usable as simple coroutines.
 
+   :pep:`0380` - Syntax for Delegating to a Subgenerator
+      The proposal to introduce the :token:`yield_from` syntax, making delegation
+      to sub-generators easy.
+
 
 .. _primaries:
 

Doc/reference/simple_stmts.rst

@@ -425,10 +425,10 @@ When :keyword:`return` passes control out of a :keyword:`try` statement with a
 :keyword:`finally` clause, that :keyword:`finally` clause is executed before
 really leaving the function.
 
-In a generator function, the :keyword:`return` statement is not allowed to
-include an :token:`expression_list`.  In that context, a bare :keyword:`return`
-indicates that the generator is done and will cause :exc:`StopIteration` to be
-raised.
+In a generator function, the :keyword:`return` statement indicates that the
+generator is done and will cause :exc:`StopIteration` to be raised. The returned
+value (if any) is used as an argument to construct :exc:`StopIteration` and
+becomes the :attr:`StopIteration.value` attribute.
 
 
 .. _yield:
@@ -450,6 +450,7 @@ The :keyword:`yield` statement is only used when defining a generator function,
 and is only used in the body of the generator function. Using a :keyword:`yield`
 statement in a function definition is sufficient to cause that definition to
 create a generator function instead of a normal function.
+
 When a generator function is called, it returns an iterator known as a generator
 iterator, or more commonly, a generator.  The body of the generator function is
 executed by calling the :func:`next` function on the generator repeatedly until
@@ -469,14 +470,25 @@ resumed before it is finalized (by reaching a zero reference count or by being
 garbage collected), the generator-iterator's :meth:`close` method will be
 called, allowing any pending :keyword:`finally` clauses to execute.
 
+When ``yield from expression`` is used, it treats the supplied expression as
+a subiterator, producing values from it until the underlying iterator is
+exhausted.
+
+For full details of :keyword:`yield` semantics, refer to the :ref:`yieldexpr`
+section.
+
 .. seealso::
 
    :pep:`0255` - Simple Generators
       The proposal for adding generators and the :keyword:`yield` statement to Python.
 
    :pep:`0342` - Coroutines via Enhanced Generators
-      The proposal that, among other generator enhancements, proposed allowing
-      :keyword:`yield` to appear inside a :keyword:`try` ... :keyword:`finally` block.
+      The proposal to enhance the API and syntax of generators, making them
+      usable as simple coroutines.
+
+   :pep:`0380` - Syntax for Delegating to a Subgenerator
+      The proposal to introduce the :token:`yield_from` syntax, making delegation
+      to sub-generators easy.
 
 
 .. _raise:

Doc/whatsnew/3.3.rst

@@ -195,6 +195,22 @@ inspection of exception attributes::
         print("You are not allowed to read document.txt")
 
 
+PEP 380: Syntax for Delegating to a Subgenerator
+================================================
+
+PEP 380 adds the ``yield from`` expression, allowing a generator to delegate
+part of its operations to another generator. This allows a section of code
+containing 'yield' to be factored out and placed in another generator.
+Additionally, the subgenerator is allowed to return with a value, and the
+value is made available to the delegating generator.
+While designed primarily for use in delegating to a subgenerator, the ``yield
+from`` expression actually allows delegation to arbitrary subiterators.
+
+(Implementation by Greg Ewing, integrated into 3.3 by Renaud Blanch, Ryan
+Kelly and Nick Coghlan, documentation by Zbigniew Jędrzejewski-Szmek and
+Nick Coghlan)
+
+
 PEP 3155: Qualified name for classes and functions
 ==================================================
 
@@ -208,7 +224,6 @@ it provides better information about where they were actually defined, and
 how they might be accessible from the global scope.
 
 Example with (non-bound) methods::
-
    >>> class C:
    ...     def meth(self):
    ...         pass

Grammar/Grammar

@@ -121,7 +121,7 @@ arglist: (argument ',')* (argument [',']
                          |'**' test)
 # The reason that keywords are test nodes instead of NAME is that using NAME
 # results in an ambiguity. ast.c makes sure it's a NAME.
-argument: test [comp_for] | test '=' test  # Really [keyword '='] test
+argument: (test) [comp_for] | test '=' test  # Really [keyword '='] test
 comp_iter: comp_for | comp_if
 comp_for: 'for' exprlist 'in' or_test [comp_iter]
 comp_if: 'if' test_nocond [comp_iter]
@@ -129,4 +129,5 @@ comp_if: 'if' test_nocond [comp_iter]
 # not used in grammar, but may appear in "node" passed from Parser to Compiler
 encoding_decl: NAME
 
-yield_expr: 'yield' [testlist]
+yield_expr: 'yield' [yield_arg]
+yield_arg: 'from' test | testlist

Include/Python-ast.h

@@ -245,6 +245,7 @@ struct _expr {
                 } GeneratorExp;
                 
                 struct {
+                        int is_from;
                         expr_ty value;
                 } Yield;
                 
@@ -487,8 +488,9 @@ expr_ty _Py_DictComp(expr_ty key, expr_ty value, asdl_seq * generators, int
 #define GeneratorExp(a0, a1, a2, a3, a4) _Py_GeneratorExp(a0, a1, a2, a3, a4)
 expr_ty _Py_GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int
                          col_offset, PyArena *arena);
-#define Yield(a0, a1, a2, a3) _Py_Yield(a0, a1, a2, a3)
-expr_ty _Py_Yield(expr_ty value, int lineno, int col_offset, PyArena *arena);
+#define Yield(a0, a1, a2, a3, a4) _Py_Yield(a0, a1, a2, a3, a4)
+expr_ty _Py_Yield(int is_from, expr_ty value, int lineno, int col_offset,
+                  PyArena *arena);
 #define Compare(a0, a1, a2, a3, a4, a5) _Py_Compare(a0, a1, a2, a3, a4, a5)
 expr_ty _Py_Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators,
                     int lineno, int col_offset, PyArena *arena);

Include/frameobject.h

@@ -9,45 +9,46 @@ extern "C" {
 #endif
 
 typedef struct {
-    int b_type;			/* what kind of block this is */
-    int b_handler;		/* where to jump to find handler */
-    int b_level;		/* value stack level to pop to */
+    int b_type;                 /* what kind of block this is */
+    int b_handler;              /* where to jump to find handler */
+    int b_level;                /* value stack level to pop to */
 } PyTryBlock;
 
 typedef struct _frame {
     PyObject_VAR_HEAD
-    struct _frame *f_back;	/* previous frame, or NULL */
-    PyCodeObject *f_code;	/* code segment */
-    PyObject *f_builtins;	/* builtin symbol table (PyDictObject) */
-    PyObject *f_globals;	/* global symbol table (PyDictObject) */
-    PyObject *f_locals;		/* local symbol table (any mapping) */
-    PyObject **f_valuestack;	/* points after the last local */
+    struct _frame *f_back;      /* previous frame, or NULL */
+    PyCodeObject *f_code;       /* code segment */
+    PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */
+    PyObject *f_globals;        /* global symbol table (PyDictObject) */
+    PyObject *f_locals;         /* local symbol table (any mapping) */
+    PyObject **f_valuestack;    /* points after the last local */
     /* Next free slot in f_valuestack.  Frame creation sets to f_valuestack.
        Frame evaluation usually NULLs it, but a frame that yields sets it
        to the current stack top. */
     PyObject **f_stacktop;
-    PyObject *f_trace;		/* Trace function */
-
-	/* In a generator, we need to be able to swap between the exception
-	   state inside the generator and the exception state of the calling
-	   frame (which shouldn't be impacted when the generator "yields"
-	   from an except handler).
-	   These three fields exist exactly for that, and are unused for
-	   non-generator frames. See the SAVE_EXC_STATE and SWAP_EXC_STATE
-	   macros in ceval.c for details of their use. */
+    PyObject *f_trace;          /* Trace function */
+    PyObject *f_yieldfrom;      /* Iterator being delegated to by yield from */
+
+        /* In a generator, we need to be able to swap between the exception
+           state inside the generator and the exception state of the calling
+           frame (which shouldn't be impacted when the generator "yields"
+           from an except handler).
+           These three fields exist exactly for that, and are unused for
+           non-generator frames. See the SAVE_EXC_STATE and SWAP_EXC_STATE
+           macros in ceval.c for details of their use. */
     PyObject *f_exc_type, *f_exc_value, *f_exc_traceback;
 
     PyThreadState *f_tstate;
-    int f_lasti;		/* Last instruction if called */
+    int f_lasti;                /* Last instruction if called */
     /* Call PyFrame_GetLineNumber() instead of reading this field
        directly.  As of 2.3 f_lineno is only valid when tracing is
        active (i.e. when f_trace is set).  At other times we use
        PyCode_Addr2Line to calculate the line from the current
        bytecode index. */
-    int f_lineno;		/* Current line number */
-    int f_iblock;		/* index in f_blockstack */
+    int f_lineno;               /* Current line number */
+    int f_iblock;               /* index in f_blockstack */
     PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */
-    PyObject *f_localsplus[1];	/* locals+stack, dynamically sized */
+    PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */
 } PyFrameObject;
 
 

Include/genobject.h

@@ -11,20 +11,20 @@ extern "C" {
 struct _frame; /* Avoid including frameobject.h */
 
 typedef struct {
-	PyObject_HEAD
-	/* The gi_ prefix is intended to remind of generator-iterator. */
+        PyObject_HEAD
+        /* The gi_ prefix is intended to remind of generator-iterator. */
 
-	/* Note: gi_frame can be NULL if the generator is "finished" */
-	struct _frame *gi_frame;
+        /* Note: gi_frame can be NULL if the generator is "finished" */
+        struct _frame *gi_frame;
 
-	/* True if generator is being executed. */
-	int gi_running;
+        /* True if generator is being executed. */
+        int gi_running;
     
-	/* The code object backing the generator */
-	PyObject *gi_code;
+        /* The code object backing the generator */
+        PyObject *gi_code;
 
-	/* List of weak reference. */
-	PyObject *gi_weakreflist;
+        /* List of weak reference. */
+        PyObject *gi_weakreflist;
 } PyGenObject;
 
 PyAPI_DATA(PyTypeObject) PyGen_Type;
@@ -34,6 +34,7 @@ PyAPI_DATA(PyTypeObject) PyGen_Type;
 
 PyAPI_FUNC(PyObject *) PyGen_New(struct _frame *);
 PyAPI_FUNC(int) PyGen_NeedsFinalizing(PyGenObject *);
+PyAPI_FUNC(int) PyGen_FetchStopIterationValue(PyObject **);
 
 #ifdef __cplusplus
 }

Include/graminit.h

@@ -81,3 +81,4 @@
 #define comp_if 334
 #define encoding_decl 335
 #define yield_expr 336
+#define yield_arg 337

Include/opcode.h

@@ -7,116 +7,117 @@ extern "C" {
 
 /* Instruction opcodes for compiled code */
 
-#define POP_TOP		1
-#define ROT_TWO		2
-#define ROT_THREE	3
-#define DUP_TOP		4
+#define POP_TOP         1
+#define ROT_TWO         2
+#define ROT_THREE       3
+#define DUP_TOP         4
 #define DUP_TOP_TWO     5
-#define NOP		9
+#define NOP             9
 
-#define UNARY_POSITIVE	10
-#define UNARY_NEGATIVE	11
-#define UNARY_NOT	12
+#define UNARY_POSITIVE  10
+#define UNARY_NEGATIVE  11
+#define UNARY_NOT       12
 
-#define UNARY_INVERT	15
+#define UNARY_INVERT    15
 
-#define BINARY_POWER	19
+#define BINARY_POWER    19
 
-#define BINARY_MULTIPLY	20
+#define BINARY_MULTIPLY 20
 
-#define BINARY_MODULO	22
-#define BINARY_ADD	23
-#define BINARY_SUBTRACT	24
-#define BINARY_SUBSCR	25
+#define BINARY_MODULO   22
+#define BINARY_ADD      23
+#define BINARY_SUBTRACT 24
+#define BINARY_SUBSCR   25
 #define BINARY_FLOOR_DIVIDE 26
 #define BINARY_TRUE_DIVIDE 27
 #define INPLACE_FLOOR_DIVIDE 28
 #define INPLACE_TRUE_DIVIDE 29
 
-#define STORE_MAP	54
-#define INPLACE_ADD	55
-#define INPLACE_SUBTRACT	56
-#define INPLACE_MULTIPLY	57
-
-#define INPLACE_MODULO	59
-#define STORE_SUBSCR	60
-#define DELETE_SUBSCR	61
-
-#define BINARY_LSHIFT	62
-#define BINARY_RSHIFT	63
-#define BINARY_AND	64
-#define BINARY_XOR	65
-#define BINARY_OR	66
-#define INPLACE_POWER	67
-#define GET_ITER	68
-#define STORE_LOCALS	69
-#define PRINT_EXPR	70
+#define STORE_MAP       54
+#define INPLACE_ADD     55
+#define INPLACE_SUBTRACT        56
+#define INPLACE_MULTIPLY        57
+
+#define INPLACE_MODULO  59
+#define STORE_SUBSCR    60
+#define DELETE_SUBSCR   61
+
+#define BINARY_LSHIFT   62
+#define BINARY_RSHIFT   63
+#define BINARY_AND      64
+#define BINARY_XOR      65
+#define BINARY_OR       66
+#define INPLACE_POWER   67
+#define GET_ITER        68
+#define STORE_LOCALS    69
+#define PRINT_EXPR      70
 #define LOAD_BUILD_CLASS 71
-
-#define INPLACE_LSHIFT	75
-#define INPLACE_RSHIFT	76
-#define INPLACE_AND	77
-#define INPLACE_XOR	78
-#define INPLACE_OR	79
-#define BREAK_LOOP	80
+#define YIELD_FROM      72
+
+#define INPLACE_LSHIFT  75
+#define INPLACE_RSHIFT  76
+#define INPLACE_AND     77
+#define INPLACE_XOR     78
+#define INPLACE_OR      79
+#define BREAK_LOOP      80
 #define WITH_CLEANUP    81
 
-#define RETURN_VALUE	83
-#define IMPORT_STAR	84
+#define RETURN_VALUE    83
+#define IMPORT_STAR     84
 
-#define YIELD_VALUE	86
-#define POP_BLOCK	87
-#define END_FINALLY	88
-#define POP_EXCEPT	89
+#define YIELD_VALUE     86
+#define POP_BLOCK       87
+#define END_FINALLY     88
+#define POP_EXCEPT      89
 
-#define HAVE_ARGUMENT	90	/* Opcodes from here have an argument: */
+#define HAVE_ARGUMENT   90      /* Opcodes from here have an argument: */
 
-#define STORE_NAME	90	/* Index in name list */
-#define DELETE_NAME	91	/* "" */
-#define UNPACK_SEQUENCE	92	/* Number of sequence items */
-#define FOR_ITER	93
+#define STORE_NAME      90      /* Index in name list */
+#define DELETE_NAME     91      /* "" */
+#define UNPACK_SEQUENCE 92      /* Number of sequence items */
+#define FOR_ITER        93
 #define UNPACK_EX       94      /* Num items before variable part +
                                    (Num items after variable part << 8) */
 
-#define STORE_ATTR	95	/* Index in name list */
-#define DELETE_ATTR	96	/* "" */
-#define STORE_GLOBAL	97	/* "" */
-#define DELETE_GLOBAL	98	/* "" */
-
-#define LOAD_CONST	100	/* Index in const list */
-#define LOAD_NAME	101	/* Index in name list */
-#define BUILD_TUPLE	102	/* Number of tuple items */
-#define BUILD_LIST	103	/* Number of list items */
-#define BUILD_SET	104     /* Number of set items */
-#define BUILD_MAP	105	/* Always zero for now */
-#define LOAD_ATTR	106	/* Index in name list */
-#define COMPARE_OP	107	/* Comparison operator */
-#define IMPORT_NAME	108	/* Index in name list */
-#define IMPORT_FROM	109	/* Index in name list */
-
-#define JUMP_FORWARD	110	/* Number of bytes to skip */
-#define JUMP_IF_FALSE_OR_POP 111	/* Target byte offset from beginning of code */
-#define JUMP_IF_TRUE_OR_POP 112	/* "" */
-#define JUMP_ABSOLUTE	113	/* "" */
-#define POP_JUMP_IF_FALSE 114	/* "" */
-#define POP_JUMP_IF_TRUE 115	/* "" */
-
-#define LOAD_GLOBAL	116	/* Index in name list */
-
-#define CONTINUE_LOOP	119	/* Start of loop (absolute) */
-#define SETUP_LOOP	120	/* Target address (relative) */
-#define SETUP_EXCEPT	121	/* "" */
-#define SETUP_FINALLY	122	/* "" */
-
-#define LOAD_FAST	124	/* Local variable number */
-#define STORE_FAST	125	/* Local variable number */
-#define DELETE_FAST	126	/* Local variable number */
-
-#define RAISE_VARARGS	130	/* Number of raise arguments (1, 2 or 3) */
+#define STORE_ATTR      95      /* Index in name list */
+#define DELETE_ATTR     96      /* "" */
+#define STORE_GLOBAL    97      /* "" */
+#define DELETE_GLOBAL   98      /* "" */
+
+#define LOAD_CONST      100     /* Index in const list */
+#define LOAD_NAME       101     /* Index in name list */
+#define BUILD_TUPLE     102     /* Number of tuple items */
+#define BUILD_LIST      103     /* Number of list items */
+#define BUILD_SET       104     /* Number of set items */
+#define BUILD_MAP       105     /* Always zero for now */
+#define LOAD_ATTR       106     /* Index in name list */
+#define COMPARE_OP      107     /* Comparison operator */
+#define IMPORT_NAME     108     /* Index in name list */
+#define IMPORT_FROM     109     /* Index in name list */
+
+#define JUMP_FORWARD    110     /* Number of bytes to skip */
+#define JUMP_IF_FALSE_OR_POP 111        /* Target byte offset from beginning of code */
+#define JUMP_IF_TRUE_OR_POP 112 /* "" */
+#define JUMP_ABSOLUTE   113     /* "" */
+#define POP_JUMP_IF_FALSE 114   /* "" */
+#define POP_JUMP_IF_TRUE 115    /* "" */
+
+#define LOAD_GLOBAL     116     /* Index in name list */
+
+#define CONTINUE_LOOP   119     /* Start of loop (absolute) */
+#define SETUP_LOOP      120     /* Target address (relative) */
+#define SETUP_EXCEPT    121     /* "" */
+#define SETUP_FINALLY   122     /* "" */
+
+#define LOAD_FAST       124     /* Local variable number */
+#define STORE_FAST      125     /* Local variable number */
+#define DELETE_FAST     126     /* Local variable number */
+
+#define RAISE_VARARGS   130     /* Number of raise arguments (1, 2 or 3) */
 /* CALL_FUNCTION_XXX opcodes defined below depend on this definition */
-#define CALL_FUNCTION	131	/* #args + (#kwargs<<8) */
-#define MAKE_FUNCTION	132	/* #defaults + #kwdefaults<<8 + #annotations<<16 */
-#define BUILD_SLICE 	133	/* Number of items */
+#define CALL_FUNCTION   131     /* #args + (#kwargs<<8) */
+#define MAKE_FUNCTION   132     /* #defaults + #kwdefaults<<8 + #annotations<<16 */
+#define BUILD_SLICE     133     /* Number of items */
 
 #define MAKE_CLOSURE    134     /* same as MAKE_FUNCTION */
 #define LOAD_CLOSURE    135     /* Load free variable from closure */
@@ -126,9 +127,9 @@ extern "C" {
 
 /* The next 3 opcodes must be contiguous and satisfy
    (CALL_FUNCTION_VAR - CALL_FUNCTION) & 3 == 1  */
-#define CALL_FUNCTION_VAR          140	/* #args + (#kwargs<<8) */
-#define CALL_FUNCTION_KW           141	/* #args + (#kwargs<<8) */
-#define CALL_FUNCTION_VAR_KW       142	/* #args + (#kwargs<<8) */
+#define CALL_FUNCTION_VAR          140  /* #args + (#kwargs<<8) */
+#define CALL_FUNCTION_KW           141  /* #args + (#kwargs<<8) */
+#define CALL_FUNCTION_VAR_KW       142  /* #args + (#kwargs<<8) */
 
 #define SETUP_WITH 143
 
@@ -148,7 +149,7 @@ extern "C" {
 
 
 enum cmp_op {PyCmp_LT=Py_LT, PyCmp_LE=Py_LE, PyCmp_EQ=Py_EQ, PyCmp_NE=Py_NE, PyCmp_GT=Py_GT, PyCmp_GE=Py_GE,
-	     PyCmp_IN, PyCmp_NOT_IN, PyCmp_IS, PyCmp_IS_NOT, PyCmp_EXC_MATCH, PyCmp_BAD};
+             PyCmp_IN, PyCmp_NOT_IN, PyCmp_IS, PyCmp_IS_NOT, PyCmp_EXC_MATCH, PyCmp_BAD};
 
 #define HAS_ARG(op) ((op) >= HAVE_ARGUMENT)
 

Include/pyerrors.h

@@ -51,6 +51,11 @@ typedef struct {
     Py_ssize_t written;   /* only for BlockingIOError, -1 otherwise */
 } PyOSErrorObject;
 
+typedef struct {
+    PyException_HEAD
+    PyObject *value;
+} PyStopIterationObject;
+
 /* Compatibility typedefs */
 typedef PyOSErrorObject PyEnvironmentErrorObject;
 #ifdef MS_WINDOWS
@@ -380,6 +385,8 @@ PyAPI_FUNC(int) PyUnicodeTranslateError_SetReason(
     const char *reason          /* UTF-8 encoded string */
     );
 
+/* create a StopIteration exception with the given value */
+PyAPI_FUNC(PyObject *) PyStopIteration_Create(PyObject *);
 
 /* These APIs aren't really part of the error implementation, but
    often needed to format error messages; the native C lib APIs are

Lib/opcode.py

@@ -87,6 +87,7 @@ def_op('STORE_LOCALS', 69)
 
 def_op('PRINT_EXPR', 70)
 def_op('LOAD_BUILD_CLASS', 71)
+def_op('YIELD_FROM', 72)
 
 def_op('INPLACE_LSHIFT', 75)
 def_op('INPLACE_RSHIFT', 76)

Lib/test/test_ast.py

@@ -813,7 +813,8 @@ class ASTValidatorTests(unittest.TestCase):
         self._check_comprehension(factory)
 
     def test_yield(self):
-        self.expr(ast.Yield(ast.Name("x", ast.Store())), "must have Load")
+        self.expr(ast.Yield(0, ast.Name("x", ast.Store())), "must have Load")
+        self.expr(ast.Yield(1, ast.Name("x", ast.Store())), "must have Load")
 
     def test_compare(self):
         left = ast.Name("x", ast.Load())

Lib/test/test_generators.py

@@ -728,29 +728,6 @@ Ye olde Fibonacci generator, tee style.
 
 syntax_tests = """
 
->>> def f():
-...     return 22
-...     yield 1
-Traceback (most recent call last):
-  ..
-SyntaxError: 'return' with argument inside generator
-
->>> def f():
-...     yield 1
-...     return 22
-Traceback (most recent call last):
-  ..
-SyntaxError: 'return' with argument inside generator
-
-"return None" is not the same as "return" in a generator:
-
->>> def f():
-...     yield 1
-...     return None
-Traceback (most recent call last):
-  ..
-SyntaxError: 'return' with argument inside generator
-
 These are fine:
 
 >>> def f():
@@ -866,20 +843,6 @@ These are fine:
 >>> type(f())
 <class 'generator'>
 
-
->>> def f():
-...     if 0:
-...         lambda x:  x        # shouldn't trigger here
-...         return              # or here
-...         def f(i):
-...             return 2*i      # or here
-...         if 0:
-...             return 3        # but *this* sucks (line 8)
-...     if 0:
-...         yield 2             # because it's a generator (line 10)
-Traceback (most recent call last):
-SyntaxError: 'return' with argument inside generator
-
 This one caused a crash (see SF bug 567538):
 
 >>> def f():
@@ -1566,11 +1529,6 @@ Traceback (most recent call last):
   ...
 SyntaxError: 'yield' outside function
 
->>> def f(): return lambda x=(yield): 1
-Traceback (most recent call last):
-  ...
-SyntaxError: 'return' with argument inside generator
-
 >>> def f(): x = yield = y
 Traceback (most recent call last):
   ...

Lib/test/test_grammar.py

@@ -458,7 +458,39 @@ class GrammarTests(unittest.TestCase):
         check_syntax_error(self, "class foo:return 1")
 
     def test_yield(self):
+        # Allowed as standalone statement
+        def g(): yield 1
+        def g(): yield from ()
+        # Allowed as RHS of assignment
+        def g(): x = yield 1
+        def g(): x = yield from ()
+        # Ordinary yield accepts implicit tuples
+        def g(): yield 1, 1
+        def g(): x = yield 1, 1
+        # 'yield from' does not
+        check_syntax_error(self, "def g(): yield from (), 1")
+        check_syntax_error(self, "def g(): x = yield from (), 1")
+        # Requires parentheses as subexpression
+        def g(): 1, (yield 1)
+        def g(): 1, (yield from ())
+        check_syntax_error(self, "def g(): 1, yield 1")
+        check_syntax_error(self, "def g(): 1, yield from ()")
+        # Requires parentheses as call argument
+        def g(): f((yield 1))
+        def g(): f((yield 1), 1)
+        def g(): f((yield from ()))
+        def g(): f((yield from ()), 1)
+        check_syntax_error(self, "def g(): f(yield 1)")
+        check_syntax_error(self, "def g(): f(yield 1, 1)")
+        check_syntax_error(self, "def g(): f(yield from ())")
+        check_syntax_error(self, "def g(): f(yield from (), 1)")
+        # Not allowed at top level
+        check_syntax_error(self, "yield")
+        check_syntax_error(self, "yield from")
+        # Not allowed at class scope
         check_syntax_error(self, "class foo:yield 1")
+        check_syntax_error(self, "class foo:yield from ()")
+
 
     def test_raise(self):
         # 'raise' test [',' test]

Lib/test/test_parser.py

@@ -50,6 +50,10 @@ class RoundtripLegalSyntaxTestCase(unittest.TestCase):
         self.check_suite("def f(): (yield 1)*2")
         self.check_suite("def f(): return; yield 1")
         self.check_suite("def f(): yield 1; return")
+        self.check_suite("def f(): yield from 1")
+        self.check_suite("def f(): x = yield from 1")
+        self.check_suite("def f(): f((yield from 1))")
+        self.check_suite("def f(): yield 1; return 1")
         self.check_suite("def f():\n"
                          "    for x in range(30):\n"
                          "        yield x\n")
@@ -621,7 +625,6 @@ class OtherParserCase(unittest.TestCase):
         with self.assertRaises(TypeError):
             parser.expr("a", "b")
 
-
 def test_main():
     support.run_unittest(
         RoundtripLegalSyntaxTestCase,

Lib/test/test_sys.py

@@ -727,7 +727,7 @@ class SizeofTest(unittest.TestCase):
         nfrees = len(x.f_code.co_freevars)
         extras = x.f_code.co_stacksize + x.f_code.co_nlocals +\
                   ncells + nfrees - 1
-        check(x, size(vh + '12P3i' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P'))
+        check(x, size(vh + '13P3i' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P'))
         # function
         def func(): pass
         check(func, size(h + '12P'))

Misc/ACKS

@@ -96,6 +96,7 @@ Dominic Binks
 Philippe Biondi
 Stuart Bishop
 Roy Bixler
+Renaud Blanch
 Mike Bland
 Martin Bless
 Pablo Bleyer
@@ -482,6 +483,7 @@ Geert Jansen
 Jack Jansen
 Bill Janssen
 Thomas Jarosch
+Zbigniew Jędrzejewski-Szmek
 Julien Jehannet
 Drew Jenkins
 Flemming Kjær Jensen

Misc/NEWS

@@ -10,6 +10,10 @@ What's New in Python 3.3 Alpha 1?
 Core and Builtins
 -----------------
 
+- PEP 380, Issue #11682: Add "yield from <x>" to support easy delegation to
+  subgenerators (initial patch by Greg Ewing, integration into 3.3 by
+  Renaud Blanch, Ryan Kelly, Zbigniew Jędrzejewski-Szmek and Nick Coghlan)
+
 - Issue #13748: Raw bytes literals can now be written with the ``rb`` prefix
   as well as ``br``.