Issue #17810: Implement PEP 3154, pickle protocol 4.

Most of the work is by Alexandre.

Doc/library/pickle.rst

@@ -459,12 +459,29 @@ implementation of this behaviour::
 Classes can alter the default behaviour by providing one or several special
 methods:
 
+.. method:: object.__getnewargs_ex__()
+
+   In protocols 4 and newer, classes that implements the
+   :meth:`__getnewargs_ex__` method can dictate the values passed to the
+   :meth:`__new__` method upon unpickling.  The method must return a pair
+   ``(args, kwargs)`` where *args* is a tuple of positional arguments
+   and *kwargs* a dictionary of named arguments for constructing the
+   object.  Those will be passed to the :meth:`__new__` method upon
+   unpickling.
+
+   You should implement this method if the :meth:`__new__` method of your
+   class requires keyword-only arguments.  Otherwise, it is recommended for
+   compatibility to implement :meth:`__getnewargs__`.
+
+
 .. method:: object.__getnewargs__()
 
-   In protocol 2 and newer, classes that implements the :meth:`__getnewargs__`
-   method can dictate the values passed to the :meth:`__new__` method upon
-   unpickling.  This is often needed for classes whose :meth:`__new__` method
-   requires arguments.
+   This method serve a similar purpose as :meth:`__getnewargs_ex__` but
+   for protocols 2 and newer.  It must return a tuple of arguments `args`
+   which will be passed to the :meth:`__new__` method upon unpickling.
+
+   In protocols 4 and newer, :meth:`__getnewargs__` will not be called if
+   :meth:`__getnewargs_ex__` is defined.
 
 
 .. method:: object.__getstate__()
@@ -496,10 +513,10 @@ the methods :meth:`__getstate__` and :meth:`__setstate__`.
 
    At unpickling time, some methods like :meth:`__getattr__`,
    :meth:`__getattribute__`, or :meth:`__setattr__` may be called upon the
-   instance.  In case those methods rely on some internal invariant being true,
-   the type should implement :meth:`__getnewargs__` to establish such an
-   invariant; otherwise, neither :meth:`__new__` nor :meth:`__init__` will be
-   called.
+   instance.  In case those methods rely on some internal invariant being
+   true, the type should implement :meth:`__getnewargs__` or
+   :meth:`__getnewargs_ex__` to establish such an invariant; otherwise,
+   neither :meth:`__new__` nor :meth:`__init__` will be called.
 
 .. index:: pair: copy; protocol
 
@@ -511,7 +528,7 @@ objects. [#]_
 
 Although powerful, implementing :meth:`__reduce__` directly in your classes is
 error prone.  For this reason, class designers should use the high-level
-interface (i.e., :meth:`__getnewargs__`, :meth:`__getstate__` and
+interface (i.e., :meth:`__getnewargs_ex__`, :meth:`__getstate__` and
 :meth:`__setstate__`) whenever possible.  We will show, however, cases where
 using :meth:`__reduce__` is the only option or leads to more efficient pickling
 or both.

Doc/whatsnew/3.4.rst

@@ -109,6 +109,7 @@ New expected features for Python implementations:
 Significantly Improved Library Modules:
 
 * Single-dispatch generic functions in :mod:`functoools` (:pep:`443`)
+* New :mod:`pickle` protocol 4 (:pep:`3154`)
 * SHA-3 (Keccak) support for :mod:`hashlib`.
 * TLSv1.1 and TLSv1.2 support for :mod:`ssl`.
 * :mod:`multiprocessing` now has option to avoid using :func:`os.fork`
@@ -285,6 +286,20 @@ described in the PEP.  Existing importers should be updated to implement
 the new methods.
 
 
+Pickle protocol 4
+=================
+
+The new :mod:`pickle` protocol addresses a number of issues that were present
+in previous protocols, such as the serialization of nested classes, very
+large strings and containers, or classes whose :meth:`__new__` method takes
+keyword-only arguments.  It also brings a couple efficiency improvements.
+
+.. seealso::
+
+   :pep:`3154` - Pickle protocol 4
+      PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti.
+
+
 Other Language Changes
 ======================
 

Lib/copyreg.py

@@ -87,6 +87,12 @@ def _reduce_ex(self, proto):
 def __newobj__(cls, *args):
     return cls.__new__(cls, *args)
 
+def __newobj_ex__(cls, args, kwargs):
+    """Used by pickle protocol 4, instead of __newobj__ to allow classes with
+    keyword-only arguments to be pickled correctly.
+    """
+    return cls.__new__(cls, *args, **kwargs)
+
 def _slotnames(cls):
     """Return a list of slot names for a given class.
 

Misc/NEWS

@@ -68,6 +68,8 @@ Core and Builtins
 Library
 -------
 
+- Issue #17810: Implement PEP 3154, pickle protocol 4.
+
 - Issue #19668: Added support for the cp1125 encoding.
 
 - Issue #19689: Add ssl.create_default_context() factory function. It creates

Objects/classobject.c

@@ -69,6 +69,30 @@ PyMethod_New(PyObject *func, PyObject *self)
     return (PyObject *)im;
 }
 
+static PyObject *
+method_reduce(PyMethodObject *im)
+{
+    PyObject *self = PyMethod_GET_SELF(im);
+    PyObject *func = PyMethod_GET_FUNCTION(im);
+    PyObject *builtins;
+    PyObject *getattr;
+    PyObject *funcname;
+    _Py_IDENTIFIER(getattr);
+
+    funcname = _PyObject_GetAttrId(func, &PyId___name__);
+    if (funcname == NULL) {
+        return NULL;
+    }
+    builtins = PyEval_GetBuiltins();
+    getattr = _PyDict_GetItemId(builtins, &PyId_getattr);
+    return Py_BuildValue("O(ON)", getattr, self, funcname);
+}
+
+static PyMethodDef method_methods[] = {
+    {"__reduce__", (PyCFunction)method_reduce, METH_NOARGS, NULL},
+    {NULL, NULL}
+};
+
 /* Descriptors for PyMethod attributes */
 
 /* im_func and im_self are stored in the PyMethod object */
@@ -367,7 +391,7 @@ PyTypeObject PyMethod_Type = {
     offsetof(PyMethodObject, im_weakreflist), /* tp_weaklistoffset */
     0,                                          /* tp_iter */
     0,                                          /* tp_iternext */
-    0,                                          /* tp_methods */
+    method_methods,                             /* tp_methods */
     method_memberlist,                          /* tp_members */
     method_getset,                              /* tp_getset */
     0,                                          /* tp_base */

Objects/descrobject.c

@@ -398,6 +398,24 @@ descr_get_qualname(PyDescrObject *descr)
     return descr->d_qualname;
 }
 
+static PyObject *
+descr_reduce(PyDescrObject *descr)
+{
+    PyObject *builtins;
+    PyObject *getattr;
+    _Py_IDENTIFIER(getattr);
+
+    builtins = PyEval_GetBuiltins();
+    getattr = _PyDict_GetItemId(builtins, &PyId_getattr);
+    return Py_BuildValue("O(OO)", getattr, PyDescr_TYPE(descr),
+                         PyDescr_NAME(descr));
+}
+
+static PyMethodDef descr_methods[] = {
+    {"__reduce__", (PyCFunction)descr_reduce, METH_NOARGS, NULL},
+    {NULL, NULL}
+};
+
 static PyMemberDef descr_members[] = {
     {"__objclass__", T_OBJECT, offsetof(PyDescrObject, d_type), READONLY},
     {"__name__", T_OBJECT, offsetof(PyDescrObject, d_name), READONLY},
@@ -494,7 +512,7 @@ PyTypeObject PyMethodDescr_Type = {
     0,                                          /* tp_weaklistoffset */
     0,                                          /* tp_iter */
     0,                                          /* tp_iternext */
-    0,                                          /* tp_methods */
+    descr_methods,                              /* tp_methods */
     descr_members,                              /* tp_members */
     method_getset,                              /* tp_getset */
     0,                                          /* tp_base */
@@ -532,7 +550,7 @@ PyTypeObject PyClassMethodDescr_Type = {
     0,                                          /* tp_weaklistoffset */
     0,                                          /* tp_iter */
     0,                                          /* tp_iternext */
-    0,                                          /* tp_methods */
+    descr_methods,                              /* tp_methods */
     descr_members,                              /* tp_members */
     method_getset,                              /* tp_getset */
     0,                                          /* tp_base */
@@ -569,7 +587,7 @@ PyTypeObject PyMemberDescr_Type = {
     0,                                          /* tp_weaklistoffset */
     0,                                          /* tp_iter */
     0,                                          /* tp_iternext */
-    0,                                          /* tp_methods */
+    descr_methods,                              /* tp_methods */
     descr_members,                              /* tp_members */
     member_getset,                              /* tp_getset */
     0,                                          /* tp_base */
@@ -643,7 +661,7 @@ PyTypeObject PyWrapperDescr_Type = {
     0,                                          /* tp_weaklistoffset */
     0,                                          /* tp_iter */
     0,                                          /* tp_iternext */
-    0,                                          /* tp_methods */
+    descr_methods,                              /* tp_methods */
     descr_members,                              /* tp_members */
     wrapperdescr_getset,                        /* tp_getset */
     0,                                          /* tp_base */
@@ -1085,6 +1103,23 @@ wrapper_repr(wrapperobject *wp)
                                wp->self);
 }
 
+static PyObject *
+wrapper_reduce(wrapperobject *wp)
+{
+    PyObject *builtins;
+    PyObject *getattr;
+    _Py_IDENTIFIER(getattr);
+
+    builtins = PyEval_GetBuiltins();
+    getattr = _PyDict_GetItemId(builtins, &PyId_getattr);
+    return Py_BuildValue("O(OO)", getattr, wp->self, PyDescr_NAME(wp->descr));
+}
+
+static PyMethodDef wrapper_methods[] = {
+    {"__reduce__", (PyCFunction)wrapper_reduce, METH_NOARGS, NULL},
+    {NULL, NULL}
+};
+
 static PyMemberDef wrapper_members[] = {
     {"__self__", T_OBJECT, offsetof(wrapperobject, self), READONLY},
     {0}
@@ -1193,7 +1228,7 @@ PyTypeObject _PyMethodWrapper_Type = {
     0,                                          /* tp_weaklistoffset */
     0,                                          /* tp_iter */
     0,                                          /* tp_iternext */
-    0,                                          /* tp_methods */
+    wrapper_methods,                            /* tp_methods */
     wrapper_members,                            /* tp_members */
     wrapper_getsets,                            /* tp_getset */
     0,                                          /* tp_base */