- Issue #10181: New memoryview implementation fixes multiple ownership

  and lifetime issues of dynamically allocated Py_buffer members (#9990)
  as well as crashes (#8305, #7433). Many new features have been added
  (See whatsnew/3.3), and the documentation has been updated extensively.
  The ndarray test object from _testbuffer.c implements all aspects of
  PEP-3118, so further development towards the complete implementation
  of the PEP can proceed in a test-driven manner.

  Thanks to Nick Coghlan, Antoine Pitrou and Pauli Virtanen for review
  and many ideas.

- Issue #12834: Fix incorrect results of memoryview.tobytes() for
  non-contiguous arrays.

- Issue #5231: Introduce memoryview.cast() method that allows changing
  format and shape without making a copy of the underlying memory.

Doc/c-api/memoryview.rst

@@ -17,16 +17,19 @@ any other object.
 
    Create a memoryview object from an object that provides the buffer interface.
    If *obj* supports writable buffer exports, the memoryview object will be
-   readable and writable, otherwise it will be read-only.
+   read/write, otherwise it may be either read-only or read/write at the
+   discretion of the exporter.
 
+.. c:function:: PyObject *PyMemoryView_FromMemory(char *mem, Py_ssize_t size, int flags)
+
+   Create a memoryview object using *mem* as the underlying buffer.
+   *flags* can be one of :c:macro:`PyBUF_READ` or :c:macro:`PyBUF_WRITE`.
 
 .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view)
 
    Create a memoryview object wrapping the given buffer structure *view*.
-   The memoryview object then owns the buffer represented by *view*, which
-   means you shouldn't try to call :c:func:`PyBuffer_Release` yourself: it
-   will be done on deallocation of the memoryview object.
-
+   For simple byte buffers, :c:func:`PyMemoryView_FromMemory` is the preferred
+   function.
 
 .. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order)
 
@@ -43,10 +46,16 @@ any other object.
    currently allowed to create subclasses of :class:`memoryview`.
 
 
-.. c:function:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *obj)
+.. c:function:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *mview)
+
+   Return a pointer to the memoryview's private copy of the exporter's buffer.
+   *mview* **must** be a memoryview instance; this macro doesn't check its type,
+   you must do it yourself or you will risk crashes.
+
+.. c:function:: Py_buffer *PyMemoryView_GET_BASE(PyObject *mview)
 
-   Return a pointer to the buffer structure wrapped by the given
-   memoryview object.  The object **must** be a memoryview instance;
-   this macro doesn't check its type, you must do it yourself or you
-   will risk crashes.
+   Return either a pointer to the exporting object that the memoryview is based
+   on or *NULL* if the memoryview has been created by one of the functions
+   :c:func:`PyMemoryView_FromMemory` or :c:func:`PyMemoryView_FromBuffer`.
+   *mview* **must** be a memoryview instance.
 

Doc/c-api/typeobj.rst

@@ -1198,46 +1198,74 @@ Buffer Object Structures
 
 .. sectionauthor:: Greg J. Stein <greg@lyra.org>
 .. sectionauthor:: Benjamin Peterson
+.. sectionauthor:: Stefan Krah
 
+.. c:type:: PyBufferProcs
 
-The :ref:`buffer interface <bufferobjects>` exports a model where an object can expose its internal
-data.
+   This structure holds pointers to the functions required by the
+   :ref:`Buffer protocol <bufferobjects>`. The protocol defines how
+   an exporter object can expose its internal data to consumer objects.
 
-If an object does not export the buffer interface, then its :attr:`tp_as_buffer`
-member in the :c:type:`PyTypeObject` structure should be *NULL*.  Otherwise, the
-:attr:`tp_as_buffer` will point to a :c:type:`PyBufferProcs` structure.
+.. c:member:: getbufferproc PyBufferProcs.bf_getbuffer
 
+   The signature of this function is::
 
-.. c:type:: PyBufferProcs
+       int (PyObject *exporter, Py_buffer *view, int flags);
+
+   Handle a request to *exporter* to fill in *view* as specified by *flags*.
+   A standard implementation of this function will take these steps:
+
+   - Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`,
+     set :c:data:`view->obj` to *NULL* and return -1.
+
+   - Fill in the requested fields.
+
+   - Increment an internal counter for the number of exports.
+
+   - Set :c:data:`view->obj` to *exporter* and increment :c:data:`view->obj`.
+
+   - Return 0.
+
+   The individual fields of *view* are described in section
+   :ref:`Buffer structure <buffer-structure>`, the rules how an exporter
+   must react to specific requests are in section
+   :ref:`Buffer request types <buffer-request-types>`.
+
+   All memory pointed to in the :c:type:`Py_buffer` structure belongs to
+   the exporter and must remain valid until there are no consumers left.
+   :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`,
+   :c:member:`~Py_buffer.suboffsets` and :c:member:`~Py_buffer.internal`
+   are read-only for the consumer.
+
+   :c:func:`PyBuffer_FillInfo` provides an easy way of exposing a simple
+   bytes buffer while dealing correctly with all request types.
+
+   :c:func:`PyObject_GetBuffer` is the interface for the consumer that
+   wraps this function.
+
+.. c:member:: releasebufferproc PyBufferProcs.bf_releasebuffer
+
+   The signature of this function is::
+
+       void (PyObject *exporter, Py_buffer *view);
 
-   Structure used to hold the function pointers which define an implementation of
-   the buffer protocol.
+   Handle a request to release the resources of the buffer. If no resources
+   need to be released, this field may be *NULL*. A standard implementation
+   of this function will take these steps:
 
-   .. c:member:: getbufferproc bf_getbuffer
+   - Decrement an internal counter for the number of exports.
 
-      This should fill a :c:type:`Py_buffer` with the necessary data for
-      exporting the type.  The signature of :data:`getbufferproc` is ``int
-      (PyObject *obj, Py_buffer *view, int flags)``.  *obj* is the object to
-      export, *view* is the :c:type:`Py_buffer` struct to fill, and *flags* gives
-      the conditions the caller wants the memory under.  (See
-      :c:func:`PyObject_GetBuffer` for all flags.)  :c:member:`bf_getbuffer` is
-      responsible for filling *view* with the appropriate information.
-      (:c:func:`PyBuffer_FillView` can be used in simple cases.)  See
-      :c:type:`Py_buffer`\s docs for what needs to be filled in.
+   - If the counter is 0, free all memory associated with *view*.
 
+   The exporter MUST use the :c:member:`~Py_buffer.internal` field to keep
+   track of buffer-specific resources (if present). This field is guaranteed
+   to remain constant, while a consumer MAY pass a copy of the original buffer
+   as the *view* argument.
 
-   .. c:member:: releasebufferproc bf_releasebuffer
 
-      This should release the resources of the buffer.  The signature of
-      :c:data:`releasebufferproc` is ``void (PyObject *obj, Py_buffer *view)``.
-      If the :c:data:`bf_releasebuffer` function is not provided (i.e. it is
-      *NULL*), then it does not ever need to be called.
+   This function MUST NOT decrement :c:data:`view->obj`, since that is
+   done automatically in :c:func:`PyBuffer_Release`.
 
-      The exporter of the buffer interface must make sure that any memory
-      pointed to in the :c:type:`Py_buffer` structure remains valid until
-      releasebuffer is called.  Exporters will need to define a
-      :c:data:`bf_releasebuffer` function if they can re-allocate their memory,
-      strides, shape, suboffsets, or format variables which they might share
-      through the struct bufferinfo.
 
-      See :c:func:`PyBuffer_Release`.
+   :c:func:`PyBuffer_Release` is the interface for the consumer that
+   wraps this function.

Doc/whatsnew/3.3.rst

@@ -49,6 +49,62 @@
 This article explains the new features in Python 3.3, compared to 3.2.
 
 
+.. _pep-3118:
+
+PEP 3118: New memoryview implementation and buffer protocol documentation
+=========================================================================
+
+:issue:`10181` - memoryview bug fixes and features.
+  Written by Stefan Krah.
+
+The new memoryview implementation comprehensively fixes all ownership and
+lifetime issues of dynamically allocated fields in the Py_buffer struct
+that led to multiple crash reports. Additionally, several functions that
+crashed or returned incorrect results for non-contiguous or multi-dimensional
+input have been fixed.
+
+The memoryview object now has a PEP-3118 compliant getbufferproc()
+that checks the consumer's request type. Many new features have been
+added, most of them work in full generality for non-contiguous arrays
+and arrays with suboffsets.
+
+The documentation has been updated, clearly spelling out responsibilities
+for both exporters and consumers. Buffer request flags are grouped into
+basic and compound flags. The memory layout of non-contiguous and
+multi-dimensional NumPy-style arrays is explained.
+
+Features
+--------
+
+* All native single character format specifiers in struct module syntax
+  (optionally prefixed with '@') are now supported.
+
+* With some restrictions, the cast() method allows changing of format and
+  shape of C-contiguous arrays.
+
+* Multi-dimensional list representations are supported for any array type.
+
+* Multi-dimensional comparisons are supported for any array type.
+
+* All array types are hashable if the exporting object is hashable
+  and the view is read-only.
+
+* Arbitrary slicing of any 1-D arrays type is supported. For example, it
+  is now possible to reverse a memoryview in O(1) by using a negative step.
+
+API changes
+-----------
+
+* The maximum number of dimensions is officially limited to 64.
+
+* The representation of empty shape, strides and suboffsets is now
+  an empty tuple instead of None.
+
+* Accessing a memoryview element with format 'B' (unsigned bytes)
+  now returns an integer (in accordance with the struct module syntax).
+  For returning a bytes object the view must be cast to 'c' first.
+
+
 .. _pep-393:
 
 PEP 393: Flexible String Representation

Include/abstract.h

@@ -559,7 +559,7 @@ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx*/
     /* Copy the data from the src buffer to the buffer of destination
      */
 
-     PyAPI_FUNC(int) PyBuffer_IsContiguous(Py_buffer *view, char fort);
+     PyAPI_FUNC(int) PyBuffer_IsContiguous(const Py_buffer *view, char fort);
 
 
      PyAPI_FUNC(void) PyBuffer_FillContiguousStrides(int ndims,

Include/memoryobject.h

@@ -6,70 +6,64 @@
 extern "C" {
 #endif
 
+#ifndef Py_LIMITED_API
+PyAPI_DATA(PyTypeObject) _PyManagedBuffer_Type;
+#endif
 PyAPI_DATA(PyTypeObject) PyMemoryView_Type;
 
 #define PyMemoryView_Check(op) (Py_TYPE(op) == &PyMemoryView_Type)
 
 #ifndef Py_LIMITED_API
-/* Get a pointer to the underlying Py_buffer of a memoryview object. */
+/* Get a pointer to the memoryview's private copy of the exporter's buffer. */
 #define PyMemoryView_GET_BUFFER(op) (&((PyMemoryViewObject *)(op))->view)
-/* Get a pointer to the PyObject from which originates a memoryview object. */
+/* Get a pointer to the exporting object (this may be NULL!). */
 #define PyMemoryView_GET_BASE(op) (((PyMemoryViewObject *)(op))->view.obj)
 #endif
 
-
-PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base, 
-						  int buffertype, 
-						  char fort);
-
-    /* Return a contiguous chunk of memory representing the buffer
-       from an object in a memory view object.  If a copy is made then the
-       base object for the memory view will be a *new* bytes object. 
-       
-       Otherwise, the base-object will be the object itself and no 
-       data-copying will be done. 
-
-       The buffertype argument can be PyBUF_READ, PyBUF_WRITE,
-       PyBUF_SHADOW to determine whether the returned buffer
-       should be READONLY, WRITABLE, or set to update the
-       original buffer if a copy must be made.  If buffertype is
-       PyBUF_WRITE and the buffer is not contiguous an error will
-       be raised.  In this circumstance, the user can use
-       PyBUF_SHADOW to ensure that a a writable temporary
-       contiguous buffer is returned.  The contents of this
-       contiguous buffer will be copied back into the original
-       object after the memoryview object is deleted as long as
-       the original object is writable and allows setting an
-       exclusive write lock. If this is not allowed by the
-       original object, then a BufferError is raised.
-       
-       If the object is multi-dimensional and if fortran is 'F',
-       the first dimension of the underlying array will vary the
-       fastest in the buffer.  If fortran is 'C', then the last
-       dimension will vary the fastest (C-style contiguous).  If
-       fortran is 'A', then it does not matter and you will get
-       whatever the object decides is more efficient.  
-
-       A new reference is returned that must be DECREF'd when finished.
-    */
-
 PyAPI_FUNC(PyObject *) PyMemoryView_FromObject(PyObject *base);
-
+PyAPI_FUNC(PyObject *) PyMemoryView_FromMemory(char *mem, Py_ssize_t size,
+                                               int flags);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) PyMemoryView_FromBuffer(Py_buffer *info);
-    /* create new if bufptr is NULL 
-        will be a new bytesobject in base */
 #endif
+PyAPI_FUNC(PyObject *) PyMemoryView_GetContiguous(PyObject *base,
+                                                  int buffertype,
+                                                  char order);
 
 
-/* The struct is declared here so that macros can work, but it shouldn't
-   be considered public. Don't access those fields directly, use the macros
+/* The structs are declared here so that macros can work, but they shouldn't
+   be considered public. Don't access their fields directly, use the macros
    and functions instead! */
 #ifndef Py_LIMITED_API
+#define _Py_MANAGED_BUFFER_RELEASED    0x001  /* access to exporter blocked */
+#define _Py_MANAGED_BUFFER_FREE_FORMAT 0x002  /* free format */
 typedef struct {
     PyObject_HEAD
-    Py_buffer view;
-    Py_hash_t hash;
+    int flags;          /* state flags */
+    Py_ssize_t exports; /* number of direct memoryview exports */
+    Py_buffer master; /* snapshot buffer obtained from the original exporter */
+} _PyManagedBufferObject;
+
+
+/* static storage used for casting between formats */
+#define _Py_MEMORYVIEW_MAX_FORMAT 3 /* must be >= 3 */
+
+/* memoryview state flags */
+#define _Py_MEMORYVIEW_RELEASED    0x001  /* access to master buffer blocked */
+#define _Py_MEMORYVIEW_C           0x002  /* C-contiguous layout */
+#define _Py_MEMORYVIEW_FORTRAN     0x004  /* Fortran contiguous layout */
+#define _Py_MEMORYVIEW_SCALAR      0x008  /* scalar: ndim = 0 */
+#define _Py_MEMORYVIEW_PIL         0x010  /* PIL-style layout */
+
+typedef struct {
+    PyObject_VAR_HEAD
+    _PyManagedBufferObject *mbuf; /* managed buffer */
+    Py_hash_t hash;               /* hash value for read-only views */
+    int flags;                    /* state flags */
+    Py_ssize_t exports;           /* number of buffer re-exports */
+    Py_buffer view;               /* private copy of the exporter's view */
+    char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* used for casting */
+    Py_ssize_t ob_array[1];       /* shape, strides, suboffsets */
 } PyMemoryViewObject;
 #endif
 

Include/object.h

@@ -186,15 +186,16 @@ typedef struct bufferinfo {
     Py_ssize_t *shape;
     Py_ssize_t *strides;
     Py_ssize_t *suboffsets;
-    Py_ssize_t smalltable[2];  /* static store for shape and strides of
-                                  mono-dimensional buffers. */
     void *internal;
 } Py_buffer;
 
 typedef int (*getbufferproc)(PyObject *, Py_buffer *, int);
 typedef void (*releasebufferproc)(PyObject *, Py_buffer *);
 
-    /* Flags for getting buffers */
+/* Maximum number of dimensions */
+#define PyBUF_MAX_NDIM 64
+
+/* Flags for getting buffers */
 #define PyBUF_SIMPLE 0
 #define PyBUF_WRITABLE 0x0001
 /*  we used to include an E, backwards compatible alias  */

Lib/ctypes/test/test_pep3118.py

@@ -25,14 +25,17 @@ class Test(unittest.TestCase):
             v = memoryview(ob)
             try:
                 self.assertEqual(normalize(v.format), normalize(fmt))
-                if shape is not None:
+                if shape:
                     self.assertEqual(len(v), shape[0])
                 else:
                     self.assertEqual(len(v) * sizeof(itemtp), sizeof(ob))
                 self.assertEqual(v.itemsize, sizeof(itemtp))
                 self.assertEqual(v.shape, shape)
-                # ctypes object always have a non-strided memory block
-                self.assertEqual(v.strides, None)
+                # XXX Issue #12851: PyCData_NewGetBuffer() must provide strides
+                #     if requested. memoryview currently reconstructs missing
+                #     stride information, so this assert will fail.
+                # self.assertEqual(v.strides, ())
+
                 # they are always read/write
                 self.assertFalse(v.readonly)
 
@@ -52,14 +55,15 @@ class Test(unittest.TestCase):
             v = memoryview(ob)
             try:
                 self.assertEqual(v.format, fmt)
-                if shape is not None:
+                if shape:
                     self.assertEqual(len(v), shape[0])
                 else:
                     self.assertEqual(len(v) * sizeof(itemtp), sizeof(ob))
                 self.assertEqual(v.itemsize, sizeof(itemtp))
                 self.assertEqual(v.shape, shape)
-                # ctypes object always have a non-strided memory block
-                self.assertEqual(v.strides, None)
+                # XXX Issue #12851
+                # self.assertEqual(v.strides, ())
+
                 # they are always read/write
                 self.assertFalse(v.readonly)
 
@@ -110,34 +114,34 @@ native_types = [
 
     ## simple types
 
-    (c_char,                    "<c",                   None,           c_char),
-    (c_byte,                    "<b",                   None,           c_byte),
-    (c_ubyte,                   "<B",                   None,           c_ubyte),
-    (c_short,                   "<h",                   None,           c_short),
-    (c_ushort,                  "<H",                   None,           c_ushort),
+    (c_char,                    "<c",                   (),           c_char),
+    (c_byte,                    "<b",                   (),           c_byte),
+    (c_ubyte,                   "<B",                   (),           c_ubyte),
+    (c_short,                   "<h",                   (),           c_short),
+    (c_ushort,                  "<H",                   (),           c_ushort),
 
     # c_int and c_uint may be aliases to c_long
-    #(c_int,                     "<i",                   None,           c_int),
-    #(c_uint,                    "<I",                   None,           c_uint),
+    #(c_int,                     "<i",                   (),           c_int),
+    #(c_uint,                    "<I",                   (),           c_uint),
 
-    (c_long,                    "<l",                   None,           c_long),
-    (c_ulong,                   "<L",                   None,           c_ulong),
+    (c_long,                    "<l",                   (),           c_long),
+    (c_ulong,                   "<L",                   (),           c_ulong),
 
     # c_longlong and c_ulonglong are aliases on 64-bit platforms
     #(c_longlong,                "<q",                   None,           c_longlong),
     #(c_ulonglong,               "<Q",                   None,           c_ulonglong),
 
-    (c_float,                   "<f",                   None,           c_float),
-    (c_double,                  "<d",                   None,           c_double),
+    (c_float,                   "<f",                   (),           c_float),
+    (c_double,                  "<d",                   (),           c_double),
     # c_longdouble may be an alias to c_double
 
-    (c_bool,                    "<?",                   None,           c_bool),
-    (py_object,                 "<O",                   None,           py_object),
+    (c_bool,                    "<?",                   (),           c_bool),
+    (py_object,                 "<O",                   (),           py_object),
 
     ## pointers
 
-    (POINTER(c_byte),           "&<b",                  None,           POINTER(c_byte)),
-    (POINTER(POINTER(c_long)),  "&&<l",                 None,           POINTER(POINTER(c_long))),
+    (POINTER(c_byte),           "&<b",                  (),           POINTER(c_byte)),
+    (POINTER(POINTER(c_long)),  "&&<l",                 (),           POINTER(POINTER(c_long))),
 
     ## arrays and pointers
 
@@ -145,32 +149,32 @@ native_types = [
     (c_float * 4 * 3 * 2,       "(2,3,4)<f",            (2,3,4),        c_float),
     (POINTER(c_short) * 2,      "(2)&<h",               (2,),           POINTER(c_short)),
     (POINTER(c_short) * 2 * 3,  "(3,2)&<h",             (3,2,),         POINTER(c_short)),
-    (POINTER(c_short * 2),      "&(2)<h",               None,           POINTER(c_short)),
+    (POINTER(c_short * 2),      "&(2)<h",               (),           POINTER(c_short)),
 
     ## structures and unions
 
-    (Point,                     "T{<l:x:<l:y:}",        None,           Point),
+    (Point,                     "T{<l:x:<l:y:}",        (),           Point),
     # packed structures do not implement the pep
-    (PackedPoint,               "B",                    None,           PackedPoint),
-    (Point2,                    "T{<l:x:<l:y:}",        None,           Point2),
-    (EmptyStruct,               "T{}",                  None,           EmptyStruct),
+    (PackedPoint,               "B",                    (),           PackedPoint),
+    (Point2,                    "T{<l:x:<l:y:}",        (),           Point2),
+    (EmptyStruct,               "T{}",                  (),           EmptyStruct),
     # the pep does't support unions
-    (aUnion,                    "B",                    None,           aUnion),
+    (aUnion,                    "B",                    (),           aUnion),
 
     ## pointer to incomplete structure
-    (Incomplete,                "B",                    None,           Incomplete),
-    (POINTER(Incomplete),       "&B",                   None,           POINTER(Incomplete)),
+    (Incomplete,                "B",                    (),           Incomplete),
+    (POINTER(Incomplete),       "&B",                   (),           POINTER(Incomplete)),
 
     # 'Complete' is a structure that starts incomplete, but is completed after the
     # pointer type to it has been created.
-    (Complete,                  "T{<l:a:}",             None,           Complete),
+    (Complete,                  "T{<l:a:}",             (),           Complete),
     # Unfortunately the pointer format string is not fixed...
-    (POINTER(Complete),         "&B",                   None,           POINTER(Complete)),
+    (POINTER(Complete),         "&B",                   (),           POINTER(Complete)),
 
     ## other
 
     # function signatures are not implemented
-    (CFUNCTYPE(None),           "X{}",                  None,           CFUNCTYPE(None)),
+    (CFUNCTYPE(None),           "X{}",                  (),           CFUNCTYPE(None)),
 
     ]
 
@@ -186,10 +190,10 @@ class LEPoint(LittleEndianStructure):
 # and little endian machines.
 #
 endian_types = [
-    (BEPoint,                   "T{>l:x:>l:y:}",        None,           BEPoint),
-    (LEPoint,                   "T{<l:x:<l:y:}",        None,           LEPoint),
-    (POINTER(BEPoint),          "&T{>l:x:>l:y:}",       None,           POINTER(BEPoint)),
-    (POINTER(LEPoint),          "&T{<l:x:<l:y:}",       None,           POINTER(LEPoint)),
+    (BEPoint,                   "T{>l:x:>l:y:}",        (),           BEPoint),
+    (LEPoint,                   "T{<l:x:<l:y:}",        (),           LEPoint),
+    (POINTER(BEPoint),          "&T{>l:x:>l:y:}",       (),           POINTER(BEPoint)),
+    (POINTER(LEPoint),          "&T{<l:x:<l:y:}",       (),           POINTER(LEPoint)),
     ]
 
 if __name__ == "__main__":

Lib/test/test_memoryview.py

@@ -24,15 +24,14 @@ class AbstractMemoryTests:
         return filter(None, [self.ro_type, self.rw_type])
 
     def check_getitem_with_type(self, tp):
-        item = self.getitem_type
         b = tp(self._source)
         oldrefcount = sys.getrefcount(b)
         m = self._view(b)
-        self.assertEqual(m[0], item(b"a"))
-        self.assertIsInstance(m[0], bytes)
-        self.assertEqual(m[5], item(b"f"))
-        self.assertEqual(m[-1], item(b"f"))
-        self.assertEqual(m[-6], item(b"a"))
+        self.assertEqual(m[0], ord(b"a"))
+        self.assertIsInstance(m[0], int)
+        self.assertEqual(m[5], ord(b"f"))
+        self.assertEqual(m[-1], ord(b"f"))
+        self.assertEqual(m[-6], ord(b"a"))
         # Bounds checking
         self.assertRaises(IndexError, lambda: m[6])
         self.assertRaises(IndexError, lambda: m[-7])
@@ -76,7 +75,9 @@ class AbstractMemoryTests:
         b = self.rw_type(self._source)
         oldrefcount = sys.getrefcount(b)
         m = self._view(b)
-        m[0] = tp(b"0")
+        m[0] = ord(b'1')
+        self._check_contents(tp, b, b"1bcdef")
+        m[0:1] = tp(b"0")
         self._check_contents(tp, b, b"0bcdef")
         m[1:3] = tp(b"12")
         self._check_contents(tp, b, b"012def")
@@ -102,10 +103,17 @@ class AbstractMemoryTests:
         # Wrong index/slice types
         self.assertRaises(TypeError, setitem, 0.0, b"a")
         self.assertRaises(TypeError, setitem, (0,), b"a")
+        self.assertRaises(TypeError, setitem, (slice(0,1,1), 0), b"a")
+        self.assertRaises(TypeError, setitem, (0, slice(0,1,1)), b"a")
+        self.assertRaises(TypeError, setitem, (0,), b"a")
         self.assertRaises(TypeError, setitem, "a", b"a")
+        # Not implemented: multidimensional slices
+        slices = (slice(0,1,1), slice(0,1,2))
+        self.assertRaises(NotImplementedError, setitem, slices, b"a")
         # Trying to resize the memory object
-        self.assertRaises(ValueError, setitem, 0, b"")
-        self.assertRaises(ValueError, setitem, 0, b"ab")
+        exc = ValueError if m.format == 'c' else TypeError
+        self.assertRaises(exc, setitem, 0, b"")
+        self.assertRaises(exc, setitem, 0, b"ab")
         self.assertRaises(ValueError, setitem, slice(1,1), b"a")
         self.assertRaises(ValueError, setitem, slice(0,2), b"a")
 
@@ -175,7 +183,7 @@ class AbstractMemoryTests:
         self.assertEqual(m.shape, (6,))
         self.assertEqual(len(m), 6)
         self.assertEqual(m.strides, (self.itemsize,))
-        self.assertEqual(m.suboffsets, None)
+        self.assertEqual(m.suboffsets, ())
         return m
 
     def test_attributes_readonly(self):
@@ -209,12 +217,16 @@ class AbstractMemoryTests:
                 # If tp is a factory rather than a plain type, skip
                 continue
 
+            class MyView():
+                def __init__(self, base):
+                    self.m = memoryview(base)
             class MySource(tp):
                 pass
             class MyObject:
                 pass
 
-            # Create a reference cycle through a memoryview object
+            # Create a reference cycle through a memoryview object.
+            # This exercises mbuf_clear().
             b = MySource(tp(b'abc'))
             m = self._view(b)
             o = MyObject()
@@ -226,6 +238,17 @@ class AbstractMemoryTests:
             gc.collect()
             self.assertTrue(wr() is None, wr())
 
+            # This exercises memory_clear().
+            m = MyView(tp(b'abc'))
+            o = MyObject()
+            m.x = m
+            m.o = o
+            wr = weakref.ref(o)
+            m = o = None
+            # The cycle must be broken
+            gc.collect()
+            self.assertTrue(wr() is None, wr())
+
     def _check_released(self, m, tp):
         check = self.assertRaisesRegex(ValueError, "released")
         with check: bytes(m)
@@ -283,9 +306,12 @@ class AbstractMemoryTests:
         i = io.BytesIO(b'ZZZZ')
         self.assertRaises(TypeError, i.readinto, m)
 
+    def test_getbuf_fail(self):
+        self.assertRaises(TypeError, self._view, {})
+
     def test_hash(self):
         # Memoryviews of readonly (hashable) types are hashable, and they
-        # hash as the corresponding object.
+        # hash as hash(obj.tobytes()).
         tp = self.ro_type
         if tp is None:
             self.skipTest("no read-only type to test")

Lib/test/test_sys.py

@@ -773,8 +773,8 @@ class SizeofTest(unittest.TestCase):
         check(int(PyLong_BASE), size(vh) + 2*self.longdigit)
         check(int(PyLong_BASE**2-1), size(vh) + 2*self.longdigit)
         check(int(PyLong_BASE**2), size(vh) + 3*self.longdigit)
-        # memory (Py_buffer + hash value)
-        check(memoryview(b''), size(h + 'PP2P2i7P' + 'P'))
+        # memoryview
+        check(memoryview(b''), size(h + 'PPiP4P2i5P3cP'))
         # module
         check(unittest, size(h + '3P'))
         # None

Misc/ACKS

@@ -1041,6 +1041,7 @@ John Viega
 Kannan Vijayan
 Kurt Vile
 Norman Vine
+Pauli Virtanen
 Frank Visser
 Johannes Vogel
 Sjoerd de Vries

Misc/NEWS

@@ -10,6 +10,23 @@ What's New in Python 3.3 Alpha 1?
 Core and Builtins
 -----------------
 
+- Issue #10181: New memoryview implementation fixes multiple ownership
+  and lifetime issues of dynamically allocated Py_buffer members (#9990)
+  as well as crashes (#8305, #7433). Many new features have been added
+  (See whatsnew/3.3), and the documentation has been updated extensively.
+  The ndarray test object from _testbuffer.c implements all aspects of
+  PEP-3118, so further development towards the complete implementation
+  of the PEP can proceed in a test-driven manner.
+
+  Thanks to Nick Coghlan, Antoine Pitrou and Pauli Virtanen for review
+  and many ideas.
+
+- Issue #12834: Fix incorrect results of memoryview.tobytes() for
+  non-contiguous arrays.
+
+- Issue #5231: Introduce memoryview.cast() method that allows changing
+  format and shape without making a copy of the underlying memory.
+
 - Issue #14084: Fix a file descriptor leak when importing a module with a
   bad encoding.
 

Misc/valgrind-python.supp

@@ -412,4 +412,15 @@
    fun:SHA1_Update
 }
 
+{
+   test_buffer_non_debug
+   Memcheck:Addr4
+   fun:PyUnicodeUCS2_FSConverter
+}
+
+{
+   test_buffer_non_debug
+   Memcheck:Addr4
+   fun:PyUnicode_FSConverter
+}
 

Modules/_testcapimodule.c

@@ -275,95 +275,6 @@ test_lazy_hash_inheritance(PyObject* self)
 }
 
 
-/* Issue #7385: Check that memoryview() does not crash
- *   when bf_getbuffer returns an error
- */
-
-static int
-broken_buffer_getbuffer(PyObject *self, Py_buffer *view, int flags)
-{
-    PyErr_SetString(
-        TestError,
-        "test_broken_memoryview: expected error in bf_getbuffer");
-    return -1;
-}
-
-static PyBufferProcs memoryviewtester_as_buffer = {
-    (getbufferproc)broken_buffer_getbuffer,     /* bf_getbuffer */
-    0,          /* bf_releasebuffer */
-};
-
-static PyTypeObject _MemoryViewTester_Type = {
-    PyVarObject_HEAD_INIT(NULL, 0)
-    "memoryviewtester",         /* Name of this type */
-    sizeof(PyObject),           /* Basic object size */
-    0,                          /* Item size for varobject */
-    (destructor)PyObject_Del, /* tp_dealloc */
-    0,                          /* tp_print */
-    0,                          /* tp_getattr */
-    0,                          /* tp_setattr */
-    0,                          /* tp_compare */
-    0,                          /* tp_repr */
-    0,                          /* tp_as_number */
-    0,                          /* tp_as_sequence */
-    0,                          /* tp_as_mapping */
-    0,                          /* tp_hash */
-    0,                          /* tp_call */
-    0,                          /* tp_str */
-    PyObject_GenericGetAttr,  /* tp_getattro */
-    0,                          /* tp_setattro */
-    &memoryviewtester_as_buffer,                        /* tp_as_buffer */
-    Py_TPFLAGS_DEFAULT,         /* tp_flags */
-    0,                          /* tp_doc */
-    0,                          /* tp_traverse */
-    0,                          /* tp_clear */
-    0,                          /* tp_richcompare */
-    0,                          /* tp_weaklistoffset */
-    0,                          /* tp_iter */
-    0,                          /* tp_iternext */
-    0,                          /* tp_methods */
-    0,                          /* tp_members */
-    0,                          /* tp_getset */
-    0,                          /* tp_base */
-    0,                          /* tp_dict */
-    0,                          /* tp_descr_get */
-    0,                          /* tp_descr_set */
-    0,                          /* tp_dictoffset */
-    0,                          /* tp_init */
-    0,                          /* tp_alloc */
-    PyType_GenericNew,                  /* tp_new */
-};
-
-static PyObject*
-test_broken_memoryview(PyObject* self)
-{
-    PyObject *obj = PyObject_New(PyObject, &_MemoryViewTester_Type);
-    PyObject *res;
-
-    if (obj == NULL) {
-        PyErr_Clear();
-        PyErr_SetString(
-            TestError,
-            "test_broken_memoryview: failed to create object");
-        return NULL;
-    }
-
-    res = PyMemoryView_FromObject(obj);
-    if (res || !PyErr_Occurred()){
-        PyErr_SetString(
-            TestError,
-            "test_broken_memoryview: memoryview() didn't raise an Exception");
-        Py_XDECREF(res);
-        Py_DECREF(obj);
-        return NULL;
-    }
-
-    PyErr_Clear();
-    Py_DECREF(obj);
-    Py_RETURN_NONE;
-}
-
-
 /* Tests of PyLong_{As, From}{Unsigned,}Long(), and (#ifdef HAVE_LONG_LONG)
    PyLong_{As, From}{Unsigned,}LongLong().
 
@@ -2421,7 +2332,6 @@ static PyMethodDef TestMethods[] = {
     {"test_list_api",           (PyCFunction)test_list_api,      METH_NOARGS},
     {"test_dict_iteration",     (PyCFunction)test_dict_iteration,METH_NOARGS},
     {"test_lazy_hash_inheritance",      (PyCFunction)test_lazy_hash_inheritance,METH_NOARGS},
-    {"test_broken_memoryview",          (PyCFunction)test_broken_memoryview,METH_NOARGS},
     {"test_long_api",           (PyCFunction)test_long_api,      METH_NOARGS},
     {"test_long_and_overflow", (PyCFunction)test_long_and_overflow,
      METH_NOARGS},
@@ -2684,7 +2594,6 @@ PyInit__testcapi(void)
         return NULL;
 
     Py_TYPE(&_HashInheritanceTester_Type)=&PyType_Type;
-    Py_TYPE(&_MemoryViewTester_Type)=&PyType_Type;
 
     Py_TYPE(&test_structmembersType)=&PyType_Type;
     Py_INCREF(&test_structmembersType);

Objects/abstract.c

@@ -340,7 +340,7 @@ PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags)
 }
 
 static int
-_IsFortranContiguous(Py_buffer *view)
+_IsFortranContiguous(const Py_buffer *view)
 {
     Py_ssize_t sd, dim;
     int i;
@@ -361,7 +361,7 @@ _IsFortranContiguous(Py_buffer *view)
 }
 
 static int
-_IsCContiguous(Py_buffer *view)
+_IsCContiguous(const Py_buffer *view)
 {
     Py_ssize_t sd, dim;
     int i;
@@ -382,16 +382,16 @@ _IsCContiguous(Py_buffer *view)
 }
 
 int
-PyBuffer_IsContiguous(Py_buffer *view, char fort)
+PyBuffer_IsContiguous(const Py_buffer *view, char order)
 {
 
     if (view->suboffsets != NULL) return 0;
 
-    if (fort == 'C')
+    if (order == 'C')
         return _IsCContiguous(view);
-    else if (fort == 'F')
+    else if (order == 'F')
         return _IsFortranContiguous(view);
-    else if (fort == 'A')
+    else if (order == 'A')
         return (_IsCContiguous(view) || _IsFortranContiguous(view));
     return 0;
 }
@@ -651,7 +651,7 @@ int
 PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len,
               int readonly, int flags)
 {
-    if (view == NULL) return 0;
+    if (view == NULL) return 0; /* XXX why not -1? */
     if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) &&
         (readonly == 1)) {
         PyErr_SetString(PyExc_BufferError,

Objects/object.c

@@ -1650,6 +1650,9 @@ _Py_ReadyTypes(void)
     if (PyType_Ready(&PyProperty_Type) < 0)
         Py_FatalError("Can't initialize property type");
 
+    if (PyType_Ready(&_PyManagedBuffer_Type) < 0)
+        Py_FatalError("Can't initialize managed buffer type");
+
     if (PyType_Ready(&PyMemoryView_Type) < 0)
         Py_FatalError("Can't initialize memoryview type");
 

PCbuild/_testbuffer.vcproj

+<?xml version="1.0" encoding="Windows-1252"?>
+<VisualStudioProject
+	ProjectType="Visual C++"
+	Version="9.00"
+	Name="_testbuffer"
+	ProjectGUID="{A2697BD3-28C1-4AEC-9106-8B748639FD16}"
+	RootNamespace="_testbuffer"
+	Keyword="Win32Proj"
+	TargetFrameworkVersion="196613"
+	>
+	<Platforms>
+		<Platform
+			Name="Win32"
+		/>
+		<Platform
+			Name="x64"
+		/>
+	</Platforms>
+	<ToolFiles>
+	</ToolFiles>
+	<Configurations>
+		<Configuration
+			Name="Debug|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd_d.vsprops"
+			CharacterSet="0"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Debug|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd_d.vsprops;.\x64.vsprops"
+			CharacterSet="0"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Release|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Release|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGInstrument|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\pginstrument.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGInstrument|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pginstrument.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+				TargetMachine="17"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGUpdate|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\pgupdate.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGUpdate|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pgupdate.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				BaseAddress="0x1e1F0000"
+				TargetMachine="17"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+	</Configurations>
+	<References>
+	</References>
+	<Files>
+		<Filter
+			Name="Source Files"
+			>
+			<File
+				RelativePath="..\Modules\_testbuffer.c"
+				>
+			</File>
+		</Filter>
+	</Files>
+	<Globals>
+	</Globals>
+</VisualStudioProject>

PCbuild/readme.txt

@@ -92,6 +92,9 @@ _socket
 _testcapi
     tests of the Python C API, run via Lib/test/test_capi.py, and
     implemented by module Modules/_testcapimodule.c
+_testbuffer
+    buffer protocol tests, run via Lib/test/test_buffer.py, and
+    implemented by module Modules/_testbuffer.c
 pyexpat
     Python wrapper for accelerated XML parsing, which incorporates stable
     code from the Expat project:  http://sourceforge.net/projects/expat/

setup.py

@@ -530,6 +530,8 @@ class PyBuildExt(build_ext):
         # Python C API test module
         exts.append( Extension('_testcapi', ['_testcapimodule.c'],
                                depends=['testcapi_long.h']) )
+        # Python PEP-3118 (buffer protocol) test module
+        exts.append( Extension('_testbuffer', ['_testbuffer.c']) )
         # profiler (_lsprof is for cProfile.py)
         exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) )
         # static Unicode character database