Issue #20699: Document that “io” methods should accept memoryview

This matches the usage by BufferedReader, BufferedWriter, etc. Also document
and test that the write() methods should only access their argument before
they return.

Doc/library/io.rst

@@ -225,10 +225,15 @@ I/O Base Classes
    support are called.
 
    The basic type used for binary data read from or written to a file is
-   :class:`bytes` (also known as :class:`str`).  :class:`bytearray`\s are
-   accepted too, and in some cases (such as :class:`readinto`) required.
+   :class:`bytes` (also known as :class:`str`).  Method arguments may
+   also be :class:`bytearray` or :class:`memoryview` of arrays of bytes.
+   In some cases, such as :meth:`~RawIOBase.readinto`, a writable object
+   such as :class:`bytearray` is required.
    Text I/O classes work with :class:`unicode` data.
 
+   .. versionchanged:: 2.7
+      Implementations should support :class:`memoryview` arguments.
+
    Note that calling any method (even inquiries) on a closed stream is
    undefined.  Implementations may raise :exc:`IOError` in this case.
 
@@ -383,18 +388,24 @@ I/O Base Classes
 
    .. method:: readinto(b)
 
-      Read up to len(b) bytes into bytearray *b* and return the number
-      of bytes read.  If the object is in non-blocking mode and no
+      Read up to len(b) bytes into *b*, and return the number
+      of bytes read.  The object *b* should be a pre-allocated, writable
+      array of bytes, either :class:`bytearray` or :class:`memoryview`.
+      If the object is in non-blocking mode and no
       bytes are available, ``None`` is returned.
 
    .. method:: write(b)
 
-      Write the given bytes or bytearray object, *b*, to the underlying raw
-      stream and return the number of bytes written.  This can be less than
+      Write *b* to the underlying raw stream, and return the
+      number of bytes written.  The object *b* should be an array
+      of bytes, either :class:`bytes`, :class:`bytearray`, or
+      :class:`memoryview`.  The return value can be less than
       ``len(b)``, depending on specifics of the underlying raw stream, and
       especially if it is in non-blocking mode.  ``None`` is returned if the
       raw stream is set not to block and no single byte could be readily
-      written to it.
+      written to it.  The caller may release or mutate *b* after
+      this method returns, so the implementation should only access *b*
+      during the method call.
 
 
 .. class:: BufferedIOBase
@@ -465,8 +476,9 @@ I/O Base Classes
 
    .. method:: readinto(b)
 
-      Read up to len(b) bytes into bytearray *b* and return the number of bytes
-      read.
+      Read up to len(b) bytes into *b*, and return the number of bytes read.
+      The object *b* should be a pre-allocated, writable array of bytes,
+      either :class:`bytearray` or :class:`memoryview`.
 
       Like :meth:`read`, multiple reads may be issued to the underlying raw
       stream, unless the latter is 'interactive'.
@@ -476,9 +488,11 @@ I/O Base Classes
 
    .. method:: write(b)
 
-      Write the given bytes or bytearray object, *b* and return the number
-      of bytes written (never less than ``len(b)``, since if the write fails
-      an :exc:`IOError` will be raised).  Depending on the actual
+      Write *b*, and return the number of bytes written
+      (always equal to ``len(b)``, since if the write fails
+      an :exc:`IOError` will be raised).  The object *b* should be
+      an array of bytes, either :class:`bytes`, :class:`bytearray`,
+      or :class:`memoryview`.  Depending on the actual
       implementation, these bytes may be readily written to the underlying
       stream, or held in a buffer for performance and latency reasons.
 
@@ -486,6 +500,9 @@ I/O Base Classes
       data needed to be written to the raw stream but it couldn't accept
       all the data without blocking.
 
+      The caller may release or mutate *b* after this method returns,
+      so the implementation should only access *b* during the method call.
+
 
 Raw File I/O
 ------------
@@ -535,7 +552,8 @@ than raw I/O does.
    A stream implementation using an in-memory bytes buffer.  It inherits
    :class:`BufferedIOBase`.
 
-   The argument *initial_bytes* is an optional initial :class:`bytes`.
+   The optional argument *initial_bytes* is a :class:`bytes` object that
+   contains initial data.
 
    :class:`BytesIO` provides or overrides these methods in addition to those
    from :class:`BufferedIOBase` and :class:`IOBase`:
@@ -611,8 +629,10 @@ than raw I/O does.
 
    .. method:: write(b)
 
-      Write the bytes or bytearray object, *b* and return the number of bytes
-      written.  When in non-blocking mode, a :exc:`BlockingIOError` is raised
+      Write *b*, and return the number of bytes written.
+      The object *b* should be an array of bytes, either
+      :class:`bytes`, :class:`bytearray`, or :class:`memoryview`.
+      When in non-blocking mode, a :exc:`BlockingIOError` is raised
       if the buffer needs to be written out but the raw stream blocks.
 
 

Lib/_pyio.py

@@ -277,8 +277,9 @@ class IOBase:
     may raise a IOError when operations they do not support are called.
 
     The basic type used for binary data read from or written to a file is
-    bytes. bytearrays are accepted too, and in some cases (such as
-    readinto) needed. Text I/O classes work with str data.
+    the bytes type. Method arguments may also be bytearray or memoryview of
+    arrays of bytes. In some cases, such as readinto, a writable object such
+    as bytearray is required. Text I/O classes work with unicode data.
 
     Note that calling any method (even inquiries) on a closed stream is
     undefined. Implementations may raise IOError in this case.
@@ -649,7 +650,6 @@ class BufferedIOBase(IOBase):
         Raises BlockingIOError if the underlying raw stream has no
         data at the moment.
         """
-        # XXX This ought to work with anything that supports the buffer API
         data = self.read(len(b))
         n = len(data)
         try:
@@ -664,8 +664,7 @@ class BufferedIOBase(IOBase):
     def write(self, b):
         """Write the given buffer to the IO stream.
 
-        Return the number of bytes written, which is never less than
-        len(b).
+        Return the number of bytes written, which is always len(b).
 
         Raises BlockingIOError if the buffer is full and the
         underlying raw stream cannot accept more data at the moment.

Lib/test/test_io.py

@@ -54,6 +54,9 @@ except ImportError:
 __metaclass__ = type
 bytes = support.py3k_bytes
 
+def byteslike(*pos, **kw):
+    return memoryview(bytearray(*pos, **kw))
+
 def _default_chunk_size():
     """Get the default TextIOWrapper chunk size"""
     with io.open(__file__, "r", encoding="latin1") as f:
@@ -273,7 +276,9 @@ class IOTest(unittest.TestCase):
         self.assertEqual(f.tell(), 6)
         self.assertEqual(f.seek(-1, 1), 5)
         self.assertEqual(f.tell(), 5)
-        self.assertEqual(f.write(bytearray(b" world\n\n\n")), 9)
+        buffer = bytearray(b" world\n\n\n")
+        self.assertEqual(f.write(buffer), 9)
+        buffer[:] = b"*" * 9  # Overwrite our copy of the data
         self.assertEqual(f.seek(0), 0)
         self.assertEqual(f.write(b"h"), 1)
         self.assertEqual(f.seek(-1, 2), 13)
@@ -286,20 +291,21 @@ class IOTest(unittest.TestCase):
     def read_ops(self, f, buffered=False):
         data = f.read(5)
         self.assertEqual(data, b"hello")
-        data = bytearray(data)
+        data = byteslike(data)
         self.assertEqual(f.readinto(data), 5)
-        self.assertEqual(data, b" worl")
+        self.assertEqual(data.tobytes(), b" worl")
+        data = bytearray(5)
         self.assertEqual(f.readinto(data), 2)
         self.assertEqual(len(data), 5)
         self.assertEqual(data[:2], b"d\n")
         self.assertEqual(f.seek(0), 0)
         self.assertEqual(f.read(20), b"hello world\n")
         self.assertEqual(f.read(1), b"")
-        self.assertEqual(f.readinto(bytearray(b"x")), 0)
+        self.assertEqual(f.readinto(byteslike(b"x")), 0)
         self.assertEqual(f.seek(-6, 2), 6)
         self.assertEqual(f.read(5), b"world")
         self.assertEqual(f.read(0), b"")
-        self.assertEqual(f.readinto(bytearray()), 0)
+        self.assertEqual(f.readinto(byteslike()), 0)
         self.assertEqual(f.seek(-6, 1), 5)
         self.assertEqual(f.read(5), b" worl")
         self.assertEqual(f.tell(), 10)
@@ -649,6 +655,16 @@ class IOTest(unittest.TestCase):
             support.gc_collect()
         self.assertEqual(recorded, [])
 
+    def test_buffered_readinto_mixin(self):
+        # Test the implementation provided by BufferedIOBase
+        class Stream(self.BufferedIOBase):
+            def read(self, size):
+                return b"12345"
+        stream = Stream()
+        buffer = byteslike(5)
+        self.assertEqual(stream.readinto(buffer), 5)
+        self.assertEqual(buffer.tobytes(), b"12345")
+
 
 class CIOTest(IOTest):
 
@@ -1111,6 +1127,11 @@ class BufferedWriterTest(unittest.TestCase, CommonBufferedTests):
         bufio = self.tp(writer, 8)
         bufio.write(b"abc")
         self.assertFalse(writer._write_stack)
+        buffer = bytearray(b"def")
+        bufio.write(buffer)
+        buffer[:] = b"***"  # Overwrite our copy of the data
+        bufio.flush()
+        self.assertEqual(b"".join(writer._write_stack), b"abcdef")
 
     def test_write_overflow(self):
         writer = self.MockRawIO()
@@ -1440,9 +1461,9 @@ class BufferedRWPairTest(unittest.TestCase):
     def test_readinto(self):
         pair = self.tp(self.BytesIO(b"abcdef"), self.MockRawIO())
 
-        data = bytearray(5)
+        data = byteslike(5)
         self.assertEqual(pair.readinto(data), 5)
-        self.assertEqual(data, b"abcde")
+        self.assertEqual(data.tobytes(), b"abcde")
 
     def test_write(self):
         w = self.MockRawIO()
@@ -1450,7 +1471,9 @@ class BufferedRWPairTest(unittest.TestCase):
 
         pair.write(b"abc")
         pair.flush()
-        pair.write(b"def")
+        buffer = bytearray(b"def")
+        pair.write(buffer)
+        buffer[:] = b"***"  # Overwrite our copy of the data
         pair.flush()
         self.assertEqual(w._write_stack, [b"abc", b"def"])
 

Lib/test/test_memoryio.py

@@ -396,6 +396,7 @@ class MemoryTestMixin:
 
 
 class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
+    # Test _pyio.BytesIO; class also inherited for testing C implementation
 
     UnsupportedOperation = pyio.UnsupportedOperation
 

Modules/_io/bufferedio.c

@@ -125,8 +125,7 @@ bufferediobase_read1(PyObject *self, PyObject *args)
 PyDoc_STRVAR(bufferediobase_write_doc,
     "Write the given buffer to the IO stream.\n"
     "\n"
-    "Returns the number of bytes written, which is never less than\n"
-    "len(b).\n"
+    "Returns the number of bytes written, which is always len(b).\n"
     "\n"
     "Raises BlockingIOError if the buffer is full and the\n"
     "underlying raw stream cannot accept more data at the moment.\n");

Modules/_io/bytesio.c

@@ -392,7 +392,7 @@ bytesio_readlines(bytesio *self, PyObject *args)
 }
 
 PyDoc_STRVAR(readinto_doc,
-"readinto(bytearray) -> int.  Read up to len(b) bytes into b.\n"
+"readinto(b) -> int.  Read up to len(b) bytes into b.\n"
 "\n"
 "Returns number of bytes read (0 for EOF), or None if the object\n"
 "is set not to block and has no data to read.");

Modules/_io/fileio.c

@@ -969,7 +969,7 @@ PyDoc_STRVAR(readall_doc,
 "or None if no data is available.  On end-of-file, returns ''.");
 
 PyDoc_STRVAR(write_doc,
-"write(b: bytes) -> int.  Write bytes b to file, return number written.\n"
+"write(b) -> int.  Write array of bytes b, return number written.\n"
 "\n"
 "Only makes one system call, so not all of the data may be written.\n"
 "The number of bytes actually written is returned.  In non-blocking mode,\n"

Modules/_io/iobase.c

@@ -38,8 +38,10 @@ PyDoc_STRVAR(iobase_doc,
     "may raise a IOError when operations they do not support are called.\n"
     "\n"
     "The basic type used for binary data read from or written to a file is\n"
-    "bytes. bytearrays are accepted too, and in some cases (such as\n"
-    "readinto) needed. Text I/O classes work with str data.\n"
+    "the bytes type. Method arguments may also be bytearray or memoryview\n"
+    "of arrays of bytes. In some cases, such as readinto, a writable\n"
+    "object such as bytearray is required. Text I/O classes work with\n"
+    "unicode data.\n"
     "\n"
     "Note that calling any method (except additional calls to close(),\n"
     "which are ignored) on a closed stream should raise a ValueError.\n"