Commit be5a9d54 authored by Jason Madden's avatar Jason Madden Committed by Jim Fulton

Use a higher pickle protocol for serializing objects on Python 2 (#179)

* Use a higher pickle protocol (2) for serializing objects on Python 2

Previously protocol 1 was used. This is more efficient for new-style
classes (all persistent objects are new-style), according to the docs,
at the cost of being very slightly less space efficient for old-style
classes.

In tests of a persistent object with two trivial numeric attributes,
the higher protocol was 12 bytes smaller, and serialized and
deserialized 1us faster. Introducing a reference to another new-style
class for a more realistic test made the higher protocol twice as fast
to serialize (20.5 vs 10.3us), almost half the size (215 vs 142
bytes), and it deserialized 30% faster (6.5 vs 4.6us).

On Python 2, this will now allow open ``file`` objects to be
pickled (loading the object will result in a closed file); previously
this would result in a ``TypeError`` (as does under Python 3). We had
tests that you couldn't do that with a BlobFile so I had to update it
to still make that true.

I wouldn't recommend serializing arbitrary open files under Python
2 (for one thing, they can't trivially be deserialized in Python 3),
but I didn't take any steps to prevent it either. Since this hasn't
been possible, there shouldn't be code in the wild that is trying to
do it---and it wouldn't be forward compatible with Python 3 either.
parent b340b651
...@@ -12,6 +12,19 @@ ...@@ -12,6 +12,19 @@
- Ensure that the ``HistoricalStorageAdapter`` forwards the ``release`` method to - Ensure that the ``HistoricalStorageAdapter`` forwards the ``release`` method to
its base instance. See `issue 78 <https://github.com/zopefoundation/ZODB/issues/788>`_. its base instance. See `issue 78 <https://github.com/zopefoundation/ZODB/issues/788>`_.
- Use a higher pickle protocol (2) for serializing objects on Python
2; previously protocol 1 was used. This is *much* more efficient for
new-style classes (all persistent objects are new-style), at the
cost of being very slightly less efficient for old-style classes.
.. note:: On Python 2, this will now allow open ``file`` objects
(but **not** open blobs or sockets) to be pickled (loading
the object will result in a closed file); previously this
would result in a ``TypeError``. Doing so is not
recommended as they cannot be loaded in Python 3.
See `issue 179 <https://github.com/zopefoundation/ZODB/pull/179>`_.
5.2.4 (2017-05-17) 5.2.4 (2017-05-17)
================== ==================
......
...@@ -34,7 +34,7 @@ if not PY3: ...@@ -34,7 +34,7 @@ if not PY3:
HIGHEST_PROTOCOL = cPickle.HIGHEST_PROTOCOL HIGHEST_PROTOCOL = cPickle.HIGHEST_PROTOCOL
IMPORT_MAPPING = {} IMPORT_MAPPING = {}
NAME_MAPPING = {} NAME_MAPPING = {}
_protocol = 1 _protocol = 2
FILESTORAGE_MAGIC = b"FS21" FILESTORAGE_MAGIC = b"FS21"
else: else:
# Python 3.x: can't use stdlib's pickle because # Python 3.x: can't use stdlib's pickle because
......
...@@ -338,6 +338,16 @@ class BlobFile(file): ...@@ -338,6 +338,16 @@ class BlobFile(file):
self.blob.closed(self) self.blob.closed(self)
super(BlobFile, self).close() super(BlobFile, self).close()
def __reduce__(self):
# Python 3 cannot pickle an open file with any pickle protocol
# because of the underlying _io.BufferedReader/Writer object.
# Python 2 cannot pickle a file with a protocol < 2, but
# protocol 2 *can* pickle an open file; the result of unpickling
# is a closed file object.
# It's pointless to do that with a blob, so we make sure to
# prohibit it on all versions.
raise TypeError("Pickling a BlobFile is not allowed")
_pid = str(os.getpid()) _pid = str(os.getpid())
def log(msg, level=logging.INFO, subsys=_pid, exc_info=False): def log(msg, level=logging.INFO, subsys=_pid, exc_info=False):
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment