Convert unittest tests into a single doctest file.

I've included some code to convert a text file containing doctest
tests into a unittest.TestCase.  This code wants to be generic, but
I'm not sure if it's simple enough yet.

src/persistent/tests/persistent.txt

+Tests for persistent.Persistent
+===============================
+
+This document is an extended doc test that covers the basics of the
+Persistent base class.  It depends on a few base classes defined in
+test_persistent.py.
+
+Test Persistent without Data Manager
+------------------------------------
+
+First do some simple tests of a Persistent instance that does not have
+a data manager (_p_jar).
+
+>>> p = P()
+>>> p.x
+0
+>>> p._p_changed
+False
+>>> p._p_state
+0
+>>> p._p_jar
+>>> p._p_oid
+
+Verify that modifications have no effect on _p_state of _p_changed.
+
+>>> p.inc()
+>>> p.inc()
+>>> p.x
+2
+>>> p._p_changed
+False
+>>> p._p_state
+0
+
+Try all sorts of different ways to change the object's state.
+
+>>> p._p_deactivate()
+>>> p._p_state
+0
+>>> p._p_changed = True
+>>> p._p_state
+0
+>>> del p._p_changed
+>>> p._p_changed
+False
+>>> p._p_state
+0
+>>> p.x
+2
+
+Test Persistent with Data Manager
+---------------------------------
+
+Next try some tests of an object with a data manager.  The DM class is
+a simple testing stub.
+    
+>>> p = P()
+>>> dm = DM()
+>>> p._p_oid = oid
+>>> p._p_jar = dm
+>>> p._p_changed
+0
+>>> dm.called
+0
+
+Modifying the object marks it as changed and registers it with the
+data manager.  Subsequent modifications don't have additional
+side-effects.
+
+>>> p.inc()
+>>> p._p_changed
+1
+>>> dm.called
+1
+>>> p.inc()
+>>> p._p_changed
+1
+>>> dm.called
+1
+
+It's not possible to deactivate a modified object.
+
+>>> p._p_deactivate()
+>>> p._p_changed
+1
+
+It is possible to invalidate it.  That's the key difference
+between deactivation and invalidation.
+
+>>> p._p_invalidate()
+>>> p._p_state
+-1
+
+Now that the object is a ghost, any attempt to modify it will
+require that it be unghosted first.  The test data manager
+has the odd property that it sets the object's 'x' attribute
+to 42 when it is unghosted.
+
+>>> p.inc()
+>>> p.x
+43
+>>> dm.called
+2
+
+You can manually reset the changed field to False, although
+it's not clear why you would want to do that.  The object
+changes to the UPTODATE state but retains its modifications.
+
+>>> p._p_changed = False
+>>> p._p_state
+0
+>>> p._p_changed
+False
+>>> p.x
+43
+
+>>> p.inc()
+>>> p._p_changed
+True
+>>> dm.called
+3
+
+__getstate__() and __setstate__()
+---------------------------------
+
+The next several tests cover the __getstate__() and __setstate__()
+implementations.
+
+>>> p = P()
+>>> p.__getstate__()
+{'x': 0}
+>>> p._p_state
+0
+
+Calling setstate always leaves the object in the uptodate state?
+(I'm not entirely clear on this one.)
+
+>>> p.__setstate__({'x': 5})
+>>> p._p_state
+0
+
+Assigning to a volatile attribute has no effect on the object state.
+
+>>> p._v_foo = 2
+>>> p.__getstate__()
+{'x': 5}
+>>> p._p_state
+0
+
+The _p_serial attribute is not affected by calling setstate.
+
+>>> p._p_serial = "00000012"
+>>> p.__setstate__(p.__getstate__())
+>>> p._p_serial
+'00000012'
+
+Change Ghost test
+-----------------
+
+If an object is a ghost and it's _p_changed is set to True, it should
+have no effect.
+
+>>> p = P()
+>>> p._p_jar = DM()
+>>> p._p_oid = 1
+>>> p._p_deactivate()
+>>> p._p_changed
+>>> p._p_state
+-1
+>>> p._p_changed = True
+>>> p._p_changed
+>>> p._p_state
+-1
+
+Activate, deactivate, and invalidate
+------------------------------------
+
+Some of these tests are redundant, but are included to make sure there
+are explicit and simple tests of _p_activate(), _p_deactivate(), and
+_p_invalidate().
+
+>>> p = P()
+>>> p._p_oid = 1
+>>> p._p_jar = DM()
+>>> p._p_deactivate()
+>>> p._p_state
+-1
+>>> p._p_activate()
+>>> p._p_state
+0
+>>> p.x
+42
+>>> p.inc()
+>>> p.x
+43
+>>> p._p_state
+1
+>>> p._p_invalidate()
+>>> p._p_state
+-1
+>>> p.x
+42
+
+Test failures
+-------------
+
+The following tests cover various errors cases.
+
+When an object is modified, it registers with its data manager.  If
+that registration fails, the exception is propagated and the object
+stays in the up-to-date state.  It shouldn't change to the modified
+state, because it won't be saved when the transaction commits.
+
+>>> p = P()
+>>> p._p_oid = 1
+>>> p._p_jar = BrokenDM()
+>>> p._p_state
+0
+>>> p._p_jar.called
+0
+>>> p._p_changed = 1
+Traceback (most recent call last):
+  ...
+NotImplementedError
+>>> p._p_jar.called
+1
+>>> p._p_state
+0
+
+Make sure that exceptions that occur inside the data manager's
+setstate() method propagate out to the caller.
+
+>>> p = P()
+>>> p._p_oid = 1
+>>> p._p_jar = BrokenDM()
+>>> p._p_deactivate()
+>>> p._p_state
+-1
+>>> p._p_activate()
+Traceback (most recent call last):
+  ...
+NotImplementedError
+>>> p._p_state
+-1
+
+
+Special test to cover layout of __dict__
+----------------------------------------
+
+We once had a bug in the Persistent class that calculated an incorrect
+offset for the __dict__ attribute.  It assigned __dict__ and _p_jar to
+the same location in memory.  This is a simple test to make sure they
+have different locations.
+
+>>> p = P()
+>>> p.inc()
+>>> p.inc()
+>>> p.__dict__
+{'x': 2}
+>>> p._p_jar
+
+
+Inheritance and metaclasses
+---------------------------
+
+Simple tests to make sure it's possible to inherit from the Persistent
+base class multiple times.  There used to be metaclasses involved in
+Persistent that probably made this a more interesting test.
+
+>>> class A(Persistent):
+...     pass
+>>> class B(Persistent):
+...     pass
+>>> class C(A, B):
+...     pass
+>>> class D(object):
+...     pass
+>>> class E(D, B):
+...     pass
+>>> a = A()
+>>> b = B()
+>>> c = C()
+>>> d = D()
+>>> e = E()
+
+Also make sure that it's possible to define Persistent classes that
+have a custom metaclass.
+
+>>> class alternateMeta(type):
+...     type
+>>> class alternate(object):
+...     __metaclass__ = alternateMeta
+>>> class mixedMeta(alternateMeta, type):
+...     pass
+>>> class mixed(alternate, Persistent):
+...     pass
+>>> class mixed(Persistent, alternate):
+...     pass
+
+
+Basic type structure
+--------------------
+
+>>> Persistent.__dictoffset__ 
+0
+>>> Persistent.__weakrefoffset__
+0
+>>> Persistent.__basicsize__ > object.__basicsize__
+True
+>>> P.__dictoffset__ > 0
+True
+>>> P.__weakrefoffset__ > 0
+True
+>>> P.__dictoffset__ < P.__weakrefoffset__
+True
+>>> P.__basicsize__ > Persistent.__basicsize__
+True
+
+
+Slots
+-----
+
+These are some simple tests of classes that have an __slots__
+attribute.  Some of the classes should have slots, others shouldn't.
+
+>>> class noDict(object):
+...     __slots__ = ['foo']
+>>> class p_noDict(Persistent):
+...     __slots__ = ['foo']
+>>> class p_shouldHaveDict(p_noDict):
+...     pass
+
+>>> p_noDict.__dictoffset__
+0
+>>> x = p_noDict()
+>>> x.foo = 1
+>>> x.foo
+1
+>>> x.bar = 1
+Traceback (most recent call last):
+  ...
+AttributeError: 'p_noDict' object has no attribute 'bar'
+>>> x._v_bar = 1
+Traceback (most recent call last):
+  ...
+AttributeError: 'p_noDict' object has no attribute '_v_bar'
+>>> x.__dict__
+Traceback (most recent call last):
+  ...
+AttributeError: 'p_noDict' object has no attribute '__dict__'
+
+The various _p_ attributes are unaffected by slots.
+>>> p._p_oid
+>>> p._p_jar
+>>> p._p_state
+0
+
+If the most-derived class does not specify 
+
+>>> p_shouldHaveDict.__dictoffset__ > 0
+True
+>>> x = p_shouldHaveDict()
+>>> x.__dict__
+{}
+
+
+Pickling
+--------
+
+There's actually a substantial effort involved in making subclasses of
+Persistent work with plain-old pickle.  The ZODB serialization layer
+never calls pickle on an object; it pickles the object's class
+description and its state as two separate pickles.
+
+>>> import pickle
+>>> p = P()
+>>> p.inc()
+>>> p2 = pickle.loads(pickle.dumps(p))
+>>> p2.__class__ is P
+True
+>>> p2.x == p.x
+True
+
+The P2 class has a custom __getstate__ and __setstate__.
+
+>>> p = P2()
+>>> p2 = pickle.loads(pickle.dumps(p))
+>>> p2.__class__ is P2
+True
+>>> p2.__dict__
+{'v': 42}

src/persistent/tests/test_persistent.py

@@ -11,10 +11,15 @@
 # FOR A PARTICULAR PURPOSE.
 #
 ##############################################################################
+import doctest
+import new
+import os
+import sys
 import unittest
 
 from persistent import Persistent
 from persistent.interfaces import IPersistent
+import persistent.tests
 
 try:
     import zope.interface
@@ -23,269 +28,7 @@ except ImportError:
 else:
     interfaces = True
 
-class Test(unittest.TestCase):
-
-    klass = None # override in subclass
-
-    def testSaved(self):
-        p = self.klass()
-        p._p_oid = '\0\0\0\0\0\0hi'
-        dm = DM()
-        p._p_jar = dm
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(dm.called, 0)
-        p.inc()
-        self.assertEqual(p._p_changed, 1)
-        self.assertEqual(dm.called, 1)
-        p.inc()
-        self.assertEqual(p._p_changed, 1)
-        self.assertEqual(dm.called, 1)
-        p._p_deactivate()
-        self.assertEqual(p._p_changed, 1)
-        self.assertEqual(dm.called, 1)
-        p._p_deactivate()
-        self.assertEqual(p._p_changed, 1)
-        self.assertEqual(dm.called, 1)
-        del p._p_changed
-        # XXX deal with current cPersistence implementation
-        if p._p_changed != 3:
-            self.assertEqual(p._p_changed, None)
-        self.assertEqual(dm.called, 1)
-        p.inc()
-        self.assertEqual(p.x, 43)
-        self.assertEqual(p._p_changed, 1)
-        self.assertEqual(dm.called, 2)
-        p._p_changed = 0
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(dm.called, 2)
-        self.assertEqual(p.x, 43)
-        p.inc()
-        self.assertEqual(p._p_changed, 1)
-        self.assertEqual(dm.called, 3)
-
-    def testUnsaved(self):
-        p = self.klass()
-
-        self.assertEqual(p.x, 0)
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(p._p_jar, None)
-        self.assertEqual(p._p_oid, None)
-        p.inc()
-        p.inc()
-        self.assertEqual(p.x, 2)
-        self.assertEqual(p._p_changed, 0)
-
-        p._p_deactivate()
-        self.assertEqual(p._p_changed, 0)
-        p._p_changed = 1
-        self.assertEqual(p._p_changed, 0)
-        p._p_deactivate()
-        self.assertEqual(p._p_changed, 0)
-        del p._p_changed
-        self.assertEqual(p._p_changed, 0)
-        if self.has_dict:
-            self.failUnless(p.__dict__)
-        self.assertEqual(p.x, 2)
-
-    def testState(self):
-        p = self.klass()
-        self.assertEqual(p.__getstate__(), {'x': 0})
-        self.assertEqual(p._p_changed, 0)
-        p.__setstate__({'x':5})
-        self.assertEqual(p._p_changed, 0)
-        if self.has_dict:
-            p._v_foo = 2
-        self.assertEqual(p.__getstate__(), {'x': 5})
-        self.assertEqual(p._p_changed, 0)
-
-    def testSetStateSerial(self):
-        p = self.klass()
-        p._p_serial = '00000012'
-        p.__setstate__(p.__getstate__())
-        self.assertEqual(p._p_serial, '00000012')
-
-    def testDirectChanged(self):
-        p = self.klass()
-        p._p_oid = 1
-        dm = DM()
-        p._p_jar = dm
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(dm.called, 0)
-        p._p_changed = 1
-        self.assertEqual(dm.called, 1)
-
-    def testGhostChanged(self):
-        # An object is a ghost, and it's _p_changed it set to True.
-        # This assignment should have no effect.
-        p = self.klass()
-        p._p_oid = 1
-        dm = DM()
-        p._p_jar = dm
-        p._p_deactivate()
-        self.assertEqual(p._p_changed, None)
-        p._p_changed = True
-        self.assertEqual(p._p_changed, None)
-
-    def testRegistrationFailure(self):
-        p = self.klass()
-        p._p_oid = 1
-        dm = BrokenDM()
-        p._p_jar = dm
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(dm.called, 0)
-        try:
-            p._p_changed = 1
-        except NotImplementedError:
-            pass
-        else:
-            raise AssertionError("Exception not propagated")
-        self.assertEqual(dm.called, 1)
-        self.assertEqual(p._p_changed, 0)
-
-    def testLoadFailure(self):
-        p = self.klass()
-        p._p_oid = 1
-        dm = BrokenDM()
-        p._p_jar = dm
-        p._p_deactivate()  # make it a ghost
-
-        try:
-            p._p_activate()
-        except NotImplementedError:
-            pass
-        else:
-            raise AssertionError("Exception not propagated")
-        self.assertEqual(p._p_changed, None)
-
-    def testActivate(self):
-        p = self.klass()
-        dm = DM()
-        p._p_oid = 1
-        p._p_jar = dm
-        p._p_changed = 0
-        p._p_deactivate()
-        # XXX does this really test the activate method?
-        p._p_activate()
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(p.x, 42)
-
-    def testDeactivate(self):
-        p = self.klass()
-        dm = DM()
-        p._p_oid = 1
-        p._p_deactivate() # this deactive has no effect
-        self.assertEqual(p._p_changed, 0)
-        p._p_jar = dm
-        p._p_changed = 0
-        p._p_deactivate()
-        self.assertEqual(p._p_changed, None)
-        p._p_activate()
-        self.assertEqual(p._p_changed, 0)
-        self.assertEqual(p.x, 42)
-
-    if interfaces:
-        def testInterface(self):
-            self.assert_(IPersistent.isImplementedByInstancesOf(Persistent),
-                         "%s does not implement IPersistent" % Persistent)
-            p = Persistent()
-            self.assert_(IPersistent.isImplementedBy(p),
-                         "%s does not implement IPersistent" % p)
-
-            self.assert_(IPersistent.isImplementedByInstancesOf(P),
-                         "%s does not implement IPersistent" % P)
-            p = self.klass()
-            self.assert_(IPersistent.isImplementedBy(p),
-                         "%s does not implement IPersistent" % p)
-
-    def testDataManagerAndAttributes(self):
-        # Test to cover an odd bug where the instance __dict__ was
-        # set at the same location as the data manager in the C type.
-        p = P()
-        p.inc()
-        p.inc()
-        self.assert_('x' in p.__dict__)
-        self.assert_(p._p_jar is None)
-
-    def testMultipleInheritance(self):
-        # make sure it is possible to inherit from two different
-        # subclasses of persistent.
-        class A(Persistent):
-            pass
-        class B(Persistent):
-            pass
-        class C(A, B):
-            pass
-        class D(object):
-            pass
-        class E(D, B):
-            pass
-
-    def testMultipleMeta(self):
-        # make sure it's possible to define persistent classes
-        # with a base whose metaclass is different
-        class alternateMeta(type):
-            pass
-        class alternate(object):
-            __metaclass__ = alternateMeta
-        class mixedMeta(alternateMeta, type):
-            pass
-        class mixed(alternate,Persistent):
-            __metaclass__ = mixedMeta
-
-    def testSlots(self):
-        # Verify that Persistent classes behave the same way
-        # as pure Python objects where '__slots__' and '__dict__'
-        # are concerned.
-
-        class noDict(object):
-            __slots__ = ['foo']
-
-        class shouldHaveDict(noDict):
-            pass
-
-        class p_noDict(Persistent):
-            __slots__ = ['foo']
-
-        class p_shouldHaveDict(p_noDict):
-            pass
-
-        self.assertEqual(noDict.__dictoffset__, 0)
-        self.assertEqual(p_noDict.__dictoffset__, 0)
-
-        self.assert_(shouldHaveDict.__dictoffset__ <> 0)
-        self.assert_(p_shouldHaveDict.__dictoffset__ <> 0)
-
-    def testBasicTypeStructure(self):
-        # test that a persistent class has a sane C type structure
-        # use P (defined below) as simplest example
-        self.assertEqual(Persistent.__dictoffset__, 0)
-        self.assertEqual(Persistent.__weakrefoffset__, 0)
-        self.assert_(Persistent.__basicsize__ > object.__basicsize__)
-        self.assert_(P.__dictoffset__)
-        self.assert_(P.__weakrefoffset__)
-        self.assert_(P.__dictoffset__ < P.__weakrefoffset__)
-        self.assert_(P.__basicsize__ > Persistent.__basicsize__)
-
-    def testDeactivateErrors(self):
-        p = self.klass()
-        p._p_oid = '\0\0\0\0\0\0hi'
-        dm = DM()
-        p._p_jar = dm
-
-        def typeerr(*args, **kwargs):
-            self.assertRaises(TypeError, p, *args, **kwargs)
-
-        typeerr(1)
-        typeerr(1, 2)
-        typeerr(spam=1)
-        typeerr(spam=1, force=1)
-
-        p._p_changed = True
-        class Err(object):
-            def __nonzero__(self):
-                raise RuntimeError
-
-        typeerr(force=Err())
+oid = "\0\0\0\0\0\0hi"
 
 class P(Persistent):
     def __init__(self):
@@ -332,42 +75,42 @@ class BrokenDM(DM):
     def setstate(self,ob):
         raise NotImplementedError
 
-class PersistentTest(Test):
-    klass = P
-    has_dict = 1
-
-    def testPicklable(self):
-        import pickle
-
-        p = self.klass()
-        p.inc()
-        p2 = pickle.loads(pickle.dumps(p))
-        self.assertEqual(p2.__class__, self.klass)
+class Test(unittest.TestCase):
 
-        # verify that the inc is reflected:
-        self.assertEqual(p2.x, p.x)
-        
-        # This assertion would be invalid.  Interfaces
-        # are compared by identity and copying doesn't
-        # preserve identity. We would get false negatives due
-        # to the differing identities of the original and copied
-        # PersistentInterface:
-        # self.assertEqual(p2.__dict__, p.__dict__)
+    # XXX This is the only remaining unittest.  Figure out how to move
+    # this into doctest?
 
-    def testPicklableWCustomState(self):
-        import pickle
+    if interfaces:
+        def testInterface(self):
+            self.assert_(IPersistent.isImplementedByInstancesOf(Persistent),
+                         "%s does not implement IPersistent" % Persistent)
+            p = Persistent()
+            self.assert_(IPersistent.isImplementedBy(p),
+                         "%s does not implement IPersistent" % p)
 
-        p = P2()
-        p2 = pickle.loads(pickle.dumps(p))
-        self.assertEqual(p2.__class__, P2);
-        self.assertEqual(p2.__dict__, {'v': 42})
+            self.assert_(IPersistent.isImplementedByInstancesOf(P),
+                         "%s does not implement IPersistent" % P)
+            p = self.klass()
+            self.assert_(IPersistent.isImplementedBy(p),
+                         "%s does not implement IPersistent" % p)
 
-class BasePersistentTest(Test):
-    klass = B
-    has_dict = 0
+def DocFileSuite(path):
+    # It's not entirely obvious how to connection this single string
+    # with unittest.  For now, re-use the _utest() function that comes
+    # standard with doctest in Python 2.3.  One problem is that the
+    # error indicator doesn't point to the line of the doctest file
+    # that failed.
+    source = open(path).read()
+    t = doctest.Tester(globs=sys._getframe(1).f_globals)
+    def runit():
+        doctest._utest(t, path, source, path, 0)
+    f = unittest.FunctionTestCase(runit, description="doctest from %s" % path)
+    suite = unittest.TestSuite()
+    suite.addTest(f)
+    return suite
 
 def test_suite():
-    s = unittest.TestSuite()
-    for klass in PersistentTest, BasePersistentTest:
-        s.addTest(unittest.makeSuite(klass))
+    p = os.path.join(persistent.tests.__path__[0], "persistent.txt")
+    s = unittest.makeSuite(Test)
+    s.addTest(DocFileSuite(p))
     return s