- "Bucket finalization" is now done more aggressively. Instead of waiting

  until a bucket is garbage collected (which may be much later than the
  expiration of that bucket), we "finalize" a bucket as soon as possible
  after it gets expired.  This effectively means that the "delete notifier"
  will be called much closer to the time that a transient object actually
  expires.
                                                                                
- Add a "_last_finalized_timeslice" counter; this counter keeps track of 
  the bucket which was finalized last.  Set the initial value of
  "_last_finalized_timeslice" to -period; this services the unit tests for
  finalization, where there can actually be a timeslice that is 0 and we 
   need to finalize that bucket.
                                                                                
- Add a series of locks to prevent finalization, replentishment, and
  garbage collection from being attempted by more than one thread
  simultaneously.
                                                                                
- Add "Fake" module for interactive testing purposes (swap out BTree for
  simpler object during stress tests for isolation purposes).
                                                                                
- Allow DATA_CLASS to be specified (this is the main data structure
  class).

- Update docs and tests to account for new finalization strategy.

lib/python/Products/Transience/Fake.py

+##############################################################################
+#
+# Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+"""
+Module used for testing transience (BTree-API-conforming data structure)
+"""
+
+from Persistence.mapping import PersistentMapping
+import sys
+
+class FakeIOBTree(PersistentMapping):
+    def keys(self, min, max):
+        L = []
+        if min is None:
+            min = 0
+
+        if max is None:
+            max = sys.maxint
+
+        for k in self.data:
+            if min <= k <= max:
+                L.append(k)
+        return L
+    

lib/python/Products/Transience/HowTransienceWorks.stx

@@ -32,7 +32,7 @@ Timeslices
 
 Data Structures Maintained by a Transient Object Container
 
-  The TOC maintains five important kinds of data structures:
+  The TOC maintains three important kinds of data structures:
 
   - a "_data" structure, which is an IOBTree mapping a "timeslice"
     integer to a "bucket" (see next bullet for definition of bucket).
@@ -43,8 +43,11 @@ Data Structures Maintained by a Transient Object Container
     "current" bucket, which is the bucket that is contained within the
     _data structured with a key equal to the "current" timeslice.
 
-  - A "max_timeslice" integer, which is equal to the "largest" timeslice
-    for which there exists a bucket in the _data structure.
+  - A "max_timeslice" integer, which is equal to the "largest"
+    timeslice for which there exists a bucket in the _data structure.
+    This is an optimization given that key operations against BTrees
+    can be slow and could cause conflicts (the same could be achieved
+    via _data.maxKey() otherwise).
 
   When a Transient Object is created via new_or_existing, it is added
   to the "current" bucket.  As time goes by, the bucket to which the
@@ -67,25 +70,25 @@ How the TransientObjectContainer Determines if a TransientObject is "Current"
   All "current" timeslice buckets (as specified by the timeout) are
   searched for the transient object, most recent bucket first.
 
-Housekeeping: Notification, Garbage Collection, and Bucket
+Housekeeping: Finalization, Garbage Collection, and Bucket
 Replentishing
 
-  The TOC performs "notification", "garbage collection", and "bucket
+  The TOC performs "finalization", "garbage collection", and "bucket
   replentishing".  It performs these tasks "in-band".  This means that
   the TOC does not maintain a separate thread that wakes up every so
   often to do these housekeeping tasks.  Instead, during the course of
   normal operations, the TOC opportunistically performs them.
 
+  Finalization is defined as optionally calling a function at bucket
+  expiration time against all transient objects contained within that
+  bucket.  The optional function call is user-defined, but it is
+  managed by the "notifyDel" method of the TOC.
+
   Garbage collection is defined as deleting "expired" buckets in the
   _data structure (the _data structure maps a timeslice to a bucket).
   Typically this is done by throwing away one or more buckets in the
   _data structure after they expire.
 
-  Notification is defined as optionally calling a function at TOC
-  finalization time against individual transient object contained
-  within a bucket.  The optional function call is user-defined, but it
-  is managed by the "notifyDel" method of the TOC.
-
   Bucket replentishing is defined as the action of (opportunistically)
   creating more buckets to insert into the the _data structure,
   replacing ones that are deleted during garbage collection.  The act
@@ -93,7 +96,9 @@ Replentishing
   will be immediately created thereafter.  We create new buckets in
   batches to reduce the possibility of conflicts.
 
-  Housekeeping is performed on a somewhat random basis to avoid
+  Finalization is attempted on every call to the transience machinery
+  to make TOs appear to expire "on time".  Garbage collection and
+  replentishment is performed on a somewhat random basis to avoid
   unnecessary conflicts.
 
 Goals

lib/python/Products/Transience/tests/testTimeoutRelated.py

@@ -109,10 +109,9 @@ class TestNotifications(TestBase):
         self.app.sm.setDelNotificationTarget(delNotificationTarget)
         sdo = self.app.sm.new_or_existing('TempObject')
         timeout = self.timeout * 60
-        fauxtime.sleep(timeout + (timeout * .75))
+        # sleep 2X longer than timeout?  doesnt work at 1.1X, 1.5X?
+        fauxtime.sleep(timeout * 2) 
         sdo1 = self.app.sm.get('TempObject')
-        # force the sdm to do housekeeping
-        self.app.sm._gc()
         now = fauxtime.time()
         k = sdo.get('endtime')
         self.assertEqual(type(k), type(now))

lib/python/Products/Transience/tests/testTransientObjectContainer.py

@@ -249,7 +249,7 @@ class TestTransientObjectContainer(TestBase):
         self.assertEqual(len(self.t.keys()), 100)
 
         # call _gc just to make sure __len__ gets changed after a gc
-        self.t._gc()
+        #self.t._gc()
         self.assertEqual(len(self.t), 100)
 
         # we should still have 100 - 199