The cache simulation seems good enough to be useful now,
although the toy app I wrote to generate a 500MB trace
file doesn't use MVCC in an essential way (some of the
MVCC simulation code is nevertheless exercised, since
an invalidation of current data in the presence of MVCC
always creates a cache record for the newly non-current
revision).

Still puzzling over what to do when the trace file records
a load hit but the simulated cache gets a miss.  The
old simulation code seemed to assume that a store for the same
oid would show up in the trace file next, and it could get
the info it needed about the missing object from the store
trace.  But that isn't true:  precisely because the load was
a hit in the trace file, the object isn't going to be stored
again "soon" in the trace file.

Here are some actual-vs-simulated hit rate results, for a
20MB cache, with a trace file covering about 9 million
loads, over 3 ZEO client (re)starts:

actual   simulated
------   ---------
  93.1        92.7
  79.8        79.0
  68.0        69.1
      
  81.4        81.1  overall

Since the simulated hit rates are both higher and lower
than the actual hit rates, that argues against a gross
systematic bias in the simulation (although there may
be several systematic biases in opposite directions).

Massive rewrite to make simul.py's circular cache aware of
MVCC.  It isn't blowing up, but it thinks the hit rate is
100% -- might have missed something there <wink>.

More steps on the way toward having useful cache simulation.

I think the primary hangup now is that simul.py doesn't know
anything about MVCC.  As a result, it thinks there's significatly
more free space in the cache than there really is, and that
probably accounts for it predicting significantly higher hit
rates than I actually see.

Lots of fiddling to work w/ current trace file format.

Output isn't obviously wholly insane, but isn't a reasonably
close enough match to stats.py output either yet.

stats.py:  cut the # of file reads by 1/3, and the # of
struct.unpack() calls by 1/2.

Various bugfixes elsewhere.

Many little bugfixes and improvements in stats.py.
This has survived several 100 MB of trace files
I generated over the last few days, so it's solid
now if not necessarily perfect.

Replaced simul.py with the much broader-ranging code
Jeremy and I were working on a couple years ago,
although it can't work with the current trace file
format (no real loss there -- the simul.py it's
replacing can't work with the current format either).

Stitch in ZConfig's docs from ZConfig.

Remove this -- it's a way out-of-date copy of the ZConfig
docs.  I'll stitch them back in from the ZConfig repo next,
via svn:externals.  Ideally, ZODB shouldn't have this as
part of its checkout tree at all, but that also has implications
for the way ZRS gets packaged, etc.

A large number of changes so that cache tracing isn't
100% broken -- I can get a trace file now, and stats.py
doesn't blow up anymore.  Don't yet know whether it's
reporting sensible things, but it's not obviously insane.
simul.py doesn't know about the current cache implementation
either yet.

ClientCache:  made it a new-style class.

FileCache:  Removed the reuse= argument.  IIRC, it made some sense
when different cache schemes were first being tried, but complicated
the logic now to no good end.

Nuked the maddening log warnings about reuse=True.  They were never
helpful and were often confusing.

Added info-level log messages to record the path of the client cache
file, whether it's persistent or temporary, and if it's persistent
whether we're creating or reusing it.

Minor changes to comments.

Massive rewrite of cache.txt, to match current reality.

Collector 1843:  IISet.keys: bad exception handling.

Bucket_rangeSearch():  De-obfuscated the calls to
Bucket_findRangeEnd(), so that they stop ignoring
the latter's error returns (a mind-bending combination
of embedded assignment nested in an UNLESS macro,
seemingly copy+paste'd so that the error occurred twice).

Port from ZODB 3.2.

Collector #1846:  If an uncommitted transaction was found, fsrecover.py
fell into an infinite loop.  Fixed that, and added a new test
(testUncommittedAtEnd) to ensure this stays fixed.

Subtransactions are deprecated in 3.5, and will go away
in 3.7.

Convert internal uses of subtxns to use savepoints instead.

I suspect BTrees/convert.py should be removed instead.

Revert extra copying in Connection._rollback().

Since TmpStore.reset() had to change too to copy the index anyway,
it no longer hurts that _rollback() clears the index (as a side
effect of self._cache.invalidate(src.index)).

Merge tim-savepoint branch.

Allow rollback to a given savepoint multiple times.

transaction/_transaction.py
    _remove_and_invalidate_after():  Don't remove the
    argument savepoint from the savepoint stack anymore.

    abort():  abort(True) relied on the subtxn savepoint
    being invalidated as a side effect of rolling back
    to it.  Invalidate it explicitly now.

    Savepoint.rollback():  Don't invalidate self.

ZODB/Connection.py
    Connection._rollback():  Pass a temp container to
    _cache.invalidate(), because that method clears the
    container as a side effect.  Clearing the index
    in a savepoint was a disaster if the savepoint was
    used again.

    TmpStore.reset():  Make a copy of the argument index.
    An alternative would be to make all callers make
    copies, but the only caller now thinks of its `state`
    argument as being opaque.

transaction/savepoint.txt
ZODB/tests/{testSubTransaction.py, testConnectionSavepoint.txt}
    Test the new semantics, and beef up the tests.

transaction/tests/savepointsample.py
    _rollback_savepoint():  Needed to copy the data.

English repairs.

Simplify.

Primarily, _tindex should have been a dict all along, pack()
worked too hard at the end, and loadEx() is a minor variant
of load().

load() and loadEx():  eliminate gratuitous differences.

Ideally, load() should call loadEx() instead, but we really
don't want "an extra" Python-level call here (heavily used).

loadEx():  deleted pointless call of self._read_txn_header().

ServerStub.loadEx() comments:  these were obviously wrong in
several ways, but I don't know the full truth.  Better to
say so up front than to leave them clearly wrong, though.

Collector 1831.

The BTree minKey() and maxKey() methods gave a misleading message if no key
satisfying the constraints existed in a non-empty tree.

Collector #1830.

In some error cases when reading a FileStorage index, the
code referenced an undefined global.

Get rid of all code defining or referencing t32.

Such code never made sense; it was all of the form

    t32 = 1L << 32
    if something < 0:
        something = t32 - something

which is a way to change negative integers into
gigantic positive integers greater than 4 billion, and
never made sense in context.  Possibly

        something = t32 + something

was intended, but that never made sense in context either.

In any case, `something` is invariably obtained from
struct.unpack using the "H" format code, and that never
returns a negative integer to begin with.

read_index():  Mechanical reformat for easier reading.

Collector 1829.

Clarified that the ``minKey()`` and ``maxKey()`` methods raise an exception
if no key exists satsifying the constraints.

Also improved the English in other interface docstrings.

Collector 1822.

Make undo{Log,Info} arguments act like Python slice indices when
both are non-negative.  The code used to do that before ZODB 3.4a9,
but changed to match ZODB's UML documentation.  Alas, some
(untested) code in Zope relied on the actual behavior (see the
collector report).  Changed code, docs, and tests to bless the
old behavior in these cases.

DemoStorage.UndoLog:  this was wrong in several ways.  I'm still
unsure about why it skips "packed" transactions.  That doesn't seem
right, but I don't have time to wonder about that now.

_triggerbase:  Make new-style class, for better debugability.

Windows trigger.__init__:  Simplify more.  The trigger can connect
after the other end has done  bind() and listen(); no need to wait
for an accept() too, and so no need to fiddle with blocking/
non-blocking or "expected" socket.error's either.

Largely rewritten.

remove_loop_callback():  There were two definitions of this function.
Deleted one, changed the other to use enumerate().

poll():  This was trying to repair a bug in Python 2.2's asyncore.poll().
2.2 is no longer supported, so removed this poll().

_start_loop(), _stop_loop():  Removed; functionality folded into
reworked loop().

loop():  The signature of asyncore.loop() changed in 2.4, but this
still had "the old" signature.  Instead of sniffing the signature
of the Python in use, this does a saner thing:  the original
asyncore.loop is captured, and called from the body of this loop().
That way, (A) we don't care what asyncore.loop's signature is; and,
(B) we still call the original Python code, so debugging prints and
breakpoints (etc) stuffed into Python's asyncore no longer "vanish
by magic" when ZEO is used.

Massive refactoring, to move the bulk of the trigger code into
an OS-indepedent base class.

__repr__:  Use the postive_id function to embed the machine address.
Addresses with the high bit set trigger warnings before Python 2.4,
and come out as negative numbers in 2.4+.

Windows trigger.__init__:  Don't make 50 guesses at a port number
to use, let Windows pick an available port for us.  Also documented
the baffling single-thread socket setup dance, which took an hour to
reverse-engineer (in large part because it used a bare "except" w/ no
clue as to why).

reachable from multiple connections.  Previously, we thought that we
could limit the time that an object was new to a single savepoint, but
that didn't work because savepoints of different connections are too
independent.   Now an object is considered new for the full extent of
the transaction in which it was created.

Made it possible to use connection add methods to explicitly control
the database an object is added too.

TmpStore needs to delegate loadBefore().