ClientStorage constructor called with both wait=1 and
read_only_fallback=1 should return, indicating its readiness, when a
read-only connection was made.  This is done by calling
connect(sync=1).  Previously this waited for the ConnectThread to
finish, but that thread doesn't finish until it's made a read-write
connection, so a different mechanism is needed.

I ended up doing a major overhaul of the interfaces between
ClientStorage, ConnectionManager, ConnectThread/ConnectWrapper, and
even ManagedConnection.  Changes:

ClientStorage.py:

  ClientStorage:

  - testConnection() now returns just the preferred flag; stubs are
    cheap and I like to have the notifyConnected() signature be the
    same for clients and servers.

  - notifyConnected() now takes a connection (to match the signature
    of this method in StorageServer), and creates a new stub.  It also
    takes care of the reconnect business if the client was already
    connected, rather than the ClientManager.  It stores the
    connection as self._connection so it can close the previous one.
    This is also reset by notifyDisconnected().

zrpc/client.py:

  ConnectionManager:

  - Changed self.thread_lock into a condition variable.  It now also
    protects self.connection.  The condition is notified when
    self.connection is set to a non-None value in connect_done();
    connect(sync=1) waits for it.  The self.connected variable is no
    more; we test "self.connection is not None" instead.

  - Tried to made close() reentrant.  (There's a trick: you can't set
    self.connection to None, conn.close() ends up calling close_conn()
    which does this.)

  - Renamed notify_closed() to close_conn(), for symmetry with the
    StorageServer API.

  - Added an is_connected() method so ConnectThread.try_connect()
    doesn't have to dig inside the manager's guts to find out if the
    manager is connected (important for the disposition of fallback
    wrappers).

  ConnectThread and ConnectWrapper:

  - Follow above changes in the ClientStorage and ConnectionManager
    APIs: don't close the manager's connection when reconnecting, but
    leave that up to notifyConnected(); ConnectWrapper no longer
    manages the stub.

  - ConnectWrapper sets self.sock to None once it's created a
    ManagedConnection -- from there on the connection is is charge of
    closing the socket.

zrpc/connection.py:

  ManagedServerConnection:

  - Changed the order in which close() calls things; super_close()
    should be last.

  ManagedConnection:

  - Ditto, and call the manager's close_conn() instead of
    notify_closed().

tests/testZEO.py:

  - In checkReconnectSwitch(), we can now open the client storage with
    wait=1 and read_only_fallback=1.

constructor signature backwards compatible with ZEO 1.  This means
adding wait_for_server_on_startup and debug options.
wait_for_server_on_startup is an alias for wait, which makes the
argument decoding for these two a little tricky.  debug is ignored.

Also change the default of wait to True, like it was in ZEO 1.  This
is less likely to screw naive customers.

of 30 seconds.  There's nothing wrong with the code, it's just slow.
So increase the timeout to 60 seconds.

On Windows, port+1 is used as well, so we don't want accidentally to
allocate two adjacent ports when we ask for multiple ports.

"if wait" rather than "if wait is not None".  Also change the default
to 0.

until I added an is_connected() test to testConnection() is solved.

After the ConnectThread has switched the client to the new, read-write
connection, it closes the read-only connection(s) that it was saving
up in case there was no read-write connection.  But closing a
ManagedConnection calls notify_closed() on the manager, which
disconnected the manager and the client from its brand new read-write
connection.  The mistake here is that this should only be done when
closing the manager's current connection!

The fix was to add an argument to notify_closed() that passes the
connection object being closed; notify_closed() returns without doing
a thing when that is not the current connection.

I presume this didn't happen on Linux because there the sockets
happened to connect in a different order, and there was no read-only
connection to close yet (just a socket trying to connect).

I'm taking out the previous "fix" to ClientStorage, because that only
masked the problem in this relatively simple test case.  The problem
could still occur when both a read-only and a read-write server are up
initially, and the read-only server connects first; once the
read-write server connects, the read-write connection is installed,
and then the saved read-only connection is closed which would again
mistakenly disconnect the read-write connection.

Another (related) fix is not to call self.mgr.notify_closed() but to
call self.mgr.connection.close() when reconnecting.  (Hmm, I wonder if
it would make more sense to have an explicit reconnect callback to the
manager and the client?  Later.)

the socket's __str__ due to a __getattr__ method in asyncore's
dispatcher base class that everybody hates but nobody dares take away.

Add the pid to the label.

When load() finds a hit in the non-current file, copy it to the
current file (except when this would cause a cache flip).

It is hoped that this improves the cache performance.

(Now merged into the trunk!  Jeremy & Jim want this in the ZEO and ZODB3 beta2 releases.)

The fix is to change testConnection() to only retry the register()
call with the read-only flag set when we're not alrady connected.
But I don't understand why this fix is needed; I'll keep debugging.

calls.  If multiple threads sharing a ZEO connection want to make
overlapping calls, they can do that now.  This is mostly useful when
one thread is waiting for a long-running pack() or undo*() call -- the
other thread can now proceed.

Jeremy & I did a review of the StorageServer code and found no place
where overlapping incoming calls from the same connection could do any
harm -- given that the only places where incoming calls can be handled
are those places where the server makes a callback to the client.

Cleanup comments for Managed*Connection.

Whitespace normalization.

- It never uses self.conn.  So don't store it.

- Make a check in register() that register() isn't called twice.  If
  it is, log a message and raise ValueError.

trying to fake a tpc_abort() call with the right transaction id, call
a new method abort(), which resets some instance variables and then
calls the strategy's abort() method.  The immediate strategy's abort()
aborts its own transaction and then calls the zeo_storage's
handle_waiting() method to restart the next blocked transaction from
the _waiting list; the delayed strategy's abort() removes the
zeo_storage from the _waiting list.

- Remove unused imports.

- Got rid of \ continuation lines.

- Remove leading underscores from all names of methods and instance
  variables of ZEOStorage.  There was no usage consistency and I don't
  see any other reason to maintain the existing names.

a variety of exceptions.  Catch all of them.  (The only ones I've seen
are Disconnected and ReadOnlyError, but since we're disconnected
abruptly as part of the switch, it's possible to see any of the
exceptions that checkReconnection() catches.)

read-write server now works!!!

unlocking bug in new_oid().

the _server.new_oids() raising Disconnected or ReadOnlyError).  Fixed
by adding a try/finally.

randrange(25000, 30000) to get one random port each time _getAddr() is
called.

Otherwise, when running the tests in an endless loop with "test.py
-L", eventually tests start failing because they're out of ports.

the client, don't log it at the ERROR level.  If it really was a
disaster, the client should log it.  But if the client was expecting
the exception, the esrver shouldn't get all upset about it.  Change
this to the INFO level.  (When it *is* considered an error by the
client, it's useful to be able to see the server-side traceback in the
log.)

hits found in the non-current file into the current file.

For the 40-hour trace starting on Sept 4 at 6pm, this increases the
hit rate from 83.2% to 87.0% (IOW a decrease in miss rate from 16.8%
to 13%):

ZEOCacheSimulation, cache size 200,000,000 bytes
  START TIME  DURATION    LOADS     HITS INVALS WRITES  FLIPS HITRATE
Sep  4 18:00  40:36:34  1420070  1181176  16198  10014     15  83.2%

AltZEOCacheSimulation, cache size 200,000,000 bytes
  START TIME  DURATION    LOADS     HITS INVALS WRITES  FLIPS HITRATE
Sep  4 18:00  40:36:34  1420070  1235770  15384  10014     25  87.0%

any difference on Win2k.  Add XXX comment with question about
WSAEALREADY.

Made progress with the reconnect tests.  The simple ones now work.
The trick was to call asyncore.poll() until the client is
(dis)connected before going to the next phase of the test; I added
pollUp() and pollDown() helper methods for this purpose.

The reconnect "upgrade" test doesn't work yet.  That is, a client is
created in fallback mode, initially connecting to a read-only server.
So far so good.  Then the server is shut down.  Somehow an attempt to
store once the server is down simply *hangs*.

No progress with the multiple server fallback test.

(now redundant) pid=%s part from that first log message.

Change label to ZSS:pid.  Re-initialize pid every time a StorageServer
is created.  XXX This is sufficient as long as a single process
doesn't create more than one StorageServer instance.  I haven't found
any code that does.

Use log() function for self.log() method of ZEOStorage, so it ends up
sharing the ZSS:pid prefix.

Caught by Guido, but why not be my compiler?

It was calling repr() twice on the elements of a tuple.  Fix by
calling repr() explicitly on each branch of the type-test code.

Add comment about how short_repr() is used.

Reformat all the code to use Python style.

We got an undiagnosed bug report for coptimization, which lead to an
attempt at code review.

Break up a the large new_persistent_id() function into several helpers
-- get_class(), get_class_tuple(), and set_oid().  Each function now
does a little less.  Don't reuse variable names for random purposes.