Invalidation age ================ When a ZEO client with a non-empty cache connects to the server, it needs to verify whether the data in its cache is current. It does this in one of 2 ways: quick verification It gets a list of invalidations from the server since the last transaction the client has seen and applies those to it's disk and in-memory caches. This is only possible if there haven't been too many transactions since the client was last connected. full verification If quick verification isn't possible, the client iterates through it's disk cache asking the server to verify whether each current entry is valid. Unfortunately, for large caches, full verification is soooooo not quick that it is impractical. Quick verificatioin is highly desireable. To support quick verification, the server keeps a list of recent invalidations. The size of this list is controlled by the invalidation_queue_size parameter. If there is a lot of database activity, the size might need to be quite large to support having clients be disconnected for more than a few minutes. A very large invalidation queue size can use a lot of memory. To suppliment the invalidation queue, you can also specify an invalidation_age parameter. When a client connects and presents the last transaction id it has seen, we first check to see if the invalidation queue has that transaction id. It it does, then we send all transactions since that id. Otherwise, we check to see if the difference between storage's last transaction id and the given id is less than or equal to the invalidation age. If it is, then we iterate over the storage, starting with the given id, to get the invalidations since the given id. NOTE: This assumes that iterating from a point near the "end" of a database is inexpensive. Don't use this option for a storage for which that is not the case. Here's an example. We set up a server, using an invalidation-queue-size of 5: >>> addr, admin = start_server(zeo_conf=dict(invalidation_queue_size=5), ... keep=True) Now, we'll open a client with a persistent cache, set up some data, and then close client: >>> import ZEO, transaction >>> db = ZEO.DB(addr, client='test') >>> conn = db.open() >>> for i in range(9): ... conn.root()[i] = conn.root().__class__() ... conn.root()[i].x = 0 >>> transaction.commit() >>> db.close() We'll open another client, and commit some transactions: >>> db = ZEO.DB(addr) >>> conn = db.open() >>> import transaction >>> for i in range(2): ... conn.root()[i].x = 1 ... transaction.commit() >>> db.close() If we reopen the first client, we'll do quick verification. We'll turn on logging so we can see this: >>> import logging, sys >>> old_logging_level = logging.getLogger().getEffectiveLevel() >>> logging.getLogger().setLevel(logging.INFO) >>> handler = logging.StreamHandler(sys.stdout) >>> logging.getLogger().addHandler(handler) >>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS ('localhost', ... ('localhost', ...) Recovering 2 invalidations >>> logging.getLogger().removeHandler(handler) >>> [v.x for v in db.open().root().values()] [1, 1, 0, 0, 0, 0, 0, 0, 0] Now, if we disconnect and commit more than 5 transactions, we'll see that verification is necessary: >>> db.close() >>> db = ZEO.DB(addr) >>> conn = db.open() >>> import transaction >>> for i in range(9): ... conn.root()[i].x = 2 ... transaction.commit() >>> db.close() >>> logging.getLogger().addHandler(handler) >>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS ('localhost', ... ('localhost', ...) Verifying cache ('localhost', ...) endVerify finishing ('localhost', ...) endVerify finished >>> logging.getLogger().removeHandler(handler) >>> [v.x for v in db.open().root().values()] [2, 2, 2, 2, 2, 2, 2, 2, 2] >>> db.close() But if we restart the server with invalidation-age set, we can do quick verification: >>> stop_server(admin) >>> addr, admin = start_server(zeo_conf=dict(invalidation_queue_size=5, ... invalidation_age=100)) >>> db = ZEO.DB(addr) >>> conn = db.open() >>> import transaction >>> for i in range(9): ... conn.root()[i].x = 3 ... transaction.commit() >>> db.close() >>> logging.getLogger().addHandler(handler) >>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS ('localhost', ... ('localhost', ...) Recovering 9 invalidations >>> logging.getLogger().removeHandler(handler) >>> [v.x for v in db.open().root().values()] [3, 3, 3, 3, 3, 3, 3, 3, 3] >>> db.close() >>> logging.getLogger().setLevel(old_logging_level)