Or else such iteration can race with deletion of elements in that list
on client disconnection. StorageServer.close was already iteration over
snapshot, but StorageServer.invalidate was not.

As the result sending out invalidations were subject to race wrt client
disconnections, and some invalidations could be skipped to be delivered
causing further data corruption.

Fixes Bug2 from https://github.com/zopefoundation/ZEO/issues/209

Tests for this fix were developed as part of ZODB5:

    https://github.com/zopefoundation/ZODB/pull/368

    https://github.com/zopefoundation/ZODB/commit/942099c6c430
    https://github.com/zopefoundation/ZODB/commit/a03c47787698
    https://github.com/zopefoundation/ZODB/commit/c593c7d5baf1
    https://github.com/zopefoundation/ZODB/commit/fa8441591074
    https://github.com/zopefoundation/ZODB/commit/fc20f3aaddda
    https://github.com/zopefoundation/ZODB/commit/a94d63c552d6

and are too big and probably not worth it to backport.

( Upstream commit c8f1c523 )

If a transaction only adds objects (like the transaction that ZODB.DB
uses to create the root objects, we still need to send invalidations so clients see the new tid, because MVCC.

This is ZEO4 backport of commit 8d7b1ceb ("Fixed: were sending out of
date size info on commit", 2016-06-21) needed to get tests passing after
previous patch.

It is needed because BasicStorage.checkLen was previously passing just
by luck because transactions in that test only create objects, and the
test is tolerating storages that do not update their len at all.

Before previous patch, if ZEOStorage.invalidated was == [], neither
invalidateTransaction, _nor_ info messages were sent (which is incorrect
since when object is created len has to be adjusted too). After previous
patch we always delve into sending invalidateTransaction and info, but,
since ZEOStorage._invalidate is called from under tpc_finish, storage
sizes are not yet updated and get_info retrieves len from previous - not
for committed - transaction, leading to errors like:

    Failure in test checkLen (ZEO.tests.testZEO.DemoStorageTests)
    Traceback (most recent call last):
      File "/usr/lib/python2.7/unittest/case.py", line 329, in run
        testMethod()
      File "/home/kirr/src/wendelin/z/ZODB4/src/ZODB/tests/BasicStorage.py", line 175, in checkLen
        self.assertIn(len(self._storage), [0,2])
      File "/usr/lib/python2.7/unittest/case.py", line 804, in assertIn
        self.fail(self._formatMessage(msg, standardMsg))
      File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
        raise self.failureException(msg)
    AssertionError: 1 not found in [0, 2]

Original description follows:

---- 8< ---- (original description)

Because we were computing the size info in the commit callback, which
might be before before the inderlying data were committed.  This caused the computation to be wrong in some cases.

Instead we get and send the size information after committing.

( Upstream commit ab86bd72 )

This is ZEO4 backport of https://github.com/zopefoundation/ZEO/pull/160

It changes ZEO4 to both include created objects into invalidation
messages, and, in turn, not to skip sending invalidation message at
all if committed transaction only creates objects. Please see original
description for details below.

Extra changes compared to ZEO5 patch:

- tests/servertesting.py: provide callAsyncNoSend to avoid the following crash:

  File ".../ZEO/tests/testZEO2.py", line 156, in ZEO.tests.testZEO2.proper_handling_of_errors_in_restart
  Failed example:
      zs1.tpc_finish('1').set_sender(0, conn1)
  Exception raised:
      Traceback (most recent call last):
        File "/usr/lib/python2.7/doctest.py", line 1315, in __run
          compileflags, 1) in test.globs
        File "<doctest ZEO.tests.testZEO2.proper_handling_of_errors_in_restart[18]>", line 1, in <module>
          zs1.tpc_finish('1').set_sender(0, conn1)
        File ".../ZEO/StorageServer.py", line 408, in tpc_finish
          self.storage.tpc_finish(self.transaction, self._invalidate)
        File ".../ZODB/FileStorage/FileStorage.py", line 741, in tpc_finish
          f(self._tid)
        File ".../ZEO/StorageServer.py", line 418, in _invalidate
          self.invalidated, self.get_size_info())
        File ".../ZEO/StorageServer.py", line 1110, in invalidate
          p.client.invalidateTransaction(tid, invalidated)
        File ".../ZEO/StorageServer.py", line 1466, in invalidateTransaction
          self.rpc.callAsyncNoSend('invalidateTransaction', tid, args)
      AttributeError: Connection instance has no attribute 'callAsyncNoSend'

- testZEO2.proper_handling_of_errors_in_restart: adjust it since now an
  invalidation message is sent and previously it was completely avoided
  because objects in that test are only created.

---- 8< ---- (original description)

Starting from 1999 (b3805a2f "just getting started") only modified - not
just created - objects were included into ZEO invalidation messages:

https://github.com/zopefoundation/ZEO/commit/b3805a2f#diff-52fb76aaf08a1643cdb8fdaf69e37802R126-R127

In 2000 this behaviour was further changed to not send invalidation
message at all if the only objects a transaction has were the created ones:

https://github.com/zopefoundation/ZEO/commit/230ffbe8#diff-52fb76aaf08a1643cdb8fdaf69e37802L163-R163

In 2016 the latter was reconsidered as bug and fixed in ZEO5 because
ZODB5 relies more heavily on MVCC semantic and needs to be notified
about every transaction committed to storage to be able to properly
update ZODB.Connection view:

https://github.com/zopefoundation/ZEO/commit/02943acd#diff-52fb76aaf08a1643cdb8fdaf69e37802L889-R834
https://github.com/zopefoundation/ZEO/commit/9613f09b

In 2020, with this patch, I'm proposing to reconsider initial "send only
modified, not created objects" as bug, and include both modified and
just created objects into invalidation messages at least for the
following reasons:

- a ZODB client (not necessarily native ZODB/py client) can maintain
  raw cache for the storage. If such client tries to load an oid at
  database view when that object did not existed yet, gets "no object"
  reply and stores that information into raw cache, to properly invalidate
  the cache it needs an invalidation message from ZODB server that
  _includes_ created object.

- tools like `zodb watch` [1,2,3] don't work properly (give incorrect output)
  if not all objects modified/created by a transaction are included into
  invalidation messages.

- similarly to `zodb watch`, a monitoring tool, that would want to be
  notified of all created/modified objects, won't see full
  database-change picture, and so won't work properly without knowing
  which objects were created.

- wendelin.core 2 - which builds data from ZODB BTrees and data objects
  into virtual filesystem - needs to get invalidation messages with both
  modified and created objects to properly implement its own lazy
  invalidation and isolation protocol for file blocks in OS cache: when
  a block of file is accessed, all clients, that have this block mmaped,
  need to be notified and asked to remmap that block into particular
  revision of the file depending on a client's view of the filesystem and
  database [4,5].

  To compute to where a client needs to remmap the block, WCFS server
  (that in turn acts as ZODB client wrt ZEO/NEO server), needs to be able
  to see whether client's view of the filesystem is before object creation
  (and then ask that client to pin that block to hole), or after creation
  (and then ask the client to pin that block to corresponding revision).

  This computation needs ZODB server to send invalidation messages in
  full: with both modified and just created objects.

The patch is simple - it removes `if serial != b"\0\0\0\0\0\0\0\0"`
before queuing oid into ZEOStorage.invalidated, and adjusts the tests
correspondingly. From my point of view and experience, in practice, this
patch should not cause any compatibility break nor performance regressions.

Thanks beforehand,
Kirill

/cc @jimfulton

[1] https://lab.nexedi.com/kirr/neo/blob/ea53a795/go/zodb/zodbtools/watch.go
[2] kirr/neo@e0d59f5d
[3] kirr/neo@c41c2907

[4] https://lab.nexedi.com/kirr/wendelin.core/blob/1efb5876/wcfs/wcfs.go#L94-182
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/1efb5876/wcfs/client/wcfs.h#L20-71

See commit 3d886d42.

----

Originally included in slapos: slapos@5cf4cf1f

ZEO4 is considered "dead" by upstream:

	https://github.com/zopefoundation/ZEO/pull/161#pullrequestreview-447245642

Keep on patching ZEO4 in our local fork until ZEO4 is used.

Mainly to have a backup cache for new branches (although this won't
help that much because of the differences between ZEO4 and ZEO5,
still...

Also stop testing on mac. I do that enough already. :)

Also, use a simpler leaner install script.

Convert cache_stats and cache_simul from getopt to argparse.

Fix tracing of loadBefore with empty tids on Py3.

Update zeo-client-cache-tracing to use accurate script names.

I haven't actually compared the output snippets (yet), but at least the
script references are right now.

Fix connecting to 'localhost' on Windows / Solaris / Android.

Closes #8.

Closes #30.

Close opened-file leak in blob cache size check.

Closes #27.

Changed tests to not depend on ZODB underware

Zodb5

This is required (or maybe strongly implied) by the IStorage interface.

Also, rearrange the order of operations in _process_invalidations.

Each of these changes avoid a potential race when a connection starts
a transaction while invalidateTransaction is being called. Basically,
we don't want a connection to get a lastTransaction corresponding to
invalidations it hasn't processed.

This neither fixes nor provokes any test failures. :) The need for
this change was discovered via code inspection. It's hard to see how
to test the race without getting being insanely whitebox.

To avoid an error on tearDown.

Fix #22

Adding osx back.

Also adding back -j99, in hopes of getting quicker runs and making
travis nor hate us so much.  But then, we'll probably get more
failures. Maybe the best option will be to just make test pass even
when starved. :(

traviiiiiiiiis