( This is ZODB4 backport of upstream commit fc4c86e6 )
Co-authored-by: ale-rt <alessandro.pisa@gmail.com>

... and to provide correct "current" view for load().

This is ZODB4 backport of https://github.com/zopefoundation/ZODB/pull/313,
which itself is just a more explicit language of
https://github.com/zopefoundation/ZODB/commit/4a6b0283#diff-881ceb274f9e538d4144950eefce8682R685-R695

It has been amended with ZODB4-specific

      It is guaranteed that after lastTransaction returns, "current" view of
      the storage as observed by load() is ≥ returned tid.

because on ZODB<5 - contrary to ZODB5 - there is "load current" for
which correct semantic must be also provided:
https://github.com/zopefoundation/ZODB/pull/307#discussion_r436662348

Original description follows:

---- 8< ----

Because if lastTransaction() returns tid, for which local database
handle has not yet been updated with invalidations, it could lead to
data corruption due to concurrency issues similar to
https://github.com/zopefoundation/ZODB/issues/290:

- DB refreshes a Connection for new transaction;
- zstor.lastTransaction() is called to obtain database view for this connection.
- objects in live-cache for this Connection are invalidated with
  invalidations that were queued through DB.invalidate() calls from
  storage.
- if lastTransaction does not guarantee that all DB invalidations for
  transactions with ID ≤ returned tid have been completed, it can be
  that:

	incomplete set of objects are invalidated in live cache

  i.e. data corruption.

This particular data corruption has been hit when working on core of
ZODB and was not immediately noticed:

https://github.com/zopefoundation/ZODB/pull/307#pullrequestreview-423017996

this fact justifies the importance of explicitly stating what IStorage
guarantees are / must be in the interface.

This guarantee

- already holds for FileStorage (no database mutations from outside of
  single process);
- is already true for ZEO4 and ZEO5
  https://github.com/zopefoundation/ZODB/pull/307#pullrequestreview-423017996
  https://github.com/zopefoundation/ZODB/pull/307#discussion_r434166238
- holds for RelStorage because it implements IMVCCStorage natively;
- is *not* currently true for NEO because NEO sets zstor.last_tid before
  calling DB.invalidate:

  https://lab.nexedi.com/nexedi/neoppod/blob/fc58c089/neo/client/handlers/master.py#L109-124

However NEO is willing to change and already prepared the fix to provide
this guarantee because of data corruption scenario that can happen
without it:

  https://github.com/zopefoundation/ZODB/pull/307#discussion_r436662348
  nexedi/neoppod@a7d101ec
  nexedi/neoppod@a7d101ec (comment 112238)
  nexedi/neoppod@a7d101ec (comment 113331)

In other words all storages that, to my knowledge, are in current use
are either already providing specified semantic, or will be shortly
fixed to provide it.

This way we can fix up the interface and make the semantic clear.

/cc @jamadden, @vpelletier, @arnaud-fontaine, @jwolf083, @klawlf82, @gitzit, @jimfulton

Zope.testrunner is used to run ZODB tests (see e.g. tox.ini). It was
specified as explicit dependency in tox.ini->deps and in buildout.cfg.
However if in SlapOS we install just ZODB[test] zope.testrunner won't be
installed.

-> Fix is by moving zope.testrunner dependency into setup, so that
ZODB[test] installs it. This is the same as what ZODB5 does:

https://github.com/zopefoundation/ZODB/blob/5.6.0-14-g0eae10cd0/setup.py#L48

Upstream considers 4 flavour to be "dead":

	https://github.com/zopefoundation/ZEO/pull/161#pullrequestreview-447245642

Let's keep on 4-nxd branch in our fork until ZODB4 is used on our side.

[ This is ZODB4 backport of commit bb9bf539
  (https://github.com/zopefoundation/ZODB/pull/298) ]

ZODB tries to avoid saving empty transactions to storage on
`transaction.commit()`. The way it works is: if no objects were changed
during ongoing transaction, ZODB.Connection does not join current
TransactionManager, and transaction.commit() performs two-phase commit
protocol only on joined DataManagers. In other words if no objects were
changed, no tpc_*() methods are called at all on ZODB.Connection at
transaction.commit() time.

This way application servers like Zope/ZServer/ERP5/... can have
something as

    try:
        # process incoming request
        transaction.commit()    # processed ok
    except:
        transaction.abort()
        # problem: log + reraise

in top-level code to process requests without creating many on-disk
transactions with empty data changes just because read-only requests
were served.

Everything is working as intended.

However at storage level, FileStorage currently also checks whether
transaction that is being committed also comes with empty data changes,
and _skips_ saving transaction into disk *at all* for such cases, even
if it has been explicitly told to commit the transaction via two-phase
commit protocol calls done at storage level.

This creates the situation, where contrary to promise in
ZODB/interfaces.py(*), after successful tpc_begin/tpc_vote/tpc_finish()
calls made at storage level, transaction is _not_ made permanent,
despite tid of "committed" transaction being returned to caller. In other
words FileStorage, when asked to commit a transaction, even if one with
empty data changes, reports "ok" and gives transaction ID to the caller,
without creating corresponding transaction record on disk.

This behaviour is

a) redundant to application-level avoidance to create empty transaction
   on storage described in the beginning, and

b) creates problems:

The first problem is that application that works at storage-level might
be interested in persisting transaction, even with empty changes to
data, just because it wants to save the metadata similarly to e.g.
`git commit --allow-empty`.

The other problem is that an application view and data in database
become inconsistent: an application is told that a transaction was
created with corresponding transaction ID, but if the storage is
actually inspected, e.g. by iteration, the transaction is not there.
This, in particular, can create problems if TID of committed transaction
is reported elsewhere and that second database client does not find the
transaction it was told should exist.

I hit this particular problem with wendelin.core. In wendelin.core,
there is custom virtual memory layer that keeps memory in sync with
data in ZODB. At commit time, the memory is inspected for being dirtied,
and if a page was changed, virtual memory layer joins current
transaction _and_ forces corresponding ZODB.Connection - via which it
will be saving data into ZODB objects - to join the transaction too,
because it would be too late to join ZODB.Connection after 2PC process
has begun(+). One of the format in which data are saved tries to
optimize disk space usage, and it actually might happen, that even if
data in RAM were dirtied, the data itself stayed the same and so nothing
should be saved into ZODB. However ZODB.Connection is already joined
into transaction and it is hard not to join it because joining a
DataManager when the 2PC is already ongoing does not work.

This used to work ok with wendelin.core 1, but with wendelin.core 2 -
where separate virtual filesystem is also connected to the database to
provide base layer for arrays mappings - this creates problem, because
when wcfs (the filesystem) is told to synchronize to view the database
@tid of committed transaction, it can wait forever waiting for that, or
later, transaction to appear on disk in the database, creating
application-level deadlock.

I agree that some more effort might be made at wendelin.core side to
avoid committing transactions with empty data at storage level.

However the most clean way to fix this problem in my view is to fix
FileStorage itself, because if at storage level it was asked to commit
something, it should not silently skip doing so and dropping even non-empty
metadata + returning ok and committed transaction ID to the caller.

As described in the beginning this should not create problems for
application-level ZODB users, while at storage-level the implementation
is now consistently matching interface and common sense.

----

(*) tpc_finish: Finish the transaction, making any transaction changes permanent.
    Changes must be made permanent at this point.
    ...

    https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L828-L831

(+) https://lab.nexedi.com/kirr/wendelin.core/blob/9ff5ed32/bigfile/file_zodb.py#L788-822

Tests running and passing on windows.

(cherry picked from commit 87fd29eb)

This backports the following commits:
  6d673057
  244bb92b

... instead of silently creating empty database on such opens.

Use-case for this are utilities like e.g. zodbdump and zodbcmp which
expect such storage opens to fail so that the tool can know there is no
such storage and report it to user.

In contrast current state is: read-only opens get created-on-the-fly
empty storage with no content, but which can be iterated over without
getting any error.

This way e.g. `zodbdump non-existent.fs` produces empty output _and_
exit code 0 which is not what caller expects.

(cherry picked from commit 30bbabf1)

This backports a change from commit 227953b9.

NEO, as well as ZEO+server_sync (ERP5 backports this feature with a
monkey-patch), pings the server (primary master node in the case of NEO) on
new transactions. However, this round-trip is actually performed by the thread
that also does tasks requiring to lock the DB, like processing of invalidations.

Since transaction 1.6.1 (more precisely commit e581a120a6), IStorage.sync()
is called indirectly by DB.open() when a transaction has already begun,
and the DB must not be locked when this happens.

Doing so leads to race conditions.

In particular, there can be an AttributeError.

See https://github.com/zodb/zodbshootout/issues/26 for details.

(cherry picked from commit f8cf23ec)

Don't require persistent at setup time. We don't build native code that needs those headers anymore. Fixes #119.

(cherry picked from commit d7dae8b1)

This is also a workaround for pypa/setuptools#864, since the default pypy3 on
Travis is quite old.

(cherry picked from commit def73970
                       and 5dcca55e)

Fixed to work with transaction 2.0.3.

Simplify the README file to avoid out of date information

We have the same information scattered around in different places,
which increases the chance that some of it will be out of date.

See the end of:
https://groups.google.com/forum/#!topic/zodb/fy6RRVAF9-s

I want to try to avoid duplicating zodb.org.

Clear Connection.transaction_manager on close. Fixes #114 for ZODB 4.

Call _p_resolveConflict() even if a conflicting change doesn't change the state

This reverts to the behaviour of 3.10.3 and older.

checkTransactionalUndoIterator: do not expect iterator to return sorted oids

fstail: print the txn offset and header size, instead of only the data offset

Before:

    2016-07-01 09:41:50.416574: hash=d7101c5ee7b8e412d7b6d54873204421e09b7f34
    user='' description='' length=1629 offset=58990284

After:

    2016-07-01 09:41:50.416574: hash=d7101c5ee7b8e412d7b6d54873204421e09b7f34
    user='' description='' length=1629 offset=58990261 (+23)

The structure of a FileStorage DB is such that it's easy to revert the last
transactions, by truncating the file at the right offset. With the above
change, `fstail` can now be used to get this offset.

In the above example:

    truncate -s 58990261 Data.fs

would delete the transaction and all those after.

Refactored FileStorage transactional undo

Fix handle_all_serials for the new and old protocols.

As part of a project to provide object-level commit locks for ZEO, I'm
refactiring FileStorage to maintain transaction-specific data in
Tranaction.data.  This involved undo.  In trying to figure this out, I
found:

- A bug in _undoDataInfo, which I verified with some tests and

- _transactionalUndoRecord was maddeningly difficult to reason about
  (and thus change).

I was concerned less by the bug than my inability to know whether a
change to the code would be correct.

So I refactored the code, mainly transactionalUndoRecord, to make the
code easier to understand, fixing some logic errors (I'm pretty sure)
along the way.  This included lots of comments. (Comments are much
easier to compose when you're working out logic you didn't
understand.)

In addition to makeing the code cleaner, it allows undo to be handled
in cases that weren't handled before.