ashex gives bytes, whereas reference_tid was str.

The sequence cannot be randomly accessed, e.g.

    In [5]: d = {1:2}

    In [6]: kv = d.keys()

    In [7]: kv
    Out[7]: dict_keys([1])

    In [8]: kv[0]
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-8-643f90e1910b> in <module>()
    ----> 1 kv[0]

    TypeError: 'dict_keys' object is not subscriptable

-> Use list(dict.keys()) in places where we need random access.

Otherwise it breaks with str on py3:

	In [1]: from io import BytesIO

	In [2]: BytesIO("abc")
	---------------------------------------------------------------------------
	TypeError                                 Traceback (most recent call last)
	<ipython-input-2-52a130edd46d> in <module>()
	----> 1 BytesIO("abc")

	TypeError: a bytes-like object is required, not 'str'

Zodbdump format is text-binary and is saved into files opened in binary
mode. -> We have to emit bytes - not strings - into it, since otherwise
on Python3 it would break.

This needs qq support from pygolang[1] to be able to use qq with both
string and bytestring format, e.g. for

	 "hello %s" % qq(name),	and
	b"hello %s" % qq(name)

to give the same output irregardless of whether name is str or bytes.

[1] nexedi/pygolang!1

Zodbdump format is already described as semi text-binary in top-level
zodbdump.py documentation. However zdump() docstring was referring to it
as "text". Fix it and use binary to handle places where zdump is
loaded/saved.

%r has different output for strings and bytes on python3:

	In [1]: a = 'hello'
	In [2]: b = b'hello'

	In [3]: repr(a)
	Out[3]: "'hello'"

	In [4]: repr(b)
	Out[4]: "b'hello'"

-> Use qq whose output is stable irregardless of whether input is string or bytes.

zodbinfo: Provide "head" as command to query DB head; Turn "last_tid" into deprecated alias for head

Similarly to go version: neo@151d8b79.

Starting with upcoming ZODB 5.5.2 ZODB tries to preserve
`extension_bytes` transaction metadata property in the raw form as it
was stored on disk in the database:

    https://github.com/zopefoundation/ZODB/commit/2f8cc67a

However now when running test/gen_testdata.py with ZODB with that patch (and
gen_testdata.py refuses to work if it detects that ZODB does not properly
supports .extension_bytes property because we want it to be present in the
generated test database [1,2]) it now breaks:

    $ ./gen_testdata.py
    Traceback (most recent call last):
      File "./gen_testdata.py", line 230, in <module>
        main()
      File "./gen_testdata.py", line 224, in main
        gen_testdb("%s.fs" % dbname, zext=zext)
      File "./gen_testdata.py", line 194, in gen_testdb
        stor.tpc_begin(txn)
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/BaseStorage.py", line 193, in tpc_begin
        ext = transaction.extension_bytes
    AttributeError: 'Transaction' object has no attribute 'extension_bytes'

The breakage is because, as specified in ZODB interfaces[3,4], storage requires
ZODB.IStorageTransactionMetaData, not transaction.ITransaction instance
gen_testdata.py was using. The script used to work before just by luck.

The fix is to convert transaction instance into storage transaction metadata
object for the place where we talk to storage at raw level.

HOWEVER, when checking regenerated database and its dump I noticed:

ZODB >= 5.4.0 uses pickle protocol 3 on both python2 and python3

    https://github.com/zopefoundation/ZODB/commit/12ee41c4

In other words it saves e.g. OID of an object as pickle binary, which decodes
as bytes on py3 and zodbpickle.binary on py2 when decoding via zodbpickle.
However it will result in *DecodeError* when decoding on py2 with standard
pickle module. The latter means that ZODB3 will _fail_ to load data from test
database, because ZODB3 - contrary to ZODB4 and ZODB5 - uses std pickle module,
not zodbpickle.

We still care about ZODB3 and in particular it is included into
zodbtools test matrix:

    https://lab.nexedi.com/nexedi/zodbtools/blob/7bc0385e/tox.ini#L9-14

so we cannot break it.

-> Temporarily patch ZODB at runtime to make sure it emits data with
older protocol and without using zodbpickle.binary for oid, so that
generated test database could be loaded on ZODB3 as well.

gen_testdata.py now works with latest ZODB, but produces exactly the
same bit-to-bit output as before.

[1] https://lab.nexedi.com/nexedi/zodbtools/blob/7bc0385e/zodbtools/test/gen_testdata.py#L215-217
[2] https://lab.nexedi.com/nexedi/zodbtools/blob/7bc0385e/zodbtools/test/testutil.py#L31-63
[3] https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L815-L818
[4] https://github.com/zopefoundation/ZODB/blob/5.5.1-35-gb5895a5c2/src/ZODB/interfaces.py#L538-L575

/reviewed-on nexedi/zodbtools!15

Zodbdump format is mixed text+binary so dumping to unicode stdout won't
work.

Based on patch by Jérome Perrin.

Because on Py3:

        def test_dumpreader():
            in_ = b"""\
        txn 0123456789abcdef " "
        user "my name"
        description "o la-la..."
        extension "zzz123 def"
        obj 0000000000000001 delete
        obj 0000000000000002 from 0123456789abcdee
        obj 0000000000000003 54 adler32:01234567 -
        obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
        ZZZZ
        obj 0000000000000005 9 crc32:52fdeac5
        ABC

        DEF!

        txn 0123456789abcdf0 " "
        user "author2"
        description "zzz"
        extension "qqq"

        """

            r = DumpReader(BytesIO(in_))
            t1 = r.readtxn()
            assert isinstance(t1, Transaction)
    >       assert t1.tid == '0123456789abcdef'.decode('hex')
    E       AttributeError: 'str' object has no attribute 'decode'

    test/test_dump.py:77: AttributeError

Based on patch by Jérome Perrin.

	self = <zodbtools.util.CRC32Hasher object at 0x7f887ae465f8>

	    def __init__(self):
	>       self._h = crc32('')
	E       TypeError: a bytes-like object is required, not 'str'

	util.py:208: TypeError

Based on patch by Jérome Perrin.

	data = 'data1'

	    def sha1(data):
	        m = hashlib.sha1()
	>       m.update(data)
	E       TypeError: Unicode-objects must be encoded before hashing

	zodbtools/util.py:38: TypeError

Based on patch by Jérome Perrin.

	s = b'\x03\xc4\x85v\x00\x00\x00\x00'

	    def ashex(s):
	>       return s.encode('hex')
	E       AttributeError: 'bytes' object has no attribute 'encode'

	zodbtools/util.py:29: AttributeError

s.encode('hex') used to work on Py2 but fails on Py3:

	In [1]: s = "abc"

	In [2]: b = b"def"

	In [3]: s.encode('hex')
	---------------------------------------------------------------------------
	LookupError                               Traceback (most recent call last)
	<ipython-input-3-75ae843597fe> in <module>()
	----> 1 s.encode('hex')

	LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs

	In [4]: b.encode('hex')
	---------------------------------------------------------------------------
	AttributeError                            Traceback (most recent call last)
	<ipython-input-4-ec2fccff20bc> in <module>()
	----> 1 b.encode('hex')

	AttributeError: 'bytes' object has no attribute 'encode'

	In [5]: import codecs

	In [6]: codecs.encode(b, 'hex')
	Out[6]: b'646566'

	In [7]: codecs.encode(s, 'hex')
	---------------------------------------------------------------------------
	TypeError                                 Traceback (most recent call last)
	/usr/lib/python3.7/encodings/hex_codec.py in hex_encode(input, errors)
	     14     assert errors == 'strict'
	---> 15     return (binascii.b2a_hex(input), len(input))
	     16

	TypeError: a bytes-like object is required, not 'str'

	The above exception was the direct cause of the following exception:

	TypeError                                 Traceback (most recent call last)
	<ipython-input-7-7fcb16cead4f> in <module>()
	----> 1 codecs.encode(s, 'hex')

	TypeError: encoding with 'hex' codec failed (TypeError: a bytes-like object is required, not 'str')

After the patch it works with bytes and raises for str.
Fromhex does not need to be changed - it already uses codecs.decode way as
originally added in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump
stream).

Based on patch by Jérome Perrin.

There is no cStringIO on Python3:

	test_dump.py:26: in <module>
	    from cStringIO import StringIO
	E   ModuleNotFoundError: No module named 'cStringIO'

Based on patch by Jérome Perrin.

This makes zodb command driver tests added in the previous patch to pass
on both python2 and python3.

----

kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`.

Added test currently passes on py2, but fails on py3:

	out = <_io.TextIOWrapper encoding='UTF-8'>

	    def usage(out):
	        print("""\
	    Zodb is a tool for managing ZODB databases.

	    Usage:

	        zodb command [arguments]

	    The commands are:
	    """, file=out)

	        cmdv = command_dict.keys()
	>       cmdv.sort()
	E       AttributeError: 'dict_keys' object has no attribute 'sort'

	zodbtools/zodb.py:55: AttributeError

It will be fixed in the next patch.

----

kirr: use loggging as log and keep emitting warnings on one line.

this silents a warning about \w being unknown escape sequence

----

kirr: preserved _obj_re definition to be on 1 line.

maxtid is in ZODB.utils starting only from ZODB5.

ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
txn.extension.

IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in
dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream).

Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
extensions, which works if ZODB is patched for txn.extension_bytes
support, but fails on pristine ZODB.

Support for txn.extension_bytes cannot get into upstream ZODB for
more than a year:

https://github.com/zopefoundation/ZODB/pull/183
https://github.com/zopefoundation/ZODB/pull/207

and  even if it somehow will make it, it will likely be only in ZODB5,
while we still care to support ZODB4 and ZODB3.

Skipping zodbdump / zodbcommit tests, if a ZODB does not have
txn.extension_bytes support, would result in significant reduction of
zodbtools test coverage, because practically that is the current
situation with all upstream ZODB{3,4,5}. Dropping test coverage for
non-empty extensions is neither a good option.

For those reason, let's rework the tests and test both zodbdump and
zodbcommit with two scenarios:

1. on a test database where transactions extensions are always empty.
   This should work on all ZODB irregardless of whether
   txn.extension_bytes patch is there or not.

2. on a test database where transactions extensions are present.
   This should work if ZODB has txn.extension_bytes support, but if not,
   we can mark this case as xfail, since the failure is expected.

This way we make the testsuite pass irregardless of whether
txn.extension_bytes support is there, and we don't abandon dump/commit
testing coverage.

/helped-by Jérome Perrin <jerome@nexedi.com>

To keep a consistent output.

and use six.moves for python3 compatibility.

Previously we were using "anydbm" which selects dbhash, gdbm or dbm, but
opening the db with the f flag that's only valid for gdm, so de-facto we
were supporting only gdbm.

this also solves the following error on python3:

 AttributeError: 'dict_keys' object has no attribute 'sort'

- add help tidrange topic.
- change all commands to refer to it.
- add TODO to parse tid from absolute and relative dates (e.g.
  1.month.ago, similarly to how git can do). Dateparser

	https://dateparser.readthedocs.io/

  will probably be of help here.

/reviewed-on nexedi/zodbtools!7

Currently zodbanalyze analyzes whole storage. However it becomes
non practical to make a full zodbanalyze run on whole storage because
usually there are many transactions and objects and the time to run full
zodbanalyze is huge.

However, similarly to zodbdump, we can teach zodbanalyze to analyze a
particular range of transactions. This should help to analyze a range of
changes for e.g.  yesterday, or for last week or similar.

/reviewed-on nexedi/zodbtools!7

Analyze uses regular ZODB storage API: .iterator() & friends. This way
it should be possible apply it not only to FileStorage, but to other
type of storages as well - for example to NEO and ZEO.

Use zodbtools.util.storageFromURL to open a storage by knowing its URL.
Preserve support to directly apply zodbanalyze to FileStorage deltas.

/reviewed-on nexedi/zodbtools!7

Storages need to be closed to indicate a clean access end. If a storage is
not closed cleanly it might require to spend time and resources on next
open. For example FileStorage might need to recompute the index.

/reviewed-on nexedi/zodbtools!6

Zodbcommit reads transaction description from stdin and commits read data into
ZODB. The transaction to be committed is read in zodbdump format, but without
first 'txn' header line. For example:

    user "author"
    description "change 123"
    extension ""
    obj 0000000000000001 4 null:00
    ZZZZ

This tool could be useful for testing and for low-level database
maintenance. Please see zodbcommit.py docstring for more details.

We will likely need this reader for `zodb restore` in the future.
We will also use this reader for `zodb commit` in the next patch.

pygolang dependency v↑ becuase we use recently introduced
golang.strconv to unquote user/desc/extension strings.

Python2 works. Python3 support is only minimal and incomplete.

hashRegistry will be needed for zodbdump reader to create a hasher by
its name. Since std hashlib does not have adler32/crc32 nor null - let's
add our own implementation for those hashers too.

The code is based on

	neo@a60c472c, and
	neo@3f578560

Before now we were emitting extra LF only in between transactions as a
separator. However the dump format states LF always goes after
transaction, and there is a reason for it:

	without LF in the end, it becomes ambiguous at EOF - whether it
	is a proper transaction end, or the transaction was cut.

So avoid the ambiguity by always emitting trailing LF after transaction
record.

I originally added escapeqq as part of 75c03368 (zodbdump: Start to
stabilize output format) with the task for this utility to quote string
into valid "..." string always quoted with ".

This utility was later copied to pygolang:

	pygolang@afa46cf5

and then further improved there to work under both Python2 and Python3
and to not escape printable UTF-8 characters:

	pygolang@02dddb97

So stop the duplication and simply switch to the better version.