Hello up there.

Recently with Ivan we needed a way to obtain last transaction ID of a ZODB storage for resiliency checking. For this `zodb info` utility to print general information about a ZODB database is introduced. Along the way, as it was already planned, all zodbtools utilities are now covered by only one `zodb` command which can invoke other subcommands and show help topics. I also used the occassion to switch how storages are specified from being  ZConfig-file-always to be specified by URL e.g.

     neo://neo1@127.0.0.1:24573
     zeo://...
     file://...

without loosing generality because zconfig:// scheme is also supported.

Please find more details about all changes in individal commit messages.

Thanks for feedback,  
Kirill

/cc @kazuhiko, @jm, @vpelletier, @jerome, @Tyagov, @klaus, @alain.takoudjou, @rafael
/reviewed-on nexedi/zodbtools!2

Either all general parameters at once:

    $ zodb info neo://neo1@127.0.0.1:24573
    name=NEOStorage(neo1)
    size=0
    last_tid=03be7484ddc7f6ee

or one particular parameter:

    $ zodb info neo://neo1@127.0.0.1:24573 last_tid
    03be7484ddc7f6ee

Most of our tools need only read access for working. However e.g.
FileStorage, when opened in read-write mode, automatically creates
database file and index.

This way if database is opened in read-write mode a simple typo in path,
e.g. to `zodb dump path` would lead to:

- new database at path will be created
- the dump will print nothing (empty database)
- exit status will be 0 (ok) and no error will be reported.

For this reason it is better tools declare access level they need so for
read-only access request we can catch it with an error from storage.

This, however, requires quite recent ZODB to work:

https://github.com/zopefoundation/ZODB/pull/153

P.S.

We don't want to force users to always specify read-only in URLs or
zconf files because:

- this is error prone
- URL or zconf can be though as of file
- when a program opens a file the program, not file, declares which type
  of access it wants.

That's why access mode declaration has to be internal.

Previously for specifying a storage we were requiring users to create
zconfig file and put information about storage there. This is not always
convenient e.g. for quickly inspecting a file storage or a NEO instance
here and there for which address and name is already at hand.

So thanks to zodburi we can switch to specifying storages by URL without
loosing generality as there is still zconfig:// schema which allows to
configure storages zconfig way.

P.S. for convenience we allow paths without schema to be treated as
    FileStorage (i.e. file:// schema is implicitly used).

P.P.S. zodbanalyze is not affected (for now ?) as it currently
    works with FileStorage only.

[1] http://docs.pylonsproject.org/projects/zodburi/

We already have 3 commands in zodbtools suite (zodbanalyze, zodbdump &
zodbcmp) and this is going to grow. And it was already noted some time
ago with TODO (in 66946b8d) that we need only one command driver to
invoke everything.

So do it: introduce `zodb` command which can invoke other subcommands
and show general help or help for subcommand or a topic.

The structure is modelled after `git` and `go` commands. Help topics
are for now empty but we'll add one help topic in the next patch.

- Nexedi adheres to GPLv3+ with wide exception for Free/Open-source
  software. Use this for our original work.

- Zodbanalyze was initially developed by Zope corp & co. See

  d86d04dc	(initial copy of ZODB3-3.9.7's ZODB/scripts/analyze.py.)
  ab17cf2d	(Hook in Nexedi's zodbanalyze)

  So follow original Zope's licensing terms for zodbanalyze.

The package was named zodbtools from start (see fd6ad1b9 "Start of
zodbtools.git"). However in 66a03ae5 I've made a thinko naming some
parts as zodbtool.

Fix the confusion and name things uniformly as zodbtools.

/reviewed-by TrustMe

zodbanalyze was added in 1e506a81 and ab17cf2d.

/reviewed-on !1
/see-also slapos!116

This is enhanced version of ZODB original scripts/analyze.py .
Nexedi version was originally developed as part of ERP5 here:

https://lab.nexedi.com/nexedi/erp5/commits/master/erp5/util/zodbanalyze

It was agreed to move zodbanalyze to zodbutils

/see nexedi/slapos!116

* it is faster.
* it does not require Products.

(where `python setup.py sdist` puts result tarballs)

WARNING output format is not stable yet.

This is a tool to compare two ZODB databases in between tidmin..tidmax
transaction range with default range being -∞..+∞ - (whole database).

For comparision both databases are scanned at storage layer and every
transaction content is compared bit-to-bit between the two. The program stops
either at first difference found, or when whole requested transaction range is
scanned with no difference detected.

Database storages are specified in files with ZConfig-based storage definition, e.g.

    %import neo.client
    <NEOStorage>
        master_nodes    ...
        name            ...
    </NEOStorage>

Please see neoppod!4 for
one of possible contexts.

The tool is generic though and is not NEO-specific. It should be able to
even check two different storages like ZEO & NEO, or FileStorage and NEO
etc and thus can be handy.

TODO tests

Project to be a place for various ZODB utilitieis. Jim Fulton suggested
to have it separate from ZODB/scripts/ [1] and now we have it here.

[1] https://github.com/zopefoundation/ZODB/pull/128#issuecomment-260970932