Before the patch if storage.watcher fails, storage.driver.Close is not
called, and so the driver will continue to send to .drvWatchq, but noone
is receiving from it.

a5dbb92b (go/zodb: Require drivers to close watchq on Close), provides
the guarantee that the driver will stop sending on drvWatchq right after
drv.Close call.

Provide guaranty that Close forces the driver to stop sending to watchq
and to close it. See a5dbb92b ("go/zodb: Require drivers to close watchq
on Close") for details.

Without the fix TestWatch fails with test timeout:

    panic: test timed out after 30s

    # Close waits for serve to stop
    goroutine 93 [semacquire]:
    sync.runtime_Semacquire(0xc000152170)
            /home/kirr/src/tools/go/go/src/runtime/sema.go:56 +0x45
    sync.(*WaitGroup).Wait(0xc000152168)
            /home/kirr/src/tools/go/go/src/sync/waitgroup.go:130 +0x65
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zLink).Close(0xc0001520f0, 0x1313, 0x1)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zrpc.go:159 +0x47
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zeo).Close(0xc000313680, 0xc000107c78, 0x1)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo.go:526 +0x2e
    lab.nexedi.com/kirr/neo/go/internal/xtesting.DrvTestWatch(0xc000082c00, 0xc0000aa2a0, 0x24, 0x6a4a38)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/internal/xtesting/xtesting.go:442 +0xdb5
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.TestWatch.func1(0xc000082c00, 0x6e3498, 0xc00009a380)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:270 +0x99
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func2.1(0xc0000a4168, 0x16)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:207 +0xfb
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func1(0xc000082c00, 0xc00009c180)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:186 +0x129
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.withZEOSrv.func2(0xc000082c00)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo_test.go:199 +0x10e
    testing.tRunner(0xc000082c00, 0xc00009c160)
            /home/kirr/src/tools/go/go/src/testing/testing.go:1194 +0xef
    created by testing.(*T).Run
            /home/kirr/src/tools/go/go/src/testing/testing.go:1239 +0x2b3

    # serve is stuck in invalidateTransaction doing watchq<-
    goroutine 26 [chan send]:
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zeo).invalidateTransaction(0xc000313680, 0x6417e0, 0xc000323b60, 0x0, 0x0)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zeo.go:176 +0x373
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zLink).serveRecv1(0xc0001520f0, 0xc000393890, 0x0, 0x0)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zrpc.go:225 +0x4b4
    lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zLink).serveRecv(0xc0001520f0)
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zrpc.go:176 +0x8d
    created by lab.nexedi.com/kirr/neo/go/zodb/storage/zeo.(*zLink).start
            /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/zeo/zrpc.go:99 +0xc8

Provide guaranty that Close forces the driver to stop sending to watchq
and to close it. See a5dbb92b ("go/zodb: Require drivers to close watchq
on Close") for details.

If we don't require drivers to stop sending to watchq after Close, there
could be many deadlock scenarios, for example:

- client called drv.Close(); no longer listens to watchq; driver is
  stuck sending to watchq, or

- client called drv.Close(), which itself waits for tasks spawned inside
  driver to complete, which are stuck sending to watchq because client
  no longer does <-watchq.

The change is similar in spirit to safety guaranty provided by high-level
Watcher where after DelWatch call it is guaranteed that there will be no
more sends to subscribed watchq (see c41c2907 "go/zodb: High-level
watching - initial draft") for details.

All drivers don't provide requested guarantee yet.
We'll be fixing them one-by-one in followup commits.

Else, on an error, it is the lineno of `t.Fatal(err)` inside FatalIf
that is printed, not the line number inside user test.

A data record with len(data)=0 and backpointer=0 is considered by
FileStorage/py as "no data":

https://github.com/zopefoundation/ZODB/blob/5.6.0-15-g22d1405d4/src/ZODB/FileStorage/FileStorage.py#L576-L582

Even though currently it is not possible to create such data record via
FileStorage(py).deleteObject (becase it raises POSKeyError if there is
no previous object revision), being able to use such data records is
semantically useful in overlayed DemoStorage settings, where δ part
marks an object that exists only in base with delete record whiteout.

It is also generally meaningfull to be able to create "delete" record
even if object was not previously existing: "deleteObject" is actually
similar to "store" (and so should be better named as "storeDelete"). If
one wants to store deletion, there should not be a reason to reject it,
because deleteObject already has seatbelt in the form of oldserial, and
if the user calls deleteObject(oid, oldserial=z64), he/she is already
telling that "I know that this object does not exist in this storage
(oldserial=z64), but still please create a deletion record for it". Once
again this is useful in overlayed DemoStorage settings described above.

For the reference, such whiteout deletion records pass ZODB/scripts/fstest just fine.

Even though FileStorage/py loads such data records just fine, on
FileStorage/go side it was not the case - DataHeader.LoadBackRef, even
with backpointer=0, was verifying that backpointer to be valid and
failing seeing it might overlap with current transaction:

    === RUN   TestLoadWhiteout
    2021/03/17 06:40:58 index load: open testdata/whiteout.fs.index: no such file or directory
    2021/03/17 06:40:58 testdata/whiteout.fs: index rebuild...
        filestorage_test.go:398: load 0000000000000017:0000000000000001: bad err:
            have: testdata/whiteout.fs: load 0000000000000017:0000000000000001: testdata/whiteout.fs: data record @27: check: backpointer (0) overlaps with txn (4)
            want: testdata/whiteout.fs: load 0000000000000017:0000000000000001: 0000000000000001: object was not yet created

It was a thinko: backPos==0 was already kind of handled in LoadBackRef,
but only in one verification case. -> Fix all checks not to trigger when
seeing backPos=0. DataHeader.LoadBack - the caller of LoadBackRef -
already handles returned backPos=0 as "no data".

We were reusing Dumper instance in between testDump subtests. This was
not noticed, as it was only "1" and then "empty" case, because "emtpy"
has no transactions. However in the next patch we'll add another
subcase, and if the dumper instance is not reset, it will think that it
starts from transaction number non-zero, which would differ from fresh
dumper output.

-> Fix it.

As Zodbtools dropped ZODB3 support its run_with_zodb3py2_compat was
renamed to run_with_zodb4py2_compat:

nexedi/zodbtools@c59a54ca

Encode/decode was deprecated and removed in recent github.com/shamaton/msgpack.

NEO/py switched from UUID to 4-byte NodeID long time ago in
nexedi/neoppod@23fad3af and has a TODO to
rename uuid to nid.

NEO protocol also talks about renaming uuid to nid:

    note: the uuid variable should be renamed into nid
    https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow#/9/5

NEO/go can do the rename now.

It is the same as regular SHA1 but returns all-zeros for empty input.
Besides there is runtime knob to skip checksum computation to benchmark
how much time and latency SHA1 computation introduces.

See the following links for some preliminary history:

- bc2cddfc
- https://navytux.spb.ru/~kirr/neo.html#results-and-discussion

See previous commits.

To use something like

	defer task.Running(&ctx, "my task")(&err)

that will create the task, trace its begin/end and integrate with return
error.

Probably this package should be integrated one way or another with
go/internal/xcontext/task.

Package log provides logging with severity levels and integration with tasks.

Package task provides primitives to track tasks via contexts.

NEO/go server accepts preferred encoding of client from start, but
NEO/py server implements either N or M encoding and disconnects client
silently if handshake is not exactly what is expected.

However we can teach Dial to retry with different preferred options and
this way when connection to a NEO/py server, it will essentially
autodetect which encoding is used and accepted by remote peer.

This patch adds support for serializing packet frames with M encoding on
the wire. To do so it follows rules defined in

    	nexedi/neoppod@9d0bf97a
    	( nexedi/neoppod!11 )

Server handshake is reworked to autodetect client's preferred encoding.
Client always prefers 'N' for now.

This patch adds proto.Encoding('M') and teaches it to encode/decode
messages via MessagePack by the rules defined in

	nexedi/neoppod@9d0bf97a
	( nexedi/neoppod!11 )

It only adds support for messages serialization, without changing
proto.go to match changes in e.g. enums reordering, and without adding
support for MessagePack at link layer in neonet.

M-encoding was only tested for NEO/go-NEO/go, and was not yet tested for
NEO/go-NEO/py interoperation. There will be likely small mistakes
discovered on my side that should be hopefully easy to fix step by step
once we get to that phase.

It complements github.com/tinylib/msgp with things that neo/proto needs
to encode/decode messages.

It is noop for 'N' encoding, but will be non-empty for MessagePack.

genSliceHead and genMapHead will be reused for MsgPack encoding.
Changes in generated code are mostly due to move of where things are
typecasted to e.g. uint32 without any change in semantic.

For decode-family of functions protogen.go sticks to convention to name
target variable to where to assign the result as "assignto", not "path".

No change in generated code.

Encoding specifies a way to encode/decode NEO messages and packets.
Current way of how messages were encoded is called to be 'N' encoding.

This patch:

- adds proto.Encoding type
- changes MsgEncode and MsgDecode to be methods of Encoding
- renames thigs that are specific to 'N' encoding to have 'N' suffix
- changes tests to run a testcase agains vector of provided encodings.
  That vector is currently only ['N'].

And provide only single top-level entry-points to encode/decode
messages. As of now the entry points are just plain forwarding, but with
introducing of msgpack and encodings, they will take into account
through which encoding a message has to be encoded/decoded.

- Regrouping messages would change their message codes, thus introducing
  backward compatibility breakage.

- The protocol itself is currently kind-of documented in separate
  document here:

  https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow

It is internal detail that ideally should not get outside of proto
package. Don't clutter documentation of all messages with it.

-> For now access message code via proto.MsgCode(msg) from outside.

Generating code through sizer can already give answer if a type is fixed
size or not. And in practice we don't care at all if it will be a bit
slower compared to previous explicit type switch.

For current code it is just a small cleanup, and it will be handy to
support messagepack encoding as well.

No change in generated code.

Not used yet, but they will be needed for msgpack codec.

Besides proto.customCodec we will need to lookup more objects.

We previously used criteria that sizeof(typ.Elem()) == 1, but it would
also trigger on e.g. `struct { x byte }` which we would not want to
consider as byte array.

-> Clarify the test to explicitly test for typ.Elem() == byte.
No change in generated code.

Without the fix any pointer field of a struct would be silently skipped
without any serialization code generated for it. With the patch default
case triggers in and reports detail where such type is and in which
struct/field it is used.

Besides *mem.Buf - that is handled specially - we don't have pointers in
proto.go, so this currently does not affect generated code at all.

Noticed accidentally.

Without that, when lonet will be used in tests, port won't be correctly
extracted from an address.

With upcoming msgpack encoding packets might be with data < PktHeaderSize.
Adjust pktAlloc to unconditionally provide the invariant.

Preparatory step before we introduce support for msgpack encoding.

It is not used anywhere and would have to be reworked with upcoming
introduction of msgpack encoding. -> Simply remove that.

Just renaming.

This represents a neonet testing environment and in the future will be
used to run tests under different protocol versions, protocol encodings, etc.

For now it is noop wrapper around testing.T + t.bin that wraps []byte
and will be amended to return something different depending on t's
encoding.

Previously we were doing handshake symmetrically: both client and server
were transmitting hello and receiving peer's hello simultaneously.
However this does not allow server to adjust its behaviour depending on
which client (protocol version, protocol encoding, ...) is connecting to it.

-> Rework handshake so that client always sends its hello first, and
only then the server side replies. This matches actual NEO/py behaviour:

https://lab.nexedi.com/nexedi/neoppod/blob/v1.12-67-g261dd4b4/neo/lib/connector.py#L293-294

even though the "NEO protocol" states that

	Handshake transmissions are not ordered with respect to each other and can go in parallel.

	( https://neo.nexedi.com/P-NEO-Protocol.Specification.2019?portal_skin=CI_slideshow#/9/2 )

If I recall correctly that sentence was authored by me in 2018 based on
previous understanding of should-be full symmetry in-between client and
server.

However soon we are going to teach server sides to autodetect client
encoding and adjust server to talk to client via its preferred way.
This needs handshake for client and server to be differentiated.

The protocol needs to be adjusted as well. However I'm not sure it is
going to happen...

This will be handy to test underlying error for e.g. EOF with
errors.Cause or errors.Is in case that underlying error is also error
wrapper.