FileStorage testdata now lives in different place(s) and is different
because gen_testdata.py in zodbtools changed.

-> Adjust fs1 testdata path to load and regenerate neo/storage/sqlite from it.

* y/py3: (32 commits)
  go/zodb: Fix _TestPersistentMapListLoad on py3_pickle3
  go/zodbpickle: Switch to unpickle with PyDict=y mode
  go/go.mode: v↑ github.com/kisielk/og-rek to pre-release with PyDict patches
  go/zodb/btree: go/zodb/fs1tools: Test it on all py2/py3 ZODB kinds of data we care about
  go/zodb/fs1: Remove old testdata files
  go/internal/xtesting: Switch to updated FileStorage testdata to validate LoadDBHistory
  go/zodbpickle: Switch to pickling with protocol=3
  go/zodb/zeo: Fix ZRPC/pickle binary and string encoding/decoding wrt py3
  go/zodb/zeo: Test it on all py2/py3 ZODB kinds of data we care about and wrt both ZEO/py2 and ZEO/py3
  go/zodb/demo: Test it on all py2/py3 ZODB kinds of data we care about
  go/zodb/zodbtools: Test it on all py2/py3 ZODB kinds of data we care about
  go/zodb/fs1tools: test: Normalize dumpOk from py3 to py2 syntax
  go/zodb/fs1tools: Test it on all py2/py3 ZODB kinds of data we care about
  go/zodb/fs1tools: DumperFsDump: Print size of every transaction record
  go/zodb/fs1tools: Regenerate testdata with ZODB 5.8.1
  go/zodb/fs1: Fix index save wrt py3
  go/zodb/fs1: Fix index load from py3 data
  go/zodb/fs1: Accept both FS21 and FS30 magics
  go/zodb/fs1: Test FileStorage on all py2/py3 ZODB kinds of data we care about
  go/zodb: Fix PyData.ClassName for py3_pickle3
  ...

plist is generated to come with str in [0]. On py2 that is bytestring,
while on py3 it is unicode. However were trying to decode it as
bytestring only and so it was failing as

    --- FAIL: TestPersistentMapListLoad/py3_pickle3 (0.73s)
        persistent_x_test.go:82: plist[0]: got string  ; expect str

-> Fix it by using ogórek.AsString helper as suggested by
StrictUnicode=y mode documentation in https://github.com/kisielk/og-rek/commit/b28613c2.

Previously we were unpickling dicts into map[any]any and it was making a
difference from which Python version the dict was saved: if it was saved
from py2 with str keys, then the dict needed to be accessed with ByteString
keys, while if it was saved from py3 with str keys, the dict needed to
be accessed with string keys. This peculiarity is one of the reasons why
Map test currently fails on py3_pickle3 - because there - both in the
test and inside Map.PySetState, dictionaries are accessed via ByteString
keys which works only wrt py2-saved data.

-> Fix it by switching the unpickler to work in PyDict=y mode(*) which
results in dicts unpickled as ogórek.Dict instead of map[any]any
with the ogórek.Dict considering ByteString to be equal to both string and Bytes
with the same underlying content. This allows programs to access Dict via
string/bytes keys following Python3 model, while still being able to handle
dictionaries generated from under Python2.

For ZODB/go this is change in behaviour exposed to outside. However
there is currently only one known ZODB/go user - WCFS in Wendelin.core -
and that user will be updated correspondingly as well.

The change fixes the following failure in go/zodb tests

    --- FAIL: TestPersistentMapListLoad/py3_pickle3 (0.70s)
        persistent_x_test.go:57: persistent.mapping.PersistentMapping(0000000000000000): activate: pysetstate: noone of ["data" "_container"] is present in state dict

but one more remains there which will be addressed in the next patch.

(*) see https://github.com/kisielk/og-rek/pull/75 for details.

We will need PyDict mode in the next patch to fix Map test on
py3_pickle3 and so that working with unpickled dictionaries is easy
independently from which Python version - py2 or py3 - they come.

PyDict mode patch: https://github.com/kisielk/og-rek/commit/3bf6c92d
		  (https://github.com/kisielk/og-rek/pull/75)

Whole set of brought changes:

	https://github.com/kisielk/og-rek/compare/b1bd3f2e3d5e...e691997e3596

Previously, like with FileStorage, we were testing btree/go only with
data generated by python2 and pickle protocol=2. However even on py2
there are more pickle protocols that are in use, and also there is python3.

-> Modernize testdata/gen-testdata to use run_with_all_zodb_pickle_kinds
   that was recently added as part of nexedi/zodbtools@f9d36ba7
   and generate test data with both python2 and python3. It is handy to
   use py2py3-venv(*) to prepare python environment to do that.

   Adjust tests on Go side to verify how btree handles all generated zkinds.

All py2_pickle1, py2_pickle2, py2_pickle3 and py3_pickle3 are handled well out of the box.

(*) see nexedi/zodbtools@fac2f190

We recently modrnized FileStorage/go tests to have all ZODB kinds of
data which live in testdata/<zkind>/ for example testdata/py2_pickle3/ -
see recent patch "go/zodb/fs1: Test FileStorage on all py2/py3 ZODB
kinds of data we care about".

We also updated all other packages to use that new FileStorage testdata.

-> Now is the time for old testdata to go.

Old fs1/testdata/1.fs will be going away -> we need to switch to using
newly generated multi-zkind testdata. However for this test using only
one zkind should be enough. Let's pick what we currently primarily use:
py2_pickle3.

Pickle protocol 3 allows to natively support all py bytes and string
types and is supported by zodbpickle/py on both py2 and py3 out of the
box. ZODB/py3 is using protocol=3 since 2013 and ZODB/py2 since 2018. In
other words nowdays protocol=3 is the default protocol with which
ZODB emits pickles, and it can be read practically everywhere.

The difference in between protocol=2 and protocol=3 is addition of
BINBYTES and SHORT_BINBYTES opcodes to represent bytes. Without those
opcodes bytes are encoded as

    `_codecs.encode(byt.decode('latin1'), 'latin1')`

which, when unpickled, results in bytes on py3 and str on py2.

Compared to using `BINBYTES byt` this form is much bigger in size, but
what is more important might turn bytes back into str when decoded and
reencoded on py2.

This form of bytes encoding is also not accepted by ZEO/py which rejects
it with

    2024-07-18T20:44:39 ERROR ZEO.asyncio.server Can't deserialize message
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/server.py", line 100, in message_received
        message_id, async_, name, args = self.decode(message)
      File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/marshal.py", line 119, in pickle_server_decode
        return unpickler.load()  # msgid, flags, name, args
      File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/marshal.py", line 176, in server_find_global
        raise ImportError("Module not allowed: %s" % (module,))
    ImportError: Module not allowed: _codecs

All in all using pickle protocol=3 is ok from both py2 and py3 point of
view, brings size optimization and correctness, and fixes ZEO and
probably other issues.

So let the pickles ZODB/go saves and other Go places emit be encoded with protocol=3 now as well.

For the reference, the

    // TODO 2 -> 3 since ZODB5 switched to it and uses zodbpickle.

is from 2019 added in a16c9e06 (go/zodb: Teach Persistent to serialize
itself).

ZEO/go client was asserting "method" field to be bytestring. As the
result interoperation with ZEO/py3 server was failing as e.g.

    2024-07-18T20:41:14 INFO ZEO.asyncio.server received handshake 'Z5'
    2024/07/18 20:41:14 zlink @ - /tmp/zeo2281880331/1.fs.zeosock: serveRecv1: decode: .0: method: got string; expected str
        zeo_test.go:221: openzeo /tmp/zeo2281880331/1.fs.zeosock: zeo: open /tmp/zeo2281880331/1.fs.zeosock: zlink @ - /tmp/zeo2281880331/1.fs.zeosock: call register: zlink is closed

because ZEO/py3 sends "method" as unicode.

-> Fix it by using pickle.AsString when decoding "method" field.

Similarly ZEO/go client was sending strings as bytestr, which on py3 decode to
unicode ok only if they are ASCII. Adjust string encoding to be emitted
as unicode always which works wrt both ZEO/py3 and ZEO/py2.

--------

ZEO/go client was sending binary ZRPC arguments as bytestr as well,
which was resulting in the following exception on ZEO/py3 side:

    2024-07-18T20:42:31 ERROR ZEO.asyncio.server Can't deserialize message
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/z/ZEO/src/ZEO/asyncio/server.py", line 102, in message_received
        message_id, async_, name, args = self.decode(message)
      File "/home/kirr/src/wendelin/z/ZEO/src/ZEO/asyncio/marshal.py", line 122, in pickle_server_decode
        return unpickler.load()  # msgid, flags, name, args
    UnicodeDecodeError: 'ascii' codec can't decode byte 0x85 in position 1: ordinal not in range(128)
    2024/07/18 20:42:31 zlink @ - /tmp/zeo627526011/1.fs.zeosock: recvPkt: EOF
        xtesting.go:361: load 0285cbac70a3d733:0000000000000001: returned err unexpected:
            have: /tmp/zeo627526011/1.fs.zeosock: load 0285cbac70a3d733:0000000000000001: zlink @ - /tmp/zeo627526011/1.fs.zeosock: call loadBefore: zlink is closed
            want: nil

-> Fix it by using pickle.Bytes which should be encoded as bytes in the
   pickle stream.

This fix is ok but depends on the next patch to upgrade ZODB
serialization protocol to 3, because on pickle protocol 2 bytes are
serialized as `_codecs.encode(byt.decode('latin1'), 'latin1')` and
ZEO/py forbids import of _codecs:

    2024-07-18T20:44:39 ERROR ZEO.asyncio.server Can't deserialize message
    Traceback (most recent call last):
      File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/server.py", line 100, in message_received
        message_id, async_, name, args = self.decode(message)
      File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/marshal.py", line 119, in pickle_server_decode
        return unpickler.load()  # msgid, flags, name, args
      File "/home/kirr/src/wendelin/z/ZEO5/src/ZEO/asyncio/marshal.py", line 176, in server_find_global
        raise ImportError("Module not allowed: %s" % (module,))
    ImportError: Module not allowed: _codecs

We will address this in the next patch.

go/zodb/zeo: Test it on all py2/py3 ZODB kinds of data we care about and wrt both ZEO/py2 and ZEO/py3

Similarly to FileStorage, fs1tools and other Go packages previously we
were testing ZEO/go client only with old FileStorage testdata generated
by python2 and pickle protocol=2. However even on py2 there are more
pickle protocols that are in use, and also there is python3.

We were also testing our ZEO/go client only wrt ZEO/py2 but there is
also ZEO/py3.

-> Adjust ZEO/go testing to automatically load and test against all ZODB
kinds from recently updated FileStorage testdata and wrt both ZEO/py2
and ZEO/py3.

All py2_pickle1, py2_pickle2, py2_pickle3 and py3_pickle3 are handled well out of the box.
However only ZEO/py2 succeeds: tests wrt ZEO/py3 server currently fail
and so are marked with "xfail".

We will fix tests for ZEO/py3 in the next patch.

Similarly to FileStorage, fs1tools and zodbtools previously we were testing
demo only with old FileStorage testdata generated by python2 and pickle
protocol=2. However even on py2 there are more pickle protocols that
are in use, and also there is python3.

-> Adjust demo testing to automatically load and test agains all ZODB
kinds from recently updated FileStorage testdata.

All py2_pickle1, py2_pickle2, py2_pickle3 and py3_pickle3 are handled well out of the box.

Like with FileStorage and fs1tools previously we were testing zodbtools/go only
with data generated by python2 and pickle protocol=2. However even on py2 there
are more pickle protocols that are in use, and also there is python3.

-> Modernize testdata generation to automatically pick up all ZODB kinds
   from recently-updated FileStorage/go testdata.

   Adjust tests on Go side to verify how zodbtools/go handle all generated zkinds.

All py2_pickle1, py2_pickle2, py2_pickle3 and py3_pickle3 are handled well out of the box.

Because else, py3_pickle3 tests fail e.g. as

        --- FAIL: TestFsDump/py3_pickle3/db=1 (0.00s)
            dump_test.go:70: fsdump: dump different:
                 Trans #00000 tid=0285cbac70a3d733 size=211 time=1979-01-03 21:00:26.400001 offset=117
                -    status='p' user=b'user0.12' description=b'step 0.12'
                +    status='p' user='user0.12' description='step 0.12'

        --- FAIL: TestFsDumpv/py3_pickle3/db=1 (0.00s)
            dump_test.go:70: fsdumpv: dump different:
                 ************************************************************
                -file identifier: b'FS30'
                +file identifier: 'FS30'

        ...

fs1tools/go sticks to py2 syntax because in my view it is more user
friendly. The preference of py2 or py3 syntax needs to be made anyway
because for fs1tools/go it is not possible to know beforehand to which
py kind its output will be compared against.

I think it is not very good for fstools/py output to be dependent on
python version. For the reference on this topic: output of zodbtools/py
is designed to be the same on both py2 and py3.

Like with FileStorage previously we were testing fs1tools only with data
generated by python2 and pickle protocol=2. However even on py2 there are
more pickle protocols that are in use, and also there is python3.

-> Modernize testdata generation to automatically pick up all ZODB kinds
   from recently-updated FileStorage/go testdata.

   Adjust tests on Go side to verify how fs1tools handles all generated zkinds.

py2_pickle1, py2_pickle2 and py2_pickle3 are handled well.
Tests for py3_pickle3 currently fail and so are marked with "xfail".

We will fix tests for py3_pickle3 in follow-up patches.

Like fsdump/py does.

https://github.com/zopefoundation/ZODB/commit/403f9869 started to print
size of every transaction record saying that it readded it. And indeed
this printing was there starting from
https://github.com/zopefoundation/ZODB/commit/06e757b3 authored in 2003.

Both commits use `size(txn) = trans._tend - trans._tpos` which is wrong
by 8 because during file iteration tpos points to the beginning of
transaction and tend points to tpos + tlen, but tlen is full
transaction length - 8:

https://github.com/zopefoundation/ZODB/blob/5.8.1/src/ZODB/FileStorage/FileStorage.py#L1997-L1998
https://github.com/zopefoundation/ZODB/blob/5.8.1/src/ZODB/FileStorage/format.py#L28

Mimic fsdump/py behaviour exactly for compatibility, even if it is a bit
buggy, since it was there for such a long time.

Without the fix TestFsDump was failing like this:

    --- FAIL: TestFsDump/db=1 (0.00s)
        dump_test.go:70: fsdump: dump different:
            -Trans #00000 tid=0285cbac258bf266 size=151 time=1979-01-03 21:00:08.800000 offset=52
            +Trans #00000 tid=0285cbac258bf266 time=1979-01-03 21:00:08.800000 offset=52
                 status=' ' user='' description='initial database creation'
               data #00000 oid=0000000000000000 size=61 class=persistent.mapping.PersistentMapping
             ...

https://github.com/zopefoundation/ZODB/commit/403f9869 adjusted fsdump
to emit size of transaction record. Everything else remains unchanged.

Our fsdump becomes broken, because it does not yet emit that size(txn).
We will fix it in the next patch.

We were saving index data with bytestrings inside index records. This
works for py2 but on py3 it causes failure on unpickling because, by default,
*STRING opcodes are unpickled into unicode on py3:

    === RUN   TestIndexSaveToPy/py2_pickle1/py3
    UnicodeDecodeError: 'ascii' codec can't decode byte 0x8d in position 25: ordinal not in range(128)

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/./py/indexcmp", line 43, in <module>
        main()
      File "/home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/./py/indexcmp", line 30, in main
        d1 = fsIndex.load(path1)
      File "/home/kirr/src/wendelin/z/ZODB/src/ZODB/fsIndex.py", line 138, in load
        v = unpickler.load()
    SystemError: <built-in method read of _io.BufferedReader object at 0x7fb631a86670> returned a result with an error set
        index_test.go:229: zodb/py read/compare index: exit status 1

-> Fix it by explicitly emitting index entries in binary form.

To be able to compare "index from py2 -> loaded into go -> saved by go ->
compared by py3" implement a bit of backward compatibility for  loading py2
index files on py3. Do this compatibility only for known-good py2 files, not
index produced by go code which is loaded by py3 without any compatibility
shims.

Python3 emits index entries using bytes, but index.go was expecting only
bytestrings. As the result loading index produced by py3 was failing
e.g. as

    index_test.go:201: index load: testdata/py3_pickle3/1.fs.index: pickle @6: invalid oidPrefix: type ogórek.Bytes

-> Fix it by accepting both bytes and bytestrings inside the index.

FileStorage/py2 saves data with FS21 magic, while FileStorage/py3 with
FS30 magic:

https://github.com/zopefoundation/ZODB/blob/5.8.1/src/ZODB/_compat.py#L25-L77

Up till now FileStorage/go was accepting only FS21 and so with
py3-generated data it was leading to e.g. the following failure:

    === RUN   TestEmptyDB/py3_pickle3
        filestorage_test.go:90: testdata/py3_pickle3/empty.fs: invalid fs1 magic "FS30"

-> Fix it by accepting both py2 and py3 FileStorage magics.

We do accept them both without any other change because FileStorage
format, even when it comes with those two different magic, is really the
same.

Previously we were testing FileStorage/go only with data generated by
python2 and pickle protocol=2. However even on py2 there are more pickle
protocols that are in use, and also there is python3.

-> Modernize py/gen-testdata to use run_with_all_zodb_pickle_kinds
   that was recently added as part of nexedi/zodbtools@f9d36ba7
   and generate test data with both python2 and python3. It is handy to
   use py2py3-venv(*) to prepare python environment to do that.

   Adjust tests on Go side to verify how FileStorage handles all generated zkinds.

py2_pickle1, py2_pickle2 and py2_pickle3 are handled well.
Tests for py3_pickle3 currently fail and so are marked with "xfail".

We will fix tests for py3_pickle3 in follow-up patches.

Old testdata are not yet removed because e.g. fs1tools and zodbdump
tests depend on them. We will remove old fs1 testdata after adjusting
tests in dependent packages step-by-step.

(*) see nexedi/zodbtools@fac2f190

Test for PyData.ClassName currently fails for py3_pickle3 zkind:

    === RUN   TestPyClassName/py3_pickle3
        pydata_test.go:50: class name for "\x80\x03X\x18\x00\x00\x00ZODB.tests.testSerializeq\x00X\x13\x00\x00\x00ClassWithoutNewargsq\x01\x86q\x02N\x86q\x03.":
            have: "?.?"
            want: "ZODB.tests.testSerialize.ClassWithoutNewargs"
        pydata_test.go:50: class name for "\x80\x03X\x18\x00\x00\x00ZODB.tests.testSerializeq\x00X\x10\x00\x00\x00ClassWithNewargsq\x01\x86q\x02K\x01\x85q\x03\x86q\x04.":
            have: "?.?"
            want: "ZODB.tests.testSerialize.ClassWithNewargs"

This happens because classname decoder was expecting class and module
names to come as py2 bytestrings, while py3 emits them as unicode.

-> Fix it by accepting both py2 bytestring and unicode for class and
module names.

Previously we were testing PyData and Map/List only with data generated by python2
and pickle protocol=2. However even on py2 there are more pickle
protocols that are in use, and also there is python3.

-> Modernize py/pydata-gen-testdata to use run_with_all_zodb_pickle_kinds
   that was recently added as part of nexedi/zodbtools@f9d36ba7
   and generate test data with both python2 and python3. It is handy to
   use py2py3-venv(*) to prepare python environment to do that.

   Adjust tests on Go side to verify how PyData and Map/List handle all generated
   zkinds.

py2_pickle1, py2_pickle2 and py2_pickle3 are handled well.
Tests for py3_pickle3 currently fail and so are marked with "xfail".

We will fix tests for py3_pickle3 in the next patches.

(*) see nexedi/zodbtools@fac2f190

- go 1.14 -> 1.18 as we now use generics
- `go get go4.org/unsafe/assume-no-moving-gc` since weak package uses that
- `go mod tidy` as go build/test now was requesting this

This replaces previous attempts to fix zodb/internal/weak with patches
from master.

* master:
  go/zodb/internal/weak: Disable support for weak references
  go/zodb/internal/weak: Assert that the object was not moved in the finalizer
  go/zodb/internal/weak: Try to fix GC crashes via reworking Ref to keep only one word instead of two

Experience shows that we cannot provide reliably working weak references
support without crashing GC because there needs to be a coordination in
between object resurrection and at least GC mark phase[1,2] with [3] and
[4] showing an idea what kind it coordination it needs to be.

And since support for weak references is upcoming in standard library
with hopefully Go 1.24 [5], it does not make sense to continue pushing
hard to make weak references work on our standalone side.

By disabling support for custom weak references we will avoid crashes,
but will start to leak unused objects in zodb.Connection live cache.
This should be relatively ok for now as WCFS in wendelin.core is
currently the only known ZODB/go user and it cares to drop ZBlk* state
after loading the data. The leak, thus, should be modest with,
hopefully, not creating problems in practice.

And once std package weak is out there we will switch to that restoring
automatic LiveCache cleanup.

[1] https://github.com/golang/go/issues/41303
[2] wendelin.core@9b44fc23
[3] https://github.com/golang/go/commit/dfc86e922cd0
[4] https://github.com/golang/go/commit/79fd633632cd
[5] https://github.com/golang/go/issues/67552

/reviewed-by @levin.zimmermann
/reviewed-on !11

This is good to do since we are relying on non-moving property of
current GC. It never triggered but it removed one uncertainty while
debugging GC crash issue. Good to have it in place.

/reviewed-by @levin.zimmermann
/reviewed-on !11

We are observing garbage-collector crashes due to weak package under
load(*) with GC crashing as e.g.

    runtime: full=0xc0001f10000005 next=205 jobs=204 nDataRoots=1 nBSSRoots=1 nSpanRoots=16 nStackRoots=184
    panic: non-empty mark queue after concurrent mark
    fatal error: panic on system stack

    runtime stack:
    runtime.throw({0x5c60fe?, 0x601d70?})
            /home/kirr/src/tools/go/go1.21/src/runtime/panic.go:1077 +0x5c fp=0xc000051e88 sp=0xc000051e58 pc=0x436efc
    panic({0x585100?, 0x601d70?})
            /home/kirr/src/tools/go/go1.21/src/runtime/panic.go:840 +0x6ea fp=0xc000051f38 sp=0xc000051e88 pc=0x436e0a
    runtime.gcMark(0x118946?)
            /home/kirr/src/tools/go/go1.21/src/runtime/mgc.go:1464 +0x40c fp=0xc000051fb0 sp=0xc000051f38 pc=0x41bd6c
    runtime.gcMarkTermination.func1()
            /home/kirr/src/tools/go/go1.21/src/runtime/mgc.go:962 +0x17 fp=0xc000051fc8 sp=0xc000051fb0 pc=0x41b277
    runtime.systemstack()
            /home/kirr/src/tools/go/go1.21/src/runtime/asm_amd64.s:509 +0x4a fp=0xc000051fd8 sp=0xc000051fc8 pc=0x468eea

One problem in current implementation is that weak.Ref keeps two words
for the copy of original interface object and recreates that interface
object on Ref.Get from those two words _nonatomically_. Which is
explicitly documented by Go memory model as something that can lead to
corrupted memory and crashes. From https://go.dev/ref/mem#restrictions:

    This means that races on multiword data structures can lead to
    inconsistent values not corresponding to a single write. When the values
    depend on the consistency of internal (pointer, length) or (pointer,
    type) pairs, as can be the case for interface values, maps, slices, and
    strings in most Go implementations, such races can in turn lead to
    arbitrary memory corruption.

We can avoid doing multiword writes during object resurrection by using
concrete type T* instead of interface{}.

Unfortunately as wendelin.core@9b44fc23
shows it does not help with the above issue and the GC continues to
crash with the same "panic: non-empty mark queue after concurrent mark"
message. This is because weak.Ref implementation needs tight integration
with concurrent GC that Go does and in practice we are unable to do that
from outside of Go runtime internals. See e.g.
https://github.com/golang/go/commit/dfc86e922cd0 and
https://github.com/golang/go/commit/79fd633632cd to get an idea what
kind of integration that needs to be.

Anyway before disabling weak references support I wanted to commit this
change to show that one-word approach was tried and it does not work.
This leaves a trace in the history.

On the good side we are going to use standard package weak in the future
hopefully (https://github.com/golang/go/issues/67552). That needs to
wait for at least Go 1.24 though.

(*) see https://github.com/golang/go/issues/41303 for details

/reviewed-by @levin.zimmermann
/reviewed-on !11

* master:
  go/neo/proto: Update 'Compression' to int to support different compression algorithms

With nexedi/neoppod@fd80cc30 NEO/py added support to encode the compression
algorithm with the 'Compression' parameter. Before this, compression could
only be true (= with compression) or false (= without compression). Now
the absence of compression is encoded with 0. Any other number than 0
encodes a compression algorithm. The mapping is currently:

	1 = zlib

In the future, 2 could mean zstd [1].

[1] https://github.com/facebook/zstd/issues/1134

/reviewed-by @kirr
/reviewed-on !6

With nexedi/neoppod@fd80cc30 NEO/py added support to encode the compression
algorithm with the 'Compression' parameter. Before this, compression could
only be true (= with compression) or false (= without compression). Now
the absence of compression is encoded with 0. Any other number than 0
encodes a compression algorithm. The mapping is currently:

	1 = zlib

In the future, 2 could mean zstd [1].

[1] https://github.com/facebook/zstd/issues/1134

/reviewed-by @kirr
/reviewed-on !6

When a node tries to connect to another node it initially sends a
'RequestIdentification' packet. The other node can either reply with
'AcceptIdentification' or in case of a secondary master with
'NotPrimaryMaster'.

In the second case the message id differs from the initial requests
message id. This makes it difficult in a multi-threaded implementation
to proceed this answer: due to the different msg/connection - id the
multi-threaded implementation tries to proceed this incoming message in
a different thread, while the requesting thread waits forever for its
peers reply.

The most straightforward solution is to use the same connection - id for
both possible answers to the 'RequestIdentification' packet. This
doesn't break given NEO/py implementation and is only a small patch.
A workaround in a multi-threaded implementation on the other hand
seems to be much more complicated and time-consuming. Finally it also makes
sense semantically, because "Message IDs are used to identify response
packets" and in the given context the 'NotPrimaryMaster' *is* the de facto
response of a 'RequestIdentification'.

/suggested-at !2
/proposed-for-review-at nexedi/neoppod!20

Tests, that want to verify something wrt both py2 and py3 will do
WithEachPy(tested_func) which will run tested_func two times:

1) under environment where python on $PATH points to python2, and
2) under environment where python on $PATH points to python3.

Client tests will now need to use just "python" instead of e.g.
"python2" or "python3" to run some python program, and each time
"python" will correspond to current phase of WithEachPy execution.

This will be soon handy to test things wrt e.g. ZEO/py2 and ZEO/py3
servers and similar situations.

Tests that merely want to use some python program just for their inner
working, for example to run `zodb commit`, no longer indicate their
preference for py2. For such tests it is a matter of preference in
pre-setup environment to where "python" points.

For the reference under environments created with py2py3-venv(*) default
"python" currently points to "python2".

(*) see nexedi/zodbtools@fac2f190

ZTestData is Go counterpart of ztestdata on zodbtools/py side from
https://lab.nexedi.com/nexedi/zodbtools/-/blob/f9d36ba7/zodbtools/test/conftest.py#L31-65
that was recently added in nexedi/zodbtools@bf772ce0.

On Go side each ZTestData describes testdata files on the filesystem +
carry arbitrary extra information. For example FileStorage tests will
use it to place there "what is expected to load" information
corresponding to on-filesystem database.

Currently with the only utility to retrieve map keys in sorted order.

We will need this in several tests in follow-up patches.

Similarly to ZODB/py which provides u64 and p64 utilities. Add
zodbpickle.Pack64 for symmetry.

Because without StrictUnicode both bytestrings (str from py2) and
unicode (unicode from py2 and str from py3) load into the same Go type
string so it becomes impossible to distinguish them from each other.
Re-saving data, thus, also generally introduces changes as e.g.
string loaded via bytestring will be saved as unicode when pickling with
protocol 3. Or a loaded unicode will be saved as bytestring with
pickling via protocol=2.

-> Switching to StrictUnicode mode solves all those problems.

Please see updated documentatin for zodbpickle.go and
https://github.com/kisielk/og-rek/commit/b28613c2 for more details about
StrictUnicode mode.

For ZODB/go this is change in behaviour exposed to outside. However
there is currently only one known ZODB/go user - WCFS in Wendelin.core -
and that user will be updated correspondingly as well.

We already have 3 places where we create picklers and unpicklers:
zodb/pydata.go, fs1/index.go and zeo/proto.go and they are already
diverging a bit: for example pydata was explicitly specifying protocol
for pickle encoder, while other places were using defaults.

We will soon want to use StrictUnicode option for all picklers and
unpicklers we create, which means it is better to keep this
customization only in one place instead of copy-pasting it in between
callsites increasing the possibility for divergence and errors.

-> Move the code to create pickler and unpickler to internal zodbpickle
package as a preparatory step for that.

For the reference: this package is merely a wrapper around ogórek, not a
fork of any code like zodbpickle/py is.

We will soon want to use generics and `any` which are available only for go >= 1.18 .

The other changes in go.sum and go.mod are there because after just

    -go 1.14
    +go 1.18

the build stopped to work and asked to do `go mod tidy`

ogórek.AsInt64 was added in https://github.com/kisielk/og-rek/commit/010fbd2e
with functionality similar to what we had in pickletools.Xint64 before.