WIP: python 3 compatibility
This the current state of my work on making this python3 compatible.
The approach I took was a mix of running the test and using type annotations and mypy, then add some
.decode()here and there until error disappear ... it still needs a lot of cleanups. It's still messy and too early for review. Anyway, I'll try to extract the "simple" fixes to merge first what's can already be merged.
tests using a pre-created database have not been updated yet and they fail with
FileStorageFormatErroras the test databases was created with python2. Except this, the rest seems working.
tox has been extended, also to run tests with https://github.com/zopefoundation/ZODB/pull/183
@jerome, thanks. I cherry-picked some first patches, but had not had time to dig into the reset. I suggest, when you have time, to split type annotations from
bytes vs stras type annotations require more thought maybe and byts vs str should be easy to get, at least in most of the part.
Thanks again for the fixups & rawext tox coverage. Looking forward for further updates (when you'll have the time).
( I hope it is ok that I did small amendments to the patches )
@kirr thank you ! of course it was OK to amend the patches. Thanks for moving this forward.
I agree that we'll leave the type annotations and the mypy stuff out before merging. I was more interested in trying type annotations in a "real case". I hit quickly a problem that type annotations for standard library are not really complete yet ( with
codecs.encodeand hex encoding - https://github.com/python/typeshed/issues/300 ) and that annotating types is "not so easy" to do in a "good way", for example the :
obj = None # type: Optional[Union[ObjectDelete, ObjectCopy, ObjectData]]
is ugly ( it should be possible to define a parent type for
Object*though, I did not try). Also for the different "hashers", I did not find a way to define an "interface" with type annotations for
hexdigestmethods only in one central place and have a way to check that the classes respect the interface - but again I have not tried much.
( edited to fix grammar mistakes)
@jerome, thanks for feedback and for typing links. Yes, py typing will require more thought. What is easily possible with interfaces in Go, or e.g. with interfaces with zope.interfaces, seems to be not so straightforward with current state of py typing.
I had not studied current py typing landscape yet and for now I suggest we do not delve too much into annotating everything - e.g. if we can properly annotate a parameter to be bytes - that's ok. If we cannot deal with interfaces - we just leave a human readable comment.
Guido is known to delay typing (and other bits) in Python for ages. Most of them started to appear in workable state in around 2000 but it was more than 10 years since only some of them got accepted at least somehow and even today it does not all properly work. Frankly I do not trust today's Python leadership. I will thus consider all possibilities, including moving e.g. to Cython/Pyrex.
While you are here, can you please take a quick look at jerome/zodbtools@e92cb115 ? I remember I was stuck on that and wanted to ask you...
When I tried to re-run
gen_testdata.pyI did not manage to get the same output as yours, even though my output was stable (if I remember correctly). I figure out that I had to set
TZthis strptime is timezone dependent, but that was still not enough. Can you just try again to run
gen_test_data.pyand let me know if your output is still the same ?
@jerome, thanks for feedback and asking. Sure, it would be better to have well-defined and well-understood procedure to regenerate files. First of all about time: I believe effective
TZon my side when I run the generator is
TZ=Europe/Moscow ./gen_testdata.py- this is what I have in
/etc/timezoneand this is what https://en.wikipedia.org/wiki/List_of_tz_database_time_zones suggests (see "TZ database name" column). If I run
TZ=MSKI too get differeces:
(neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/z/zodbtools/zodbtools/test$ TZ=MSK ./gen_testdata.py WARNING:ZODB.FileStorage:Ignoring index for /home/kirr/src/wendelin/z/zodbtools/zodbtools/test/testdata/1.fs WARNING:ZODB.FileStorage:Ignoring index for /home/kirr/src/wendelin/z/zodbtools/zodbtools/test/testdata/1_!zext.fs (neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/z/zodbtools/zodbtools/test$ git st На ветке master Ваша ветка опережает «kirr/master» на 9 коммитов. (используйте «git push», чтобы опубликовать ваши локальные коммиты) Изменения, которые не в индексе для коммита: (используйте «git add <файл>…», чтобы добавить файл в индекс) (используйте «git checkout -- <файл>…», чтобы отменить изменения в рабочем каталоге) изменено: testdata/1.fs изменено: testdata/1.zdump.ok изменено: testdata/1_!zext.fs изменено: testdata/1_!zext.zdump.ok diff --git a/zodbtools/test/testdata/1.zdump.ok b/zodbtools/test/testdata/1.zdump.ok index 2f199ec..9581830 100644 --- a/zodbtools/test/testdata/1.zdump.ok +++ b/zodbtools/test/testdata/1.zdump.ok @@ -1,4 +1,4 @@ -txn 0285cbac258bf266 " " +txn 0285cc60258bf266 " " user "" description "initial database creation" extension "" @@ -7,7 +7,7 @@ obj 0000000000000000 61 sha1:664e6de0f153d8eaeda638d616a320c6e3c5feb1 PersistentMapping q^A.<80>^B}q^BU^Ddataq^C}q^Ds. -txn 0285cbac3d0369e6 " " +txn 0285cc603d0369e6 " " user "user0.0" description "step 0.0" extension "\x80\x02}q\x01(U\tx-cookieSU\x05RF9IEU\x0bx-generatorq\x02U\x0czodb/py2 (f)u." @@ -22,7 +22,7 @@ obj 0000000000000001 33 sha1:1e4a07534b581b8c84c6d887042b3a3469bd4a5c Object ...
if needed we can change the generator to use UTC timezone.
The next point is which version of ZODB is used to run gen_testdata. From looking at
git show --text e92cb1157798172bc20e4e3c95484039c18ee29eI believe you were using ZODB@7f5a67b3e0d2dcf2d13f18a42a82e526f6d4055c = y/rawext3 of navytux/ZODB. That branch comes after https://github.com/zopefoundation/ZODB/commit/12ee41c4 which changed pickle protocol ZODB uses for serialzation and that is what we are seeing in the diff, for example:
--- a/zodbtools/test/testdata/1.zdump.ok +++ b/zodbtools/test/testdata/1.zdump.ok @@ -1,572 +1,567 @@ -txn 0285cbac258bf266 " " +txn 0285ce04258bf266 " " user "" description "initial database creation" extension "" -obj 0000000000000000 61 sha1:664e6de0f153d8eaeda638d616a320c6e3c5feb1 -<80>^Bcpersistent.mapping +obj 0000000000000000 61 sha1:8002575e7e680f98bc228ebcb31efb56487b955f +<80>^Ccpersistent.mapping
<80>). I was keeping my ZODB checkout at d5d43ba0e3e6f266df6bfc14eb2181dee9fca9dc = y/rawext2 of navytux/ZODB and since that comes before https://github.com/zopefoundation/ZODB/commit/12ee41c4 that explains the difference in pickled data.
We can make a better decision to which ZODB version to use to generate the files and check that in gen_testdata similarly to how we are already checking support for
Hope it clarifies things a bit,
Thanks, I can confirm that when running with ZODB at https://github.com/zopefoundation/ZODB/commit/d5d43ba0e3e6f266df6bfc14eb2181dee9fca9dc the dump are same, this was the missing part.
I agree that we could make the gen script check it's using the expected ZODB version (and also make it
Isn't everything blocked on 207 ?
I agree on the path to improve the generation script. Could you please do it when/if you'll have time and interest? Otherwise I will try to improve it the next time I will do something on zodbtools in likely distant future.
Thanks, I'll try to improve this script when I work on this, but I have lots of things on the TODO list.