Add StrictUnicode mode
Up till now ogórek works relatively well for me but experience gained with ZODB/go and Wendelin.core highlighted two problems with strings: 1) loading, then re-saving string data is not generally identity, and 2) there is no way to distinguish binary data saved via py2 str from unicode strings saved via either py2 unicode or py3 str. Let me explain those problems: 1) Loading, then re-saving string data is not generally identity ---------------------------------------------------------------- On decoding ogórek currently loads both byte-string (*STRING opcodes) and unicode string (*UNICODE opcodes) into the same Go type string. And on encoding that string type is encoded as *STRING for protocol <= 2 and as *UNICODE for protocol >= 3. It was me to implement that encoding logic in 2018 in e7d96969 (encoder: Fix string wrt protocol version) where in particular I did - if protocol >= 3 we have to emit the string as unicode pickle object the same way as Python3 does. If we don't do - Python3 won't be generally able to load our pickle ... with the idea that protocol=3 always means that the pickle is intended to be for Python3. But there I missed that zodbpickle can use and generate pickles with protocol=3 even on py2 and that ZODB/py2 actually uses this protocol=3 mode since https://github.com/zopefoundation/ZODB/commit/12ee41c4 authored in the same 2018. So, there can be pickles saved under protocol=3 that do contain both *STRING and *UNICODE opcodes, and when ogórek sees those it loads that *STRING and *UNICODE data as just string loosing the information about type of particular variant. And then on encoding the data is saved as all UNICODE, or all STRING, thus breaking decode/encode=identity property. This breakage is there even with plain pickle and protocol=2 - when both *STRING and *UNICODE are in the database, ogórek loads them both as the same Go type string, and then saving back under the same protocol=2 goes as all STRING resulting in unicode objects becoming str on resave without any intended change. 2) there is no way to distinguish binary data saved via py2 str from unicode strings saved via either py2 unicode or py3 str ---------------------------------------------------------------------------------------------------------------------------- Continuing above example of py2 database with both *STRING and *UNICODE opcodes present there is currently no way for application to distinguish those from each other. In other words there is currently no way for the application to distinguish whether it is binary data coming from py2 protocol <= 2 era, from unicode text. The latter problem I hit for real: with Wendelin.core we have lots of data saved from under Python2 as just str. And the Go part of Wendlin.core, upon loading block of data, wants to accept only binary - either bytes from py3 or bytestring from py2, but not unicode, because it indicates a mistake if e.g. a ZBlk object would come with unicode data https://lab.nexedi.com/nexedi/wendelin.core/-/blob/07087ec8/bigfile/file_zodb.py#L267-300 https://lab.nexedi.com/nexedi/wendelin.core/-/blob/07087ec8/wcfs/internal/zdata/zblk.go#L31-32 https://lab.nexedi.com/nexedi/wendelin.core/-/blob/07087ec8/wcfs/internal/zdata/zblk.go#L98-107 but there is currently no way to distinguish whether it was unicode or bytestring saved into the database becase they both are represented as the same Go string type after decoding. ---------------------------------------- So to solve those problems I thought it over and understood that the issues start to appear becase we let *STRING and *UNICODE to become mixed into the same entity on loading. This behaviour is there from ogórek beginning and the intention, it seems, was to get the data from the pickle stream in an easily-accepted form on the Go side. However the convenience turned out to come with cost of loosing some correctness in the general case as explained above. So if we are to fix the correctness we need to change that behaviour and load *STRING and *UNICODE opcodes into distinct types, so that the information about what was what is preserved and it becomes possible to distinguish bytestring from unicode strings and resave the data in exactly the same form as loaded. Though we can do this only under opt-in option with default behaviour staying as it was before to preserve backward compatibility. -> Do it. Below is excerpt from doc.go and DecoderConfig and EncoderConfig changes that describe the new system: For strings there are two modes. In the first, default, mode both py2/py3 str and py2 unicode are decoded into string with py2 str being considered as UTF-8 encoded. Correspondingly for protocol ≤ 2 Go string is encoded as UTF-8 encoded py2 str, and for protocol ≥ 3 as py3 str / py2 unicode. ogórek.ByteString can be used to produce bytestring objects after encoding even for protocol ≥ 3. This mode tries to match Go string with str type of target Python depending on protocol version, but looses information after decoding/encoding cycle: py2/py3 str↔ string StrictUnicode=n mode, default py2 unicode → string py2 str ← ogórek.ByteString However with StrictUnicode=y mode there is 1-1 mapping in between py2 unicode / py3 str vs Go string, and between py2 str vs ogórek.ByteString. In this mode decoding/encoding and encoding/decoding operations are always identity with respect to strings: py2 unicode / py3 str↔ string StrictUnicode=y mode py2 str↔ ogórek.ByteString For bytes, unconditionally to string mode, there is direct 1-1 mapping in between Python and Go types: bytes↔ ogórek.Bytes (~) bytearray↔ []byte -------- type DecoderConfig struct { // StrictUnicode, when true, requests to decode to Go string only // Python unicode objects. Python2 bytestrings (py2 str type) are // decoded into ByteString in this mode... StrictUnicode bool } type EncoderConfig struct { // StrictUnicode, when true, requests to always encode Go string // objects as Python unicode independently of used pickle protocol... StrictUnicode bool } Since strings are now split into two types, string and ByteString, and ByteString can either mean text or binary data, new AsString and AsBytes helpers are also added to handle string and binary data in uniform way supporting both py2 and py3 databases. Corresponding excerpts from doc.go and typeconv.go changes with the description of those helpers come below: On Python3 strings are unicode strings and binary data is represented by bytes type. However on Python2 strings are bytestrings and could contain both text and binary data. In the default mode py2 strings, the same way as py2 unicode, are decoded into Go strings. However in StrictUnicode mode py2 strings are decoded into ByteString - the type specially dedicated to represent them on Go side. There are two utilities to help programs handle all those bytes/string data in the pickle stream in uniform way: - the program should use AsString if it expects text data - either unicode string, or byte string. - the program should use AsBytes if it expects binary data - either bytes, or byte string. Using the helpers fits into Python3 strings/bytes model but also allows to handle the data generated from under Python2. -------- // AsString tries to represent unpickled value as string. // // It succeeds only if the value is either string, or ByteString. // It does not succeed if the value is Bytes or any other type. // // ByteString is treated related to string because ByteString represents str // type from py2 which can contain both string and binary data. func AsString(x interface{}) (string, error) { // AsBytes tries to represent unpickled value as Bytes. // // It succeeds only if the value is either Bytes, or ByteString. // It does not succeed if the value is string or any other type. // // ByteString is treated related to Bytes because ByteString represents str // type from py2 which can contain both string and binary data. func AsBytes(x interface{}) (Bytes, error) { ZODB/go and Wendelin.core intend to switch to using StrictUnicode mode while leaving ogórek to remain 100% backward-compatible in its default mode for other users.
Showing
This diff is collapsed.
Please register or sign in to comment