• Kirill Smelkov's avatar
    encoder: Fix data corruption when encoding uint64 · 773b7a86
    Kirill Smelkov authored
    I discovered that uint64 is wrongly encoded. For example
    uint64(0x80000000ffffffff) is encoded as
    
    	"I-9223372032559808513\n."
    
    instead of
    
    	"I9223372041149743103\n."
    
    which results in wrong data to be loaded on Python side:
    
    	In [1]: import pickle
    
    	In [2]: pickle.loads(b"I-9223372032559808513\n.")
    	Out[2]: -9223372032559808513
    
    	In [3]: pickle.loads(b"I9223372041149743103\n.")
    	Out[3]: 9223372041149743103
    
    Similarly uint64(0xffffffffffffffff) is wrongly encoded as
    
    	"I-1\n."
    
    instead of
    
    	"I18446744073709551615\n."
    
    with the same effect of misinforming peer:
    
    	In [4]: pickle.loads(b"I-1\n.")
    	Out[4]: -1
    
    	In [5]: pickle.loads(b"\x80\x02I18446744073709551615\n.")
    	Out[5]: 18446744073709551615
    
    in general any uint64 that is greater than max(int64) is encoded
    wrongly.
    
    -> Fix that by encoding those integer range properly and with INT
       opcode. Unfortunately there is no BININT* variant to handle integers
       out of int32 range so we need to use text encoding for large
       uint64 similarly to how we already do for large int64.
    
    For symmetry we need to adjust the decoder as well to handle those I...
    with numbers out of signed int64 range, because without the adjustment
    it fails as e.g.
    
        --- FAIL: TestDecode/uint(0x80000000ffffffff)/StrictUnicode=n/"I9223372041149743103\n." (0.00s)
            ogorek_test.go:619: strconv.ParseInt: parsing "9223372041149743103": value out of range
            ogorek_test.go:623: decode:
                have: <nil>
                want: 9223372041149743103
    
    Since Python handles such input just fine (see In[3]/Out[3] above) we
    should as well. The number cannot be represented as int64 though, so if
    the number coming with INT opcode is out of fixed int64 range, we decode
    it into big.Int .
    773b7a86
ogorek_test.go 40 KB