We recently moved our custom UTF-8 encoding/decoding routines to Cython. Now we can start taking speedup advantage on C level to make our own UTF-8 decoder a bit less horribly slow on py2:

name       old time/op  new time/op  delta
stddecode   752ns ± 0%   743ns ± 0%   -1.19%  (p=0.000 n=9+10)
udecode     216µs ± 0%    75µs ± 0%  -65.19%  (p=0.000 n=9+10)
stdencode   328ns ± 2%   327ns ± 1%     ~     (p=0.252 n=10+9)
bencode    34.1µs ± 1%  32.1µs ± 1%   -5.92%  (p=0.000 n=10+10)

So it is ~ 3x speedup for u(), but still significantly slower compared to std unicode.decode('utf-8').

Only low-hanging fruit here to make _utf_decode_rune a bit more prompt, since it sits in the most inner loop. In the future _utf8_decode_surrogateescape might be reworked as well to avoid constructing resulting unicode via py-level list of py-unicode character objects. And similarly for _utf8_encode_surrogateescape.

On py3 the performance of std and u/b decode/encode is approximately the same.

/cc @jerome