PyArena_New(); unsure what the intent is.

but without a specified encoding: decoding_fgets() (and decoding_feof()) can
return NULL and fiddle with the 'tok' struct, making tok->buf NULL. This is
okay in the other cases of calls to decoding_*(), it seems, but not in this
one.

This should get a test added, somewhere, but the testsuite doesn't seem to
test encoding anywhere (although plenty of tests use it.)

It seems to me that decoding errors in other places in the code (like at the
start of a token, instead of in the middle of one) make the code end up
adding small integers to NULL pointers, but happen to check for error states
before using the calculated new pointers. I haven't been able to trigger any
other crashes, in any case.

I would nominate this file for a comlete rewrite for Py3k. The whole
decoding trick is too bolted-on for my tastes.

(Should 'code' be cast to the right pointer type?)

to an unsigned int (and back again) on 64-bit machines, even though the
actual value of the Py_ssize_t variable is way below 31 bits. I suspect
compiler-error.

The culprit was an expression-less yield -- the first apparently in
the standard library.  I added a unit test for this.
Also removed the hack to force compilation of test_with.py.

convince gcc (4.0.x) the variables are never used uninitialized (and raising
a proper exception if they ever are.)

(PyMarshal_ReadShortFromFile() is only used in zipimport.c, I don't believe
the extra initializations will matter one way or another.)

everywhere else?)

the sentinel value in the main function, rather than the helper. This
function could possibly do with an early-out if any of the helper calls ends
up with a len of 0, but I doubt it really matters (how common are malformed
hangul syllables, really?)