Made test_tokenize discoverable.

fix a resource warning.

Indexing bytes retiurns an integer, not bytes.

Original patches by A. Kuchling and G. Rees (#12691).

constructed examples and all but 7 of the test/test_*.py files (run with -ucpu)
pass. Remove those that fail the new test from the selection list.
Patch partly based on patches by G. Brandl (#8478) and G. Rees (#12691).

Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.

Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.

* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.

now detect Python source code encoding only in comment lines.

#16152: fix tokenize to ignore whitespace at the end of the code when no newline is found.  Patch by Ned Batchelder.

#16152: fix tokenize to ignore whitespace at the end of the code when no newline is found.  Patch by Ned Batchelder.

Patch by Serhiy Storchaka.

Patch by Serhiy Storchaka.

tokenizer.detect_encoding() (when available).

if the first two lines have non-UTF-8 characters without an encoding declaration.

(Patch by Robert Xiao)

svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r86596 | ezio.melotti | 2010-11-20 21:04:17 +0200 (Sat, 20 Nov 2010) | 1 line

  #9424: Replace deprecated assert* methods in the Python test suite.
........

tokenize.detect_encoding() and open it in read only mode.

svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r85990 | brian.curtin | 2010-10-30 16:35:28 -0500 (Sat, 30 Oct 2010) | 2 lines

  Fix #10258 - clean up resource warning
........

svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r85482 | antoine.pitrou | 2010-10-14 17:34:31 +0200 (jeu., 14 oct. 2010) | 4 lines

  Replace the "compiler" resource with the more generic "cpu", so
  as to mark CPU-heavy tests.
........

svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r85482 | antoine.pitrou | 2010-10-14 17:34:31 +0200 (jeu., 14 oct. 2010) | 4 lines

  Replace the "compiler" resource with the more generic "cpu", so
  as to mark CPU-heavy tests.
........

as to mark CPU-heavy tests.

svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r84364 | benjamin.peterson | 2010-08-30 09:41:20 -0500 (Mon, 30 Aug 2010) | 1 line

  handle names starting with non-ascii characters correctly #9712
........

(And similarly for complex numbers.)

svn+ssh://pythondev@svn.python.org/python/trunk

........
  r82356 | mark.dickinson | 2010-06-29 08:37:25 +0100 (Tue, 29 Jun 2010) | 1 line

  Spelling.
........