Copy Lib/test/test_robotparser.py from master to 2.7 and adapt it for
Python 2.7:

* Replace urllib.robotparser with robotparser
* Adjust HTTPServer import
* Replace io.StringIO with StringIO.StringIO
* Remove tests on crawl_delay() and request_rate() since these
  methods were added to Python 3
* Remove subTest()
* Add test_main() which explicitly lists all test cases

Patch based on the commit 4da0fd06
written by Berker Peksag.
Co-Authored-By: Berker Peksag <berker.peksag@gmail.com>

* Rename again Lib/test/bisectcmd.py to Lib/test/bisect.py
* regrtest now removes '' and Lib/test/ from sys.path
* Use absolute import in test_bisect

Revert test_bisect.py change:
remove "from __future__ import absolute_import".

Sleep 1 ms instead of 0.1 ms to workaround a rounding issue on
Windows. On Windows, time.sleep(0.0001) sleeps 0 ms, so
collect_in_thread() calls gc.collect() in a loop and tests using this
thread takes too long. Sleep 1 ms so time.sleep() sleeps 15.6 ms on
Windows.

FXpy doesn't have a Python 3 port and it only
supports Python 2.2 and older versions.

Reported by Alex Walters on docs@p.o.

(cherry picked from commit d3ed2877)

Write header and "Using seed ..." after checking for --list-cases and
--list-tests.

I thought I had run the full test suite before the last checkin, but
obviously I didn't.  test_multibytecodec_support.py isn't really a test file,
it is a support file that contains a base test class.  Rename it to
multibytecodec_support so that regrtest test discovery doesn't think it is a
test file that should be run..
(cherry picked from commit 75d9aca9)

* bpo-29512, bpo-30764: Backport regrtest enhancements from 3.5 to 2.7

* bpo-29512: Add test.bisect, bisect failing tests (#2452)

Add a new "python3 -m test.bisect" tool to bisect failing tests.

It can be used to find which test method(s) leak references, leak
files, etc.

* bpo-30764: Fix regrtest --fail-env-changed --forever (#2536) (#2539)

--forever now stops if a fail changes the environment.

* Fix test_bisect: use absolute import

Add missing "import subprocess".

Oh, this is my fault too. I missed this mistake. Thank you for the correction.

* bpo-30807: signal.setitimer() may disable the timer by mistake

* Add NEWS blurb.
(cherry picked from commit 729780a8)

* Clear potential ref cycle between Process and Process target

Besides Process.join() not being called, this was an indirect cause of bpo-30775.
The threading module already does this.

* Add issue reference.
(cherry picked from commit 79d37ae9)

Clarify that the break statement breaks out of the innermost enclosing for or while loop.
(cherry picked from commit 36fc8967)

* bpo-30523: regrtest --list-cases --match (#2401)

* regrtest --list-cases now supports --match and --match-file options.
  Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
  messages to stdout when loading test modules.
* Add support._match_test() private function.
(cherry picked from commit ace56d58)

(cherry picked from commit 36946c06a3bdb1104c53f0fe535086572ffbdd9e)

* bpo-30764: regrtest: add --fail-env-changed option (#2402)

* bpo-30764: regrtest: change exit code on failure

* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted

* bpo-30764: regrtest: add --fail-env-changed option

If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.

(cherry picked from commit 63f54c68)
(cherry picked from commit 1f33857a3677de84bdc6d80c39f577eb081ed85d)

* bpo-30776: reduce regrtest -R false positives (#2422)

* Change the regrtest --huntrleaks checker to decide if a test file
  leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
  and not used anymore to compute the sum.
* Add an unit test for a reference leak.

Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):

    [3, 0, 0]
    [0, 1, 0]
    [8, -8, 1]
(cherry picked from commit 48b5c422)

(cherry picked from commit e0f8b43a46850aa5e5992465cab684496f102fe5)

* Add test.support._crash_python() which triggers a crash but uses
  test.support.SuppressCrashReport() to prevent a crash report from
  popping up.
* Modify test_child_terminated_in_stopped_state() of test_subprocess
  and test_crashed() of test_regrtest to use _crash_python().

* Add Windows support to test.support.SuppressCrashReport: call
  SetErrorMode() and CrtSetReportMode().
* _testcapi: add CrtSetReportMode() and CrtSetReportFile() functions
  and CRT_xxx and CRTDBG_xxx constants needed by SuppressCrashReport.

* bpo-30764: Backport support.SuppressCrashReport

Backport test.support.SuppressCrashReport context-manager from
master. Drop the Windows implementation since it depends on
msvcrt.CrtSetReportMode() which isn't available on Python 2.7.

* bpo-30764: test_subprocess uses SuppressCrashReport (#2405)

bpo-30764, bpo-29335: test_child_terminated_in_stopped_state() of
test_subprocess now uses support.SuppressCrashReport() to prevent the
creation of a core dump on FreeBSD.
(cherry picked from commit cdee3f14)

* [2.7] bpo-30765: Avoid blocking when PyThread_acquire_lock() is asked not to (GH-2403)

* bpo-30765: Avoid blocking when PyThread_acquire_lock() is asked not to lock

This is especially important if PyThread_acquire_lock() is called reentrantly
(for example from a signal handler).

* Update 2017-06-26-14-29-50.bpo-30765.Q5iBmf.rst

* Avoid core logic when taking the mutex failed.
(cherry picked from commit f84ac420)

* Remove test undef

in `os.putenv()` and `os.spawn*()`..
(cherry picked from commit 77703942)

[2.7] bpo-30730: Prevent environment variables injection in subprocess on Windows. (GH-2325) (#2372)

Prevent passing other invalid environment variables and command arguments..
(cherry picked from commit d174d24a)

(cherry picked from commit 32cb968a)

(cherry picked from commit b066edfb)

CPython workflow is changing!  We're going to start using "blurb"
to manage Misc/NEWS entries:
    https://github.com/python/core-workflow
(This will be a big win for release managers, honest.)

This checkin simply populates the "Misc/NEWS.d" subdirectory tree
so that people can start putting their news entries in there.
No other changes (yet).

delaying the main thread so that it doesn't race ahead of the workers.

(cherry picked from commit 020af2a2)

* Silence warnings caused by duplicated defines from Modules\expat\winconfig.h
* Add WIN32 define to VS9.0 project files to match MSBuild project files.

* bpo-30694: Upgrade Modules/expat/ to libexpat 2.2.1 (#2300)

New file: Modules/expat/siphash.h.
(cherry picked from commit 5ff71323)

* siphash.h: Address lack of stdint.h for Visual Studio 2003-2008

(cherry picked from libexpat commit c74d380b8f0b5215153d840d2af9455d9c53b960)

* bpo-30368: Update build_ssl.py to restore Perl-less building

OpenSSL 1.0.2 releases changed how files are copied in the makefile,
thus causing Perl to be required even for Python's "prepared" OpenSSL.
Now build_ssl.py does the requisite copies before running nmake.

* bpo-30368: Update build_ssl.py to use prepared OpenSSL

* Updates SSL-linking projects to use the new include{suffix} directory
* build_ssl.py now only copies those files not handled by prepare_ssl.py

* * bpo-30368: Update build_ssl.py to use prepared OpenSSL

* Update SSL-linking projects to use the new include{suffix} directory

* Move comment to following line

The current regex based splitting produces a wrong result. For example::

  http://abc#@def

Web browsers parse that URL as ``http://abc/#@def``, that is, the host
is ``abc``, the path is ``/``, and the fragment is ``#@def``.
(cherry picked from commit 90e01e50)

[2.7] bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest

regrtest now warms up caches: create explicitly all internal
singletons which are created on demand to prevent false positives
when checking for reference leaks.

* bpo-29591: Upgrade Modules/expat to libexpat 2.2

* bpo-29591: Restore Python changes on expat

* bpo-29591: Remove expat config of unsupported platforms

Remove the configuration (Modules/expat/*config.h) of unsupported
platforms:

* Amiga
* MacOS Classic on PPC32
* Open Watcom

* bpo-29591: Remove useless XML_HAS_SET_HASH_SALT

The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became
useless since our local expat copy was upgrade to expat 2.1 (it's now
expat 2.2.0).

(cherry picked from commit 23ec4b57)

(cherry picked from commit 184bd82b)

Replace platform with platforms.
(cherry picked from commit 4ebf03d1)

* bpo-24484: Avoid race condition in multiprocessing cleanup

The finalizer registry can be mutated while inspected by multiprocessing
at process exit.

* Use test.support.start_threads()

* Add Misc/NEWS.
(cherry picked from commit 1eb6c007)