Skip to content

C
cpython
Commit d419a93f authored by Georg Brandl's avatar Georg Brandl
Browse files
Options

Apply patch #1489784 from Michael Foord.

parent c01b6aee
Show whitespace changes
Inline Side-by-side
Showing with 62 additions and 39 deletions
Doc/howto/urllib2.rst
View file @ d419a93f
==============================================
HOWTO Fetch Internet Resources Using urllib2
==============================================
------------------------------------------
----------------------------
Fetching URLs With Python
------------------------------------------
----------------------------
.. note::
......@@ -30,19 +30,18 @@ Introduction
This HOWTO is written by `Michael Foord
<http://www.voidspace.org.uk/python/index.shtml>`_.
**urllib2** is a Python_ module for fetching URLs (Uniform Resource
Locators). It offers a very simple interface, in the form of the
*urlopen* function. This is capable of fetching URLs using a variety
**urllib2** is a `Python <http://www.python.org>`_ module for fetching URLs
(Uniform Resource Locators). It offers a very simple interface, in the form of
the *urlopen* function. This is capable of fetching URLs using a variety
of different protocols. It also offers a slightly more complex
interface for handling common situations - like basic authentication,
cookies, proxies, and so on. These are provided by objects called
cookies, proxies and so on. These are provided by objects called
handlers and openers.
While urllib2 supports fetching URLs for many "URL schemes"
(identified by the string before the ":" in URL - e.g. "ftp" is the
URL scheme of "ftp://python.org/") using their associated network
protocols (e.g. FTP, HTTP), this tutorial focuses on the most common
case, HTTP.
urllib2 supports fetching URLs for many "URL schemes" (identified by the string
before the ":" in URL - for example "ftp" is the URL scheme of
"ftp://python.org/") using their associated network protocols (e.g. FTP, HTTP).
This tutorial focuses on the most common case, HTTP.
For straightforward situations *urlopen* is very easy to use. But as
soon as you encounter errors or non-trivial cases when opening HTTP
......@@ -51,7 +50,8 @@ Protocol. The most comprehensive and authoritative reference to HTTP
is :RFC:`2616`. This is a technical document and not intended to be
easy to read. This HOWTO aims to illustrate using *urllib2*, with
enough detail about HTTP to help you through. It is not intended to
replace the `urllib2 docs`_ , but is supplementary to them.
replace the `urllib2 docs <http://docs.python.org/lib/module-urllib2.html>`_ ,
but is supplementary to them.
Fetching URLs
......@@ -122,8 +122,9 @@ the ``data`` argument. The encoding is done using a function from the
the_page = response.read()
Note that other encodings are sometimes required (e.g. for file upload
from HTML forms - see `HTML Specification, Form Submission`_ for more
details).
from HTML forms - see
`HTML Specification, Form Submission <http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_
for more details).
If you do not pass the ``data`` argument, urllib2 uses a **GET**
request. One way in which GET and POST requests differ is that POST
......@@ -134,7 +135,25 @@ Though the HTTP standard makes it clear that POSTs are intended to
*always* cause side-effects, and GET requests *never* to cause
side-effects, nothing prevents a GET request from having side-effects,
nor a POST requests from having no side-effects. Data can also be
passed in an HTTP request by encoding it in the URL itself.
passed in an HTTP GET request by encoding it in the URL itself.
This is done as follows::
>>> import urllib2
>>> import urllib
>>> data = {}
>>> data['name'] = 'Somebody Here'
>>> data['location'] = 'Northampton'
>>> data['language'] = 'Python'
>>> url_values = urllib.urlencode(data)
>>> print url_values
name=Somebody+Here&language=Python&location=Northampton
>>> url = 'http://www.example.com/example.cgi'
>>> full_url = url + '?' + url_values
>>> data = urllib2.open(full_url)
Notice that the full URL is created by adding a ``?`` to the URL, followed by
the encoded values.
Headers
-------
......@@ -355,7 +374,7 @@ Number 2
::
from urllib2 import Request, urlopen
from urllib2 import Request, urlopen, URLError
req = Request(someurl)
try:
response = urlopen(req)
......@@ -386,15 +405,17 @@ page fetched, particularly the headers sent by the server. It is
currently an ``httplib.HTTPMessage`` instance.
Typical headers include 'Content-length', 'Content-type', and so
on. See the `Quick Reference to HTTP Headers`_ for a useful listing of
HTTP headers with brief explanations of their meaning and use.
on. See the
`Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_
for a useful listing of HTTP headers with brief explanations of their meaning
and use.
Openers and Handlers
====================
When you fetch a URL you use an opener (an instance of the perhaps
confusingly-named urllib2.OpenerDirector). Normally we have been using
confusingly-named ``urllib2.OpenerDirector``). Normally we have been using
the default opener - via ``urlopen`` - but you can create custom
openers. Openers use handlers. All the "heavy lifting" is done by the
handlers. Each handler knows how to open URLs for a particular URL
......@@ -458,7 +479,7 @@ header sent by the server), then you can use a
that case, it is convenient to use
``HTTPPasswordMgrWithDefaultRealm``. This allows you to specify a
default username and password for a URL. This will be supplied in the
absence of yoou providing an alternative combination for a specific
absence of you providing an alternative combination for a specific
realm. We indicate this by providing ``None`` as the realm argument to
the ``add_password`` method.
......@@ -557,19 +578,21 @@ Footnotes
This document was reviewed and revised by John Lee.
.. [#] For an introduction to the CGI protocol see `Writing Web Applications in Python`_.
.. [#] Like Google for example. The *proper* way to use google from a program is to use PyGoogle_ of course. See `Voidspace Google`_ for some examples of using the Google API.
.. [#] Browser sniffing is a very bad practise for website design - building sites using web standards is much more sensible. Unfortunately a lot of sites still send different versions to different browsers.
.. [#] The user agent for MSIE 6 is *'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'*
.. [#] For details of more HTTP request headers, see `Quick Reference to HTTP Headers`_.
.. [#] In my case I have to use a proxy to access the internet at work. If you attempt to fetch *localhost* URLs through this proxy it blocks them. IE is set to use the proxy, which urllib2 picks up on. In order to test scripts with a localhost server, I have to prevent urllib2 from using the proxy.
.. _Python: http://www.python.org
.. _urllib2 docs: http://docs.python.org/lib/module-urllib2.html
.. _HTML Specification, Form Submission: http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13
.. _Quick Reference to HTTP Headers: http://www.cs.tut.fi/~jkorpela/http.html
.. _PyGoogle: http://pygoogle.sourceforge.net
.. _Voidspace Google: http://www.voidspace.org.uk/python/recipebook.shtml#google
.. _Writing Web Applications in Python: http://www.pyzine.com/Issue008/Section_Articles/article_CGIOne.html
.. _Basic Authentication Tutorial: http://www.voidspace.org.uk/python/articles/authentication.shtml
.. [#] For an introduction to the CGI protocol see
`Writing Web Applications in Python <http://www.pyzine.com/Issue008/Section_Articles/article_CGIOne.html>`_.
.. [#] Like Google for example. The *proper* way to use google from a program
is to use `PyGoogle <http://pygoogle.sourceforge.net>_ of course. See
`Voidspace Google <http://www.voidspace.org.uk/python/recipebook.shtml#google>`_
for some examples of using the Google API.
.. [#] Browser sniffing is a very bad practise for website design - building
sites using web standards is much more sensible. Unfortunately a lot of
sites still send different versions to different browsers.
.. [#] The user agent for MSIE 6 is
*'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'*
.. [#] For details of more HTTP request headers, see
`Quick Reference to HTTP Headers`_.
.. [#] In my case I have to use a proxy to access the internet at work. If you
attempt to fetch *localhost* URLs through this proxy it blocks them. IE
is set to use the proxy, which urllib2 picks up on. In order to test
scripts with a localhost server, I have to prevent urllib2 from using
the proxy.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7