Add some doc about using valgrind

Misc/README

@@ -24,8 +24,10 @@ pymemcompat.h	Memory interface compatibility file.
 python.man	UNIX man page for the python interpreter
 python-mode.el	Emacs mode for editing Python programs
 README		The file you're reading now
+README.valgrind	Information for Valgrind users, see valgrind-python.supp
 RFD		Request For Discussion about a Python newsgroup
 RPM		(Old) tools to build RPMs
 SpecialBuilds.txt     Describes extra symbols you can set for debug builds
 setuid-prog.c	C helper program for set-uid Python scripts
 vgrindefs	Python configuration for vgrind (a generic pretty printer)
+valgrind-python.supp	Valgrind suppression file, see README.valgrind

Misc/README.valgrind

+This document describes some caveats about the use of Valgrind with
+Python.  Valgrind is used periodically by Python developers to try 
+to ensure there are no memory leaks or invalid memory reads/writes.
+
+If you don't want to read about the details of using Valgrind, there
+are still two things you must do to suppress the warnings.  First, 
+you must use a suppressions file.  One is supplied in
+Misc/valgrind-python.supp.  Second, you must do one of the following:
+
+  * Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c,
+    then rebuild Python 
+  * Uncomment the lines in Misc/valgrind-python.supp that 
+    suppress the warnings for PyObject_Free and PyObject_Realloc
+
+Details:
+--------
+Python uses its own allocation scheme on top of malloc called PyMalloc.
+Valgrind my show some unexpected results when PyMalloc is used.
+Starting with Python 2.3, PyMalloc is used by default.  You can disable
+PyMalloc when configuring python by adding the --without-pymalloc option.
+If you disable PyMalloc, most of the information in this document and 
+the supplied suppressions file will not be useful.
+
+If you use valgrind on a default build of Python,  you will see
+many errors like:
+
+        ==6399== Use of uninitialised value of size 4
+        ==6399== at 0x4A9BDE7E: PyObject_Free (obmalloc.c:711)
+        ==6399== by 0x4A9B8198: dictresize (dictobject.c:477)
+
+These are expected and not a problem.  Tim Peters explains
+the situation:
+
+        PyMalloc needs to know whether an arbitrary address is one
+	that's managed by it, or is managed by the system malloc.  
+	The current scheme allows this to be determined in constant
+	time, regardless of how many memory areas are under pymalloc's
+	control.
+
+        The memory pymalloc manages itself is in one or more "arenas",
+	each a large contiguous memory area obtained from malloc.  
+	The base address of each arena is saved by pymalloc 
+	in a vector, and a field at the start of each arena contains
+	the index of that arena's base address in that vector.
+
+        Given an arbitrary address, pymalloc computes the arena base
+	address corresponding to it, then looks at "the index" stored
+	near there.  If the index read up is out of bounds for the
+	vector of arena base addresses pymalloc maintains, then
+	pymalloc knows for certain that this address is not under
+	pymalloc's control.  Otherwise the index is in bounds, and
+	pymalloc compares
+
+            the arena base address stored at that index in the vector
+
+        to
+
+            the computed arena address
+
+        pymalloc controls this arena if and only if they're equal.
+
+        It doesn't matter whether the memory pymalloc reads up ("the
+	index") is initialized.  If it's not initialized, then
+	whatever trash gets read up will lead pymalloc to conclude
+	(correctly) that the address isn't controlled by it.
+
+        This determination has to be made on every call to one of
+	pymalloc's free/realloc entry points, so its speed is critical
+	(Python allocates and frees dynamic memory at a ferocious rate
+	-- everything in Python, from integers to "stack frames",
+	lives in the heap).

Misc/valgrind-python.supp

+#
+# This is a valgrind suppression file that should be used when using valgrind.
+#
+#  Here's an example of running valgrind:
+#
+#	cd python/dist/src
+#	valgrind --tool=memcheck --suppressions=Misc/valgrind-python.supp \
+#		./python -E -tt ./Lib/test/regrtest.py -u bsddb,network
+#
+# You must edit Objects/obmalloc.c and uncomment Py_USING_MEMORY_DEBUGGER
+# to use the preferred suppressions with Py_ADDRESS_IN_RANGE.
+#
+# If you do not want to recompile Python, you can uncomment
+# suppressions for PyObject_Free and PyObject_Realloc.
+#
+# See Misc/README.valgrind for more information.
+
+# all tool names: Addrcheck,Memcheck,cachegrind,helgrind,massif
+{
+   ADDRESS_IN_RANGE/Invalid read of size 4
+   Memcheck:Addr4
+   fun:Py_ADDRESS_IN_RANGE
+}
+
+{
+   ADDRESS_IN_RANGE/Invalid read of size 4
+   Memcheck:Value4
+   fun:Py_ADDRESS_IN_RANGE
+}
+
+{
+   ADDRESS_IN_RANGE/Conditional jump or move depends on uninitialised value
+   Memcheck:Cond
+   fun:Py_ADDRESS_IN_RANGE
+}
+
+###{
+###   ADDRESS_IN_RANGE/Invalid read of size 4
+###   Memcheck:Addr4
+###   fun:PyObject_Free
+###}
+###
+###{
+###   ADDRESS_IN_RANGE/Invalid read of size 4
+###   Memcheck:Value4
+###   fun:PyObject_Free
+###}
+###
+###{
+###   ADDRESS_IN_RANGE/Conditional jump or move depends on uninitialised value
+###   Memcheck:Cond
+###   fun:PyObject_Free
+###}
+
+###{
+###   ADDRESS_IN_RANGE/Invalid read of size 4
+###   Memcheck:Addr4
+###   fun:PyObject_Realloc
+###}
+###
+###{
+###   ADDRESS_IN_RANGE/Invalid read of size 4
+###   Memcheck:Value4
+###   fun:PyObject_Realloc
+###}
+###
+###{
+###   ADDRESS_IN_RANGE/Conditional jump or move depends on uninitialised value
+###   Memcheck:Cond
+###   fun:PyObject_Realloc
+###}
+
+###
+### All the suppressions below are for errors that occur within libraries
+### that Python uses.  The problems to not appear to be related to Python's
+### use of the libraries.
+###
+{
+   GDBM problems, see test_gdbm
+   Memcheck:Param
+   write(buf)
+   fun:write
+   fun:gdbm_open
+
+}
+
+###
+### These occur from somewhere within the SSL, when running
+###  test_socket_sll.  They are too general to leave on by default.
+###
+###{
+###   somewhere in SSL stuff
+###   Memcheck:Cond
+###   fun:memset
+###}
+###{
+###   somewhere in SSL stuff
+###   Memcheck:Value4
+###   fun:memset
+###}
+###
+###{
+###   somewhere in SSL stuff
+###   Memcheck:Cond
+###   fun:MD5_Update
+###}
+###
+###{
+###   somewhere in SSL stuff
+###   Memcheck:Value4
+###   fun:MD5_Update
+###}
+
+#
+# All of these problems come from using test_socket_ssl
+#
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:BN_bin2bn
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:BN_num_bits_word
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Value4
+   fun:BN_num_bits_word
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:BN_mod_exp_mont_word
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:BN_mod_exp_mont
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Param
+   write(buf)
+   fun:write
+   obj:/usr/lib/libcrypto.so.0.9.7
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:RSA_verify
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Value4
+   fun:RSA_verify
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Value4
+   fun:DES_set_key_unchecked
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Value4
+   fun:DES_encrypt2
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   obj:/usr/lib/libssl.so.0.9.7
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Value4
+   obj:/usr/lib/libssl.so.0.9.7
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:BUF_MEM_grow_clean
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:memcpy
+   fun:ssl3_read_bytes
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Cond
+   fun:SHA1_Update
+}
+
+{
+   from test_socket_ssl
+   Memcheck:Value4
+   fun:SHA1_Update
+}
+
+