bpo-6691: Pyclbr now reports nested classes and functions. (#2503)

 Original patch by Guilherme Polo.  Revisions by Cheryl Sabella.

Doc/library/pyclbr.rst

@@ -10,107 +10,144 @@
 
 --------------
 
-The :mod:`pyclbr` module can be used to determine some limited information
-about the classes, methods and top-level functions defined in a module.  The
-information provided is sufficient to implement a traditional three-pane
-class browser.  The information is extracted from the source code rather
-than by importing the module, so this module is safe to use with untrusted
-code.  This restriction makes it impossible to use this module with modules
-not implemented in Python, including all standard and optional extension
+The :mod:`pyclbr` module provides limited information about the
+functions, classes, and methods defined in a python-coded module.  The
+information is sufficient to implement a module browser.  The
+information is extracted from the python source code rather than by
+importing the module, so this module is safe to use with untrusted code.
+This restriction makes it impossible to use this module with modules not
+implemented in Python, including all standard and optional extension
 modules.
 
 
 .. function:: readmodule(module, path=None)
 
-   Read a module and return a dictionary mapping class names to class
-   descriptor objects.  The parameter *module* should be the name of a
-   module as a string; it may be the name of a module within a package.  The
-   *path* parameter should be a sequence, and is used to augment the value
-   of ``sys.path``, which is used to locate module source code.
+   Return a dictionary mapping module-level class names to class
+   descriptors.  If possible, descriptors for imported base classes are
+   included.  Parameter *module* is a string with the name of the module
+   to read; it may be the name of a module within a package.  If given,
+   *path* is a sequence of directory paths prepended to ``sys.path``,
+   which is used to locate the module source code.
 
 
 .. function:: readmodule_ex(module, path=None)
 
-   Like :func:`readmodule`, but the returned dictionary, in addition to
-   mapping class names to class descriptor objects, also maps top-level
-   function names to function descriptor objects.  Moreover, if the module
-   being read is a package, the key ``'__path__'`` in the returned
-   dictionary has as its value a list which contains the package search
-   path.
+   Return a dictionary-based tree containing a function or class
+   descriptors for each function and class defined in the module with a
+   ``def`` or ``class`` statement.  The returned dictionary maps
+   module-level function and class names to their descriptors.  Nested
+   objects are entered into the children dictionary of their parent.  As
+   with readmodule, *module* names the module to be read and *path* is
+   prepended to sys.path.  If the module being read is a package, the
+   returned dictionary has a key ``'__path__'`` whose value is a list
+   containing the package search path.
 
+.. versionadded:: 3.7
+   Descriptors for nested definitions.  They are accessed through the
+   new children attibute.  Each has a new parent attribute.
 
-.. _pyclbr-class-objects:
+The descriptors returned by these functions are instances of
+Function and Class classes.  Users are not expected to create instances
+of these classes.
 
-Class Objects
--------------
 
-The :class:`Class` objects used as values in the dictionary returned by
-:func:`readmodule` and :func:`readmodule_ex` provide the following data
-attributes:
+.. _pyclbr-function-objects:
 
+Function Objects
+----------------
+Class :class:`Function` instances describe functions defined by def
+statements.  They have the following attributes:
 
-.. attribute:: Class.module
 
-   The name of the module defining the class described by the class descriptor.
+.. attribute:: Function.file
 
+   Name of the file in which the function is defined.
 
-.. attribute:: Class.name
 
-   The name of the class.
+.. attribute:: Function.module
 
+   The name of the module defining the function described.
 
-.. attribute:: Class.super
 
-   A list of :class:`Class` objects which describe the immediate base
-   classes of the class being described.  Classes which are named as
-   superclasses but which are not discoverable by :func:`readmodule` are
-   listed as a string with the class name instead of as :class:`Class`
-   objects.
+.. attribute:: Function.name
+
+   The name of the function.
 
 
-.. attribute:: Class.methods
+.. attribute:: Function.lineno
+
+   The line number in the file where the definition starts.
+
+
+.. attribute:: Function.parent
+
+   For top-level functions, None.  For nested functions, the parent.
+
+   .. versionadded:: 3.7
+
+
+.. attribute:: Function.children
+
+   A dictionary mapping names to descriptors for nested functions and
+   classes.
 
-   A dictionary mapping method names to line numbers.
+   .. versionadded:: 3.7
+
+
+.. _pyclbr-class-objects:
+
+Class Objects
+-------------
+Class :class:`Class` instances describe classes defined by class
+statements.  They have the same attributes as Functions and two more.
 
 
 .. attribute:: Class.file
 
-   Name of the file containing the ``class`` statement defining the class.
+   Name of the file in which the class is defined.
 
 
-.. attribute:: Class.lineno
+.. attribute:: Class.module
 
-   The line number of the ``class`` statement within the file named by
-   :attr:`~Class.file`.
+   The name of the module defining the class described.
 
 
-.. _pyclbr-function-objects:
+.. attribute:: Class.name
 
-Function Objects
-----------------
+   The name of the class.
 
-The :class:`Function` objects used as values in the dictionary returned by
-:func:`readmodule_ex` provide the following attributes:
 
+.. attribute:: Class.lineno
 
-.. attribute:: Function.module
+   The line number in the file where the definition starts.
 
-   The name of the module defining the function described by the function
-   descriptor.
 
+.. attribute:: Class.parent
 
-.. attribute:: Function.name
+   For top-level classes, None.  For nested classes, the parent.
 
-   The name of the function.
+   .. versionadded:: 3.7
 
 
-.. attribute:: Function.file
+.. attribute:: Class.children
 
-   Name of the file containing the ``def`` statement defining the function.
+   A dictionary mapping names to descriptors for nested functions and
+   classes.
 
+   .. versionadded:: 3.7
 
-.. attribute:: Function.lineno
 
-   The line number of the ``def`` statement within the file named by
-   :attr:`~Function.file`.
+.. attribute:: Class.super
+
+   A list of :class:`Class` objects which describe the immediate base
+   classes of the class being described.  Classes which are named as
+   superclasses but which are not discoverable by :func:`readmodule_ex`
+   are listed as a string with the class name instead of as
+   :class:`Class` objects.
+
+
+.. attribute:: Class.methods
 
+   A dictionary mapping method names to line numbers.  This can be
+   derived from the newer children dictionary, but remains for
+   back-compatibility.

Lib/test/test_pyclbr.py

@@ -2,10 +2,15 @@
    Test cases for pyclbr.py
    Nick Mathewson
 '''
+
+import os
 import sys
+from textwrap import dedent
 from types import FunctionType, MethodType, BuiltinFunctionType
 import pyclbr
 from unittest import TestCase, main as unittest_main
+from test import support
+from functools import partial
 
 StaticMethodType = type(staticmethod(lambda: None))
 ClassMethodType = type(classmethod(lambda c: None))
@@ -150,6 +155,67 @@ class PyclbrTest(TestCase):
         #
         self.checkModule('test.pyclbr_input', ignore=['om'])
 
+    def test_nested(self):
+        mb = pyclbr
+        # Set arguments for descriptor creation and _creat_tree call.
+        m, p, f, t, i = 'test', '', 'test.py', {}, None
+        source = dedent("""\
+        def f0:
+            def f1(a,b,c):
+                def f2(a=1, b=2, c=3): pass
+                    return f1(a,b,d)
+            class c1: pass
+        class C0:
+            "Test class."
+            def F1():
+                "Method."
+                return 'return'
+            class C1():
+                class C2:
+                    "Class nested within nested class."
+                    def F3(): return 1+1
+
+        """)
+        actual = mb._create_tree(m, p, f, source, t, i)
+
+        # Create descriptors, linked together, and expected dict.
+        f0 = mb.Function(m, 'f0', f, 1)
+        f1 = mb._nest_function(f0, 'f1', 2)
+        f2 = mb._nest_function(f1, 'f2', 3)
+        c1 = mb._nest_class(f0, 'c1', 5)
+        C0 = mb.Class(m, 'C0', None, f, 6)
+        F1 = mb._nest_function(C0, 'F1', 8)
+        C1 = mb._nest_class(C0, 'C1', 11)
+        C2 = mb._nest_class(C1, 'C2', 12)
+        F3 = mb._nest_function(C2, 'F3', 14)
+        expected = {'f0':f0, 'C0':C0}
+
+        def compare(parent1, children1, parent2, children2):
+            """Return equality of tree pairs.
+
+            Each parent,children pair define a tree.  The parents are
+            assumed equal.  Comparing the children dictionaries as such
+            does not work due to comparison by identity and double
+            linkage.  We separate comparing string and number attributes
+            from comparing the children of input children.
+            """
+            self.assertEqual(children1.keys(), children2.keys())
+            for ob in children1.values():
+                self.assertIs(ob.parent, parent1)
+            for ob in children2.values():
+                self.assertIs(ob.parent, parent2)
+            for key in children1.keys():
+                o1, o2 = children1[key], children2[key]
+                t1 = type(o1), o1.name, o1.file, o1.module, o1.lineno
+                t2 = type(o2), o2.name, o2.file, o2.module, o2.lineno
+                self.assertEqual(t1, t2)
+                if type(o1) is mb.Class:
+                    self.assertEqual(o1.methods, o2.methods)
+                # Skip superclasses for now as not part of example
+                compare(o1, o1.children, o2, o2.children)
+
+        compare(None, actual, None, expected)
+
     def test_others(self):
         cm = self.checkModule