Skip to content

C
cpython
Commit 4bd65740 authored by Barry Warsaw's avatar Barry Warsaw
Browse files
Options

PEP 3147

parent 66506cc7
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1204 additions and 288 deletions
.bzrignore
View file @ 4bd65740
......@@ -33,3 +33,4 @@ Parser/pgen
Lib/test/data/*
Lib/lib2to3/Grammar*.pickle
Lib/lib2to3/PatternGrammar*.pickle
__pycache__
.hgignore
View file @ 4bd65740
......@@ -54,3 +54,4 @@ PCbuild/*.o
PCbuild/*.ncb
PCbuild/*.bsc
PCbuild/Win32-temp-*
__pycache__
Doc/c-api/import.rst
View file @ 4bd65740
......@@ -124,12 +124,24 @@ Importing Modules
If *name* points to a dotted name of the form ``package.module``, any package
structures not already created will still not be created.
See also :func:`PyImport_ExecCodeModuleEx` and
:func:`PyImport_ExecCodeModuleWithPathnames`.
.. cfunction:: PyObject* PyImport_ExecCodeModuleEx(char *name, PyObject *co, char *pathname)
Like :cfunc:`PyImport_ExecCodeModule`, but the :attr:`__file__` attribute of
the module object is set to *pathname* if it is non-``NULL``.
See also :func:`PyImport_ExecCodeModuleWithPathnames`.
.. cfunction:: PyObject* PyImport_ExecCodeModuleWithPathnames(char *name, PyObject *co, char *pathname, char *cpathname)
Like :cfunc:`PyImport_ExecCodeModuleEx`, but the :attr:`__cached__`
attribute of the module object is set to *cpathname* if it is
non-``NULL``. Of the three functions, this is the preferred one to use.
.. cfunction:: long PyImport_GetMagicNumber()
......@@ -138,6 +150,11 @@ Importing Modules
of the bytecode file, in little-endian byte order.
.. cfunction:: const char * PyImport_GetMagicTag()
Return the magic tag string for :pep:`3147` format Python bytecode file
names.
.. cfunction:: PyObject* PyImport_GetModuleDict()
Return the dictionary used for the module administration (a.k.a.
......
Doc/library/compileall.rst
View file @ 4bd65740
......@@ -17,9 +17,11 @@ line. If no arguments are given, the invocation is equivalent to ``-l
sys.path``. Printing lists of the files compiled can be disabled with the
:option:`-q` flag. In addition, the :option:`-x` option takes a regular
expression argument. All files that match the expression will be skipped.
The :option:`-b` flag may be given to write legacy ``.pyc`` file path names,
otherwise :pep:`3147` style byte-compiled path names are written.
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False)
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False)
Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
files along the way. The *maxlevels* parameter is used to limit the depth of
......@@ -34,12 +36,16 @@ expression argument. All files that match the expression will be skipped.
If *quiet* is true, nothing is printed to the standard output in normal
operation.
If *legacy* is true, old-style ``.pyc`` file path names are written,
otherwise (the default), :pep:`3147` style path names are written.
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False)
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False)
Byte-compile all the :file:`.py` files found along ``sys.path``. If
*skip_curdir* is true (the default), the current directory is not included in
the search. The *maxlevels* and *force* parameters default to ``0`` and are
the search. The *maxlevels* parameter defaults to ``0``, and the *force*
and *legacy* parameters default to ``False``. All are
passed to the :func:`compile_dir` function.
To force a recompile of all the :file:`.py` files in the :file:`Lib/`
......
Doc/library/imp.rst
View file @ 4bd65740
......@@ -204,8 +204,41 @@ This module provides an interface to the mechanisms used to implement the
function does nothing.
The following constants with integer values, defined in this module, are used to
indicate the search result of :func:`find_module`.
The following functions and data provide conveniences for handling :pep:`3147`
byte-compiled file paths.
.. versionadded:: 3.2
.. function:: cache_from_source(path, debug_override=None)
Return the PEP 3147 path to the byte-compiled file associated with the
source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return
value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2.
The ``cpython-32`` string comes from the current magic tag (see
:func:`get_tag`). The returned path will end in ``.pyc`` when
``__debug__`` is True or ``.pyo`` for an optimized Python
(i.e. ``__debug__`` is False). By passing in True or False for
*debug_override* you can override the system's value for ``__debug__`` for
extension selection.
*path* need not exist.
.. function:: source_from_cache(path)
Given the *path* to a PEP 3147 file name, return the associated source code
file path. For example, if *path* is
``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be
``/foo/bar/baz.py``. *path* need not exist, however if it does not conform
to PEP 3147 format, a ``ValueError`` is raised.
.. function:: get_tag()
Return the PEP 3147 magic tag string matching this version of Python's
magic number, as returned by :func:`get_magic`.
The following constants with integer values, defined in this module, are used
to indicate the search result of :func:`find_module`.
.. data:: PY_SOURCE
......
Doc/library/py_compile.rst
View file @ 4bd65740
......@@ -26,12 +26,16 @@ byte-code cache files in the directory containing the source code.
Compile a source file to byte-code and write out the byte-code cache file. The
source code is loaded from the file name *file*. The byte-code is written to
*cfile*, which defaults to *file* ``+`` ``'c'`` (``'o'`` if optimization is
enabled in the current interpreter). If *dfile* is specified, it is used as the
*cfile*, which defaults to the :PEP:`3147` path, ending in ``.pyc``
(``'.pyo`` if optimization is enabled in the current interpreter). For
example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to
``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the
name of the source file in error messages instead of *file*. If *doraise* is
true, a :exc:`PyCompileError` is raised when an error is encountered while
compiling *file*. If *doraise* is false (the default), an error string is
written to ``sys.stderr``, but no exception is raised.
written to ``sys.stderr``, but no exception is raised. This function
returns the path to byte-compiled file, i.e. whatever *cfile* value was
used.
.. function:: main(args=None)
......
Doc/library/runpy.rst
View file @ 4bd65740
......@@ -32,7 +32,8 @@ The :mod:`runpy` module provides two functions:
below are defined in the supplied dictionary, those definitions are
overridden by :func:`run_module`.
The special global variables ``__name__``, ``__file__``, ``__loader__``
The special global variables ``__name__``, ``__file__``, ``__cached__``,
``__loader__``
and ``__package__`` are set in the globals dictionary before the module
code is executed (Note that this is a minimal set of variables - other
variables may be set implicitly as an interpreter implementation detail).
......@@ -45,6 +46,8 @@ The :mod:`runpy` module provides two functions:
loader does not make filename information available, this variable is set
to :const:`None`.
``__cached__`` will be set to ``None``.
``__loader__`` is set to the PEP 302 module loader used to retrieve the
code for the module (This loader may be a wrapper around the standard
import mechanism).
......
Include/import.h
View file @ 4bd65740
......@@ -8,9 +8,12 @@ extern "C" {
#endif
PyAPI_FUNC(long) PyImport_GetMagicNumber(void);
PyAPI_FUNC(const char *) PyImport_GetMagicTag(void);
PyAPI_FUNC(PyObject *) PyImport_ExecCodeModule(char *name, PyObject *co);
PyAPI_FUNC(PyObject *) PyImport_ExecCodeModuleEx(
char *name, PyObject *co, char *pathname);
PyAPI_FUNC(PyObject *) PyImport_ExecCodeModuleWithPathnames(
char *name, PyObject *co, char *pathname, char *cpathname);
PyAPI_FUNC(PyObject *) PyImport_GetModuleDict(void);
PyAPI_FUNC(PyObject *) PyImport_AddModule(const char *name);
PyAPI_FUNC(PyObject *) PyImport_ImportModule(const char *name);
......
Lib/compileall.py
View file @ 4bd65740
......@@ -12,6 +12,7 @@ See module py_compile for details of the actual byte-compilation.
"""
import os
import errno
import sys
import py_compile
import struct
......@@ -20,7 +21,7 @@ import imp
__all__ = ["compile_dir","compile_file","compile_path"]
def compile_dir(dir, maxlevels=10, ddir=None,
force=0, rx=None, quiet=0):
force=False, rx=None, quiet=False, legacy=False):
"""Byte-compile all modules in the given directory tree.
Arguments (only dir is required):
......@@ -29,8 +30,9 @@ def compile_dir(dir, maxlevels=10, ddir=None,
maxlevels: maximum recursion level (default 10)
ddir: if given, purported directory name (this is the
directory name that will show up in error messages)
force: if 1, force compilation, even if timestamps are up-to-date
quiet: if 1, be quiet during compilation
force: if True, force compilation, even if timestamps are up-to-date
quiet: if True, be quiet during compilation
legacy: if True, produce legacy pyc paths instead of PEP 3147 paths
"""
if not quiet:
......@@ -49,24 +51,26 @@ def compile_dir(dir, maxlevels=10, ddir=None,
else:
dfile = None
if not os.path.isdir(fullname):
if not compile_file(fullname, ddir, force, rx, quiet):
if not compile_file(fullname, ddir, force, rx, quiet, legacy):
success = 0
elif maxlevels > 0 and \
name != os.curdir and name != os.pardir and \
os.path.isdir(fullname) and \
not os.path.islink(fullname):
if not compile_dir(fullname, maxlevels - 1, dfile, force, rx,
quiet):
quiet, legacy):
success = 0
return success
def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
legacy=False):
"""Byte-compile file.
file: the file to byte-compile
fullname: the file to byte-compile
ddir: if given, purported directory name (this is the
directory name that will show up in error messages)
force: if 1, force compilation, even if timestamps are up-to-date
quiet: if 1, be quiet during compilation
force: if True, force compilation, even if timestamps are up-to-date
quiet: if True, be quiet during compilation
legacy: if True, produce legacy pyc paths instead of PEP 3147 paths
"""
success = 1
......@@ -80,13 +84,22 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
if mo:
return success
if os.path.isfile(fullname):
if legacy:
cfile = fullname + ('c' if __debug__ else 'o')
else:
cfile = imp.cache_from_source(fullname)
cache_dir = os.path.dirname(cfile)
try:
os.mkdir(cache_dir)
except OSError as error:
if error.errno != errno.EEXIST:
raise
head, tail = name[:-3], name[-3:]
if tail == '.py':
if not force:
try:
mtime = int(os.stat(fullname).st_mtime)
expect = struct.pack('<4sl', imp.get_magic(), mtime)
cfile = fullname + (__debug__ and 'c' or 'o')
with open(cfile, 'rb') as chandle:
actual = chandle.read(8)
if expect == actual:
......@@ -96,14 +109,15 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
if not quiet:
print('Compiling', fullname, '...')
try:
ok = py_compile.compile(fullname, None, dfile, True)
ok = py_compile.compile(fullname, cfile, dfile, True)
except py_compile.PyCompileError as err:
if quiet:
print('*** Error compiling', fullname, '...')
else:
print('*** ', end='')
# escape non-printable characters in msg
msg = err.msg.encode(sys.stdout.encoding, errors='backslashreplace')
msg = err.msg.encode(sys.stdout.encoding,
errors='backslashreplace')
msg = msg.decode(sys.stdout.encoding)
print(msg)
success = 0
......@@ -119,15 +133,17 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
success = 0
return success
def compile_path(skip_curdir=1, maxlevels=0, force=0, quiet=0):
def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
legacy=False):
"""Byte-compile all module on sys.path.
Arguments (all optional):
skip_curdir: if true, skip current directory (default true)
maxlevels: max recursion level (default 0)
force: as for compile_dir() (default 0)
quiet: as for compile_dir() (default 0)
force: as for compile_dir() (default False)
quiet: as for compile_dir() (default False)
legacy: as for compile_dir() (default False)
"""
success = 1
......@@ -136,7 +152,8 @@ def compile_path(skip_curdir=1, maxlevels=0, force=0, quiet=0):
print('Skipping current directory')
else:
success = success and compile_dir(dir, maxlevels, None,
force, quiet=quiet)
force, quiet=quiet,
legacy=legacy)
return success
def expand_args(args, flist):
......@@ -162,10 +179,10 @@ def main():
"""Script main program."""
import getopt
try:
opts, args = getopt.getopt(sys.argv[1:], 'lfqd:x:i:')
opts, args = getopt.getopt(sys.argv[1:], 'lfqd:x:i:b')
except getopt.error as msg:
print(msg)
print("usage: python compileall.py [-l] [-f] [-q] [-d destdir] " \
print("usage: python compileall.py [-l] [-f] [-q] [-d destdir] "
"[-x regexp] [-i list] [directory|file ...]")
print("-l: don't recurse down")
print("-f: force rebuild even if timestamps are up-to-date")
......@@ -174,23 +191,27 @@ def main():
print(" if no directory arguments, -l sys.path is assumed")
print("-x regexp: skip files matching the regular expression regexp")
print(" the regexp is searched for in the full path of the file")
print("-i list: expand list with its content (file and directory names)")
print("-i list: expand list with its content "
"(file and directory names)")
print("-b: Produce legacy byte-compile file paths")
sys.exit(2)
maxlevels = 10
ddir = None
force = 0
quiet = 0
force = False
quiet = False
rx = None
flist = None
legacy = False
for o, a in opts:
if o == '-l': maxlevels = 0
if o == '-d': ddir = a
if o == '-f': force = 1
if o == '-q': quiet = 1
if o == '-f': force = True
if o == '-q': quiet = True
if o == '-x':
import re
rx = re.compile(a)
if o == '-i': flist = a
if o == '-b': legacy = True
if ddir:
if len(args) != 1 and not os.path.isdir(args[0]):
print("-d destdir require exactly one directory argument")
......@@ -207,13 +228,14 @@ def main():
for arg in args:
if os.path.isdir(arg):
if not compile_dir(arg, maxlevels, ddir,
force, rx, quiet):
force, rx, quiet, legacy):
success = 0
else:
if not compile_file(arg, ddir, force, rx, quiet):
if not compile_file(arg, ddir, force, rx,
quiet, legacy):
success = 0
else:
success = compile_path()
success = compile_path(legacy=legacy)
except KeyboardInterrupt:
print("\n[interrupt]")
success = 0
......
Lib/importlib/_bootstrap.py
View file @ 4bd65740
......@@ -488,6 +488,16 @@ class _PyPycFileLoader(PyPycLoader, _PyFileLoader):
"""Load a module from a source or bytecode file."""
def _find_path(self, ext_type):
"""Return PEP 3147 path if ext_type is PY_COMPILED, otherwise
super()._find_path() is called."""
if ext_type == imp.PY_COMPILED:
# We don't really care what the extension on self._base_path is,
# as long as it has exactly one dot.
bytecode_path = imp.cache_from_source(self._base_path + '.py')
return (bytecode_path if _path_exists(bytecode_path) else None)
return super()._find_path(ext_type)
@_check_name
def source_mtime(self, name):
"""Return the modification time of the source for the specified
......@@ -515,7 +525,16 @@ class _PyPycFileLoader(PyPycLoader, _PyFileLoader):
"""
bytecode_path = self.bytecode_path(name)
if not bytecode_path:
bytecode_path = self._base_path + _suffix_list(imp.PY_COMPILED)[0]
source_path = self.source_path(name)
bytecode_path = imp.cache_from_source(source_path)
# Ensure that the __pycache__ directory exists. We can't use
# os.path.dirname() here.
dirname, sep, basename = bytecode_path.rpartition(path_sep)
try:
_os.mkdir(dirname)
except OSError as error:
if error.errno != errno.EEXIST:
raise
try:
# Assuming bytes.
with _closing(_io.FileIO(bytecode_path, 'w')) as bytecode_file:
......
Lib/importlib/test/__main__.py
View file @ 4bd65740
......@@ -13,7 +13,12 @@ import unittest
def test_main():
start_dir = os.path.dirname(__file__)
if '__pycache__' in __file__:
parts = __file__.split(os.path.sep)
start_dir = sep.join(parts[:-2])
else:
start_dir = os.path.dirname(__file__)
# XXX 2010-03-18 barry: Fix __file__
top_dir = os.path.dirname(os.path.dirname(start_dir))
test_loader = unittest.TestLoader()
if '--builtin' in sys.argv:
......
Lib/importlib/test/source/test_file_loader.py
View file @ 4bd65740
......@@ -127,7 +127,7 @@ class BadBytecodeTest(unittest.TestCase):
except KeyError:
pass
py_compile.compile(mapping[name])
bytecode_path = source_util.bytecode_path(mapping[name])
bytecode_path = imp.cache_from_source(mapping[name])
with open(bytecode_path, 'rb') as file:
bc = file.read()
new_bc = manipulator(bc)
......@@ -226,7 +226,7 @@ class BadBytecodeTest(unittest.TestCase):
zeros = b'\x00\x00\x00\x00'
with source_util.create_modules('_temp') as mapping:
py_compile.compile(mapping['_temp'])
bytecode_path = source_util.bytecode_path(mapping['_temp'])
bytecode_path = imp.cache_from_source(mapping['_temp'])
with open(bytecode_path, 'r+b') as bytecode_file:
bytecode_file.seek(4)
bytecode_file.write(zeros)
......@@ -242,9 +242,10 @@ class BadBytecodeTest(unittest.TestCase):
def test_bad_marshal(self):
# Bad marshal data should raise a ValueError.
with source_util.create_modules('_temp') as mapping:
bytecode_path = source_util.bytecode_path(mapping['_temp'])
bytecode_path = imp.cache_from_source(mapping['_temp'])
source_mtime = os.path.getmtime(mapping['_temp'])
source_timestamp = importlib._w_long(source_mtime)
source_util.ensure_bytecode_path(bytecode_path)
with open(bytecode_path, 'wb') as bytecode_file:
bytecode_file.write(imp.get_magic())
bytecode_file.write(source_timestamp)
......@@ -260,7 +261,7 @@ class BadBytecodeTest(unittest.TestCase):
with source_util.create_modules('_temp') as mapping:
# Create bytecode that will need to be re-created.
py_compile.compile(mapping['_temp'])
bytecode_path = source_util.bytecode_path(mapping['_temp'])
bytecode_path = imp.cache_from_source(mapping['_temp'])
with open(bytecode_path, 'r+b') as bytecode_file:
bytecode_file.seek(0)
bytecode_file.write(b'\x00\x00\x00\x00')
......
Lib/importlib/test/source/test_finder.py
View file @ 4bd65740
from importlib import _bootstrap
from .. import abc
from . import util as source_util
from test.support import make_legacy_pyc
import os
import errno
import py_compile
import unittest
import warnings
......@@ -52,6 +54,14 @@ class FinderTests(abc.FinderTests):
if unlink:
for name in unlink:
os.unlink(mapping[name])
try:
make_legacy_pyc(mapping[name])
except OSError as error:
# Some tests do not set compile_=True so the source
# module will not get compiled and there will be no
# PEP 3147 pyc file to rename.
if error.errno != errno.ENOENT:
raise
loader = self.import_(mapping['.root'], test)
self.assertTrue(hasattr(loader, 'load_module'))
return loader
......@@ -60,7 +70,8 @@ class FinderTests(abc.FinderTests):
# [top-level source]
self.run_test('top_level')
# [top-level bc]
self.run_test('top_level', compile_={'top_level'}, unlink={'top_level'})
self.run_test('top_level', compile_={'top_level'},
unlink={'top_level'})
# [top-level both]
self.run_test('top_level', compile_={'top_level'})
......
Lib/importlib/test/source/test_source_encoding.py
View file @ 4bd65740
......@@ -33,7 +33,7 @@ class EncodingTest(unittest.TestCase):
def run_test(self, source):
with source_util.create_modules(self.module_name) as mapping:
with open(mapping[self.module_name], 'wb')as file:
with open(mapping[self.module_name], 'wb') as file:
file.write(source)
loader = _bootstrap._PyPycFileLoader(self.module_name,
mapping[self.module_name], False)
......
Lib/importlib/test/source/util.py
View file @ 4bd65740
from .. import util
import contextlib
import errno
import functools
import imp
import os
......@@ -26,14 +27,16 @@ def writes_bytecode_files(fxn):
return wrapper
def bytecode_path(source_path):
for suffix, _, type_ in imp.get_suffixes():
if type_ == imp.PY_COMPILED:
bc_suffix = suffix
break
else:
raise ValueError("no bytecode suffix is defined")
return os.path.splitext(source_path)[0] + bc_suffix
def ensure_bytecode_path(bytecode_path):
"""Ensure that the __pycache__ directory for PEP 3147 pyc file exists.
:param bytecode_path: File system path to PEP 3147 pyc file.
"""
try:
os.mkdir(os.path.dirname(bytecode_path))
except OSError as error:
if error.errno != errno.EEXIST:
raise
@contextlib.contextmanager
......
Lib/importlib/util.py
View file @ 4bd65740
"""Utility code for constructing importers, etc."""
from ._bootstrap import module_for_loader
from ._bootstrap import set_loader
from ._bootstrap import set_package
Lib/inspect.py
View file @ 4bd65740
......@@ -54,6 +54,7 @@ def ismodule(object):
"""Return true if the object is a module.
Module objects provide these attributes:
__cached__ pathname to byte compiled file
__doc__ documentation string
__file__ filename (missing for built-in modules)"""
return isinstance(object, types.ModuleType)
......
Lib/py_compile.py
View file @ 4bd65740
......@@ -4,6 +4,7 @@ This module has intimate knowledge of the format of .pyc files.
"""
import builtins
import errno
import imp
import marshal
import os
......@@ -37,16 +38,18 @@ class PyCompileError(Exception):
can be accesses as class variable 'file'
msg: string message to be written as error message
If no value is given, a default exception message will be given,
consistent with 'standard' py_compile output.
message (or default) can be accesses as class variable 'msg'
If no value is given, a default exception message will be
given, consistent with 'standard' py_compile output.
message (or default) can be accesses as class variable
'msg'
"""
def __init__(self, exc_type, exc_value, file, msg=''):
exc_type_name = exc_type.__name__
if exc_type is SyntaxError:
tbtext = ''.join(traceback.format_exception_only(exc_type, exc_value))
tbtext = ''.join(traceback.format_exception_only(
exc_type, exc_value))
errmsg = tbtext.replace('File "<string>"', 'File "%s"' % file)
else:
errmsg = "Sorry: %s: %s" % (exc_type_name,exc_value)
......@@ -64,7 +67,7 @@ class PyCompileError(Exception):
def wr_long(f, x):
"""Internal; write a 32-bit int to a file in little-endian order."""
f.write(bytes([x & 0xff,
f.write(bytes([x & 0xff,
(x >> 8) & 0xff,
(x >> 16) & 0xff,
(x >> 24) & 0xff]))
......@@ -72,20 +75,18 @@ def wr_long(f, x):
def compile(file, cfile=None, dfile=None, doraise=False):
"""Byte-compile one Python source file to Python bytecode.
Arguments:
file: source filename
cfile: target filename; defaults to source with 'c' or 'o' appended
('c' normally, 'o' in optimizing mode, giving .pyc or .pyo)
dfile: purported filename; defaults to source (this is the filename
that will show up in error messages)
doraise: flag indicating whether or not an exception should be
raised when a compile error is found. If an exception
occurs and this flag is set to False, a string
indicating the nature of the exception will be printed,
and the function will return to the caller. If an
exception occurs and this flag is set to True, a
PyCompileError exception will be raised.
:param file: The source file name.
:param cfile: The target byte compiled file name. When not given, this
defaults to the PEP 3147 location.
:param dfile: Purported file name, i.e. the file name that shows up in
error messages. Defaults to the source file name.
:param doraise: Flag indicating whether or not an exception should be
raised when a compile error is found. If an exception occurs and this
flag is set to False, a string indicating the nature of the exception
will be printed, and the function will return to the caller. If an
exception occurs and this flag is set to True, a PyCompileError
exception will be raised.
:return: Path to the resulting byte compiled file.
Note that it isn't necessary to byte-compile Python modules for
execution efficiency -- Python itself byte-compiles a module when
......@@ -102,7 +103,6 @@ def compile(file, cfile=None, dfile=None, doraise=False):
See compileall.py for a script/module that uses this module to
byte-compile all installed files (or all files in selected
directories).
"""
with open(file, "rb") as f:
encoding = tokenize.detect_encoding(f.readline)[0]
......@@ -122,7 +122,12 @@ def compile(file, cfile=None, dfile=None, doraise=False):
sys.stderr.write(py_exc.msg + '\n')
return
if cfile is None:
cfile = file + (__debug__ and 'c' or 'o')
cfile = imp.cache_from_source(file)
try:
os.mkdir(os.path.dirname(cfile))
except OSError as error:
if error.errno != errno.EEXIST:
raise
with open(cfile, 'wb') as fc:
fc.write(b'\0\0\0\0')
wr_long(fc, timestamp)
......@@ -130,6 +135,7 @@ def compile(file, cfile=None, dfile=None, doraise=False):
fc.flush()
fc.seek(0, 0)
fc.write(MAGIC)
return cfile
def main(args=None):
"""Compile several source files.
......
Lib/pydoc.py
View file @ 4bd65740
......@@ -159,7 +159,8 @@ def visiblename(name, all=None):
"""Decide whether to show documentation on a variable."""
# Certain special names are redundant.
_hidden_names = ('__builtins__', '__doc__', '__file__', '__path__',
'__module__', '__name__', '__slots__', '__package__')
'__module__', '__name__', '__slots__', '__package__',
'__cached__')
if name in _hidden_names: return 0
# Private names are hidden, but special names are displayed.
if name.startswith('__') and name.endswith('__'): return 1
......
Lib/runpy.py
View file @ 4bd65740
......@@ -67,6 +67,7 @@ def _run_code(code, run_globals, init_globals=None,
run_globals.update(init_globals)
run_globals.update(__name__ = mod_name,
__file__ = mod_fname,
__cached__ = None,
__loader__ = mod_loader,
__package__ = pkg_name)
exec(code, run_globals)
......@@ -130,6 +131,7 @@ def _run_module_as_main(mod_name, alter_argv=True):
At the very least, these variables in __main__ will be overwritten:
__name__
__file__
__cached__
__loader__
__package__
"""
......
Lib/site.py
View file @ 4bd65740
......@@ -74,15 +74,19 @@ def makepath(*paths):
return dir, os.path.normcase(dir)
def abs__file__():
"""Set all module' __file__ attribute to an absolute path"""
def abs_paths():
"""Set all module __file__ and __cached__ attributes to an absolute path"""
for m in set(sys.modules.values()):
if hasattr(m, '__loader__'):
continue # don't mess with a PEP 302-supplied __file__
try:
m.__file__ = os.path.abspath(m.__file__)
except AttributeError:
continue
pass
try:
m.__cached__ = os.path.abspath(m.__cached__)
except AttributeError:
pass
def removeduppaths():
......@@ -518,7 +522,7 @@ def execusercustomize():
def main():
global ENABLE_USER_SITE
abs__file__()
abs_paths()
known_paths = removeduppaths()
if (os.name == "posix" and sys.path and
os.path.basename(sys.path[-1]) == "Modules"):
......
Lib/test/script_helper.py
View file @ 4bd65740
......@@ -11,6 +11,9 @@ import contextlib
import shutil
import zipfile
from imp import source_from_cache
from test.support import make_legacy_pyc
# Executing the interpreter in a subprocess
def python_exit_code(*args):
cmd_line = [sys.executable, '-E']
......@@ -62,20 +65,18 @@ def make_script(script_dir, script_basename, source):
script_file.close()
return script_name
def compile_script(script_name):
py_compile.compile(script_name, doraise=True)
if __debug__:
compiled_name = script_name + 'c'
else:
compiled_name = script_name + 'o'
return compiled_name
def make_zip_script(zip_dir, zip_basename, script_name, name_in_zip=None):
zip_filename = zip_basename+os.extsep+'zip'
zip_name = os.path.join(zip_dir, zip_filename)
zip_file = zipfile.ZipFile(zip_name, 'w')
if name_in_zip is None:
name_in_zip = os.path.basename(script_name)
parts = script_name.split(os.sep)
if len(parts) >= 2 and parts[-2] == '__pycache__':
legacy_pyc = make_legacy_pyc(source_from_cache(script_name))
name_in_zip = os.path.basename(legacy_pyc)
script_name = legacy_pyc
else:
name_in_zip = os.path.basename(script_name)
zip_file.write(script_name, name_in_zip)
zip_file.close()
#if test.test_support.verbose:
......@@ -98,8 +99,8 @@ def make_zip_pkg(zip_dir, zip_basename, pkg_name, script_basename,
script_name = make_script(zip_dir, script_basename, source)
unlink.append(script_name)
if compiled:
init_name = compile_script(init_name)
script_name = compile_script(script_name)
init_name = py_compile(init_name, doraise=True)
script_name = py_compile(script_name, doraise=True)
unlink.extend((init_name, script_name))
pkg_names = [os.sep.join([pkg_name]*i) for i in range(1, depth+1)]
script_name_in_zip = os.path.join(pkg_names[-1], os.path.basename(script_name))
......
Lib/test/support.py
View file @ 4bd65740
......@@ -17,22 +17,25 @@ import unittest
import importlib
import collections
import re
import imp
import time
__all__ = ["Error", "TestFailed", "ResourceDenied", "import_module",
"verbose", "use_resources", "max_memuse", "record_original_stdout",
"get_original_stdout", "unload", "unlink", "rmtree", "forget",
"is_resource_enabled", "requires", "find_unused_port", "bind_port",
"fcmp", "is_jython", "TESTFN", "HOST", "FUZZ", "SAVEDCWD", "temp_cwd",
"findfile", "sortdict", "check_syntax_error", "open_urlresource",
"check_warnings", "CleanImport", "EnvironmentVarGuard",
"TransientResource", "captured_output", "captured_stdout",
"time_out", "socket_peer_reset", "ioerror_peer_reset",
"run_with_locale",
"set_memlimit", "bigmemtest", "bigaddrspacetest", "BasicTestRunner",
"run_unittest", "run_doctest", "threading_setup", "threading_cleanup",
"reap_children", "cpython_only", "check_impl_detail", "get_attribute",
"swap_item", "swap_attr"]
__all__ = [
"Error", "TestFailed", "ResourceDenied", "import_module",
"verbose", "use_resources", "max_memuse", "record_original_stdout",
"get_original_stdout", "unload", "unlink", "rmtree", "forget",
"is_resource_enabled", "requires", "find_unused_port", "bind_port",
"fcmp", "is_jython", "TESTFN", "HOST", "FUZZ", "SAVEDCWD", "temp_cwd",
"findfile", "sortdict", "check_syntax_error", "open_urlresource",
"check_warnings", "CleanImport", "EnvironmentVarGuard",
"TransientResource", "captured_output", "captured_stdout",
"time_out", "socket_peer_reset", "ioerror_peer_reset",
"run_with_locale", 'temp_umask',
"set_memlimit", "bigmemtest", "bigaddrspacetest", "BasicTestRunner",
"run_unittest", "run_doctest", "threading_setup", "threading_cleanup",
"reap_children", "cpython_only", "check_impl_detail", "get_attribute",
"swap_item", "swap_attr",
]
class Error(Exception):
......@@ -177,27 +180,50 @@ def unload(name):
def unlink(filename):
try:
os.unlink(filename)
except OSError:
pass
except OSError as error:
# The filename need not exist.
if error.errno != errno.ENOENT:
raise
def rmtree(path):
try:
shutil.rmtree(path)
except OSError as e:
except OSError as error:
# Unix returns ENOENT, Windows returns ESRCH.
if e.errno not in (errno.ENOENT, errno.ESRCH):
if error.errno not in (errno.ENOENT, errno.ESRCH):
raise
def make_legacy_pyc(source):
"""Move a PEP 3147 pyc/pyo file to its legacy pyc/pyo location.
The choice of .pyc or .pyo extension is done based on the __debug__ flag
value.
:param source: The file system path to the source file. The source file
does not need to exist, however the PEP 3147 pyc file must exist.
:return: The file system path to the legacy pyc file.
"""
pyc_file = imp.cache_from_source(source)
up_one = os.path.dirname(os.path.abspath(source))
legacy_pyc = os.path.join(up_one, source + ('c' if __debug__ else 'o'))
os.rename(pyc_file, legacy_pyc)
return legacy_pyc
def forget(modname):
'''"Forget" a module was ever imported by removing it from sys.modules and
deleting any .pyc and .pyo files.'''
"""'Forget' a module was ever imported.
This removes the module from sys.modules and deletes any PEP 3147 or
legacy .pyc and .pyo files.
"""
unload(modname)
for dirname in sys.path:
unlink(os.path.join(dirname, modname + '.pyc'))
# Deleting the .pyo file cannot be within the 'try' for the .pyc since
# the chance exists that there is no .pyc (and thus the 'try' statement
# is exited) but there is a .pyo file.
unlink(os.path.join(dirname, modname + '.pyo'))
source = os.path.join(dirname, modname + '.py')
# It doesn't matter if they exist or not, unlink all possible
# combinations of PEP 3147 and legacy pyc and pyo files.
unlink(source + 'c')
unlink(source + 'o')
unlink(imp.cache_from_source(source, debug_override=True))
unlink(imp.cache_from_source(source, debug_override=False))
def is_resource_enabled(resource):
"""Test whether a resource is enabled. Known resources are set by
......@@ -208,7 +234,9 @@ def requires(resource, msg=None):
"""Raise ResourceDenied if the specified resource is not available.
If the caller's module is __main__ then automatically return True. The
possibility of False being returned occurs when regrtest.py is executing."""
possibility of False being returned occurs when regrtest.py is
executing.
"""
# see if the caller's module is __main__ - if so, treat as if
# the resource was set
if sys._getframe(1).f_globals.get("__name__") == "__main__":
......@@ -405,6 +433,16 @@ def temp_cwd(name='tempcwd', quiet=False):
rmtree(name)
@contextlib.contextmanager
def temp_umask(umask):
"""Context manager that temporarily sets the process umask."""
oldmask = os.umask(umask)
try:
yield
finally:
os.umask(oldmask)
def findfile(file, here=__file__, subdir=None):
"""Try to find a file on sys.path and the working directory. If it is not
found the argument passed to the function is returned (this does not
......
Lib/test/test_cmd_line_script.py
View file @ 4bd65740
# Tests command line execution of scripts
# tests command line execution of scripts
import unittest
import os
import os.path
import py_compile
import test.support
from test.script_helper import (run_python,
temp_dir, make_script, compile_script,
make_pkg, make_zip_script, make_zip_pkg)
from test.script_helper import (
make_pkg, make_script, make_zip_pkg, make_zip_script, run_python,
temp_dir)
verbose = test.support.verbose
......@@ -28,6 +30,7 @@ assertEqual(result, ['Top level assignment', 'Lower level reference'])
# Check population of magic variables
assertEqual(__name__, '__main__')
print('__file__==%r' % __file__)
assertEqual(__cached__, None)
print('__package__==%r' % __package__)
# Check the sys module
import sys
......@@ -101,9 +104,10 @@ class CmdLineTest(unittest.TestCase):
def test_script_compiled(self):
with temp_dir() as script_dir:
script_name = _make_test_script(script_dir, 'script')
compiled_name = compile_script(script_name)
compiled_name = py_compile.compile(script_name, doraise=True)
os.remove(script_name)
self._check_script(compiled_name, compiled_name, compiled_name, None)
self._check_script(compiled_name, compiled_name,
compiled_name, None)
def test_directory(self):
with temp_dir() as script_dir:
......@@ -113,9 +117,10 @@ class CmdLineTest(unittest.TestCase):
def test_directory_compiled(self):
with temp_dir() as script_dir:
script_name = _make_test_script(script_dir, '__main__')
compiled_name = compile_script(script_name)
compiled_name = py_compile.compile(script_name, doraise=True)
os.remove(script_name)
self._check_script(script_dir, compiled_name, script_dir, '')
pyc_file = test.support.make_legacy_pyc(script_name)
self._check_script(script_dir, pyc_file, script_dir, '')
def test_directory_error(self):
with temp_dir() as script_dir:
......@@ -131,7 +136,7 @@ class CmdLineTest(unittest.TestCase):
def test_zipfile_compiled(self):
with temp_dir() as script_dir:
script_name = _make_test_script(script_dir, '__main__')
compiled_name = compile_script(script_name)
compiled_name = py_compile.compile(script_name, doraise=True)
zip_name, run_name = make_zip_script(script_dir, 'test_zip', compiled_name)
self._check_script(zip_name, run_name, zip_name, '')
......@@ -176,11 +181,12 @@ class CmdLineTest(unittest.TestCase):
pkg_dir = os.path.join(script_dir, 'test_pkg')
make_pkg(pkg_dir)
script_name = _make_test_script(pkg_dir, '__main__')
compiled_name = compile_script(script_name)
compiled_name = py_compile.compile(script_name, doraise=True)
os.remove(script_name)
pyc_file = test.support.make_legacy_pyc(script_name)
launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg')
self._check_script(launch_name, compiled_name,
compiled_name, 'test_pkg')
self._check_script(launch_name, pyc_file,
pyc_file, 'test_pkg')
def test_package_error(self):
with temp_dir() as script_dir:
......
Lib/test/test_compileall.py
View file @ 4bd65740
......@@ -5,22 +5,23 @@ import os
import py_compile
import shutil
import struct
import subprocess
import tempfile
from test import support
import unittest
import io
from test import support
class CompileallTests(unittest.TestCase):
def setUp(self):
self.directory = tempfile.mkdtemp()
self.source_path = os.path.join(self.directory, '_test.py')
self.bc_path = self.source_path + ('c' if __debug__ else 'o')
self.bc_path = imp.cache_from_source(self.source_path)
with open(self.source_path, 'w') as file:
file.write('x = 123\n')
self.source_path2 = os.path.join(self.directory, '_test2.py')
self.bc_path2 = self.source_path2 + ('c' if __debug__ else 'o')
self.bc_path2 = imp.cache_from_source(self.source_path2)
shutil.copyfile(self.source_path, self.source_path2)
def tearDown(self):
......@@ -65,17 +66,19 @@ class CompileallTests(unittest.TestCase):
except:
pass
compileall.compile_file(self.source_path, force=False, quiet=True)
self.assertTrue(os.path.isfile(self.bc_path) \
and not os.path.isfile(self.bc_path2))
self.assertTrue(os.path.isfile(self.bc_path) and
not os.path.isfile(self.bc_path2))
os.unlink(self.bc_path)
compileall.compile_dir(self.directory, force=False, quiet=True)
self.assertTrue(os.path.isfile(self.bc_path) \
and os.path.isfile(self.bc_path2))
self.assertTrue(os.path.isfile(self.bc_path) and
os.path.isfile(self.bc_path2))
os.unlink(self.bc_path)
os.unlink(self.bc_path2)
class EncodingTest(unittest.TestCase):
'Issue 6716: compileall should escape source code when printing errors to stdout.'
"""Issue 6716: compileall should escape source code when printing errors
to stdout."""
def setUp(self):
self.directory = tempfile.mkdtemp()
......@@ -95,9 +98,65 @@ class EncodingTest(unittest.TestCase):
finally:
sys.stdout = orig_stdout
class CommandLineTests(unittest.TestCase):
"""Test some aspects of compileall's CLI."""
def setUp(self):
self.addCleanup(self._cleanup)
self.directory = tempfile.mkdtemp()
self.pkgdir = os.path.join(self.directory, 'foo')
os.mkdir(self.pkgdir)
# Touch the __init__.py and a package module.
with open(os.path.join(self.pkgdir, '__init__.py'), 'w'):
pass
with open(os.path.join(self.pkgdir, 'bar.py'), 'w'):
pass
sys.path.insert(0, self.directory)
def _cleanup(self):
support.rmtree(self.directory)
assert sys.path[0] == self.directory, 'Missing path'
del sys.path[0]
def test_pep3147_paths(self):
# Ensure that the default behavior of compileall's CLI is to create
# PEP 3147 pyc/pyo files.
retcode = subprocess.call(
(sys.executable, '-m', 'compileall', '-q', self.pkgdir))
self.assertEqual(retcode, 0)
# Verify the __pycache__ directory contents.
cachedir = os.path.join(self.pkgdir, '__pycache__')
self.assertTrue(os.path.exists(cachedir))
ext = ('pyc' if __debug__ else 'pyo')
expected = sorted(base.format(imp.get_tag(), ext) for base in
('__init__.{}.{}', 'bar.{}.{}'))
self.assertEqual(sorted(os.listdir(cachedir)), expected)
# Make sure there are no .pyc files in the source directory.
self.assertFalse([pyc_file for pyc_file in os.listdir(self.pkgdir)
if pyc_file.endswith(ext)])
def test_legacy_paths(self):
# Ensure that with the proper switch, compileall leaves legacy
# pyc/pyo files, and no __pycache__ directory.
retcode = subprocess.call(
(sys.executable, '-m', 'compileall', '-b', '-q', self.pkgdir))
self.assertEqual(retcode, 0)
# Verify the __pycache__ directory contents.
cachedir = os.path.join(self.pkgdir, '__pycache__')
self.assertFalse(os.path.exists(cachedir))
ext = ('pyc' if __debug__ else 'pyo')
expected = [base.format(ext) for base in ('__init__.{}', 'bar.{}')]
expected.extend(['__init__.py', 'bar.py'])
expected.sort()
self.assertEqual(sorted(os.listdir(self.pkgdir)), expected)
def test_main():
support.run_unittest(CompileallTests,
EncodingTest)
support.run_unittest(
CommandLineTests,
CompileallTests,
EncodingTest,
)
if __name__ == "__main__":
......
Lib/test/test_frozen.py
View file @ 4bd65740
......@@ -11,7 +11,7 @@ class FrozenTests(unittest.TestCase):
except ImportError as x:
self.fail("import __hello__ failed:" + str(x))
self.assertEqual(__hello__.initialized, True)
self.assertEqual(len(dir(__hello__)), 6, dir(__hello__))
self.assertEqual(len(dir(__hello__)), 7, dir(__hello__))
try:
import __phello__
......@@ -19,9 +19,9 @@ class FrozenTests(unittest.TestCase):
self.fail("import __phello__ failed:" + str(x))
self.assertEqual(__phello__.initialized, True)
if not "__phello__.spam" in sys.modules:
self.assertEqual(len(dir(__phello__)), 7, dir(__phello__))
else:
self.assertEqual(len(dir(__phello__)), 8, dir(__phello__))
else:
self.assertEqual(len(dir(__phello__)), 9, dir(__phello__))
self.assertEquals(__phello__.__path__, [__phello__.__name__])
try:
......@@ -29,8 +29,8 @@ class FrozenTests(unittest.TestCase):
except ImportError as x:
self.fail("import __phello__.spam failed:" + str(x))
self.assertEqual(__phello__.spam.initialized, True)
self.assertEqual(len(dir(__phello__.spam)), 6)
self.assertEqual(len(dir(__phello__)), 8)
self.assertEqual(len(dir(__phello__.spam)), 7)
self.assertEqual(len(dir(__phello__)), 9)
try:
import __phello__.foo
......
Lib/test/test_imp.py
View file @ 4bd65740
import imp
import os
import os.path
import shutil
import sys
import unittest
from test import support
......@@ -139,7 +140,8 @@ class ImportTests(unittest.TestCase):
mod = imp.load_source(temp_mod_name, temp_mod_name + '.py')
self.assertEqual(mod.a, 1)
mod = imp.load_compiled(temp_mod_name, temp_mod_name + '.pyc')
mod = imp.load_compiled(
temp_mod_name, imp.cache_from_source(temp_mod_name + '.py'))
self.assertEqual(mod.a, 1)
if not os.path.exists(test_package_name):
......@@ -184,11 +186,132 @@ class ReloadTests(unittest.TestCase):
imp.reload(marshal)
class PEP3147Tests(unittest.TestCase):
"""Tests of PEP 3147."""
tag = imp.get_tag()
def test_cache_from_source(self):
# Given the path to a .py file, return the path to its PEP 3147
# defined .pyc file (i.e. under __pycache__).
self.assertEqual(
imp.cache_from_source('/foo/bar/baz/qux.py', True),
'/foo/bar/baz/__pycache__/qux.{}.pyc'.format(self.tag))
def test_cache_from_source_optimized(self):
# Given the path to a .py file, return the path to its PEP 3147
# defined .pyo file (i.e. under __pycache__).
self.assertEqual(
imp.cache_from_source('/foo/bar/baz/qux.py', False),
'/foo/bar/baz/__pycache__/qux.{}.pyo'.format(self.tag))
def test_cache_from_source_cwd(self):
self.assertEqual(imp.cache_from_source('foo.py', True),
os.sep.join(('__pycache__',
'foo.{}.pyc'.format(self.tag))))
def test_cache_from_source_override(self):
# When debug_override is not None, it can be any true-ish or false-ish
# value.
self.assertEqual(
imp.cache_from_source('/foo/bar/baz.py', []),
'/foo/bar/__pycache__/baz.{}.pyo'.format(self.tag))
self.assertEqual(
imp.cache_from_source('/foo/bar/baz.py', [17]),
'/foo/bar/__pycache__/baz.{}.pyc'.format(self.tag))
# However if the bool-ishness can't be determined, the exception
# propagates.
class Bearish:
def __bool__(self): raise RuntimeError
self.assertRaises(
RuntimeError,
imp.cache_from_source, '/foo/bar/baz.py', Bearish())
@unittest.skipIf(os.altsep is None,
'test meaningful only where os.altsep is defined')
def test_altsep_cache_from_source(self):
# Windows path and PEP 3147.
self.assertEqual(
imp.cache_from_source('\\foo\\bar\\baz\\qux.py', True),
'\\foo\\bar\\baz\\__pycache__\\qux.{}.pyc'.format(self.tag))
@unittest.skipIf(os.altsep is None,
'test meaningful only where os.altsep is defined')
def test_altsep_and_sep_cache_from_source(self):
# Windows path and PEP 3147 where altsep is right of sep.
self.assertEqual(
imp.cache_from_source('\\foo\\bar/baz\\qux.py', True),
'\\foo\\bar/baz\\__pycache__\\qux.{}.pyc'.format(self.tag))
@unittest.skipIf(os.altsep is None,
'test meaningful only where os.altsep is defined')
def test_sep_altsep_and_sep_cache_from_source(self):
# Windows path and PEP 3147 where sep is right of altsep.
self.assertEqual(
imp.cache_from_source('\\foo\\bar\\baz/qux.py', True),
'\\foo\\bar\\baz/__pycache__/qux.{}.pyc'.format(self.tag))
def test_source_from_cache(self):
# Given the path to a PEP 3147 defined .pyc file, return the path to
# its source. This tests the good path.
self.assertEqual(imp.source_from_cache(
'/foo/bar/baz/__pycache__/qux.{}.pyc'.format(self.tag)),
'/foo/bar/baz/qux.py')
def test_source_from_cache_bad_path(self):
# When the path to a pyc file is not in PEP 3147 format, a ValueError
# is raised.
self.assertRaises(
ValueError, imp.source_from_cache, '/foo/bar/bazqux.pyc')
def test_source_from_cache_no_slash(self):
# No slashes at all in path -> ValueError
self.assertRaises(
ValueError, imp.source_from_cache, 'foo.cpython-32.pyc')
def test_source_from_cache_too_few_dots(self):
# Too few dots in final path component -> ValueError
self.assertRaises(
ValueError, imp.source_from_cache, '__pycache__/foo.pyc')
def test_source_from_cache_too_many_dots(self):
# Too many dots in final path component -> ValueError
self.assertRaises(
ValueError, imp.source_from_cache,
'__pycache__/foo.cpython-32.foo.pyc')
def test_source_from_cache_no__pycache__(self):
# Another problem with the path -> ValueError
self.assertRaises(
ValueError, imp.source_from_cache,
'/foo/bar/foo.cpython-32.foo.pyc')
def test_package___file__(self):
# Test that a package's __file__ points to the right source directory.
os.mkdir('pep3147')
sys.path.insert(0, os.curdir)
def cleanup():
if sys.path[0] == os.curdir:
del sys.path[0]
shutil.rmtree('pep3147')
self.addCleanup(cleanup)
# Touch the __init__.py file.
with open('pep3147/__init__.py', 'w'):
pass
m = __import__('pep3147')
# Ensure we load the pyc file.
support.forget('pep3147')
m = __import__('pep3147')
self.assertEqual(m.__file__,
os.sep.join(('.', 'pep3147', '__init__.py')))
def test_main():
tests = [
ImportTests,
PEP3147Tests,
ReloadTests,
]
]
try:
import _thread
except ImportError:
......
Lib/test/test_import.py
View file @ 4bd65740
This diff is collapsed.
Lib/test/test_pkg.py
View file @ 4bd65740
......@@ -196,14 +196,14 @@ class TestPkg(unittest.TestCase):
import t5
self.assertEqual(fixdir(dir(t5)),
['__doc__', '__file__', '__name__',
['__cached__', '__doc__', '__file__', '__name__',
'__package__', '__path__', 'foo', 'string', 't5'])
self.assertEqual(fixdir(dir(t5.foo)),
['__doc__', '__file__', '__name__', '__package__',
'string'])
['__cached__', '__doc__', '__file__', '__name__',
'__package__', 'string'])
self.assertEqual(fixdir(dir(t5.string)),
['__doc__', '__file__', '__name__','__package__',
'spam'])
['__cached__', '__doc__', '__file__', '__name__',
'__package__', 'spam'])
def test_6(self):
hier = [
......@@ -218,13 +218,13 @@ class TestPkg(unittest.TestCase):
import t6
self.assertEqual(fixdir(dir(t6)),
['__all__', '__doc__', '__file__',
['__all__', '__cached__', '__doc__', '__file__',
'__name__', '__package__', '__path__'])
s = """
import t6
from t6 import *
self.assertEqual(fixdir(dir(t6)),
['__all__', '__doc__', '__file__',
['__all__', '__cached__', '__doc__', '__file__',
'__name__', '__package__', '__path__',
'eggs', 'ham', 'spam'])
self.assertEqual(dir(), ['eggs', 'ham', 'self', 'spam', 't6'])
......@@ -252,18 +252,18 @@ class TestPkg(unittest.TestCase):
t7, sub, subsub = None, None, None
import t7 as tas
self.assertEqual(fixdir(dir(tas)),
['__doc__', '__file__', '__name__',
['__cached__', '__doc__', '__file__', '__name__',
'__package__', '__path__'])
self.assertFalse(t7)
from t7 import sub as subpar
self.assertEqual(fixdir(dir(subpar)),
['__doc__', '__file__', '__name__',
['__cached__', '__doc__', '__file__', '__name__',
'__package__', '__path__'])
self.assertFalse(t7)
self.assertFalse(sub)
from t7.sub import subsub as subsubsub
self.assertEqual(fixdir(dir(subsubsub)),
['__doc__', '__file__', '__name__',
['__cached__', '__doc__', '__file__', '__name__',
'__package__', '__path__', 'spam'])
self.assertFalse(t7)
self.assertFalse(sub)
......
Lib/test/test_pkgimport.py
View file @ 4bd65740
import os, sys, string, random, tempfile, unittest
import os
import sys
import shutil
import string
import random
import tempfile
import unittest
from imp import cache_from_source
from test.support import run_unittest
class TestImport(unittest.TestCase):
......@@ -26,22 +33,17 @@ class TestImport(unittest.TestCase):
self.module_path = os.path.join(self.package_dir, 'foo.py')
def tearDown(self):
for file in os.listdir(self.package_dir):
os.remove(os.path.join(self.package_dir, file))
os.rmdir(self.package_dir)
os.rmdir(self.test_dir)
shutil.rmtree(self.test_dir)
self.assertNotEqual(sys.path.count(self.test_dir), 0)
sys.path.remove(self.test_dir)
self.remove_modules()
def rewrite_file(self, contents):
for extension in "co":
compiled_path = self.module_path + extension
if os.path.exists(compiled_path):
os.remove(compiled_path)
f = open(self.module_path, 'w')
f.write(contents)
f.close()
compiled_path = cache_from_source(self.module_path)
if os.path.exists(compiled_path):
os.remove(compiled_path)
with open(self.module_path, 'w') as f:
f.write(contents)
def test_package_import__semantics(self):
......
Lib/test/test_pydoc.py
View file @ 4bd65740
......@@ -19,8 +19,7 @@ from test import pydoc_mod
if hasattr(pydoc_mod, "__loader__"):
del pydoc_mod.__loader__
expected_text_pattern = \
"""
expected_text_pattern = """
NAME
test.pydoc_mod - This is a test module for test_pydoc
......@@ -87,8 +86,7 @@ CREDITS
Nobody
""".strip()
expected_html_pattern = \
"""
expected_html_pattern = """
<table width="100%%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="#7799ee">
<td valign=bottom>&nbsp;<br>
......@@ -186,7 +184,7 @@ war</tt></dd></dl>
\x20\x20\x20\x20
<tr><td bgcolor="#7799ee"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
<td width="100%%">Nobody</td></tr></table>
""".strip()
""".strip() # ' <- emacs turd
# output pattern for missing module
......@@ -287,7 +285,8 @@ class PyDocDocTest(unittest.TestCase):
('i_am_not_here', 'i_am_not_here'),
('test.i_am_not_here_either', 'i_am_not_here_either'),
('test.i_am_not_here.neither_am_i', 'i_am_not_here.neither_am_i'),
('i_am_not_here.{}'.format(modname), 'i_am_not_here.{}'.format(modname)),
('i_am_not_here.{}'.format(modname),
'i_am_not_here.{}'.format(modname)),
('test.{}'.format(modname), modname),
)
......@@ -304,9 +303,8 @@ class PyDocDocTest(unittest.TestCase):
fullmodname = os.path.join(TESTFN, modname)
sourcefn = fullmodname + os.extsep + "py"
for importstring, expectedinmsg in testpairs:
f = open(sourcefn, 'w')
f.write("import {}\n".format(importstring))
f.close()
with open(sourcefn, 'w') as f:
f.write("import {}\n".format(importstring))
try:
result = run_pydoc(modname).decode("ascii")
finally:
......
Lib/test/test_runpy.py
View file @ 4bd65740
......@@ -5,9 +5,10 @@ import os.path
import sys
import re
import tempfile
from test.support import verbose, run_unittest, forget
from test.script_helper import (temp_dir, make_script, compile_script,
make_pkg, make_zip_script, make_zip_pkg)
import py_compile
from test.support import forget, make_legacy_pyc, run_unittest, verbose
from test.script_helper import (
make_pkg, make_script, make_zip_pkg, make_zip_script, temp_dir)
from runpy import _run_code, _run_module_code, run_module, run_path
......@@ -45,6 +46,7 @@ class RunModuleCodeTest(unittest.TestCase):
self.assertEqual(d["result"], self.expected_result)
self.assertIs(d["__name__"], None)
self.assertIs(d["__file__"], None)
self.assertIs(d["__cached__"], None)
self.assertIs(d["__loader__"], None)
self.assertIs(d["__package__"], None)
self.assertIs(d["run_argv0"], saved_argv0)
......@@ -73,6 +75,7 @@ class RunModuleCodeTest(unittest.TestCase):
self.assertTrue(d2["run_name_in_sys_modules"])
self.assertTrue(d2["module_in_sys_modules"])
self.assertIs(d2["__file__"], file)
self.assertIs(d2["__cached__"], None)
self.assertIs(d2["run_argv0"], file)
self.assertIs(d2["__loader__"], loader)
self.assertIs(d2["__package__"], package)
......@@ -170,6 +173,7 @@ class RunModuleTest(unittest.TestCase):
del d1 # Ensure __loader__ entry doesn't keep file open
__import__(mod_name)
os.remove(mod_fname)
make_legacy_pyc(mod_fname)
if verbose: print("Running from compiled:", mod_name)
d2 = run_module(mod_name) # Read from bytecode
self.assertIn("x", d2)
......@@ -192,6 +196,7 @@ class RunModuleTest(unittest.TestCase):
del d1 # Ensure __loader__ entry doesn't keep file open
__import__(mod_name)
os.remove(mod_fname)
make_legacy_pyc(mod_fname)
if verbose: print("Running from compiled:", pkg_name)
d2 = run_module(pkg_name) # Read from bytecode
self.assertIn("x", d2)
......@@ -246,6 +251,7 @@ from ..uncle.cousin import nephew
del d1 # Ensure __loader__ entry doesn't keep file open
__import__(mod_name)
os.remove(mod_fname)
make_legacy_pyc(mod_fname)
if verbose: print("Running from compiled:", mod_name)
d2 = run_module(mod_name, run_name=run_name) # Read from bytecode
self.assertIn("__package__", d2)
......@@ -313,6 +319,7 @@ argv0 = sys.argv[0]
result = run_path(script_name)
self.assertEqual(result["__name__"], expected_name)
self.assertEqual(result["__file__"], expected_file)
self.assertEqual(result["__cached__"], None)
self.assertIn("argv0", result)
self.assertEqual(result["argv0"], expected_argv0)
self.assertEqual(result["__package__"], expected_package)
......@@ -332,7 +339,7 @@ argv0 = sys.argv[0]
with temp_dir() as script_dir:
mod_name = 'script'
script_name = self._make_test_script(script_dir, mod_name)
compiled_name = compile_script(script_name)
compiled_name = py_compile.compile(script_name, doraise=True)
os.remove(script_name)
self._check_script(compiled_name, "<run_path>", compiled_name,
compiled_name, None)
......@@ -348,9 +355,10 @@ argv0 = sys.argv[0]
with temp_dir() as script_dir:
mod_name = '__main__'
script_name = self._make_test_script(script_dir, mod_name)
compiled_name = compile_script(script_name)
compiled_name = py_compile.compile(script_name, doraise=True)
os.remove(script_name)
self._check_script(script_dir, "<run_path>", compiled_name,
legacy_pyc = make_legacy_pyc(script_name)
self._check_script(script_dir, "<run_path>", legacy_pyc,
script_dir, '')
def test_directory_error(self):
......@@ -371,8 +379,9 @@ argv0 = sys.argv[0]
with temp_dir() as script_dir:
mod_name = '__main__'
script_name = self._make_test_script(script_dir, mod_name)
compiled_name = compile_script(script_name)
zip_name, fname = make_zip_script(script_dir, 'test_zip', compiled_name)
compiled_name = py_compile.compile(script_name, doraise=True)
zip_name, fname = make_zip_script(script_dir, 'test_zip',
compiled_name)
self._check_script(zip_name, "<run_path>", fname, zip_name, '')
def test_zipfile_error(self):
......
Lib/test/test_site.py
View file @ 4bd65740
......@@ -258,19 +258,38 @@ class ImportSideEffectTests(unittest.TestCase):
"""Restore sys.path"""
sys.path[:] = self.sys_path
def test_abs__file__(self):
# Make sure all imported modules have their __file__ attribute
# as an absolute path.
# Handled by abs__file__()
site.abs__file__()
for module in (sys, os, builtins):
try:
self.assertTrue(os.path.isabs(module.__file__), repr(module))
except AttributeError:
continue
# We could try everything in sys.modules; however, when regrtest.py
# runs something like test_frozen before test_site, then we will
# be testing things loaded *after* test_site did path normalization
def test_abs_paths(self):
# Make sure all imported modules have their __file__ and __cached__
# attributes as absolute paths. Arranging to put the Lib directory on
# PYTHONPATH would cause the os module to have a relative path for
# __file__ if abs_paths() does not get run. sys and builtins (the
# only other modules imported before site.py runs) do not have
# __file__ or __cached__ because they are built-in.
parent = os.path.relpath(os.path.dirname(os.__file__))
env = os.environ.copy()
env['PYTHONPATH'] = parent
command = 'import os; print(os.__file__, os.__cached__)'
# First, prove that with -S (no 'import site'), the paths are
# relative.
proc = subprocess.Popen([sys.executable, '-S', '-c', command],
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
self.assertEqual(proc.returncode, 0)
os__file__, os__cached__ = stdout.split()
self.assertFalse(os.path.isabs(os__file__))
self.assertFalse(os.path.isabs(os__cached__))
# Now, with 'import site', it works.
proc = subprocess.Popen([sys.executable, '-c', command],
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
self.assertEqual(proc.returncode, 0)
os__file__, os__cached__ = stdout.split()
self.assertTrue(os.path.isabs(os__file__))
self.assertTrue(os.path.isabs(os__cached__))
def test_no_duplicate_paths(self):
# No duplicate paths should exist in sys.path
......
Lib/test/test_zipfile.py
View file @ 4bd65740
......@@ -6,6 +6,7 @@ except ImportError:
import io
import os
import imp
import time
import shutil
import struct
......@@ -587,7 +588,13 @@ class PyZipFileTests(unittest.TestCase):
with zipfile.PyZipFile(TemporaryFile(), "w") as zipfp:
fn = __file__
if fn.endswith('.pyc') or fn.endswith('.pyo'):
fn = fn[:-1]
path_split = fn.split(os.sep)
if os.altsep is not None:
path_split.extend(fn.split(os.altsep))
if '__pycache__' in path_split:
fn = imp.source_from_cache(fn)
else:
fn = fn[:-1]
zipfp.writepy(fn)
......
Lib/test/test_zipimport.py
View file @ 4bd65740
......@@ -48,17 +48,14 @@ NOW = time.time()
test_pyc = make_pyc(test_co, NOW)
if __debug__:
pyc_ext = ".pyc"
else:
pyc_ext = ".pyo"
TESTMOD = "ziptestmodule"
TESTPACK = "ziptestpackage"
TESTPACK2 = "ziptestpackage2"
TEMP_ZIP = os.path.abspath("junk95142.zip")
pyc_file = imp.cache_from_source(TESTMOD + '.py')
pyc_ext = ('.pyc' if __debug__ else '.pyo')
class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
......@@ -83,14 +80,11 @@ class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
stuff = kw.get("stuff", None)
if stuff is not None:
# Prepend 'stuff' to the start of the zipfile
f = open(TEMP_ZIP, "rb")
data = f.read()
f.close()
f = open(TEMP_ZIP, "wb")
f.write(stuff)
f.write(data)
f.close()
with open(TEMP_ZIP, "rb") as f:
data = f.read()
with open(TEMP_ZIP, "wb") as f:
f.write(stuff)
f.write(data)
sys.path.insert(0, TEMP_ZIP)
......@@ -180,8 +174,9 @@ class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
def testBadMTime(self):
badtime_pyc = bytearray(test_pyc)
badtime_pyc[7] ^= 0x02 # flip the second bit -- not the first as that one
# isn't stored in the .py's mtime in the zip archive.
# flip the second bit -- not the first as that one isn't stored in the
# .py's mtime in the zip archive.
badtime_pyc[7] ^= 0x02
files = {TESTMOD + ".py": (NOW, test_src),
TESTMOD + pyc_ext: (NOW, badtime_pyc)}
self.doTest(".py", files, TESTMOD)
......@@ -232,7 +227,8 @@ class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
self.assertEquals(zi.get_source(TESTPACK), None)
self.assertEquals(zi.get_source(mod_path), None)
self.assertEquals(zi.get_filename(mod_path), mod.__file__)
# To pass in the module name instead of the path, we must use the right importer
# To pass in the module name instead of the path, we must use the
# right importer
loader = mod.__loader__
self.assertEquals(loader.get_source(mod_name), None)
self.assertEquals(loader.get_filename(mod_name), mod.__file__)
......@@ -266,8 +262,10 @@ class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
mod = zi.load_module(TESTPACK2)
self.assertEquals(zi.get_filename(TESTPACK2), mod.__file__)
self.assertEquals(zi.is_package(TESTPACK2 + os.sep + '__init__'), False)
self.assertEquals(zi.is_package(TESTPACK2 + os.sep + TESTMOD), False)
self.assertEquals(
zi.is_package(TESTPACK2 + os.sep + '__init__'), False)
self.assertEquals(
zi.is_package(TESTPACK2 + os.sep + TESTMOD), False)
mod_path = TESTPACK2 + os.sep + TESTMOD
mod_name = module_path_to_dotted_name(mod_path)
......@@ -276,7 +274,8 @@ class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
self.assertEquals(zi.get_source(TESTPACK2), None)
self.assertEquals(zi.get_source(mod_path), None)
self.assertEquals(zi.get_filename(mod_path), mod.__file__)
# To pass in the module name instead of the path, we must use the right importer
# To pass in the module name instead of the path, we must use the
# right importer
loader = mod.__loader__
self.assertEquals(loader.get_source(mod_name), None)
self.assertEquals(loader.get_filename(mod_name), mod.__file__)
......
Lib/zipfile.py
View file @ 4bd65740
......@@ -3,10 +3,17 @@ Read and write ZIP files.
XXX references to utf-8 need further investigation.
"""
import struct, os, time, sys, shutil
import binascii, io, stat
import io
import os
import re
import imp
import sys
import time
import stat
import shutil
import struct
import binascii
try:
import zlib # We may need its compression method
......@@ -1303,22 +1310,42 @@ class PyZipFile(ZipFile):
file_py = pathname + ".py"
file_pyc = pathname + ".pyc"
file_pyo = pathname + ".pyo"
if os.path.isfile(file_pyo) and \
os.stat(file_pyo).st_mtime >= os.stat(file_py).st_mtime:
fname = file_pyo # Use .pyo file
elif not os.path.isfile(file_pyc) or \
os.stat(file_pyc).st_mtime < os.stat(file_py).st_mtime:
pycache_pyc = imp.cache_from_source(file_py, True)
pycache_pyo = imp.cache_from_source(file_py, False)
if (os.path.isfile(file_pyo) and
os.stat(file_pyo).st_mtime >= os.stat(file_py).st_mtime):
# Use .pyo file.
arcname = fname = file_pyo
elif (os.path.isfile(file_pyc) and
os.stat(file_pyc).st_mtime >= os.stat(file_py).st_mtime):
# Use .pyc file.
arcname = fname = file_pyc
elif (os.path.isfile(pycache_pyc) and
os.stat(pycache_pyc).st_mtime >= os.stat(file_py).st_mtime):
# Use the __pycache__/*.pyc file, but write it to the legacy pyc
# file name in the archive.
fname = pycache_pyc
arcname = file_pyc
elif (os.path.isfile(pycache_pyo) and
os.stat(pycache_pyo).st_mtime >= os.stat(file_py).st_mtime):
# Use the __pycache__/*.pyo file, but write it to the legacy pyo
# file name in the archive.
fname = pycache_pyo
arcname = file_pyo
else:
# Compile py into PEP 3147 pyc file.
import py_compile
if self.debug:
print("Compiling", file_py)
try:
py_compile.compile(file_py, file_pyc, None, True)
except py_compile.PyCompileError as err:
py_compile.compile(file_py, doraise=True)
except py_compile.PyCompileError as error:
print(err.msg)
fname = file_pyc
else:
fname = file_pyc
archivename = os.path.split(fname)[1]
fname = file_py
else:
fname = (pycache_pyc if __debug__ else pycache_pyo)
arcname = (file_pyc if __debug__ else file_pyo)
archivename = os.path.split(arcname)[1]
if basename:
archivename = "%s/%s" % (basename, archivename)
return (fname, archivename)
......
Makefile.pre.in
View file @ 4bd65740
......@@ -1161,6 +1161,7 @@ TAGS::
# files, which clobber removes as well
pycremoval:
find $(srcdir) -name '*.py[co]' -exec rm -f {} ';'
find $(srcdir) -name '__pycache__' | xargs rmdir
rmtestturds:
-rm -f *BAD *GOOD *SKIPPED
......
Python/import.c
View file @ 4bd65740
This diff is collapsed.
Python/pythonrun.c
View file @ 4bd65740
......@@ -1155,6 +1155,8 @@ PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit,
Py_DECREF(f);
return -1;
}
if (PyDict_SetItemString(d, "__cached__", Py_None) < 0)
return -1;
set_file_name = 1;
Py_DECREF(f);
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7