Commit 2bcb14ff authored by Robert Bradshaw's avatar Robert Bradshaw

Merge docs repo

parents 8128cb65 19ca6b53
syntax: glob
*.pyc
*~
.*.swp
syntax: regexp
^build/
^_build/
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d build/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html web htmlhelp latex changes linkcheck
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " web to make files usable by Sphinx.web"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " changes to make an overview over all changed/added/deprecated items"
@echo " linkcheck to check all external links for integrity"
clean:
-rm -rf build/*
html:
mkdir -p build/html build/doctrees
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) build/html
@echo
@echo "Build finished. The HTML pages are in build/html."
web:
mkdir -p build/web build/doctrees
$(SPHINXBUILD) -b web $(ALLSPHINXOPTS) build/web
@echo
@echo "Build finished; now you can run"
@echo " python -m sphinx.web build/web"
@echo "to start the server."
htmlhelp:
mkdir -p build/htmlhelp build/doctrees
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) build/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in build/htmlhelp."
latex:
mkdir -p build/latex build/doctrees
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) build/latex
@echo
@echo "Build finished; the LaTeX files are in build/latex."
@echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \
"run these through (pdf)latex."
changes:
mkdir -p build/changes build/doctrees
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) build/changes
@echo
@echo "The overview file is in build/changes."
linkcheck:
mkdir -p build/linkcheck build/doctrees
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) build/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in build/linkcheck/output.txt."
Cython's entire documentation suite is currently being overhauled.
For the time being, I'll use this page to post notes.
The previous Cython documentation files are hosted at http://hg.cython.org/cython-docs
Notes
=======
1) Some css work should definately be done.
2) Use local 'top-of-page' contents rather than the sidebar, imo.
3) Provide a link from each (sub)section to the TOC of the page.
4) Fix cython highlighter for cdef blocks
{% extends "!layout.html" %}
{% block footer %}
{{ super() }}
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-6139100-3");
pageTracker._trackPageview();
} catch(err) {}</script>
{% endblock %}
# -*- coding: utf-8 -*-
#
# Cython documentation build configuration file, created by
# sphinx-quickstart on Fri Apr 25 12:49:32 2008.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# The contents of this file are pickled, so don't put values in the namespace
# that aren't pickleable (module imports are okay, they're removed automatically).
#
# All configuration values have a default value; values that are commented out
# serve to show the default value.
import sys
# If your extensions are in another directory, add it here.
sys.path.append('sphinxext')
# Import support for ipython console session syntax highlighting (lives
# in the sphinxext directory defined above)
import ipython_console_highlighting
# General configuration
# ---------------------
# Use cython as the default syntax highlighting language, as python is a subset
# this does the right thing
highlight_language = 'cython'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['ipython_console_highlighting', 'cython_highlighting', 'sphinx.ext.pngmath', 'sphinx.ext.todo']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
exclude_patterns = ['py*', 'build']
# General substitutions.
project = 'Cython'
copyright = '2011, Stefan Behnel, Robert Bradshaw, Dag Sverre Seljebotn, Greg Ewing, William Stein, Gabriel Gellner, et al.'
# The default replacements for |version| and |release|, also used in various
# other places throughout the built documents.
#
# The short X.Y version.
version = '0.15'
# The full version, including alpha/beta/rc tags.
release = '0.15pre'
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
today_fmt = '%B %d, %Y'
# List of documents that shouldn't be included in the build.
#unused_docs = []
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# Options for HTML output
# -----------------------
# suffix for generated files
html_file_suffix = '.html'
# The style sheet to use for HTML and HTML Help pages. A file of that name
# must exist either in Sphinx' static/ path, or in one of the custom paths
# given in html_static_path.
html_style = 'default.css'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
html_last_updated_fmt = '%b %d, %Y'
# Include the Cython logo in the sidebar
html_logo = '_static/cython-logo-light.png'
# used a favicon!
html_favicon = '_static/favicon.ico'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Content template for the index page.
#html_index = ''
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
html_use_modindex = False
# Don't generate and index
html_use_index = False
# If true, the reST sources are included in the HTML build as _sources/<name>.
#html_copy_source = True
# Output file base name for HTML help builder.
htmlhelp_basename = 'Cythondoc'
# Options for LaTeX output
# ------------------------
# The paper size ('letter' or 'a4').
#latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
#latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, document class [howto/manual]).
#_stdauthor = r'Greg Ewig\\ Gabriel Gellner, editor'
_stdauthor = r'Stefan Behnel, Robert Bradshaw, William Stein\\ Gary Furnish, Dag Seljebotn, Greg Ewing\\ Gabriel Gellner, editor'
latex_documents = [
('src/reference/index', 'reference.tex',
'Cython Reference Guide', _stdauthor, 'manual'),
('src/tutorial/index', 'tutorial.tex',
'Cython Tutorial', _stdauthor, 'manual')
]
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_use_modindex = True
# todo
todo_include_todos = True
def fib(n):
"""Print the Fibonacci series up to n."""
a, b = 0, 1
while b < n:
print b,
a, b = b, a + b
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("fib", ["fib.pyx"])]
)
import math
def great_circle(lon1, lat1, lon2, lat2):
radius = 3956 # miles
x = math.pi/180.0
a = (90.0 - lat1)*x
b = (90.0 - lat2)*x
theta = (lon2 - lon1)*x
c = math.acos(math.cos(a)*math.cos(b) + math.sin(a)*math.sin(b)*math.cos(theta))
return radius*c
import math
def great_circle(double lon1, double lat1, double lon2, double lat2):
cdef double radius = 3956 # miles
cdef double x = math.pi/180.0
cdef double a, b, theta, c
a = (90.0 - lat1)*x
b = (90.0 - lat2)*x
theta = (lon2 - lon1)*x
c = math.acos(math.cos(a)*math.cos(b) + math.sin(a)*math.sin(b)*math.cos(theta))
return radius*c
import math
def great_circle(lon1, lat1, lon2, lat2):
radius = 3956 # miles
x = math.pi/180.0
a = (90.0 - lat1)*x
b = (90.0 - lat2)*x
theta = (lon2 - lon1)*x
c = math.acos(math.cos(a)*math.cos(b) + math.sin(a)*math.sin(b)*math.cos(theta))
return radius*c
def primes(kmax):
result = []
if kmax > 1000:
kmax = 1000
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
result.append(n)
n = n + 1
return result
def primes(int kmax):
cdef int n, k, i
cdef int p[1000]
result = []
if kmax > 1000:
kmax = 1000
k = 0
n = 2
while k < kmax:
i = 0
while i < k and n % p[i] != 0:
i = i + 1
if i == k:
p[k] = n
k = k + 1
result.append(n)
n = n + 1
return result
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("primes", ["primes.pyx"])]
)
Welcome to Cython's Documentation
=================================
.. toctree::
:maxdepth: 2
src/quickstart/index
src/tutorial/index
src/userguide/index
src/reference/index
import re
from pygments.lexer import Lexer, RegexLexer, ExtendedRegexLexer, \
LexerContext, include, combined, do_insertions, bygroups, using
from pygments.token import Error, Text, \
Comment, Operator, Keyword, Name, String, Number, Generic, Punctuation
from pygments.util import get_bool_opt, get_list_opt, shebang_matches
from pygments import unistring as uni
from sphinx import highlighting
line_re = re.compile('.*?\n')
class CythonLexer(RegexLexer):
"""
For `Cython <http://cython.org>`_ source code.
"""
name = 'Cython'
aliases = ['cython', 'pyx']
filenames = ['*.pyx', '*.pxd', '*.pxi']
mimetypes = ['text/x-cython', 'application/x-cython']
tokens = {
'root': [
(r'\n', Text),
(r'^(\s*)("""(?:.|\n)*?""")', bygroups(Text, String.Doc)),
(r"^(\s*)('''(?:.|\n)*?''')", bygroups(Text, String.Doc)),
(r'[^\S\n]+', Text),
(r'#.*$', Comment),
(r'[]{}:(),;[]', Punctuation),
(r'\\\n', Text),
(r'\\', Text),
(r'(in|is|and|or|not)\b', Operator.Word),
(r'(<)([a-zA-Z0-9.?]+)(>)',
bygroups(Punctuation, Keyword.Type, Punctuation)),
(r'!=|==|<<|>>|[-~+/*%=<>&^|.?]', Operator),
(r'(from)(\d+)(<=)(\s+)(<)(\d+)(:)',
bygroups(Keyword, Number.Integer, Operator, Name, Operator,
Name, Punctuation)),
include('keywords'),
(r'(def|property)(\s+)', bygroups(Keyword, Text), 'funcname'),
(r'(cp?def)(\s+)', bygroups(Keyword, Text), 'cdef'),
(r'(class|struct)(\s+)', bygroups(Keyword, Text), 'classname'),
(r'(from)(\s+)', bygroups(Keyword, Text), 'fromimport'),
(r'(c?import)(\s+)', bygroups(Keyword, Text), 'import'),
include('builtins'),
include('backtick'),
('(?:[rR]|[uU][rR]|[rR][uU])"""', String, 'tdqs'),
("(?:[rR]|[uU][rR]|[rR][uU])'''", String, 'tsqs'),
('(?:[rR]|[uU][rR]|[rR][uU])"', String, 'dqs'),
("(?:[rR]|[uU][rR]|[rR][uU])'", String, 'sqs'),
('[uU]?"""', String, combined('stringescape', 'tdqs')),
("[uU]?'''", String, combined('stringescape', 'tsqs')),
('[uU]?"', String, combined('stringescape', 'dqs')),
("[uU]?'", String, combined('stringescape', 'sqs')),
include('name'),
include('numbers'),
],
'keywords': [
(r'(assert|break|by|continue|ctypedef|del|elif|else|except\??|exec|'
r'finally|for|gil|global|if|include|lambda|nogil|pass|print|raise|'
r'return|try|while|yield|as|with)\b', Keyword),
(r'(DEF|IF|ELIF|ELSE)\b', Comment.Preproc),
],
'builtins': [
(r'(?<!\.)(__import__|abs|all|any|apply|basestring|bin|bool|buffer|'
r'bytearray|bytes|callable|chr|classmethod|cmp|coerce|compile|'
r'complex|delattr|dict|dir|divmod|enumerate|eval|execfile|exit|'
r'file|filter|float|frozenset|getattr|globals|hasattr|hash|hex|id|'
r'input|int|intern|isinstance|issubclass|iter|len|list|locals|'
r'long|map|max|min|next|object|oct|open|ord|pow|property|range|'
r'raw_input|reduce|reload|repr|reversed|round|set|setattr|slice|'
r'sorted|staticmethod|str|sum|super|tuple|type|unichr|unicode|'
r'vars|xrange|zip)\b', Name.Builtin),
(r'(?<!\.)(self|None|Ellipsis|NotImplemented|False|True|NULL'
r')\b', Name.Builtin.Pseudo),
(r'(?<!\.)(ArithmeticError|AssertionError|AttributeError|'
r'BaseException|DeprecationWarning|EOFError|EnvironmentError|'
r'Exception|FloatingPointError|FutureWarning|GeneratorExit|IOError|'
r'ImportError|ImportWarning|IndentationError|IndexError|KeyError|'
r'KeyboardInterrupt|LookupError|MemoryError|NameError|'
r'NotImplemented|NotImplementedError|OSError|OverflowError|'
r'OverflowWarning|PendingDeprecationWarning|ReferenceError|'
r'RuntimeError|RuntimeWarning|StandardError|StopIteration|'
r'SyntaxError|SyntaxWarning|SystemError|SystemExit|TabError|'
r'TypeError|UnboundLocalError|UnicodeDecodeError|'
r'UnicodeEncodeError|UnicodeError|UnicodeTranslateError|'
r'UnicodeWarning|UserWarning|ValueError|Warning|ZeroDivisionError'
r')\b', Name.Exception),
],
'numbers': [
(r'(\d+\.?\d*|\d*\.\d+)([eE][+-]?[0-9]+)?', Number.Float),
(r'0\d+', Number.Oct),
(r'0[xX][a-fA-F0-9]+', Number.Hex),
(r'\d+L', Number.Integer.Long),
(r'\d+', Number.Integer)
],
'backtick': [
('`.*?`', String.Backtick),
],
'name': [
(r'@[a-zA-Z0-9_]+', Name.Decorator),
('[a-zA-Z_][a-zA-Z0-9_]*', Name),
],
'funcname': [
('[a-zA-Z_][a-zA-Z0-9_]*', Name.Function, '#pop')
],
'cdef': [
(r'(public|readonly|extern|api|inline)\b', Keyword.Reserved),
(r'(struct|enum|union|class)\b', Keyword),
(r'([a-zA-Z_][a-zA-Z0-9_]*)(\s*)(?=[(:#=]|$)',
bygroups(Name.Function, Text), '#pop'),
(r'([a-zA-Z_][a-zA-Z0-9_]*)(\s*)(,)',
bygroups(Name.Function, Text, Punctuation)),
(r'from\b', Keyword, '#pop'),
(r'as\b', Keyword),
(r':', Punctuation, '#pop'),
(r'(?=["\'])', Text, '#pop'),
(r'[a-zA-Z_][a-zA-Z0-9_]*', Keyword.Type),
(r'.', Text),
],
'classname': [
('[a-zA-Z_][a-zA-Z0-9_]*', Name.Class, '#pop')
],
'import': [
(r'(\s+)(as)(\s+)', bygroups(Text, Keyword, Text)),
(r'[a-zA-Z_][a-zA-Z0-9_.]*', Name.Namespace),
(r'(\s*)(,)(\s*)', bygroups(Text, Operator, Text)),
(r'', Text, '#pop') # all else: go back
],
'fromimport': [
(r'(\s+)(c?import)\b', bygroups(Text, Keyword), '#pop'),
(r'[a-zA-Z_.][a-zA-Z0-9_.]*', Name.Namespace),
# ``cdef foo from "header"``, or ``for foo from 0 < i < 10``
(r'', Text, '#pop'),
],
'stringescape': [
(r'\\([\\abfnrtv"\']|\n|N{.*?}|u[a-fA-F0-9]{4}|'
r'U[a-fA-F0-9]{8}|x[a-fA-F0-9]{2}|[0-7]{1,3})', String.Escape)
],
'strings': [
(r'%(\([a-zA-Z0-9]+\))?[-#0 +]*([0-9]+|[*])?(\.([0-9]+|[*]))?'
'[hlL]?[diouxXeEfFgGcrs%]', String.Interpol),
(r'[^\\\'"%\n]+', String),
# quotes, percents and backslashes must be parsed one at a time
(r'[\'"\\]', String),
# unhandled string formatting sign
(r'%', String)
# newlines are an error (use "nl" state)
],
'nl': [
(r'\n', String)
],
'dqs': [
(r'"', String, '#pop'),
(r'\\\\|\\"|\\\n', String.Escape), # included here again for raw strings
include('strings')
],
'sqs': [
(r"'", String, '#pop'),
(r"\\\\|\\'|\\\n", String.Escape), # included here again for raw strings
include('strings')
],
'tdqs': [
(r'"""', String, '#pop'),
include('strings'),
include('nl')
],
'tsqs': [
(r"'''", String, '#pop'),
include('strings'),
include('nl')
],
}
##TODO: fix this, as shebang lines don't make sense for cython.
def analyse_text(text):
return shebang_matches(text, r'pythonw?(2\.\d)?')
highlighting.lexers['cython'] = CythonLexer()
from pygments.lexer import Lexer, do_insertions
from pygments.lexers.agile import PythonConsoleLexer, PythonLexer, \
PythonTracebackLexer
from pygments.token import Comment, Generic
from sphinx import highlighting
import re
line_re = re.compile('.*?\n')
class IPythonConsoleLexer(Lexer):
"""
For IPython console output or doctests, such as:
Tracebacks are not currently supported.
.. sourcecode:: ipython
In [1]: a = 'foo'
In [2]: a
Out[2]: 'foo'
In [3]: print a
foo
In [4]: 1 / 0
"""
name = 'IPython console session'
aliases = ['ipython']
mimetypes = ['text/x-ipython-console']
input_prompt = re.compile("(In \[[0-9]+\]: )|( \.\.\.+:)")
output_prompt = re.compile("(Out\[[0-9]+\]: )|( \.\.\.+:)")
continue_prompt = re.compile(" \.\.\.+:")
tb_start = re.compile("\-+")
def get_tokens_unprocessed(self, text):
pylexer = PythonLexer(**self.options)
tblexer = PythonTracebackLexer(**self.options)
curcode = ''
insertions = []
for match in line_re.finditer(text):
line = match.group()
input_prompt = self.input_prompt.match(line)
continue_prompt = self.continue_prompt.match(line.rstrip())
output_prompt = self.output_prompt.match(line)
if line.startswith("#"):
insertions.append((len(curcode),
[(0, Comment, line)]))
elif input_prompt is not None:
insertions.append((len(curcode),
[(0, Generic.Prompt, input_prompt.group())]))
curcode += line[input_prompt.end():]
elif continue_prompt is not None:
insertions.append((len(curcode),
[(0, Generic.Prompt, continue_prompt.group())]))
curcode += line[continue_prompt.end():]
elif output_prompt is not None:
insertions.append((len(curcode),
[(0, Generic.Output, output_prompt.group())]))
curcode += line[output_prompt.end():]
else:
if curcode:
for item in do_insertions(insertions,
pylexer.get_tokens_unprocessed(curcode)):
yield item
curcode = ''
insertions = []
yield match.start(), Generic.Output, line
if curcode:
for item in do_insertions(insertions,
pylexer.get_tokens_unprocessed(curcode)):
yield item
highlighting.lexers['ipython'] = IPythonConsoleLexer()
Building Cython code
====================
Cython code must, unlike Python, be compiled. This happens in two stages:
- A ``.pyx`` file is compiled by Cython to a ``.c`` file, containing
the code of a Python extension module
- The ``.c`` file is compiled by a C compiler to
a ``.so`` file (or ``.pyd`` on Windows) which can be
``import``-ed directly into a Python session.
There are several ways to build Cython code:
- Write a distutils ``setup.py``.
- Use ``pyximport``, importing Cython ``.pyx`` files as if they
were ``.py`` files (using distutils to compile and build the background).
- Run the ``cython`` command-line utility manually to produce the ``.c`` file
from the ``.pyx`` file, then manually compiling the ``.c`` file into a shared
object library or ``.dll`` suitable for import from Python.
(This is mostly for debugging and experimentation.)
- Use the [Sage]_ notebook which allows Cython code inline.
Currently, distutils is the most common way Cython files are built and distributed. The other methods are described in more detail in the :ref:`compilation` section of the reference manual.
Building a Cython module using distutils
----------------------------------------
Imagine a simple "hello world" script in a file ``hello.pyx``::
def say_hello_to(name):
print("Hello %s!" % name)
The following could be a corresponding ``setup.py`` script::
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("hello", ["hello.pyx"])]
setup(
name = 'Hello world app',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
To build, run ``python setup.py build_ext --inplace``. Then simply
start a Python session and do ``from hello import say_hello_to`` and
use the imported function as you see fit.
.. figure:: sage.png
The Sage notebook allows transparently editing and compiling Cython
code simply by typing ``%cython`` at the top of a cell and evaluate
it. Variables and functions defined in a Cython cell imported into
the running session.
.. [Sage] W. Stein et al., Sage Mathematics Software, http://sagemath.org
Faster code via static typing
=============================
Cython is a Python compiler. This means that it can compile normal
Python code without changes (with a few obvious exceptions of some as-yet
unsupported language features). However, for performance critical
code, it is often helpful to add static type declarations, as they
will allow Cython to step out of the dynamic nature of the Python code
and generate simpler and faster C code - sometimes faster by orders of
magnitude.
It must be noted, however, that type declarations can make the source
code more verbose and thus less readable. It is therefore discouraged
to use them without good reason, such as where benchmarks prove
that they really make the code substantially faster in a performance
critical section. Typically a few types in the right spots go a long way.
All C types are available for type declarations: integer and floating
point types, complex numbers, structs, unions and pointer types.
Cython can automatically and correctly convert between the types on
assignment. This also includes Python's arbitrary size integer types,
where value overflows on conversion to a C type will raise a Python
``OverflowError`` at runtime. (It does not, however, check for overflow
when doing arithmetic.) The generated C code will handle the
platform dependent sizes of C types correctly and safely in this case.
Types are declared via the cdef keyword.
Typing Variables
----------------
Consider the following pure Python code::
def f(x):
return x**2-x
def integrate_f(a, b, N):
s = 0
dx = (b-a)/N
for i in range(N):
s += f(a+i*dx)
return s * dx
Simply compiling this in Cython merely gives a 35% speedup. This is
better than nothing, but adding some static types can make a much larger
difference.
With additional type declarations, this might look like::
def f(double x):
return x**2-x
def integrate_f(double a, double b, int N):
cdef int i
cdef double s, dx
s = 0
dx = (b-a)/N
for i in range(N):
s += f(a+i*dx)
return s * dx
Since the iterator variable ``i`` is typed with C semantics, the for-loop will be compiled
to pure C code. Typing ``a``, ``s`` and ``dx`` is important as they are involved
in arithmetic withing the for-loop; typing ``b`` and ``N`` makes less of a
difference, but in this case it is not much extra work to be
consistent and type the entire function.
This results in a 4 times speedup over the pure Python version.
Typing Functions
----------------
Python function calls can be expensive -- in Cython doubly so because
one might need to convert to and from Python objects to do the call.
In our example above, the argument is assumed to be a C double both inside f()
and in the call to it, yet a Python ``float`` object must be constructed around the
argument in order to pass it.
Therefore Cython provides a syntax for declaring a C-style function,
the cdef keyword::
cdef double f(double) except? -2:
return x**2-x
Some form of except-modifier should usually be added, otherwise Cython
will not be able to propagate exceptions raised in the function (or a
function it calls). The ``except? -2`` means that an error will be checked
for if ``-2`` is returned (though the ``?`` indicates that ``-2`` may also
be used as a valid return value).
Alternatively, the slower ``except *`` is always
safe. An except clause can be left out if the function returns a Python
object or if it is guaranteed that an exception will not be raised
within the function call.
A side-effect of cdef is that the function is no longer available from
Python-space, as Python wouldn't know how to call it. Using the
``cpdef`` keyword instead of cdef, a Python wrapper is also created,
so that the function is available both from Cython (fast, passing
typed values directly) and from Python (wrapping values in Python
objects).
Note also that it is no longer possible to change ``f`` at runtime.
Speedup: 150 times over pure Python.
Determining where to add types
------------------------------
Because static typing is often the key to large speed gains, beginners
often have a tendency to type everything in sight. This cuts down on both
readability and flexibility. On the other hand, it is easy to kill
performance by forgetting to type a critical loop variable. Two essential
tools to help with this task are profiling and annotation.
Profiling should be the first step of any optimization effort, and can
tell you where you are spending your time. Cython's annotation can then
tell you why your code is taking time.
Using the ``-a`` switch to the ``cython`` command line program (or
following a link from the Sage notebook) results in an HTML report
of Cython code interleaved with the generated C code. Lines are
colored according to the level of "typedness" -- white lines
translates to pure C without any Python API calls. This report
is invaluable when optimizing a function for speed.
.. figure:: htmlreport.png
from time import time
from math import sin
cdef double first_time = 0
def timeit(f, label):
global first_time
t = time()
f(1.0, 2.0, 10**7)
cdef double elapsed = time() - t
if first_time == 0:
first_time = elapsed
print label, elapsed, (100*elapsed/first_time), '% or', first_time/elapsed, 'x'
# Pure Python
py_funcs = {'sin': sin}
py_funcs.update(__builtins__.__dict__)
exec """
def f(x):
return x**2-x
def integrate_f(a, b, N):
s = 0
dx = (b-a)/N
for i in range(N):
s += f(a+i*dx)
return s * dx
""" in py_funcs
timeit(py_funcs['integrate_f'], "Python")
# Just compiled
def f0(x):
return x**2-x
def integrate_f0(a, b, N):
s = 0
dx = (b-a)/N
for i in range(N):
s += f0(a+i*dx)
return s * dx
timeit(integrate_f0, "Cython")
# Typed vars
def f1(double x):
return x**2-x
def integrate_f1(double a, double b, int N):
cdef int i
cdef double s, dx
s = 0
dx = (b-a)/N
for i in range(N):
s += f1(a+i*dx)
return s * dx
timeit(integrate_f1, "Typed vars")
# Typed func
cdef double f2(double x) except? -2:
return x**2-x
def integrate_f2(double a, double b, int N):
cdef int i
cdef double s, dx
s = 0
dx = (b-a)/N
for i in range(N):
s += f2(a+i*dx)
return s * dx
timeit(integrate_f2, "Typed func")
Getting Started
===============
.. toctree::
:maxdepth: 2
overview
install
build
cythonize
Installing Cython
=================
Many scientific Python distributions, such as the Enthought Python
Distribution [EPD]_, Python(x,y) [Pythonxy]_, and Sage [Sage]_, bundle
Cython and no setup is needed. Note however that if your distribution
ships a version of Cython which is too old you can still use the
instructions below to update Cython. Everything in this tutorial
should work with Cython 0.11.2 and newer, unless a footnote says
otherwise.
Unlike most Python software, Cython requires a C compiler to be
present on the system. The details of getting a C compiler varies
according to the system used:
- **Linux** The GNU C Compiler (gcc) is usually present, or easily
available through the package system. On Ubuntu or Debian, for
instance, the command ``sudo apt-get install build-essential`` will
fetch everything you need.
- **Mac OS X** To retrieve gcc, one option is to install Apple's
XCode, which can be retrieved from the Mac OS X's install DVDs or
from http://developer.apple.com.
- **Windows** A popular option is to use the open source MinGW (a
Windows distribution of gcc). See the appendix for instructions for
setting up MinGW manually. EPD and Python(x,y) bundle MinGW, but
some of the configuration steps in the appendix might still be
necessary. Another option is to use Microsoft's Visual C. One must
then use the same version which the installed Python was compiled
with.
.. dagss tried other forms of ReST lists and they didn't look nice
.. with rst2latex.
The newest Cython release can always be downloaded from
http://cython.org. Unpack the tarball or zip file, enter the
directory, and then run::
python setup.py install
If you have Python setuptools set up on your system, you should be
able to fetch Cython from PyPI and install it using::
easy_install cython
For Windows there is also an executable installer available for
download.
.. [EPD] http://www.enthought.com/products/epd.php
.. [Pythonxy] http://www.pythonxy.com/
.. [Sage] W. Stein et al., Sage Mathematics Software, http://sagemath.org
Cython - an overview
====================
[Cython]_ is a programming language based on Python, with extra syntax
allowing for optional static type declarations. It aims to become a superset
of the [Python]_ language which gives it high-level, object-oriented,
functional, and dynamic programming. The source code gets translated
into optimized C/C++ code and compiled as Python extension modules.
This allows for both very fast program execution and tight integration
with external C libraries, while keeping up the high programmer
productivity for which the Python language is well known.
The primary Python execution environment is commonly referred to as
CPython, as it is written in C. Other major implementations use Java
(Jython [Jython]_), C# (IronPython [IronPython]_) and Python itself
(PyPy [PyPy]_). Written in C, CPython has been conducive to wrapping
many external libraries that interface through the C language. It
has, however, remained non trivial to write the necessary glue code in
C, especially for programmers who are more fluent in a high-level
language like Python than in a close-to-the-metal language like C.
Originally based on the well-known Pyrex [Pyrex]_, the Cython project has
approached this problem by means of a source code compiler that
translates Python code to equivalent C code. This code is executed
within the CPython runtime environment, but at the speed of compiled C
and with the ability to call directly into C libraries.
At the same time, it keeps the original interface of the Python
source code, which makes it directly usable from Python code. These
two-fold characteristics enable Cython's two major use cases:
extending the CPython interpreter with fast binary modules, and
interfacing Python code with external C libraries.
While Cython can compile (most) regular Python code, the generated C
code usually gains major (and sometime impressive) speed improvements
from optional static type declarations for both Python and C types.
These allow Cython to assign C semantics to parts of the code, and to
translate them into very efficient C code. Type declarations can
therefore be used for two purposes: for moving code sections from
dynamic Python semantics into static-and-fast C semantics, but also
for directly manipulating types defined in external libraries. Cython
thus merges the two worlds into a very broadly applicable programming
language.
.. [Cython] G. Ewing, R. W. Bradshaw, S. Behnel, D. S. Seljebotn et al.,
The Cython compiler, http://cython.org.
.. [IronPython] Jim Hugunin et al., http://www.codeplex.com/IronPython.
.. [Jython] J. Huginin, B. Warsaw, F. Bock, et al.,
Jython: Python for the Java platform, http://www.jython.org/
.. [PyPy] The PyPy Group, PyPy: a Python implementation written in Python,
http://codespeak.net/pypy.
.. [Pyrex] G. Ewing, Pyrex: C-Extensions for Python,
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/
.. [Python] G. van Rossum et al., The Python programming language,
http://python.org.
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d build/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html web htmlhelp latex changes linkcheck
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " web to make files usable by Sphinx.web"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " changes to make an overview over all changed/added/deprecated items"
@echo " linkcheck to check all external links for integrity"
clean:
-rm -rf build/*
html:
mkdir -p build/html build/doctrees
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) build/html
@echo
@echo "Build finished. The HTML pages are in build/html."
web:
mkdir -p build/web build/doctrees
$(SPHINXBUILD) -b web $(ALLSPHINXOPTS) build/web
@echo
@echo "Build finished; now you can run"
@echo " python -m sphinx.web build/web"
@echo "to start the server."
htmlhelp:
mkdir -p build/htmlhelp build/doctrees
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) build/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in build/htmlhelp."
latex:
mkdir -p build/latex build/doctrees
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) build/latex
@echo
@echo "Build finished; the LaTeX files are in build/latex."
@echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \
"run these through (pdf)latex."
changes:
mkdir -p build/changes build/doctrees
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) build/changes
@echo
@echo "The overview file is in build/changes."
linkcheck:
mkdir -p build/linkcheck build/doctrees
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) build/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in build/linkcheck/output.txt."
.. highlight:: cython
.. _compilation:
***********
Compilation
***********
* Cython code, unlike Python, must be compiled.
* This happens in two stages:
* A ``.pyx`` file is compiles by Cython to a ``.c`` file.
* The ``.c`` file is compiled by a C comiler to a ``.so`` file (or a ``.pyd`` file on Windows)
* The following sub-sections describe several ways to build your extension modules.
.. note:: The ``-a`` option
* Using the Cython compiler with the ``-a`` option will produce a really nice HTML file of the Cython generated ``.c`` code.
* Double clicking on the highlighted sections will expand the code to reveal what Cython has actually generated for you.
* This is very useful for understanding, optimizing or debugging your module.
=====================
From the Command Line
=====================
* Run the Cython compiler command with your options and list of ``.pyx`` files to generate::
$ cython -a yourmod.pyx
* This creates a ``yourmod.c`` file. (and the -a switch produces a generated html file)
* Compiling your ``.c`` files will vary depending on your operating system.
* Python documentation for writing extension modules should have some details for your system.
* Here we give an example on a Linux system::
$ gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python2.5 -o yourmod.so yourmod.c
* ``gcc`` will need to have paths to your included header files and paths to libraries you need to link with.
* A ``yourmod.so`` file is now in the same directory.
* Your module, ``yourmod`` is available for you to import as you normally would.
=========
Distutils
=========
* Ensure Distutils is installed in your system.
* The following assumes a Cython file to be compiled called *hello.pyx*.
* Create a ``setup.py`` script::
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("hello", ["hello.pyx"])]
setup(
name = ’Hello world app’,
cmdclass = {’build_ext’: build_ext},
ext_modules = ext_modules
)
* Run the command ``python setup.py build_ext --inplace`` in your system's command shell.
* Your done.. import your new extension module into your python shell or script as normal.
=====
SCons
=====
to be completed...
=========
Pyximport
=========
* For generating Cython code right in your pure python modulce::
>>> import pyximport; pyximport.install()
>>> import helloworld
Hello World
* Use for simple Cython builds only.
* No extra C libraries.
* No special build setup needed.
* Also has experimental compilation support for normal Python modules.
* Allows you to automatically run Cython on every ``.pyx`` and ``.py`` module that Python imports.
* This includes the standard library and installed packages.
* In the case that Cython fails to compile a Python module, *pyximport* will fall back to loading the source modules instead.
* The ``.py`` import mechanism is installed like this::
>>> pyximport.install(pyimport = True)
.. note:: Authors
Paul Prescod, Stefan Behnal
====
Sage
====
The Sage notebook allows transparently editing and
compiling Cython code simply by typing %cython at
the top of a cell and evaluate it. Variables and func-
tions defined in a Cython cell imported into the run-
ning session.
.. todo:: Provide a link to Sage docs
Compiler Directives
===================
TODO. See http://wiki.cython.org/enhancements/compilerdirectives
\ No newline at end of file
This diff is collapsed.
Reference Guide
===============
.. note::
.. todo::
Most of the **boldface** is to be changed to refs or other markup later.
Contents:
.. toctree::
:maxdepth: 2
compilation
language_basics
extension_types
interfacing_with_other_code
special_mention
limitations
directives
Indices and tables
------------------
.. toctree::
:maxdepth: 2
special_methods_table
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. highlight:: cython
.. _interfacing_with_other_code:
***************************
Interfacing with Other Code
***************************
==
C
==
===
C++
===
=======
Fortran
=======
=====
Numpy
=====
This diff is collapsed.
.. highlight:: cython
.. _limitations:
***********
Limitations
***********
.. highlight:: cython
.. _special_mention:
***************
Special Mention
***************
This diff is collapsed.
Appendix: Installing MinGW on Windows
=====================================
1. Download the MinGW installer from
http://www.mingw.org/wiki/HOWTO_Install_the_MinGW_GCC_Compiler_Suite.
(As of this
writing, the download link is a bit difficult to find; it's under
"About" in the menu on the left-hand side). You want the file
entitled "Automated MinGW Installer" (currently version 5.1.4).
2. Run it and install MinGW. Only the basic package is strictly
needed for Cython, although you might want to grab at least the
C++ compiler as well.
3. You need to set up Windows' "PATH" environment variable so that
includes e.g. "c:\\mingw\\bin" (if you installed MinGW to
"c:\\mingw"). The following web-page describes the procedure
in Windows XP (the Vista procedure is similar):
http://support.microsoft.com/kb/310519
4. Finally, tell Python to use MinGW as the default compiler
(otherwise it will try for Visual C). If Python is installed to
"c:\\Python26", create a file named
"c:\\Python26\\Lib\\distutils\\distutils.cfg" containing::
[build]
compiler = mingw32
The [WinInst]_ wiki page contains updated information about this
procedure. Any contributions towards making the Windows install
process smoother is welcomed; it is an unfortunate fact that none of
the regular Cython developers have convenient access to Windows.
.. [WinInst] http://wiki.cython.org/InstallingOnWindows
Caveats
=======
Since Cython mixes C and Python semantics, some things may be a bit
surprising or unintuitive. Work always goes on to make Cython more natural
for Python users, so this list may change in the future.
- ``10**-2 == 0``, instead of ``0.01`` like in Python.
- Given two typed ``int`` variables ``a`` and ``b``, ``a % b`` has the
same sign as the second argument (following Python semantics) rather then
having the same sign as the first (as in C). The C behavior can be
obtained, at some speed gain, by enabling the division directive.
(Versions prior to Cython 0.12. always followed C semantics.)
- Care is needed with unsigned types. ``cdef unsigned n = 10;
print(range(-n, n))`` will print an empty list, since ``-n`` wraps
around to a large positive integer prior to being passed to the
``range`` function.
- Python's ``float`` type actually wraps C ``double`` values, and
Python's ``int`` type wraps C ``long`` values.
Extension types (aka. cdef classes)
===================================
To support object-oriented programming, Cython supports writing normal
Python classes exactly as in Python::
class MathFunction(object):
def __init__(self, name, operator):
self.name = name
self.operator = operator
def __call__(self, *operands):
return self.operator(*operands)
Based on what Python calls a "built-in type", however, Cython supports
a second kind of class: *extension types*, sometimes referred to as
"cdef classes" due to the keywords used for their declaration. They
are somewhat restricted compared to Python classes, but are generally
more memory efficient and faster than generic Python classes. The
main difference is that they use a C struct to store their fields and methods
instead of a Python dict. This allows them to store arbitrary C types
in their fields without requiring a Python wrapper for them, and to
access fields and methods directly at the C level without passing
through a Python dictionary lookup.
Normal Python classes can inherit from cdef classes, but not the other
way around. Cython requires to know the complete inheritance
hierarchy in order to lay out their C structs, and restricts it to
single inheritance. Normal Python classes, on the other hand, can
inherit from any number of Python classes and extension types, both in
Cython code and pure Python code.
So far our integration example has not been very useful as it only
integrates a single hard-coded function. In order to remedy this,
without sacrificing speed, we will use a cdef class to represent a
function on floating point numbers::
cdef class Function:
cpdef double evaluate(self, double x) except *:
return 0
Like before, cpdef makes two versions of the method available; one
fast for use from Cython and one slower for use from Python. Then::
cdef class SinOfSquareFunction(Function):
cpdef double evaluate(self, double x) except *:
return sin(x**2)
Using this, we can now change our integration example::
def integrate(Function f, double a, double b, int N):
cdef int i
cdef double s, dx
if f is None:
raise ValueError("f cannot be None")
s = 0
dx = (b-a)/N
for i in range(N):
s += f.evaluate(a+i*dx)
return s * dx
print(integrate(SinOfSquareFunction(), 0, 1, 10000))
This is almost as fast as the previous code, however it is much more flexible
as the function to integrate can be changed. It is even possible to pass
in a new function defined in Python-space::
>>> import integrate
>>> class MyPolynomial(integrate.Function):
... def evaluate(self, x):
... return 2*x*x + 3*x - 10
...
>>> integrate(MyPolynomial(), 0, 1, 10000)
-7.8335833300000077
This is about 20 times slower, but still about 10 times faster than
the original Python-only integration code. This shows how large the
speed-ups can easily be when whole loops are moved from Python code
into a Cython module.
Some notes on our new implementation of ``evaluate``:
- The fast method dispatch here only works because ``evaluate`` was
declared in ``Function``. Had ``evaluate`` been introduced in
``SinOfSquareFunction``, the code would still work, but Cython
would have used the slower Python method dispatch mechanism
instead.
- In the same way, had the argument ``f`` not been typed, but only
been passed as a Python object, the slower Python dispatch would
be used.
- Since the argument is typed, we need to check whether it is
``None``. In Python, this would have resulted in an ``AttributeError``
when the ``evaluate`` method was looked up, but Cython would instead
try to access the (incompatible) internal structure of ``None`` as if
it were a ``Function``, leading to a crash or data corruption.
There is a *compiler directive* ``nonecheck`` which turns on checks
for this, at the cost of decreased speed. Here's how compiler directives
are used to dynamically switch on or off ``nonecheck``::
#cython: nonecheck=True
# ^^^ Turns on nonecheck globally
import cython
# Turn off nonecheck locally for the function
@cython.nonecheck(False)
def func():
cdef MyClass obj = None
try:
# Turn nonecheck on again for a block
with cython.nonecheck(True):
print obj.myfunc() # Raises exception
except AttributeError:
pass
print obj.myfunc() # Hope for a crash!
Attributes in cdef classes behave differently from attributes in regular classes:
- All attributes must be pre-declared at compile-time
- Attributes are by default only accessible from Cython (typed access)
- Properties can be declared to expose dynamic attributes to Python-space
::
cdef class WaveFunction(Function):
# Not available in Python-space:
cdef double offset
# Available in Python-space:
cdef public double freq
# Available in Python-space:
property period:
def __get__(self):
return 1.0 / self. freq
def __set__(self, value):
self. freq = 1.0 / value
<...>
This diff is collapsed.
{
'title': 'Cython Tutorial',
'paper_abstract': '''
Cython is a programming language based on Python with extra
syntax to provide static type declarations. This takes advantage of the
benefits of Python while allowing one to achieve the speed of C.
In this paper we describe the Cython language and show how it can
be used both to write optimized code and to interface with external
C libraries.
''',
'authors': [
{'first_names': 'Stefan',
'surname': 'Behnel',
'address': '',
'country': 'Germany',
'email_address': 'stefan\_ml@behnel.de',
'institution': ''},
{'first_names': 'Robert W.',
'surname': 'Bradshaw',
'address': '',
'country': 'USA',
'email_address': 'robertwb@math.washington.edu',
'institution': '''University of Washington\\footnote{
Department of Mathematics, University of Washington, Seattle, WA, USA
}'''},
{'first_names': 'Dag Sverre',
'surname': 'Seljebotn',
'address': '',
'country': 'Norway',
'email_address': 'dagss@student.matnat.uio.no',
# I need three institutions w/ full address... leave it
# all here until we get to editing stage
'institution': '''University of Oslo\\footnote{Institute of Theoretical Astrophysics,
University of Oslo, P.O. Box 1029 Blindern, N-0315 Oslo, Norway}\\footnote{Department
of Mathematics, University of Oslo, P.O. Box 1053 Blindern,
N-0316 Oslo, Norway}\\footnote{Centre of Mathematics for
Applications, University of Oslo, P.O. Box 1053 Blindern, N-0316
Oslo, Norway}'''}
],
}
Calling C functions
====================
This tutorial describes shortly what you need to know in order to call
C library functions from Cython code. For a longer and more
comprehensive tutorial about using external C libraries, wrapping them
and handling errors, see :doc:`clibraries`.
For simplicity, let's start with a function from the standard C
library. This does not add any dependencies to your code, and it has
the additional advantage that Cython already defines many such
functions for you. So you can just cimport and use them.
For example, let's say you need a low-level way to parse a number from
a ``char*`` value. You could use the ``atoi()`` function, as defined
by the ``stdlib.h`` header file. This can be done as follows::
from libc.stdlib cimport atoi
cdef parse_charptr_to_py_int(char* s):
assert s is not NULL, "byte string value is NULL"
return atoi(s) # note: atoi() has no error detection!
You can find a complete list of these standard cimport files in
Cython's source package ``Cython/Includes/``. It also has a complete
set of declarations for CPython's C-API. For example, to test at C
compilation time which CPython version your code is being compiled
with, you can do this::
from cpython.version cimport PY_VERSION_HEX
print PY_VERSION_HEX >= 0x030200F0 # Python version >= 3.2 final
Cython also provides declarations for the C math library::
from libc.math cimport sin
cdef double f(double x):
return sin(x*x)
However, this is a library that is not linked by default on some Unix-like
systems, such as Linux. In addition to cimporting the
declarations, you must configure your build system to link against the
shared library ``m``. For distutils, it is enough to add it to the
``libraries`` parameter of the ``Extension()`` setup::
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules=[
Extension("demo",
["demo.pyx"],
libraries=["m"]) # Unix-like specific
]
setup(
name = "Demos",
cmdclass = {"build_ext": build_ext},
ext_modules = ext_modules
)
If you want to access C code for which Cython does not provide a ready
to use declaration, you must declare them yourself. For example, the
above ``sin()`` function is defined as follows::
cdef extern from "math.h":
double sin(double)
This declares the ``sin()`` function in a way that makes it available
to Cython code and instructs Cython to generate C code that includes
the ``math.h`` header file. The C compiler will see the original
declaration in ``math.h`` at compile time, but Cython does not parse
"math.h" and requires a separate definition.
Just like the ``sin()`` function from the math library, it is possible
to declare and call into any C library as long as the module that
Cython generates is properly linked against the shared or static
library.
Tutorials
=========
.. toctree::
:maxdepth: 2
external
clibraries
cdef_classes
pxd_files
caveats
profiling_tutorial
numpy
strings
pure
readings
related_work
appendix
Using Cython with NumPy
=======================
Cython has support for fast access to NumPy arrays. To optimize code
using such arrays one must ``cimport`` the NumPy pxd file (which ships
with Cython), and declare any arrays as having the ``ndarray``
type. The data type and number of dimensions should be fixed at
compile-time and passed. For instance::
import numpy as np
cimport numpy as np
def myfunc(np.ndarray[np.float64_t, ndim=2] A):
<...>
``myfunc`` can now only be passed two-dimensional arrays containing
double precision floats, but array indexing operation is much, much faster,
making it suitable for numerical loops. Expect speed increases well
over 100 times over a pure Python loop; in some cases the speed
increase can be as high as 700 times or more. [Seljebotn09]_
contains detailed examples and benchmarks.
Fast array declarations can currently only be used with function
local variables and arguments to ``def``-style functions (not with
arguments to ``cpdef`` or ``cdef``, and neither with fields in cdef
classes or as global variables). These limitations are considered
known defects and we hope to remove them eventually. In most
circumstances it is possible to work around these limitations rather
easily and without a significant speed penalty, as all NumPy arrays
can also be passed as untyped objects.
Array indexing is only optimized if exactly as many indices are
provided as the number of array dimensions. Furthermore, all indices
must have a native integer type. Slices and NumPy "fancy indexing" is
not optimized. Examples::
def myfunc(np.ndarray[np.float64_t, ndim=1] A):
cdef Py_ssize_t i, j
for i in range(A.shape[0]):
print A[i, 0] # fast
j = 2*i
print A[i, j] # fast
k = 2*i
print A[i, k] # slow, k is not typed
print A[i][j] # slow
print A[i,:] # slow
``Py_ssize_t`` is a signed integer type provided by Python which
covers the same range of values as is supported as NumPy array
indices. It is the preferred type to use for loops over arrays.
Any Cython primitive type (float, complex float and integer types) can
be passed as the array data type. For each valid dtype in the ``numpy``
module (such as ``np.uint8``, ``np.complex128``) there is a
corresponding Cython compile-time definition in the cimport-ed NumPy
pxd file with a ``_t`` suffix [#]_. Cython structs are also allowed
and corresponds to NumPy record arrays. Examples::
cdef packed struct Point:
np.float64_t x, y
def f():
cdef np.ndarray[np.complex128_t, ndim=3] a = \
np.zeros((3,3,3), dtype=np.complex128)
cdef np.ndarray[Point] b = np.zeros(10,
dtype=np.dtype([('x', np.float64),
('y', np.float64)]))
<...>
Note that ``ndim`` defaults to 1. Also note that NumPy record arrays
are by default unaligned, meaning data is packed as tightly as
possible without considering the alignment preferences of the
CPU. Such unaligned record arrays corresponds to a Cython ``packed``
struct. If one uses an aligned dtype, by passing ``align=True`` to the
``dtype`` constructor, one must drop the ``packed`` keyword on the
struct definition.
Some data types are not yet supported, like boolean arrays and string
arrays. Also data types describing data which is not in the native
endian will likely never be supported. It is however possible to
access such arrays on a lower level by casting the arrays::
cdef np.ndarray[np.uint8, cast=True] boolarr = (x < y)
cdef np.ndarray[np.uint32, cast=True] values = \
np.arange(10, dtype='>i4')
Assuming one is on a little-endian system, the ``values`` array
can still access the raw bit content of the array (which must then
be reinterpreted to yield valid results on a little-endian system).
Finally, note that typed NumPy array variables in some respects behave
a little differently from untyped arrays. ``arr.shape`` is no longer a
tuple. ``arr.shape[0]`` is valid but to e.g. print the shape one must
do ``print (<object>arr).shape`` in order to "untype" the variable
first. The same is true for ``arr.data`` (which in typed mode is a C
data pointer).
There are many more options for optimizations to consider for Cython
and NumPy arrays. We again refer the interested reader to [Seljebotn09]_.
.. [#] In Cython 0.11.2, ``np.complex64_t`` and ``np.complex128_t``
does not work and one must write ``complex`` or
``double complex`` instead. This is fixed in 0.11.3. Cython
0.11.1 and earlier does not support complex numbers.
.. [Seljebotn09] D. S. Seljebotn, Fast numerical computations with Cython,
Proceedings of the 8th Python in Science Conference, 2009.
This diff is collapsed.
Pure Python Mode
================
Cython provides language constructs to let the same file be either interpreted
or compiled. This is accomplished by the same "magic" module ``cython`` that
directives use and which must be imported. This is available for both :file:`.py` and
:file:`.pyx` files.
This is accomplished via special functions and decorators and an (optional)
augmenting :file:`.pxd` file.
Magic Attributes
----------------
The currently supported attributes of the ``cython`` module are:
* ``declare`` declares a typed variable in the current scope, which can be used in
place of the :samp:`cdef type var [= value]` construct. This has two forms, the
first as an assignment (useful as it creates a declaration in
interpreted mode as well)::
x = cython.declare(cython.int) # cdef int x
y = cython.declare(cython.double, 0.57721) # cdef double y = 0.57721
and the second mode as a simple function call::
cython.declare(x=cython.int, y=cython.double) # cdef int x; cdef double y
* ``locals`` is a decorator that is used to specify the types of local variables
in the function body (including any or all of the argument types)::
@cython.locals(a=cython.double, b=cython.double, n=cython.p_double)
def foo(a, b, x, y):
...
* ``address`` is used in place of the ``&`` operator::
cython.declare(x=cython.int, x_ptr=cython.p_int)
x_ptr = cython.address(x)
* ``sizeof`` emulates the `sizeof` operator. It can take both types and
expressions.::
cython.declare(n=cython.longlong)
print cython.sizeof(cython.longlong), cython.sizeof(n)
* ``struct`` can be used to create struct types.::
MyStruct = cython.struct(x=cython.int, y=cython.int, data=cython.double)
a = cython.declare(MyStruct)
is equivalent to the code::
cdef struct MyStruct:
int x
int y
double data
cdef MyStruct a
* ``union`` creates union types with exactly the same syntax as ``struct``
* ``typedef`` creates a new type::
T = cython.typedef(cython.p_int) # ctypedef int* T
* ``compiled`` is a special variable which is set to ``True`` when the compiler
runs, and ``False`` in the interpreter. Thus the code::
if cython.compiled:
print "Yep, I'm compiled."
else:
print "Just a lowly interpreted script."
will behave differently depending on whether or not the code is loaded as a
compiled :file:`.so` file or a plain :file:`.py` file.
Augmenting .pxd
---------------
If a :file:`.pxd` file is found with the same name as a :file:`.py` file, it will be
searched for :keyword:`cdef` classes and :keyword:`cdef`/:keyword:`cpdef`
functions and methods. It will then convert the corresponding
classes/functions/methods in the :file:`.py` file to be of the correct type. Thus if
one had :file:`a.pxd`::
cdef class A:
cpdef foo(self, int i)
the file :file:`a.py`::
class A:
def foo(self, i):
print "Big" if i > 1000 else "Small"
would be interpreted as::
cdef class A:
cpdef foo(self, int i):
print "Big" if i > 1000 else "Small"
The special cython module can also be imported and used within the augmenting
:file:`.pxd` file. This makes it possible to add types to a pure python file without
changing the file itself. For example, the following python file
:file:`dostuff.py`::
def dostuff(n):
t = 0
for i in range(n):
t += i
return t
could be augmented with the following :file:`.pxd` file :file:`dostuff.pxd`::
import cython
@cython.locals(t = cython.int, i = cython.int)
cpdef int dostuff(int n)
Besides the ``cython.locals`` decorator, the :func:`cython.declare` function can also be
used to add types to global variables in the augmenting :file:`.pxd` file.
Note that normal Python (:keyword:`def`) functions cannot be declared in
:file:`.pxd` files, so it is currently impossible to override the types of
Python functions in :file:`.pxd` files if they use ``*args`` or ``**kwargs`` in their
signature, for instance.
Types
-----
There are numerous types built in to the cython module. One has all the
standard C types, namely ``char``, ``short``, ``int``, ``long``, ``longlong``
as well as their unsigned versions ``uchar``, ``ushort``, ``uint``, ``ulong``,
``ulonglong``. One also has ``bint`` and ``Py_ssize_t``. For each type, one
has pointer types ``p_int``, ``pp_int``, . . ., up to three levels deep in
interpreted mode, and infinitely deep in compiled mode. The Python types int,
long and bool are interpreted as C ``int``, ``long`` and ``bint``
respectively. Also, the python types ``list``, ``dict``, ``tuple``, . . . may
be used, as well as any user defined types.
Pointer types may be constructed with ``cython.pointer(cython.int)``, and
arrays as ``cython.int[10]``. A limited attempt is made to emulate these more
complex types, but only so much can be done from the Python language.
Decorators (not yet implemented)
--------------------------------
We have settled on ``@cython.cclass`` for the ``cdef class``
decorators, and ``@cython.cfunc`` and ``@cython.ccall`` for :keyword:`cdef` and
:keyword:`cpdef` functions (respectively).
http://codespeak.net/pipermail/cython-dev/2008-November/002925.html
pxd files
=========
In addition to the ``.pyx`` source files, Cython uses ``.pxd`` files
which work like C header files -- they contain Cython declarations
(and sometimes code sections) which are only meant for inclusion by
Cython modules. A ``pxd`` file is imported into a ``pyx`` module by
using the ``cimport`` keyword.
``pxd`` files have many use-cases:
1. They can be used for sharing external C declarations.
2. They can contain functions which are well suited for inlining by
the C compiler. Such functions should be marked ``inline``, example:
::
cdef inline int int_min(int a, int b):
return b if b < a else a
3. When accompanying an equally named ``pyx`` file, they
provide a Cython interface to the Cython module so that other
Cython modules can communicate with it using a more efficient
protocol than the Python one.
In our integration example, we might break it up into ``pxd`` files like this:
1. Add a ``cmath.pxd`` function which defines the C functions available from
the C ``math.h`` header file, like ``sin``. Then one would simply do
``from cmath cimport sin`` in ``integrate.pyx``.
2. Add a ``integrate.pxd`` so that other modules written in Cython
can define fast custom functions to integrate.
::
cdef class Function:
cpdef evaluate(self, double x)
cpdef integrate(Function f, double a,
double b, int N)
Note that if you have a cdef class with attributes, the attributes must
be declared in the class declaration ``pxd`` file (if you use one), not
the ``pyx`` file. The compiler will tell you about this.
cdef extern from "libcalg/queue.h":
ctypedef struct Queue:
pass
ctypedef void* QueueValue
Queue* queue_new()
void queue_free(Queue* queue)
int queue_push_head(Queue* queue, QueueValue data)
QueueValue queue_pop_head(Queue* queue)
QueueValue queue_peek_head(Queue* queue)
int queue_push_tail(Queue* queue, QueueValue data)
QueueValue queue_pop_tail(Queue* queue)
QueueValue queue_peek_tail(Queue* queue)
int queue_is_empty(Queue* queue)
cimport cqueue
cimport python_exc
cdef class Queue:
cdef cqueue.Queue* _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
if self._c_queue is NULL:
python_exc.PyErr_NoMemory()
def __dealloc__(self):
if self._c_queue is not NULL:
cqueue.queue_free(self._c_queue)
cpdef int append(self, int value) except -1:
if not cqueue.queue_push_tail(self._c_queue, <void*>value):
python_exc.PyErr_NoMemory()
return 0
cdef int extend(self, int* values, Py_ssize_t count) except -1:
cdef Py_ssize_t i
for i in xrange(count):
if not cqueue.queue_push_tail(self._c_queue, <void*>values[i]):
python_exc.PyErr_NoMemory()
return 0
cpdef int peek(self) except? 0:
cdef int value = <int>cqueue.queue_peek_head(self._c_queue)
if value == 0:
# this may mean that the queue is empty, or that it
# happens to contain a 0 value
if cqueue.queue_is_empty(self._c_queue):
raise IndexError("Queue is empty")
return value
cpdef int pop(self) except? 0:
cdef int value = <int>cqueue.queue_pop_head(self._c_queue)
if value == 0:
# this may mean that the queue is empty, or that it
# happens to contain a 0 value
if cqueue.queue_is_empty(self._c_queue):
raise IndexError("Queue is empty")
return value
def __nonzero__(self):
return not cqueue.queue_is_empty(self._c_queue)
DEF repeat_count=10000
def test_cy():
cdef int i
cdef Queue q = Queue()
for i in xrange(repeat_count):
q.append(i)
for i in xrange(repeat_count):
q.peek()
while q:
q.pop()
def test_py():
cdef int i
q = Queue()
for i in xrange(repeat_count):
q.append(i)
for i in xrange(repeat_count):
q.peek()
while q:
q.pop()
from collections import deque
def test_deque():
cdef int i
q = deque()
for i in xrange(repeat_count):
q.appendleft(i)
for i in xrange(repeat_count):
q[-1]
while q:
q.pop()
repeat = range(repeat_count)
def test_py_exec():
q = Queue()
d = dict(q=q, repeat=repeat)
exec u"""\
for i in repeat:
q.append(9)
for i in repeat:
q.peek()
while q:
q.pop()
""" in d
Further reading
===============
The main documentation is located at http://docs.cython.org/. Some
recent features might not have documentation written yet, in such
cases some notes can usually be found in the form of a Cython
Enhancement Proposal (CEP) on http://wiki.cython.org/enhancements.
[Seljebotn09]_ contains more information about Cython and NumPy
arrays. If you intend to use Cython code in a multi-threaded setting,
it is essential to read up on Cython's features for managing the
Global Interpreter Lock (the GIL). The same paper contains an
explanation of the GIL, and the main documentation explains the Cython
features for managing it.
Finally, don't hesitate to ask questions (or post reports on
successes!) on the Cython users mailing list [UserList]_. The Cython
developer mailing list, [DevList]_, is also open to everybody, but
focusses on core development issues. Feel free to use it to report a
clear bug, to ask for guidance if you have time to spare to develop
Cython, or if you have suggestions for future development.
.. [DevList] Cython developer mailing list: http://mail.python.org/mailman/listinfo/cython-devel
.. [Seljebotn09] D. S. Seljebotn, Fast numerical computations with Cython,
Proceedings of the 8th Python in Science Conference, 2009.
.. [UserList] Cython users mailing list: http://groups.google.com/group/cython-users
Related work
============
Pyrex [Pyrex]_ is the compiler project that Cython was originally
based on. Many features and the major design decisions of the Cython
language were developed by Greg Ewing as part of that project. Today,
Cython supersedes the capabilities of Pyrex by providing a
substantially higher compatibility with Python code and Python
semantics, as well as superior optimisations and better integration
with scientific Python extensions like NumPy.
ctypes [ctypes]_ is a foreign function interface (FFI) for Python. It
provides C compatible data types, and allows calling functions in DLLs
or shared libraries. It can be used to wrap these libraries in pure
Python code. Compared to Cython, it has the major advantage of being
in the standard library and being usable directly from Python code,
without any additional dependencies. The major drawback is its
performance, which suffers from the Python call overhead as all
operations must pass through Python code first. Cython, being a
compiled language, can avoid much of this overhead by moving more
functionality and long-running loops into fast C code.
SWIG [SWIG]_ is a wrapper code generator. It makes it very easy to
parse large API definitions in C/C++ header files, and to generate
straight forward wrapper code for a large set of programming
languages. As opposed to Cython, however, it is not a programming
language itself. Thin wrappers are easy to generate, but the more
functionality a wrapper needs to provide, the harder it gets to
implement it with SWIG. Cython, on the other hand, makes it very easy
to write very elaborate wrapper code specifically for the Python
language, and to make it as thin or thick as needed at any given
place. Also, there exists third party code for parsing C header files
and using it to generate Cython definitions and module skeletons.
ShedSkin [ShedSkin]_ is an experimental Python-to-C++ compiler. It
uses a very powerful whole-module type inference engine to generate a
C++ program from (restricted) Python source code. The main drawback
is that it has no support for calling the Python/C API for operations
it does not support natively, and supports very few of the standard
Python modules.
.. [ctypes] http://docs.python.org/library/ctypes.html.
.. there's also the original ctypes home page: http://python.net/crew/theller/ctypes/
.. [Pyrex] G. Ewing, Pyrex: C-Extensions for Python,
http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/
.. [ShedSkin] M. Dufour, J. Coughlan, ShedSkin,
http://code.google.com/p/shedskin/
.. [SWIG] David M. Beazley et al.,
SWIG: An Easy to Use Tool for Integrating Scripting Languages with C and C++,
http://www.swig.org.
This diff is collapsed.
This diff is collapsed.
.. highlight:: cython
.. _early-binding-for-speed:
**************************
Early Binding for Speed
**************************
As a dynamic language, Python encourages a programming style of considering
classes and objects in terms of their methods and attributes, more than where
they fit into the class hierarchy.
This can make Python a very relaxed and comfortable language for rapid
development, but with a price - the 'red tape' of managing data types is
dumped onto the interpreter. At run time, the interpreter does a lot of work
searching namespaces, fetching attributes and parsing argument and keyword
tuples. This run-time 'late binding' is a major cause of Python's relative
slowness compared to 'early binding' languages such as C++.
However with Cython it is possible to gain significant speed-ups through the
use of 'early binding' programming techniques.
For example, consider the following (silly) code example:
.. sourcecode:: cython
cdef class Rectangle:
cdef int x0, y0
cdef int x1, y1
def __init__(self, int x0, int y0, int x1, int y1):
self.x0 = x0; self.y0 = y0; self.x1 = x1; self.y1 = y1
def area(self):
area = (self.x1 - self.x0) * (self.y1 - self.y0)
if area < 0:
area = -area
return area
def rectArea(x0, y0, x1, y1):
rect = Rectangle(x0, y0, x1, y1)
return rect.area()
In the :func:`rectArea` method, the call to :meth:`rect.area` and the
:meth:`.area` method contain a lot of Python overhead.
However, in Cython, it is possible to eliminate a lot of this overhead in cases
where calls occur within Cython code. For example:
.. sourcecode:: cython
cdef class Rectangle:
cdef int x0, y0
cdef int x1, y1
def __init__(self, int x0, int y0, int x1, int y1):
self.x0 = x0; self.y0 = y0; self.x1 = x1; self.y1 = y1
cdef int _area(self):
cdef int area
area = (self.x1 - self.x0) * (self.y1 - self.y0)
if area < 0:
area = -area
return area
def area(self):
return self._area()
def rectArea(x0, y0, x1, y1):
cdef Rectangle rect
rect = Rectangle(x0, y0, x1, y1)
return rect._area()
Here, in the Rectangle extension class, we have defined two different area
calculation methods, the efficient :meth:`_area` C method, and the
Python-callable :meth:`area` method which serves as a thin wrapper around
:meth:`_area`. Note also in the function :func:`rectArea` how we 'early bind'
by declaring the local variable ``rect`` which is explicitly given the type
Rectangle. By using this declaration, instead of just dynamically assigning to
``rect``, we gain the ability to access the much more efficient C-callable
:meth:`_rect` method.
But Cython offers us more simplicity again, by allowing us to declare
dual-access methods - methods that can be efficiently called at C level, but
can also be accessed from pure Python code at the cost of the Python access
overheads. Consider this code:
.. sourcecode:: cython
cdef class Rectangle:
cdef int x0, y0
cdef int x1, y1
def __init__(self, int x0, int y0, int x1, int y1):
self.x0 = x0; self.y0 = y0; self.x1 = x1; self.y1 = y1
cpdef int area(self):
cdef int area
area = (self.x1 - self.x0) * (self.y1 - self.y0)
if area < 0:
area = -area
return area
def rectArea(x0, y0, x1, y1):
cdef Rectangle rect
rect = Rectangle(x0, y0, x1, y1)
return rect.area()
.. note::
in earlier versions of Cython, the :keyword:`cpdef` keyword is
:keyword:`rdef` - but has the same effect).
Here, we just have a single area method, declared as :keyword:`cpdef` to make it
efficiently callable as a C function, but still accessible from pure Python
(or late-binding Cython) code.
If within Cython code, we have a variable already 'early-bound' (ie, declared
explicitly as type Rectangle, (or cast to type Rectangle), then invoking its
area method will use the efficient C code path and skip the Python overhead.
But if in Pyrex or regular Python code we have a regular object variable
storing a Rectangle object, then invoking the area method will require:
* an attribute lookup for the area method
* packing a tuple for arguments and a dict for keywords (both empty in this case)
* using the Python API to call the method
and within the area method itself:
* parsing the tuple and keywords
* executing the calculation code
* converting the result to a python object and returning it
So within Cython, it is possible to achieve massive optimisations by
using strong typing in declaration and casting of variables. For tight loops
which use method calls, and where these methods are pure C, the difference can
be huge.
This diff is collapsed.
This diff is collapsed.
Cython Users Guide
======================
Contents:
.. toctree::
:maxdepth: 2
overview
tutorial
language_basics
extension_types
special_methods
sharing_declarations
external_C_code
source_files_and_compilation
wrapping_CPlusPlus
numpy_tutorial
limitations
pyrex_differences
early_binding_for_speed
debugging
Indices and tables
------------------
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. toctree::
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
.. highlight:: cython
.. _overview:
********
Overview
********
About Cython
==============
Cython is a language that makes writing C extensions for the Python language
as easy as Python itself. Cython is based on the well-known `Pyrex
<http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ language by Greg Ewing,
but supports more cutting edge functionality and optimizations [#]_.
The Cython language is very close to the Python language, but Cython
additionally supports calling C functions and declaring C types on variables
and class attributes. This allows the compiler to generate very efficient C
code from Cython code.
This makes Cython the ideal language for wrapping external C libraries,
and for fast C modules that speed up the execution of Python code.
Future Plans
============
Cython is not finished. Substantial tasks remaining. See
:ref:`cython-limitations` for a current list.
.. rubric:: Footnotes
.. [#] For differences with Pyrex see :ref:`pyrex-differences`.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment