Skip to content

C
cython
Commit 5ebe3fe5 authored by Stefan Behnel's avatar Stefan Behnel
Browse files
Options

updated C library tutorial as proposed by Terry Reedy

parent 432b797c
Hide whitespace changes
Inline Side-by-side
Showing with 153 additions and 110 deletions
src/tutorial/clibraries.rst
View file @ 5ebe3fe5
......@@ -2,8 +2,8 @@ Using C libraries
=================
Apart from writing fast code, one of the main use cases of Cython is
to call external C libraries from Python code. As seen for the C
string decoding functions above, it is actually trivial to call C
to call external C libraries from Python code. As Cython code
compiles down to C code itself, it is actually trivial to call C
functions directly in the code. The following describes what needs to
be done to use an external C library in Cython code.
......@@ -21,6 +21,8 @@ type that can encapsulate all memory management.
The C API of the queue implementation, which is defined in the header
file ``libcalg/queue.h``, essentially looks like this::
/* file: queue.h */
typedef struct _Queue Queue;
typedef void *QueueValue;
......@@ -40,6 +42,8 @@ file ``libcalg/queue.h``, essentially looks like this::
To get started, the first step is to redefine the C API in a ``.pxd``
file, say, ``cqueue.pxd``::
# file: cqueue.pxd
cdef extern from "libcalg/queue.h":
ctypedef struct Queue:
pass
......@@ -59,54 +63,64 @@ file, say, ``cqueue.pxd``::
bint queue_is_empty(Queue* queue)
Note how these declarations are almost identical to the header file
declarations, so you can often just copy them over. One exception is
the last line. The return value of the ``queue_is_empty`` method is
actually a C boolean value, i.e. it is either zero or non-zero,
indicating if the queue is empty or not. This is best expressed by
Cython's ``bint`` type, which is a normal ``int`` type when used in C
but maps to Python's boolean values ``True`` and ``False`` when
converted to a Python object. Another difference is the first
line. ``Queue`` is in this case used as an *opaque handle*; only the
library that is called know what is actually inside. Since no Cython code
needs to know the contents of the struct, we do not need to declare its contents,
so we simply provide an empty definition (as we do not want to declare
the ``_Queue`` type which is referenced in the C header) [#]_.
declarations, so you can often just copy them over. One noteworthy
difference is the first line. ``Queue`` is in this case used as an
*opaque handle*; only the library that is called knows what is really
inside. Since no Cython code needs to know the contents of the
struct, we do not need to declare its contents, so we simply provide
an empty definition (as we do not want to declare the ``_Queue`` type
which is referenced in the C header) [#]_.
.. [#] There's a subtle difference between ``cdef struct Queue: pass``
and ``ctypedef struct Queue: pass``. The former declares a type
which is referenced in C code as ``struct Queue``, while the
latter is referenced in C as ``Queue``. This is a C language
quirk that Cython is not able to hide. Most modern C libraries
use the ``ctypedef`` kind of struct.
and ``ctypedef struct Queue: pass``. The former declares a
type which is referenced in C code as ``struct Queue``, while
the latter is referenced in C as ``Queue``. This is a C
language quirk that Cython is not able to hide. Most modern C
libraries use the ``ctypedef`` kind of struct.
Another exception is the last line. The integer return value of the
``queue_is_empty`` method is actually a C boolean value, i.e. it is
either zero or non-zero, indicating if the queue is empty or not.
This is best expressed by Cython's ``bint`` type, which is a normal
``int`` type when used in C but maps to Python's boolean values
``True`` and ``False`` when converted to a Python object.
Next, we need to design the Queue class that should wrap the C queue.
It will live in a file called ``queue.pyx``. [#]_
.. [#] Note that the name of the ``.pyx`` file must be different from
the ``cqueue.pxd`` file with declarations from the C library,
as both do not describe the same code. A ``.pxd`` file next to
a ``.pyx`` file with the same name defines exported
declarations for code in the ``.pyx`` file.
Here is a first start for the Queue class::
# file: queue.pyx
cimport cqueue
cimport python_exc
cdef class Queue:
cdef cqueue.Queue _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
Note that it says ``__cinit__`` rather than ``__init__``. While
Note that it says ``__cinit__`` rather than ``__init__``. While
``__init__`` is available as well, it is not guaranteed to be run (for
instance, one could create a subclass and forget to call the ancestor
constructor). Because not initializing C pointers often leads to
crashing the Python interpreter without leaving as much as a stack
trace, Cython provides ``__cinit__`` which is *always* called on
construction. However, as ``__cinit__`` is called during object
construction, ``self`` is not fully
constructed yet, and one must avoid doing anything with ``self`` but
assigning to ``cdef`` fields.
instance, one could create a subclass and forget to call the
ancestor's constructor). Because not initializing C pointers often
leads to crashing the Python interpreter without leaving as much as a
stack trace, Cython provides ``__cinit__`` which is *always* called on
construction. However, as ``__cinit__`` is called during object
construction, ``self`` is not fully constructed yet, and one must
avoid doing anything with ``self`` but assigning to ``cdef`` fields.
Note also that the above method takes no parameters, although subtypes
may want to accept some. Although it is guaranteed to get called, the
no-arguments ``__cinit__()`` method is a special case here as it does
not prevent subclasses from adding parameters as they see fit. If
parameters are added they must match those of any declared ``__init__``
method.
not prevent subclasses from adding parameters as they see fit. If
parameters are added they must match those of any declared
``__init__`` method.
Before we continue implementing the other methods, it is important to
understand that the above implementation is not safe. In case
......@@ -117,45 +131,67 @@ only reason why the above can fail is due to insufficient memory. In
that case, it will return ``NULL``, whereas it would normally return a
pointer to the new queue.
The normal way to get out of this is to raise an exception, but
allocating a new exception instance may actually fail when we are
running out of memory. Luckily, CPython provides a function
``PyErr_NoMemory()`` that raises the right exception for us. We can
thus change the init function as follows::
The normal Python way to get out of this is to raise an exception, but
in this specific case, allocating a new exception instance may
actually fail because we are running out of memory. Luckily, CPython
provides a function ``PyErr_NoMemory()`` that safely raises the right
exception for us. We can thus change the init function as follows::
cimport cpython.exc # standard cimport from CPython's C-API
cimport cqueue
cdef class Queue:
cdef cqueue.Queue _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
if self._c_queue is NULL:
python_exc.PyErr_NoMemory()
The next thing to do is to clean up when the Queue is no longer used.
To this end, CPython provides a callback that Cython makes available
as a special method ``__dealloc__()``. In our case, all we have to do
is to free the Queue, but only if we succeeded in initialising it in
cpython.exc.PyErr_NoMemory()
The ``cpython`` package contains pre-defined ``.pxd`` files that ship
with Cython. If you need any CPython C-API functions, you can cimport
them from this package. See Cython's ``Cython/Includes/`` source
package for a complete list of ``.pxd`` files, including parts of the
standard C library.
The next thing to do is to clean up when the Queue instance is no
longer used (i.e. all references to it have been deleted). To this
end, CPython provides a callback that Cython makes available as a
special method ``__dealloc__()``. In our case, all we have to do is
to free the C Queue, but only if we succeeded in initialising it in
the init method::
def __dealloc__(self):
if self._c_queue is not NULL:
cqueue.queue_free(self._c_queue)
At this point, we have a compilable Cython module that we can test.
To compile it, we need to configure a ``setup.py`` script for
distutils. Based on the example presented earlier on, we can extend
the script to include the necessary setup for building against the
external C library. Assuming it's installed in the normal places
(e.g. under ``/usr/lib`` and ``/usr/include`` on a Unix-like system),
we could simply change the extension setup from
At this point, we have a working Cython module that we can test. To
compile it, we need to configure a ``setup.py`` script for distutils.
Reusing the basic script from the main tutorial::
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("queue", ["queue.pyx"])]
)
We can extend this script to include the necessary setup for building
against the external C library. Assuming it's installed in the normal
places (e.g. under ``/usr/lib`` and ``/usr/include`` on a Unix-like
system), we could simply change the extension setup from
::
ext_modules = [Extension("hello", ["hello.pyx"])]
ext_modules = [Extension("queue", ["queue.pyx"])]
to
::
ext_modules = [
Extension("hello", ["hello.pyx"],
Extension("queue", ["queue.pyx"],
libraries=["calg"])
]
......@@ -167,13 +203,13 @@ flags, such as::
LDFLAGS="-L/usr/local/otherdir/calg/lib" \
python setup.py build_ext -i
Once we have compiled the module for the first time, we can try to
import it::
Once we have compiled the module for the first time, we can now import
it and instantiate a new Queue::
PYTHONPATH=. python -c 'import queue.Queue as Q; Q()'
PYTHONPATH=. python -c 'import queue.Queue as Q ; Q()'
However, our class doesn't do much yet so far, so
let's make it more usable.
However, this is all our Queue class can do so far, so let's make it
more usable.
Before implementing the public interface of this class, it is good
practice to look at what interfaces Python offers, e.g. in its
......@@ -198,12 +234,12 @@ Here is a simple implementation for the ``append()`` method::
Again, the same error handling considerations as for the
``__cinit__()`` method apply, so that we end up with this
implementation::
implementation instead::
cdef append(self, int value):
if not cqueue.queue_push_tail(self._c_queue,
<void*>value):
python_exc.PyErr_NoMemory()
cpython.exc.PyErr_NoMemory()
Adding an ``extend()`` method should now be straight forward::
......@@ -214,7 +250,7 @@ Adding an ``extend()`` method should now be straight forward::
for i in range(count):
if not cqueue.queue_push_tail(
self._c_queue, <void*>values[i]):
python_exc.PyErr_NoMemory()
cpython.exc.PyErr_NoMemory()
This becomes handy when reading values from a NumPy array, for
example.
......@@ -230,15 +266,14 @@ which provide read-only and destructive read access respectively::
return <int>cqueue.queue_pop_head(self._c_queue)
Simple enough. Now, what happens when the queue is empty? According
to the documentation, the functions return a ``NULL`` pointer, which is
typically not a valid value. Since we are simply casting to and from
ints, we cannot distinguish anymore if the
return value was ``NULL`` because the queue was empty or because the
value stored in the queue was ``0``. However, in Cython code, we
would expect the first case to raise an exception, whereas the second
case should simply return ``0``. To deal with this, we need to
special case this value, and check if the queue really is empty or
not::
to the documentation, the functions return a ``NULL`` pointer, which
is typically not a valid value. Since we are simply casting to and
from ints, we cannot distinguish anymore if the return value was
``NULL`` because the queue was empty or because the value stored in
the queue was ``0``. However, in Cython code, we would expect the
first case to raise an exception, whereas the second case should
simply return ``0``. To deal with this, we need to special case this
value, and check if the queue really is empty or not::
cdef int peek(self) except? 0:
cdef int value = \
......@@ -264,30 +299,29 @@ an exception was raised, and if so, propagate the exception. This
obviously has a performance penalty. Cython therefore allows you to
indicate which value is explicitly returned in the case of an
exception, so that the surrounding code only needs to check for an
exception when receiving this special value. All other values will be
exception when receiving this exact value. All other values will be
accepted almost without a penalty.
Now that the ``peek()`` method is implemented, the ``pop()`` method is
almost identical. It only calls a different C function::
Now that the ``peek()`` method is implemented, the ``pop()`` method
also needs adaptation. Since it removes a value from the queue,
however, it is not enough to test if the queue is empty *after* the
removal. Instead, we must test it on entry::
cdef int pop(self) except? 0:
cdef int value = \
<int>cqueue.queue_pop_head(self._c_queue)
if value == 0:
# this may mean that the queue is empty, or
# that it happens to contain a 0 value
if cqueue.queue_is_empty(self._c_queue):
raise IndexError("Queue is empty")
return value
if cqueue.queue_is_empty(self._c_queue):
raise IndexError("Queue is empty")
return <int>cqueue.queue_pop_head(self._c_queue)
Lastly, we can provide the Queue with an emptiness indicator in the
normal Python way::
normal Python way by defining the ``__bool__()`` special method (note
that Python 2 calls this method ``__nonzero__``, whereas Cython code
can use both)::
def __nonzero__(self):
def __bool__(self):
return not cqueue.queue_is_empty(self._c_queue)
Note that this method returns either ``True`` or ``False`` as the
return value of the ``queue_is_empty`` function is declared as a
Note that this method returns either ``True`` or ``False`` as we
declared the return type of the ``queue_is_empty`` function as
``bint``.
Now that the implementation is complete, you may want to write some
......@@ -305,21 +339,17 @@ callable from C code with fast C semantics and without requiring
intermediate argument conversion from or to Python types.
The following listing shows the complete implementation that uses
``cpdef`` methods where possible. This feature is obviously not
available for the ``extend()`` method, as the method signature is
incompatible with Python argument types.
::
``cpdef`` methods where possible::
cimport cqueue
cimport python_exc
cimport cpython.exc
cdef class Queue:
cdef cqueue.Queue* _c_queue
def __cinit__(self):
self._c_queue = cqueue.queue_new()
if self._c_queue is NULL:
python_exc.PyErr_NoMemory()
cpython.exc.PyErr_NoMemory()
def __dealloc__(self):
if self._c_queue is not NULL:
......@@ -328,14 +358,14 @@ incompatible with Python argument types.
cpdef append(self, int value):
if not cqueue.queue_push_tail(self._c_queue,
<void*>value):
python_exc.PyErr_NoMemory()
cpython.exc.PyErr_NoMemory()
cdef extend(self, int* values, Py_ssize_t count):
cdef Py_ssize_t i
for i in range(count):
for i in xrange(count):
if not cqueue.queue_push_tail(
self._c_queue, <void*>values[i]):
python_exc.PyErr_NoMemory()
cpython.exc.PyErr_NoMemory()
cpdef int peek(self) except? 0:
cdef int value = \
......@@ -347,24 +377,37 @@ incompatible with Python argument types.
raise IndexError("Queue is empty")
return value
cpdef int pop(self) except? 0:
cdef int value = \
<int>cqueue.queue_pop_head(self._c_queue)
if value == 0:
# this may mean that the queue is empty,
# or that it happens to contain a 0 value
if cqueue.queue_is_empty(self._c_queue):
raise IndexError("Queue is empty")
return value
cdef int pop(self) except? 0:
if cqueue.queue_is_empty(self._c_queue):
raise IndexError("Queue is empty")
return <int>cqueue.queue_pop_head(self._c_queue)
def __nonzero__(self):
def __bool__(self):
return not cqueue.queue_is_empty(self._c_queue)
As a quick test with numbers from 0 to 9999 indicates, using this
Queue from Cython code with C ``int`` values is about five times as
fast as using it from Cython code with Python values, almost eight
times faster than using it from Python code in a Python loop, and
still more than twice as fast as using Python's highly optimised
``collections.deque`` type from Cython code with Python integers.
The ``cpdef`` feature is obviously not available for the ``extend()``
method, as the method signature is incompatible with Python argument
types. However, if wanted, we can rename the C-ish ``extend()``
method to e.g. ``c_extend()``, and write a new ``extend()`` method
instead that accepts an arbitrary Python iterable::
cdef c_extend(self, int* values, Py_ssize_t count):
cdef Py_ssize_t i
for i in range(count):
if not cqueue.queue_push_tail(
self._c_queue, <void*>values[i]):
cpython.exc.PyErr_NoMemory()
cpdef extend(self, values):
for value in values:
self.append(value)
As a quick test with numbers from 0 to 9999 on the author's machine
indicates, using this Queue from Cython code with C ``int`` values is
about five times as fast as using it from Cython code with Python
values, almost eight times faster than using it from Python code in a
Python loop, and still more than twice as fast as using Python's
highly optimised ``collections.deque`` type from Cython code with
Python integers.
.. [CAlg] Simon Howard, C Algorithms library, http://c-algorithms.sourceforge.net/
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7