What's new in Cython 0.27?

Cython 0.27 is freshly released and comes with several great improvements. It finally implements all major features that were added in CPython 3.6, such as async generators and variable annotations. The long list of new features and resolved issues can be found in the changelog, or in the list of resolved tickets.

Probably the biggest new feature is the support for asynchronous generators and asynchronous comprehensions, as specified in PEP 525 and PEP 530 respectively. They allow using yield inside of async coroutines and await inside of comprehensions, so that the following becomes possible:

async def generate_results(source):
    async for i in source:
        yield i ** 2
    d = {i: await result for i, result in enumerate(async_results)}
    l = [s for c in more_async_results
         for s in await c]

As usual, this feature is available in Cython compiled modules across all supported Python versions, starting with Python 2.6. However, using async cleanup in generators, e.g. in a finally-block, requires CPython 3.6 in order to remember which I/O-loop the generator must use. Async comprehensions do not suffer from this.

The next big and long awaited feature is support for PEP 484 compatible typing. Both signature annotations (PEP 484) and variable annotations (PEP 526) are now parsed for Python types and cython.* types like list or cython.int. Complex types like List[int] are not currently evaluated as the semantics are less clear in the context of static compilation. This will be added in a future release.

One special twist here is exception handling, which tries to mimic Python more closely than the defaults in Cython code. Thus, it is no longer necessary to explicitly declare an exception return value in code like this:

def add_1(x: cython.int) -> cython.int:
    if x < 0:
        raise ValueError("...")
    return x + 1

Cython will automatically return -1 (the default exception value for C integer types) when an exception is raised and check for exceptions after calling it. This is identical to the Cython signature declaration except? -1.

In cases where annotations are not meant as static type declarations during compilation, the extraction can be disabled with the compiler directive annotation_typing=False.

The new release brings another long awaited feature: automatic ``__richcmp__()`` generation. Previously, extension types required a major difference to Python classes with respect to the special methods for comparison, __eq__, __lt__ etc. Users had to implement their own special __richcmp__() method which implemented all comparisons at once. Now, Cython can automatically generate an efficient __richcmp__() method from the normal comparison methods, including inherited base type implementations. This brings Python classes and extension types another huge step closer.

To bring extension modules also closer to Python modules, Cython now implements the new extension module initialisation process of PEP 489 in CPython 3.5 and later. This makes the special global names like __file__ and __path__ correctly available to module level code and improves the support for module-level relative imports. As with most internal features, existing Cython code will benefit from this by simple recompilation.

As a last feature worth mentioning, the IPython/Jupyter magic integration gained a new option %%cython --pgo for easy profile guided optimisation. This allows the C compiler to take better decisions during its optimisation phase based on a (hopefully) realistic runtime profile. The option compiles the cell with PGO settings for the C compiler, executes it to generate the runtime profile, and then compiles it again using that profile for C compiler optimisation. This is currently only tested with gcc. Support for other compilers can easily be added to the IPythonMagic.py module and pull requests are welcome, as usual.

By design, the Jupyter cell itself is responsible for generating a suitable profile. This can be done by implementing the functions that should be optimised via PGO, and then calling them directly in the same cell on some realistic training data like this:

%%cython --pgo
def critical_function(data):
    for item in data:

# execute function several times to build profile
from somewhere import some_typical_data
for _ in range(100):

Together with the improved module initialisation in Python 3.5 and later, you can now also distinguish between the profile and non-profile runs as follows:

if "_pgo_" in __name__:
    ...  # execute critical code here

Nichts falsch machen

»Das Netz ist voll mit mal mehr, mal weniger ernst gemeinten Aufzählungen der konkreten Vor- und Nachteile einzelner Programmiersprachen. Es ist zwar verlockend, diesen Listen eine weitere hinzuzufügen und dabei unsubtil die eigenen Vorlieben und Abneigungen einfließen zu lassen. Aber die Details solcher konkreten Empfehlungen ändern sich alle paar Jahre, und Sie lernen nicht viel dazu, wenn wir Ihnen einfach raten, »Nehmen Sie einfach Python, damit machen Sie nichts falsch!« (Obwohl Sie damit sicher wirklich nichts falsch machen.)« – Kathrin Passig und Johannes Jander, Weniger schlecht programmieren.

What's new in Cython 0.26?

Cython 0.26 has finally been released and it comes with some big and several smaller new features, contributed by quite a number of non-core developers.

Probably the biggest addition, definitely codewise, is support for Pythran as a backend for NumPy array expressions, contributed by Adrien Guinet. Pythran understands many usage patterns for NumPy, including array expressions and some methods, and can directly be used from within Cython to compile NumPy using code by setting the directive np_pythran=True. Thus, if Pythran is available at compile time, users can avoid writing manual loops and instead often just use NumPy in the same way as they would from Python. Note that this does not currently work generically with Cython's memoryviews, so you need to declare the specific numpy.ndarray[] types in order to benefit from the translation. Also, this requires the C++ mode in Cython as Pythran generates C++ code.

The other major new feature in this release, and likely with an even wider impact, is pickling support for cdef classes (a.k.a. extension types). This is enabled by default for all classes with Python compatible attribute types, which explicitly excludes pointers and unions. Classes with struct type attributes are also excluded for practical reasons (such as a high code overhead), but can be enabled to support pickling with the class decorator @cython.auto_pickle(True). This was a long-standing feature for which users previously had to implement the pickle protocol themselves. Since Cython has all information about the extension type and its attributes, however, there was no technical reason why it can't also generate the support for pickling them, and it now does.

As always, there are several new optimisations including speed-ups for abs(complex), comparing strings and dispatching to specialised function implementations based on fused types arguments. Particularly interesting might be the faster GIL re-entry with the directive fast_gil=True. It tries to remember the current GIL lock state in the fast thread local storage and avoids costly calls into the thread and GIL handling APIs if possible, even when calling across multiple Cython modules.

A slightly controversial change now hides the C code lines from tracebacks by default. Cython exceptions used to show the failing line number of the generated C code in addition to the Cython module code line, now only the latter is shown, like in Python code. On the one hand, this removes clutter that is irrelevant for most users. On the other hand, it hides information that could help developers debug failures from bug reports. For debugging purposes, the re-inclusion of C code lines can now be enabled with a runtime setting as follows:

import cython_runtime

What's new in Cython 0.25?

Cython 0.25 has been released in October 2016, so here's a quick writeup of the most relevant new features in this release.

My personal favourites are the call optimisations. Victor Stinner has done a great job throughout the year to optimise and benchmark different parts of CPython, and one of the things he came up with was a faster way to process function calls internally. For this, he added a new calling convention, METH_FASTCALL, which avoids tuple creation by passing positional arguments as a C array. I've added support for this to Cython and also ported a part of the CPython implementation to speed up calls to Python functions and PyCFunction functions from Cython code.

Further optimisations speed up f-string formatting, cython.inline() and some Python long integer operations (now also in Py2.7 and not only Py3).

The next big feature, one that has been on our list essentially forever, was finally added in 0.25. If you declare a special attribute __dict__ on a cdef class (a.k.a. extension type), if will have an instance dict that allows setting arbitrary attributes on the objects. Otherwise, extension types are limited to the declared attributes and trying to access undeclared ones would result in an AttributeError.

The C++ integration has received several minor improvements, including support for calling subclass methods from C++ classes implemented in Cython code, iterating over std::string with Cython's for-loop, typedef members in class declarations, or the typeid operator.

And a final little goodie is redundant declarations for pi and e in libc.math, which makes from libc cimport math pretty much a drop-in replacement for Python's import math.

Einflussreiche Bücher bei Skoobe

Die Webseite Highly Reco präsentiert hunderte Empfehlungen von Büchern, die berühmten Persönlichkeiten aus Wirtschaft und Gesellschaft etwas bedeutet haben. Seien es Bill Gates, Jessica Livingston, Warren Buffet oder Jeff Bezos, sie alle haben aus Büchern gelernt und schreiben dort über ihre Einflüsse und Ideengeber.

Bei Skoobe finden sich viele dieser Bücher im Abo-Katalog, hier eine kleine Auswahl einflussreicher Bücher, für alle, die ihnen nacheifern möchten.

Python-Bücher bei Skoobe

Neben einer riesigen Auswahl an Belletristik hat das eBook-Abo Skoobe auch zahlreiche gute Sachbücher im Angebot. Hier ist eine kleine Auswahl empfehlenswerter Python-Bücher aus der langen Liste bei Skoobe:

Cython for async networking

EuroPython 2016 seems to have three major topics this year, two of which make heavy use of Cython. The first, and probably most wonderful topic is beginners. The conference started with a workshop day on Sunday that was split between Django Girls and (other) Python beginners. The effect on the conference is totally visible: lots of new people walking around, visibly more Python beginners, and a clearly better ratio of women to men.

The other two big topics are: async networking and machine learning. Machine learning fills several talks and tutorials, and is obviously backed by Cython implemented tools in many corners.

For async networking, however, it might seem more surprising that Cython has such a good stand. But there are good reasons for it: even mostly I/O bound applications can hugely benefit from processing speed at the different layers, as Anton and I showed in our talk on Monday (see below). The deeper you step down into the machinery, however, the more important that speed becomes. And Yury Selivanov is giving an excellent example for that with his reimplementation of the asyncio event loop in Cython, named uvloop. Here is a blog post announcing uvloop.

Since the final talk recordings are not online yet, I have to refer to the live stream dumps for now.

The talk by Anton Caceres and me (we're both working at Skoobe) on Fast Async Code with Cython and AsyncIO starts at hour/minute 2:20 in the video. We provide examples and give motivations for compiling async code to speed up the processing and cut down the overall response latency. I'm also giving a very quick "Cython in 10 Minutes" intro to the language about half way through the talk.

Yury's talk on High Performance Networking in Python starts at minute 10. He gives a couple of great testimonials for Cython along the way, describing how the async/await support in Cython and the ease of talking to C libraries has enabled him to write a tool that beats the performance of well known async libraries in Go and Javascript.

What's new in Cython 0.24

Cython 0.24 has been released after about half a year of development. Time for a writeup of the most interesting features and improvements that already anticipate some of the new language features scheduled for CPython 3.6, at the end of this year.

My personal feature favourite is PEP-498 f-strings, e.g.

>>> value = 4 * 20
>>> f'The value is {value}.'
'The value is 80.'
>>> f'Four times the value is {4*value}.'
'Four times the value is 320.'

I was initially opposed to making them a part of the Python language, but then got convinced by others that they do bring an actual improvement by making the .format() string formatting a) the One Way To Do It compared to the other, older ways, and b) actually avoidable by providing a simpler language syntax for it. Especially the indirection of mapping values to format string names is now gone. Compare:

>>> 'The value is {value}.'.format(value=value)
'The value is 80.'

Sadly, we cannot use them in Cython itself (and we do a lot of string formatting in Cython), since the Python code of the compiler still has to run on Python 2.6. But at least code written in Cython can now make use of them, even if the compiled modules need to run in Python 2.6. The only required underlying Python feature is the __format__() protocol, which is available in all Python versions that Cython supports. Now that there is a dedicated syntax for string formatting that the compiler can see and explore, work has started for optimising it further at compile time.

EDIT (2016-04-12): f-string formatting is currently about 2-4 times as fast in Cython as in CPython 3.6.

Another new feature (also from Python 3.6) is the underscore separator in number literals, as defined by PEP 515. Since Cython is used a lot for developing numeric, scientific and algorithmic code, it's certainly helpful to be able to write

n = 1_000_000
x = 3.14159_26535_89

rather than the much less readable "count my digits" forms

n = 1000000
x = 3.141592653589

IDEs like PyCharm and PyDev will still have to catch up with these PEPs, but that will come. It seems that the PyCharm developers have already started working on this.

Some people complained about the size of the C files that Cython generates, so there is now a compiler option to disable the injection of C comments that show the original source code line right before the C code that came out of it. Just start your source file with

# cython: emit_code_comments=False

or, rather, pass the option from your setup.py script. The size reduction can be quite large. There is, however, a drawback: this makes source level debugging more difficult and also prevents coverage reporting. But since these are developer features, omitting the code comments can still be a good choice for release builds.

Pure Python mode (i.e. the optimised compilation of .py files) has also seen some improvements. C-tuples are now supported, i.e. you can write code as in the following (contrieved) example, which provides efficient access to C values of a tuple:

import cython

@cython.locals(t=(cython.int,cython.double), i=cython.int)
def func(t):
    x = 1.0
    for i in range(t[0]):
        x *= t[1]

Type inference will also automatically turn Python tuples into C tuples in some cases, but that already happend in previous releases.

Cython enums were adapted to PEP 435 enums and thus use the standard Python Enum type if available.

And the @property decorator is finally supported for methods in cdef classes. This removes the need for the old special property name: syntax block that Cython inherited from Pyrex. This syntax is now officially deprecated.

Apart from these, there is the usual list of optimisations, bug fixes and general improvements that went into this release. I am happy to notice that the number of people contributing to the code base has been steadily growing over the last couple of years. Thanks to everyone who helps making Cython better with every release!

And for those interested in learning Cython from a pro, there are still places left for the extensive Cython training that I'll be giving in Leipzig (DE) next June, 16-17th. It's part of a larger series of courses throughout that week on High-Performance Computing with Python.