What's new in Cython 0.23

It was really nice to hear Cython being mentioned in pretty much every second talk at this year's EuroSciPy in Cambridge, either directly or by promoting tools that were themselves written in Cython. It seems to have become a key cornerstone in Python's ecosystem.

For me, this has been an intense conference summer overall, so I only just noticed that I hadn't written up anything about the long list of changes in Cython 0.23 yet. Well, apart from the changelog :). So here it goes.

Cython 0.23 is a major feature release. It implements the new language features of the new Python 3.5, but with the usual backports to make them available in Python 2.6 and later. My personal feature favourites are PEP 492 (async/await) and inlined generator expressions, but also the new support for Cython code coverage analysis with the coverage.py tool, that I blogged about already. Note that some Py3.5 features, such as the PEP 465 matrix multiplication operator, were already released early this year in Cython 0.22.

One of the greatest changes in Python 3.5, from a Cython perspective, is the shift from generators and coroutines as builtin types to making them ABCs, i.e. abstract base classes in the collections.abc module. This enables full interoperability of Cython's own implementations with Python's builtin types. To make this available in older Python versions, I created the backports_abc package for applications to use. Basically, you would install it and then say

import backports_abc
backports_abc.patch()

to get the Coroutine and Generator classes added to the collections.abc module if they are missing (i.e. in CPython versions before 3.5). This allows Cython to register its internal types with them and from that point on, any code using isinstance(x, Generator) or isinstance(x, Coroutine) will be able to support Cython's fast builtin types. If you find software packages that do not support this yet because they do type checks instead of ABC instance checks, please file a bug report with their projects to get them adapted. Twisted comes to mind, for example, or Tornado. [Update 2015-09-15]: The next Tornado release will have support for Cython coroutines.

CPython 3.5 itself has already been fixed up, thanks to Yury Selivanov, to natively support Cython implemented generators and coroutines. This allows asynchronous code (currently, but not limited to, asyncio) to take advantage of Cython's speed, which can easily be several times faster than Python coroutines in practice.

What else is there in Cython 0.23? Well, inlined generator expressions are back. This means that when you call builtin functions like any() or sorted() on a generator expression, Cython will inline its own (partial) implementation of that function into the generator implementation and avoid calling back and forth between the generator and its consumer. Instead, it will calculate the final result inside of the generator and only return it in one step at the very end. This can speed up the overall execution considerably.

Another feature that I invested some time into is faster numeric operations on Python's long (Py3 int) and float types. Arithmetic or shifting operations that involves integer or float constants are now faster because they avoid double unpacking of the operands and inline their constant parts.

On the C++ integration front, several minor issues have seen improvements and new standard declarations were added, which makes Cython 0.23 an even nicer and rounder tool for wrapping C++ code bases, or for making use of STL containers in Cython code. More C++ improvements are planned for the upcoming 0.24 release.

As usual, the complete list of features and fixes is much longer for such a large release. Hope you like it.