What's new in Cython 0.28?

The freshly released Cython 0.28 is another major step in the development of the Cython compiler. It comes with several big new features, some of them long awaited, as well as various little optimisations. It also improves the integration with recent Python features (including the upcoming Python 3.7) and the Pythran compiler for NumPy expressions. The Changelog has the long list of relevant changes. As always, recompiling with the latest version will make your code adapt automatically to new Python releases and take advantage of the new optimisations.

The most long requested change in this release, however, is the support for read-only memory views. You can now simply declare a memory view as const, e.g. const double[:,:], and Cython will request a read-only buffer for it. This allows interaction with bytes objects and other non-writable buffer providers. Note that this makes the item type of the memory view const, i.e. non-writable, and not just the view itself. If the item type is not just a simple numeric type, this might require minor changes to the data types used in the code reading from the view. This feature was an open issue essentially ever since memory views were first introduced into Cython, back in 2009, but during all that time, no-one stepped forward to implement it. Alongside with this improvement, users can now write view[i][j] instead of view[i,j] if they want to, without the previous slow-down due to sub-view creation.

The second very long requested feature is the support for copying C code verbatimly into the generated files. The background is that some constructs, especially C hacks and macros, but also some adaptations to C specifics used by external libraries, can really only be done in plain C in order to use them from Cython. Previously, users had to create an external C header file to implement these things and then use a cdef extern from ... block to include the file from Cython code. Cython 0.28 now allows docstrings on these cdef extern blocks (with or without a specific header file name) that can contain arbitrary C/C++ code, for example:

cdef extern from *:
    """
    #define add1(i) ((i) + 1)
    """
    cdef int add1(int x) nogil

print(add1(x=2))

This is definitely considered an expert feature. Since the code is copied verbatimly into the generated C code file, Cython has no way to apply any validation or safety checks. Use at your own risk.

Another big new feature is the support for multiple inheritance from Python classes by extension types (a.k.a. cdef classes). Previously, extension types could only inherit from other natively implemented types, inlcuding builtins. While cdef classes still cannot inherit only from Python classes, and also cannot inherit from multiple cdef classes, it is now possible to use normal Python classes as additional base classes, following an extension type as primary base. This enables use cases like Python mixins, while still keeping up the efficient memory layout and the fast C-level access to attributes and cdef methods that cdef classes provide.

A bit of work has been done to start reducing the shared library size of Cython generated extension modules. In general, Cython aims to optimise its operations (especially Python operations) for speed and extensively uses C function inlining, optimistic code branches and type specialisations for that. However, the code in the module init function is really only executed once and rarely contains any loops, certainly not time critical ones. Therefore, Cython has now started to avoid certain code intensive optimisations inside of the module init code and also uses GCC pragmas to make the C compiler optimise this specific function for smaller size instead of speed. Without making the import visibly slower, this results in a certain reduction of the overall library size, but probably still leaves some space for future improvements.

Several new optimisations for Python builtins were implemented and often contributed by users. This includes faster operations and iteration for sets and bytearrays, from which existing code can benefit through simple recompilation. We are always happy to receive these contributions, and several tickets in the bug tracker are now marked as beginner friendly "first issues".

Cython has long supported f-strings, and the new release brings another set of little performance improvements for them. More interestingly, however, several common cases of unicode string %-formatting are now mapped to the f-string builder, as long as the argument side is a literal tuple. If the template string uses no unsupported formats, Cython applies this transformation automatically, which leads to visibly faster string formatting and avoids the intermediate creation of Python number objects and the value tuple. Existing code that makes use of %-formatting, including code in compiled Python .py files that needs to stay compatible with Python 2.x, can therefore benefit directly without rewriting all the template strings. Further coverage of formatting features for this transformation is certainly possible, and contributions are welcome.

Finally, a last minute change improves the handling of string literals that are being passed into C++ functions as std::string& references. Previously, the generated code always unpacked a Python byte string and made a fresh copy of it, whereas now Cython detects const arguments and passes the string literal directly. Also in the non-const case, Cython does not follow C++ in outright rejecting the literal argument at compile time, but instead just creates a writable copy and passes it into the function. This avoids special casing in user code and leads to working code by default, as expected in Python, and Cython.