While I also provide numbers regarding the memory usage of each library in this comparison, they are not directly comparable between PyPy and CPython because of the different memory management of both platforms and because the overall memory that PyPy uses right from the start is much larger than for CPython. So the relative increase in memory may or may not be an accurate way to tell what each runtime does with the memory. However, it appears that PyPy manages to kill at least the severe memory problems of MiniDOM, as the total amount of memory used for the larger files is several times smaller than that used by CPython.

So, what do I take from this benchmark? If you have legacy MiniDOM code lying around, you want PyPy to run it. It exhibits several times better performance in terms of memory and runtime. It also performs substantially better for ElementTree than the plain Python ElementTree in CPython.

However, for fast XML processing in general, the better performance of PyPy even for plain Python ElementTree is not really all that interesting, because it is still several times slower than cElementTree or lxml.etree in CPython. That means that you will often be able to process multiple files in CPython in the time that you need for just one in PyPy, even if your actual application code that does the processing manages to get a substantial JIT speed-up in PyPy. Even worse, the GIL in PyPy will keep your code from getting a parallel speedup that you usually get with multi-threaded processing in lxml and CPython, e.g. in a web server setting.

So, as always, the decision depends on what your actual application does and which library it uses. Do your own benchmarks.

Parsing hamlet.xml in English, 274KB:

Memory usage: 7284
xml.etree.ElementTree.parse done in 0.104 seconds
Memory usage: 14240 (+6956)
xml.etree.cElementTree.parse done in 0.022 seconds
Memory usage: 9736 (+2452)
lxml.etree.parse done in 0.014 seconds
Memory usage: 11028 (+3744)
minidom tree read in 0.152 seconds
Memory usage: 30360 (+23076)

Parsing the old testament in English (ot.xml, 3.4MB) into memory:

Memory usage: 20444
xml.etree.ElementTree.parse done in 0.385 seconds
Memory usage: 46088 (+25644)
xml.etree.cElementTree.parse done in 0.056 seconds
Memory usage: 32628 (+12184)
lxml.etree.parse done in 0.041 seconds
Memory usage: 37500 (+17056)
minidom tree read in 0.672 seconds
Memory usage: 110428 (+89984)

A 25MB XML file with Slavic Unicode text content:

Memory usage: 57368
xml.etree.ElementTree.parse done in 3.274 seconds
Memory usage: 223720 (+166352)
xml.etree.cElementTree.parse done in 0.459 seconds
Memory usage: 154012 (+96644)
lxml.etree.parse done in 0.454 seconds
Memory usage: 135720 (+78352)
minidom tree read in 6.193 seconds
Memory usage: 604860 (+547492)

And a contrived 4.5MB XML file with a lot more structure than data and no whitespace at all:

Memory usage: 11600
xml.etree.ElementTree.parse done in 3.374 seconds
Memory usage: 203420 (+191820)
xml.etree.cElementTree.parse done in 0.192 seconds
Memory usage: 36444 (+24844)
lxml.etree.parse done in 0.131 seconds
Memory usage: 62648 (+51048)
minidom tree read in 5.935 seconds
Memory usage: 527684 (+516084)

I also took the last file and pretty printed it, thus adding lots of indentation whitespace that increased the file size to 6.2MB. Here are the numbers for that:

Memory usage: 13308
xml.etree.ElementTree.parse done in 4.178 seconds
Memory usage: 222088 (+208780)
xml.etree.cElementTree.parse done in 0.478 seconds
Memory usage: 103056 (+89748)
lxml.etree.parse done in 0.199 seconds
Memory usage: 101860 (+88552)
minidom tree read in 8.705 seconds
Memory usage: 810964 (+797656)

Yes, 2MB of whitespace account for almost 300MB more memory in MiniDOM.

Here are the graphs:

I think it is pretty clear that minidom has basically left the scale, whereas cElementTree and lxml.etree are pretty close to each other. lxml tends to be a tad faster, and cElementTree tends to use a little less memory.

Eines der wichtigsten Ergebnisse der Konferenz war die Gründung des Python Software Verband e.V. als Nachfolger der ehemals Zope-spezifischen DZUG. Die Neuausrichtung wird es wesentlich erleichtern, die deutschsprachige Python-Gemeinde unter ein gemeinsames Dach zu bringen, und die Python-Lobby in Deutschland, Österreich und der Schweiz zu verbessern.

Ich selbst habe zwei Vorträge über Cython und lxml gehalten, sowie ein Tutorial zu Cython. Alle wurden mit großem Interesse aufgenommen (wobei ich noch auf die konkreten Rückmeldungen zum Tutorial warte) und gaben Anlass zu einigen interessanten Diskussionen. Cython und lxml sind weiterhin zwei Best-of-Breed Tools, und große Themen in der Python-Gemeinde. Besonders lxml hat mir einiges an Schulterklopfen dafür eingebracht, dass ich es in den letzten Jahren zu dem einen großen XML-Tool für Python gemacht habe. Paul Everitt, der eine Keynote hielt und den ich eigentlich immer schon mal treffen wollte (hiermit passiert), hat sogar mitten in seinem Vortrag eine Riesenfolie aufgelegt, auf der nur zwei Namen standen - Martijn Faassen (der mit lxml angefangen hat) und ich. Werde ich also doch noch berühmt auf meine alten Tage …

Einige Zeit habe ich mit Kay Hayen diskutiert, der an einem statischen Python-Compiler namens Nuitka schreibt. Wenig überraschend stand er dabei vor etlichen der Probleme, in die wir auch mit Cython gelaufen sind. Er hat Recht damit, dass ich nicht gänzlich froh darüber bin, dass er ein separates Projekt begonnen hat anstatt uns mit Cython zu helfen, aber so ist OpenSource nun einmal. Jede/r hat das Recht, so viele Räder zu erfinden wie es Spaß macht. Soweit ich es verstehe, strebt Kay mit Nuitka eine Untermenge von dem an was wir aus Cython machen, aber kommt dabei von der anderen Seite. Cython war früher nur eine Erweiterungssprache und entwickelt sich nun zusätzlich zu einem vollwertigen Python-Compiler, wohingegen Nuitka einzig und allein die Nische des Python-Compilers füllen soll. Aber bisher hat Kay schon dabei einiges an Chuzpe und Durchhaltevermögen bewiesen, da warten vielleicht noch ein paar Überraschungen…

Es war interessant, einige Vorträge zu Themen zu sehen, mit denen sich auch mein Arbeitgeber herumschlägt - nur eben mit Python statt mit Java. So arbeitet beispielsweise eine interne Abteilung bei SAP an einer Web-basierten Client-Infrastruktur für SAP-Systeme in Python, inklusive Objekt-nach-SAP Mapper (ähnlich einem ORM), Offline-Caching-Mechanismen und so weiter. Im Vortrag sah es ganz so aus als könnte sich das allgemein als interessant für SAP-Clients erweisen, auch ganz unabhängig von Web-Anwendungen. Und es könnte bald schon OpenSource sein…

Ein weiterer Vortrag, bei dem ich mich gleich heimisch gefühlt habe, handelte von PyTAF, einem grafischen Framework zur Anwendungsintegration. Es wird intern bei der LBBW in Stuttgart entwickelt und erreicht mehr oder weniger das, was wir in Java machen. Darüber hinaus hat es ein GUI mit dem Datenintegrationsprozesse grafisch zusammengesetzt werden können, und es ist in Python geschrieben, was bei dieser Art von Software ein ernst zu nehmender Vorteil ist. Übrigens verwendet es intern lxml.objectify zur Datenverarbeitung - eine sehr gute Wahl

Die PyCon-DE im nächsten Jahr könnte ruhig wieder am selben Ort stattfinden. So gut, wie die diesjährige funktioniert hat, gibt es eigentlich keinen Grund zu wechseln. Obwohl sowas wie Berlin natürlich auch immer eine Reise wert ist …

One of the major outcomes was the formation of the “Python Software Verband e.V.” as a successor to the previous Zope centered “DZUG e.V.”. The new direction will make it much easier to gather the German speaking Python community under a common umbrella, and to enforce the Python lobbying in Germany, Austria and Switzerland.

I gave two talks on Cython and lxml, as well as a tutorial on Cython. All of them were well received (although I”m still waiting for the final feedback on the tutorial) and gave the chance for interesting discussions. Both Cython and lxml continue to be best of breed tools and hot topics in the community, and I received a lot of backslapping for making lxml the one great XML tool for Python over the last few years. One of the keynote speakers, Paul Everitt, whom I wanted to meet for a while until I finally got the chance now, even put up a huge slide right in his talk with only two names on it, that of Martijn Faassen (the original author of lxml) and mine. I”m finally getting famous.

I spent some time talking to Kay Hayen, who has written a static Python compiler called Nuitka. It was not surprising that he bumped into a lot of the problems that we met with Cython as well. He’’s right in that I”m not entirely happy about the fact that he started a completely separate project instead of helping with Cython, but that’’s OpenSource. People are free to reinvent as many wheels as they like. From what I understand, Nuitka aims to become a subset of what Cython heads for, just coming from a different side. Cython has originally been an extension language and is now additionally evolving into a Python compiler, whereas Nuitka is plainly targeted at being a Python compiler. But I wouldn”t mind getting surprised at some point. So far, Kay has certainly shown a remarkable investment and was pretty successful.

It was nice to see in a couple of presentations that the kind of things that the company I currently work for is doing in Java is done in Python in other places. For example, an internal department at SAP is developping a Web based client infrastructure for SAP systems in Python, including a transparent object-to-SAP mapper (similar to ORMs), offline caching mechanisms, etc. From the presentation, it sounded very much like this could be useful for talking to SAP in general, not only for web clients. And it may become open source at some point.

Another feel-alike talk was about PyTAF, a graphical application integration framework for financial applications that is being developped in-house at LBBW in Stuttgart. It aims to do more or less the same as the code we write in Java, but has a graphical frontend for putting together integration flows. And, it’’s Python, which is a serious advantage for this kind of software. It even uses lxml.objectify internally for data processing - best choice ever!

It may well be that next year’’s PyCon-DE will take place at the same location. It worked so well that there’’s no reason for a change. Although Berlin would also be a great location…

Something as trivial as chain() is exactly as fast as in the C implementation, but compared to the more than 60 lines of C code, it is certainly a lot more readable in Cython:

def chain(*iterables):
    """Make an iterator that returns elements from the first iterable
    until it is exhausted, then proceeds to the next iterable, until
    all of the iterables are exhausted. Used for treating consecutive
    sequences as a single sequence.
    """
    for it in iterables:
        for element in it:
            yield element

Other functions, like islice(), are faster in C, partly because CPython actually takes a couple of shortcuts, e.g. by only looking up the iterator slot method once. You cannot do that in Python code, and I wanted to keep the implementation compatible with regular Python. Specifically, the C speed advantage for islice() is currently about 30-50% in general, although the Cython implementation can also be up to 10% faster for some cases, e.g. when extracting only a couple of items from the middle of a longer sequence. The C implementation is about 90 lines, here is the Cython implementation:

import sys
import cython

# Python 2/3 compatibility
_max_size = cython.declare(cython.Py_ssize_t,
        getattr(sys, "maxsize", getattr(sys, "maxint", None)))

@cython.locals(i=cython.Py_ssize_t, nexti=cython.Py_ssize_t,
               start=cython.Py_ssize_t, stop=cython.Py_ssize_t, step=cython.Py_ssize_t)
def islice(iterable, *args):
    """Make an iterator that returns selected elements from the
    iterable. If start is non-zero, then elements from the iterable
    are skipped until start is reached. Afterward, elements are
    returned consecutively unless step is set higher than one which
    results in items being skipped. If stop is None, then iteration
    continues until the iterator is exhausted, if at all; otherwise,
    it stops at the specified position. Unlike regular slicing,
    islice() does not support negative values for start, stop, or
    step. Can be used to extract related fields from data where the
    internal structure has been flattened (for example, a multi-line
    report may list a name field on every third line).
    """
    s = slice(*args)
    start = s.start or 0
    stop = s.stop or _max_size
    step = s.step or 1
    if start < 0:
        raise ValueError("...")
    if step < 1:
        raise ValueError("...")
    if start >= stop:
        return
    nexti = start
    for i, element in enumerate(iterable):
        if i == nexti:
            yield element
            nexti += step
            if nexti >= stop or nexti < 0:
                return

Here is one that is conceptually quite simple: count(). I had to optimise it quite a bit, because the iteration code in the C code is extremely tight. Even the tuned version below runs about 10% slower than the hand tuned C version, which is about 230 lines long.

@cython.locals(i=cython.Py_ssize_t)
def count(n=0):
    """Make an iterator that returns consecutive integers starting
    with n. If not specified n defaults to zero. Often used as an argument to imap()
    to generate consecutive data points. Also, used with zip() to add
    sequence numbers.
    """
    try:
        i = n
    except OverflowError:
        i = _max_size # skip i-loop
    else:
        n = _max_size # first value after i-loop
    while i < _max_size:
        yield i
        i += 1
    while True:
        yield n
        n += 1

Note that all of the above generators execute in the order of microseconds, so even a slow-down of 50% will likely not be measurable in real world code.

So far, I did not try any of the more fancy functions in itertools (those that actually do something). The Cython project has announced a Google Summer of Code project with exactly the intent to rewrite some of the C stdlib modules of CPython in pure Python code with Cython compiler hints. So I leave this exercise to interested readers for now.

Here is a simple way to configure mod_rewrite to serve the compressed pages even if you do not have mod_deflate available.

First, compress all your HTML pages, e.g. using

find -name "*.html" | while read file; do gzip -9c < $file > $file.gz; done

Since I am using make to handle my web site, here is an extract that helps me keeping all compressed files updated when I upload the pages (also in parallel, when I pass e.g. “-j 5″):

TEXT_FILES=$(shell find -name "*.html" -o -name "*.css" -o -name "*.js")

.PHONY: copy

copy: $(addsuffix .gz, $(TEXT_FILES))
	copy_website.sh

%.gz: %
	gzip -9c $< > $@

Now we can configure mod_rewrite to prefer these compressed files for clients that support it. To do this, I put the following into my .htaccess file:

RewriteCond %{REQUEST_FILENAME}.gz -f
RewriteCond %{HTTP:Accept-Encoding}   .*gzip.*
RewriteRule ^(.*[.])(html|js|css)$        $1$2.gz      [L]

To spell this out:

1. check if there really is a compressed version of the requested file available (”-f” tests for the path being a file).
2. check if the client tells us that it accepts “gzip” compressed content
3. if both conditions hold, redirect the client to the compressed file version.

Given that the compressed files are often 5x smaller than the plain HTML version, this saves lots of bandwidth from my web site with really little effort.