A static Python compiler? What’s the point?

I’ve finally found the time to look through the talks of this year’s EuroPython (which I didn’t attend - I mean, Firenze? In plain summer? Seriously?). That made me stumble over a rather lengthy talk by Kay Hayen about his Nuitka compiler project. It took more than an hour, almost one and a half. I had to skip ahead through the video more than once. Certainly reminded me that it’s a good idea to keep my own talks short.

Apparently, there was a mixed reception of that talk. Some people seemed to be heavily impressed, others didn’t like it at all. According to the comments, Guido was more in the latter camp. I can understand that. The way Kay presented his project was not very convincing. The only “excuse” he had for its existence was basically “I do it in my spare time” and “I don’t like the alternatives”. In the stream of details that he presented, he completely failed to make the case for a static Python compiler at all. And Guido’s little remark in his keynote that “some people still try to do this” showed once again that this case must still be made.

So, what’s the problem with static Python compilers, compared to static compilers for other languages? Python can obviously be translated into static code, the mere fact that it can be interpreted shows that. Simply chaining all code that the interpreter executes will yield a static code representation. However, that doesn’t answer the question whether it’s worth doing. The interpreter in CPython is a much more compact piece of code than the result of such a translation would be, and it’s also much simpler. The trace pruning that HotPy does, according to another talk at the same conference, is a very good example for the complexity involved. The fact that ShedSkin and PyPy’s RPython do explicitly not try to implement the whole Python language speaks volumes. And the overhead of an additional compilation step is actually something that drives many people to use the Python interpreter in the first place. Static compilation is not a virtue. Thus, I would expect an excuse for writing a static translator from anyone who attempts it. The normal excuse that people bring forward is “because it’s faster”. Faster than interpretation.

Now, Python is a dynamic language, which makes static translation difficult already, but it’s a dynamic language where side-effects are the normal case rather than an exception. That means that static analysis and optimisation can never be as effective as runtime analysis and optimisation, not with a resonable effort. At least WPA (whole program analysis) would be required in order to make static optimisations as effective as runtime optimisations, but both ShedSkin and RPython make it clear that this can only be done for a limited subset of the language. And it obviously requires the whole program to be available at compile time, which is usually not the case, if only due to the excessive resource requirements of a WPA. PyPy is a great example, compiling its RPython sources takes tons of memory and a ridiculous amount of time.

That’s why I don’t think that “because it’s faster” catches it, not as plain as that. The case for a static compiler must be that “it solves a problem”. Cython does that. People don’t use Cython because it has such a great Python code optimiser. Plain, unmodified Python code compiled by Cython, while usually faster than interpretation in CPython, will often be slower and sometimes several times slower than what PyPy’s JIT driven optimiser gets out of it. No, people use Cython because it helps them solve a problem. Which is either that they want to connect to external non-Python libraries from Python code or that they want to be able to manually optimise their code, or both. It’s manual code optimisation and tuning where static compilers are great. Runtime optimisers can’t give you that and interpreters obviously won’t give you that either. The whole selling point of Cython is not that it will make Python code magically run fast all by itself, but that it allows users to tremendously expand the range of manual optimisations that they can apply to their Python code, up to the point where it’s no longer Python code but essentially C code in a Python-like syntax, or even plain C code that they interface with as if it was Python code. And this works completely seamlessly, without building new language barriers along the way.

So, the point is not that Cython is a static Python compiler, the point is that it is more than a Python compiler. It solves a problem in addition to just being a compiler. People have been trying to write static compilers for Python over and over again, but all of them fail to provide that additional feature that can make them useful to a broad audience. I don’t mind them doing that, having fun writing code is a perfectly valid reason to do it. But they shouldn’t expect others to start raving about the result, unless they can provide more than just static compilation.

14 Responses to “A static Python compiler? What’s the point?”

  1. Kay Hayen Says:

    Hello Stefan,

    thanks for your post.

    You might have noticed, that I myself don””t consider Nuitka all that useful yet. For one, the issues with Mercurial, have only been ruled out very recently. Running unmodified Mercurial test suite with success, seems in reach.

    Now, should one (unlikely, because Mercurial will be IO bound) day, Nuitka produce a Mercurial binary that runs faster, it may get used by people just because they like faster with tools that they use a lot. Of that I am pretty sure.

    I got e.g. a lot of interest from Android people. And I joined that camp recently too. Anything helpful that speeds up the whole Python module stack, would be very welcome. Of course it is, and only Nuitka aims for this.

    As for the rationale, there is plenty. Making good designs not slower than optimized code. That is something that matters to me. Being compatible, not making people wonder about the subtle differences. Having a speedup, but without a lockin. Being able to e.g. use PyPy or CPython as immediate replacements is great to have. Providing rich assertions without speed penalty and so on. Detecting unused code without code coverage tests. Detecting errors and mismatches.

    As for Guido not liking Nuitka, it”’’s true. I mean, he totally hated it. But I think that was more about lack of benchmarks, lack of focus on performance first (instead of compatibility, which he finds boring and stupid stuff), but less about lack of innovation when it comes to language. I think he would also like to see new research applied for a Python compiler. So yeah, if you ask me, he mostly hated it not being ready yet.

    I will just continue. Maybe it will remain useless. I don””t have many worries about that. But given all the progress I had with re-formulations recently, there is little doubt in my head, that it should be possible to catch up at least. And you need to point out, where I was asking anybody to rave about it. I didn””t. A few people do though.

    Yours,
    Kay

  2. Michael Says:

    To me some reasons for desiring static compilation seem obvious:
    1) ability to ship a runnable binary in a single executable file* (e.g. a Windows .exe file)
    2) to make it much harder to decompile the provided binary

    * I””m aware py2exe is supposed to do this, but it has never really worked on any moderately complex program. But this does infer that static compilation shouldn””t be strictly necessary, but that leaves out #2.

    The lack of this capability leans me toward believing the future belongs to something like go-lang.

    Thanks for a thought-provoking article.

  3. Stefan Behnel Says:

    @Michael:

    I agree with the goal to make a big binary of something. This can be done by freezing code, e.g. using py2exe or PyRun, but static compilation is another way of doing it. I””m not a big fan of obfuscation, though.

    I also fail to see why moving to “go” would change anything, substantially. It”’’s just a different language.

  4. Stefan Behnel Says:

    @Kay Hayen:

    Sure, go ahead. As I said, I don””t mind people having fun doing what they do.

    However, the way I see it, Nuitka is still way behind Cython in terms of usability. It might be a tad better in terms of Python compatibility because you put a lot of investment there, so it may or may not be easier to compile existing Python code without modifications in Nuitka than in Cython (I never tried Nuitka here, it works pretty well in Cython these days). But as I said, Nuitka, like many other compiler projects, currently fails to provide more than that. Just being able to compile code instead of interpreting it doesn””t buy anything, really. It just makes things harder to use, and often substantially so. I keep telling people that: static compilation is not a virtue. Use it when appropriate, but don””t use it unless you understand the implications and know that you have to. Premature optimisation is the root of all evil.

    BTW, I hope you are aware that bold statements like “only Nuitka aims for speeding up the whole Python module stack” are rather misplaced.

  5. Klaus Ries Says:

    Being able to ship Python Applikations is a problem, especially on HW restricted platforms (Smartphones or Embedded devices). Lua is currently the language of choice since the overall runtime is small enough to ship and speed is high. I wish we had Python in this space.

    Even in larger applications the current py2exe can be a pain for developers and users alike.

    However yet another static compiler drives the fragmentation of the python community.

  6. Stefan Behnel Says:

    @Klaus Ries:

    I agree with the fragmentation issue, maybe that”’’s my biggest problem with this in fact.

    PyRun is supposed to fix some issues with py2exe. You might want to take a look at it.

    And, yes, Lua has a head start in certain environments. It”’’s not only the runtime that”’’s small, though, it”’’s also the language. You pay very little in terms of resources, but you also only get what you””ve paid for. Using Lua means that you””re basically on your own, you have to build up everything yourself. And if you””re unlucky, others have built things differently and you can””t even exchange code with them because it uses different libraries and what not. Sure, Python would be much nicer to use here. There has been a substantial move towards that goal BTW, e.g. through kivi and friends. We””ll see how it turns out.

  7. Michael Says:

    @Stefan

    Thanks for the pointer to PyRun, don””t think I was aware of it. But their website says several times “supports *most* Python application and scripts”. So it is not complete either. And as I said, py2exe just doesn””t work. So, to be fair, these must be lumped in the same category as Nuitka.

    Go lang is obviously a fully different language, but as a python devotee I see lots to like there. And the continued popularity of C++ shows there is still a strong need for a static compiled modern high-performance language. D could also be a possibility but it is unlikely to ever get much mindshare. Go, with Google behind it, could.

    I””d just prefer Python take that spot.

  8. Kay Hayen Says:

    Hello Stefan,

    here I was leaving PyPy out of the equation, as it”’’s off topic in this post about static compilation, and they don””t really mean to support RPython anymore for others to use. As you know, PyPy really impresses me. Of course it aims at the whole Python stack too. Otherwise I am unaware of anybody else though. Does Cython nowadays aim at something as running Mercurial unmodified? There would be a bunch of things missing or so I was informed.

    If fCython does, excuse my statement as merely uninformed. I recall something to that extent, maybe even from you personally, but that may be wrong. And goals of course can change. Last time I checked, people were still focusing on creating extension modules, and not on speeding up whole programs. If that changed, great.

    Fully compatible implementations, be it Jython, PyPy, IronPython, or Nuitka, they do not cause a fragmentation issue, but only choice of implementation. I don””t want to enter a discussion, where I detail how I believe fragmentation is caused. I perfectly see the merits of Cython, so I don””t have to argue against it. But please don””t throw stones out of your fragmented community part. ;-)

    Yours,
    Kay

  9. zaur Says:

    IMHO, we really need optional static typing protocol for python program in order to make them effectively compiled into performant code.

  10. zaur Says:

    IMHO, If we like to see python programs compiled into performant code then we really need optional static typing in python.

  11. Stefan Behnel Says:

    @Kay Hayen

    It”’’s not about changing goals. There are different ways to achieve what users want, and Cython is tailored to user requirements. I think I was pretty clear in my original post as to what the goals are (and should be).

    Stefan

  12. Stefan Behnel Says:

    @zaur

    We already have optional static typing. But it is only needed under the assumption that you actually want to statically compile the code, which is not a requirement by itself and not even a good thing all by itself. I hope I was clear enough about that in my post.

    Stefan

  13. fijal Says:

    Hi Kay.

    As far as I remember Guido”’’s issues were mostly around the fact that you failed to convey the message how you””re going to tackle hard problems (like globals changing under your feet in a way that you cannot predict) without incurring massive performance penalty. That was his main concern - that you keep saying it”’’s “normal compiler technology that”’’s well known” instead of really trying to attack the hard part.

    As for mercurial, it”’’s definitely not I/O bound. Run some benchmarks.

  14. Kay Hayen Says:

    @fiijal: Well, just for the record, if the compiler cannot statically prove that it”’’s not possible, it will have guards that make it fall back to less optimized code variants that are still correct.

    There isn””t much choice to be had about this, so I don””t think that there was or could be any argument around such an issue. The PyPy JIT e.g. does the same, and it”’’s quite obvious that they do not incurr much of a performance penalty from it, do they?

    The hope is that given sufficient analysis and hints, it won””t happen at important places and won””t need guards.

    What one can do is to e.g. have a declaration that says “globals cannot be set from outside” which will cause the module to be wrapped with something that gives an error when setting attributes on it, and hand that to people importing it.

    The difficulty will be to write this so it works for CPython, which probably means to plug __import__ early on, and then Nuitka can hard code this knowledge. Maybe I am only dreaming, but it could realize that the module variables are indeed not accessible outside from the hints source code.

    Same for class decorators of the kind “@hints.nomonkeypatching” and so on. But I didn””t spend much thought on that yet. I have to beg your pardon. To me, this issue is not yet on the table, for as long as local type inference is not done at all yet.

    With due time, I will address these issues. For this evening though, I am hunting down, one error that only looked like a false alarm in Mercurial tests, but turns out to be an actual Nuitka incompatibility. Because for one thing, “wrong execution” is not allowed.

    But yeah, Guido also complained that I write too little interesting things up. In this last release cycle, so many thing happened, and I only managed to write up C+11 vs. C++03. I will probably issue a public response to this post as well.

    Personally I consider it entirely premature to discuss this, before I even start to actually try and do anything. Talk is cheap, code is golden, remember that one. And talking about how Nuitka in theory could optimize, while it doesn””t optimize much at all yet, seems inappropiate.

    Yours,
    Kay

Leave a Reply