Writing C code is a premature optimisation

It seems I can’t repeat this often enough. People who write Python wrappers for libraries in plain C “because it’s faster” tend to overestimate their C-API skills and simply have no idea how costly maintenance is. It’s like the old advice about optimisation: Don’t do it! (and, if you’re an expert: Don’t do it yet!). If you write your wrapper code in C instead of Cython, it will be

  • slower
  • less portable
  • harder to maintain
  • harder to extend
  • harder to optimise
  • harder to debug and fix

It will cost you a lot of effort, both short term and long term, that is much better spent in adding cool features and optimising the performance critical parts of your code once you got it working. Say, is your time really so cheap that you want to waste it writing C code?

7 Responses to “Writing C code is a premature optimisation”

  1. Koen Says:

    Right now your argument is based on authority. How about showing an example of wrapping some C code both ways? And pointing out some of your 6 points?

  2. Michael Droettboom Says:

    Cython is only easier to debug until something goes wrong and you have to walk through the generated code in gdb. C is a well understood transformation with lots of very mature debug tools.

    As for speed — do you have numbers or references to back that up?

  3. Stefan Behnel Says:

    @Michael Droettboom:
    Cython has source level debugging these days, actually multi-level source debugging for Python, Cython and C code, all through the same tool (which happens to be gdb, BTW).

    As for speed, I faintly recall that a thin Cython generated wrapper that I tried when I wrote the optimised argument unpacking code in Cython was somewhere around 30% faster than the corresponding “normal” C-API code that most users would write. But seriously, you shouldn””t believe me and instead come up with your own numbers. Try to prove me wrong.

  4. Stefan Behnel Says:

    @Koen: How about starting to read up on Cython? Maybe start with the documentation, or some of the blog posts that we link to from our main web site? I think saying things based on authority is fine as long as they match with experience (especially when it”’’s not just your own one).

  5. Ben Bangert Says:

    @Stefan Behnel: Sorry, your presenting an argument, its up to you to prove yourself right, not for others to prove you wrong. The burden of proof lies on the one making the claim. Otherwise I could just make the opposite argument and then ask you to prove me wrong and we””d get nowhere.

    People asking for some examples is totally fair, especially when large well-used Python libraries *do* include C extensions that make them faster (numpy is an obvious example).

  6. Stefan Behnel Says:

    @Ben Bangert

    I””ve seen a lot of such libraries and I actually see more and more of them switch to Cython by reimplementing parts of their code base. NumPy is just one example here, there are many, many others. From user stories, almost everyone who speaks up is happy about doing that switch, although some lament that if they had known about (and learned) Cython earlier, they would have saved a lot of time and would have had a lot more fun writing their code. At least they tend to have fun rewriting it at that point and that often leads to a better code base by getting rid of existing bugs and adding new features that were previously hard to implement in C. So it wasn””t entirely a waste of time for them.

    If my post can get a couple of more people to switch a bit earlier and thus save some more of their time and allow them to write better code, I think it”’’s already worth it. That”’’s the difference to you “making the opposite argument”, by the way: that wouldn””t help anyone. However, as the comments show, not everyone is ready yet. Some still seem to prefer the warm and fuzzy feeling of the cat-o””-nine-tails when they do their work.

    I didn””t give a proof for my arguments, sure, they are plain experience, both my own and from user reports. But you can””t seriously expect me to present new examples each time I mention the advantages of replacing C by Cython+Python. I think that”’’s what the documentation is for (hint, hint). Feel free to ask any of our happy users instead, or take the time to read one of their blog posts. We link to some of them on the Cython project home page. You can also read some more of my own blog posts on this topic.

  7. Dima Q Says:

    I agree with your thesis wholeheartedly.
    Indeed, why manually do the work a tool can do for you?

    Although what it takes to distribute an extension written in Cython is still not clear to me, I reckon it”’’s always a subset of the effort required to distribute an oldschool extension. After all a developer is free to run cython locally and distribute only the resulting C file.

Leave a Reply