Threads, processes and concurrency in Python: some thoughts

Threads, processes and concurrency in Python: some thoughts

I attended the EuroPython conference in Birmingham last week. Nice place and nice meeting overall. There were lots of interesting talks on many subjects. I want to focus on the talks about concurrency here. We had a keynote by Russel Winder about the "multicore revolution" and various talks about different approaches to concurrency (Python-CSP, Twisted, stackless, etc). Since this is a hot topic in Python (and in other languages) and everybody wants to have his saying, I will take the occasion to make a comment.

The multicore non revolution

First of all, I want to say that I believe in the multicore non revolution: I claim that essentially nothing will change for the average programmer with the advent of multicore machines. Actually, the multicore machines are already here and you can already see that nothing has changed.

For instance, I am interacting with my database just as before: yes, internally the database may have support for multiple cores, it may be able to perform parallel restore and other neat tricks, but as a programmer I do not see any difference in my day to day SQL programming, except (hopefully) on the performance side.

I am also writing my web application as before the revolution: perhaps internally my web server is using processes and not threads, but I do not see any difference at the web framework user level. Ditto if I am writing a desktop application: the GUI framework provides a way to launch processes or threads in the background: I just perform the high level calls and I not fiddle with locks.

At work we have a Linux cluster with hundreds of CPUs, running thousands of processes per day in parallel: still, all of the complication of scheduling and load balancing is managed by the Grid engine, and what we write is just single threaded code interacting with a database. The multicore revolution did not change anything for the way we code. On the other extreme of the spectrum, people developing for embedded platforms will just keep using platform-specific mechanisms.

The only programmers that (perhaps) may see a difference are scientific programmers, or people writing games, but they are a minority of the programmers out there. Besides, they already know how to write parallel programs, since in the scientific community people have discussed parallelization for thirty years, so no revolution for them either.

Read more: The Explorer

Posted via email from .NET Info