Playing with PyPy
I've been following the PyPy project since I first heard of it in 2003 or so. The concept behind it is fascinating; it's a Python interpreter written in (a subset of) Python. It's actually a lot more than that because the language front-ends (e.g. Python) are quite separate from the backends (e.g. C, JVM, CLI, Python). This makes it a unique platform for language research because coding Python is typically easier than C, and so much of the work is already done for you.
It's clear that PyPy is very useful for academic research, but it's also quickly becoming a practical target for developing and deploying Python code. At the PyPy Speed Center you can see that it's already several times faster than CPython, and has the potential to fix most of the more fundamental flaws of the CPython VM.
What's awesome right now:
- PyPy has a modern garbage collector, not ref counting
- PyPy's JIT can run string mangling and numerics code very quickly, which removes the need for most C extensions
- PyPy is already fast, and is getting faster all the time
How it can be more awesome (just my opinion, I don't speak for the PyPy team and their implementation goals):
- PyPy has an alpha quality cpyext that will allow you to use CPython extensions (requires a recompile), and when that's polished it will be very easy for CPython users to migrate en masse, even though they may have complicated dependencies such as NumPy, SciPy, PIL, etc.
- PyPy has the potential to eventually remove the GIL, and/or have multiple VMs in the same OS process
- PyPy could add M:N threading and concurrency constructs to the language (some stackless support already exists, but is not currently compatible with the JIT and doesn't take advantage of multiple cores)
- PyPy could simultaneously support Python 2.x and 3.x code in the same process, making it practical to actually make the transition (note: this is a crazy idea that would be terribly difficult)
Playing with PyPy
I've been working on helping the PyPy team with some real world benchmarks for JSON, and helping sort out Mac OS X issues. I've also been tuning a branch of simplejson to run efficiently on PyPy. I'll write more about this in a follow up post, but here's how you can get started.
If you're a library author or an advanced user you should be experimenting with PyPy right now. In these instructions we'll install PyPy in ~/opt and create a virtualenv for it in ~/virtualenv.
Prerequisites
Install Mercurial 1.7 or newer.
Install Xcode 3.2.x (gcc-4.0 is currently required for building PyPy).
Download virtualenv 1.5.2 (or later):
mkdir -p ~/src (cd ~/src; curl -s http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.5.2.tar.gz | tar zxf -)
Clone jitviewer and pypy:
# Make a ~/src to store all of these clones mkdir -p ~/src # Get the PyPy jitviewer application (install to ~/src/jitviewer) (cd ~/src; hg clone https://bitbucket.org/pypy/jitviewer) # Clone PyPy source, this will take a while (cd ~/src; hg clone https://bitbucket.org/pypy/pypy)
Installing from binary
Follow the link on http://pypy.org/download.html to download one of the Mac OS X binaries (either 32-bit or 64-bit), the current version at this time is 1.4.1.
This will install PyPy to ~/opt and create a virtualenv:
# Unpack PyPy and "install" to ``~/opt``
mkdir -p ~/opt
cd ~/opt
tar jxvf ~/Downloads/pypy-1.4.1-osx64.tar.bz2
# Install virtualenv to this PyPy build, and create a new virtualenv.
# I also like to create a symlink ``~/virtualenv/pypy-env`` to the
# version I am currently working with::
PKG=pypy-1.4.1-osx64
PYPY=~/opt/$PKG/bin/pypy
# install virtualenv
(cd ~/src/virtualenv-1.5.2; $PYPY setup.py install)
# create virtualenv
mkdir -p ~/virtualenv
rm -rf ~/virtualenv/$PKG
~/opt/$PKG/bin/virtualenv --distribute ~/virtualenv/$PKG
# update symlink
(cd ~/virtualenv; rm -f pypy-env; ln -s $PKG pypy-env)
# install jitviewer
(source ~/virtualenv/pypy-env/bin/activate; \
pip install flask pygments simplejson; \
cd ~/src/jitviewer; pypy setup.py develop )
When you want to use PyPy, just activate the virtualenv:
source ~/virtualenv/pypy-env/bin/activate # now you can use PyPy! both "python" and "pypy" will work
Tuning for PyPy 1.4.1
On Mac OS X, PyPy 1.4.1 (and current default) does not choose optimal tuning values for the GC. You will get ~30% better performance by setting this environment variable:
export PYPY_GC_NURSERY=1M
Note that 1M is a machine specific value, so if your Mac isn't the same model as mine there might be a better default for you. The value is very likely to be dependent on the amount of L2/L3 cache and how many physical cores you have, and you can get those values from sysctl:
$ sysctl hw.l3cachesize hw.l2cachesize hw.physicalcpu hw.l3cachesize: 4194304 hw.l2cachesize: 262144 hw.physicalcpu: 2
From the pypy source directory, with a pypy virtualenv activated, you can run this script to see what a good value might be (lowest time is best):
#!/bin/bash
for ((procs=1; procs <= 4 ; procs++)); do
for ram in 128K 256K 512K 768K 1M 2M 3M 4M; do
echo "export PYPY_GC_NURSERY=$ram # procs=$procs"
export PYPY_GC_NURSERY=$ram
for ((p=1; p <= $procs; p++)); do
(cd pypy/translator/goal; pypy gcbench.py | grep 'Completed in') &
done
wait
done
done
The PyPy team is very interested in knowing what the sysctl values are for your machine and the output of the GC benchmark, so if you get this far please send it along to me or the PyPy mailing list! Having output from many different models of Mac will help us come up with a better algorithm for choosing sane defaults.
Building PyPy from source
Make sure to install a binary first. Since translating PyPy is CPU bound, this runs a lot faster if you use PyPy.
These commands will build PyPy, create a release based on the hg revision, update ~/virtualenv/pypy-env, etc.:
# Translate PyPy (expect this a while, at least an hour for me) (cd pypy/translator/goal; pypy translate.py -Ojit) # Build the release BRANCH=$(hg branch) PKG=pypy-$(hg branches|grep "^$BRANCH " | cut -d: -f2)-osx64 mkdir -p ~/opt (cd pypy/tool/release; /usr/bin/python package.py ../../.. $PKG) rm -rf ~/opt/$PKG mv $TMPDIR/usession-$BRANCH-$USER/build/$PKG ~/opt/$PKG # install virtualenv PYPY=~/opt/$PKG/bin/pypy (cd ~/src/virtualenv-1.5.2; $PYPY setup.py install) # create virtualenv mkdir -p ~/virtualenv rm -rf ~/virtualenv/$PKG ~/opt/$PKG/bin/virtualenv --distribute ~/virtualenv/$PKG # make default (cd ~/virtualenv; rm -f pypy-env; ln -s $PKG pypy-env) # install jitviewer (source ~/virtualenv/pypy-env/bin/activate; \ pip install flask pygments simplejson; \ cd ~/src/jitviewer; pypy setup.py develop )
Running jitviewer
jitviewer is an awesome web app for reading PyPy logs, it will help you optimize your code for PyPy (once you have a basic understanding of the output, which is beyond the scope of this post).
Run your code with JIT logging turned on:
# log to pypy-jit.log PYPYLOG=jit-log-opt,jit-backend-counts:pypy-jit.log pypy benchmark.py # start the jitviewer server with pypy-jit.log PYTHONPATH=~/src/pypy jitviewer.py pypy-jit.log
After jitviewer is started, open a web browser to http://127.0.0.1:5000/