Building Python 2.5 (and 2.4, and 2.6, and 2.7, and 3.2) on Snow Leopard

The Problem

No amount of autotools-fu could get either Python 2.4 or 2.5 satisfactory built for Mark Holder and myself on Snow Leopard. Setting a slew of environmental variables, hacking the ./configure script, the ./configure.in, or the generated Make file all yielded some manner of progress, but all ultimately ended in failure.

Here is what we initially tried and the results ...

1. We downloaded and unarchived the Python 2.5.6 source, followed by:

   $ ./configure --prefix=/opt/python-2.5
   $ make
   

   which resulted in:

   gcc -c -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
   cc1: error: unrecognized command line option "-Wno-long-double"
   make: *** [Modules/python.o] Error 1
   

2. Setting some environmental variables got us past the above:

   $ MACOSX_DEPLOYMENT_TARGET=10.6 CC=gcc-4.0 ./configure --prefix=/opt/python-2.5
   $ make
   

   but resulted in:

   ...
   ...
   ...
   ./Modules/posixmodule.c: In function 'wait_helper':
   ./Modules/posixmodule.c:5550: error: 'struct rusage' has no member named 'ru_maxrss'
   ./Modules/posixmodule.c:5551: error: 'struct rusage' has no member named 'ru_ixrss'
   ./Modules/posixmodule.c:5552: error: 'struct rusage' has no member named 'ru_idrss'
   ./Modules/posixmodule.c:5553: error: 'struct rusage' has no member named 'ru_isrss'
   ./Modules/posixmodule.c:5554: error: 'struct rusage' has no member named 'ru_minflt'
   ./Modules/posixmodule.c:5555: error: 'struct rusage' has no member named 'ru_majflt'
   ./Modules/posixmodule.c:5556: error: 'struct rusage' has no member named 'ru_nswap'
   ./Modules/posixmodule.c:5557: error: 'struct rusage' has no member named 'ru_inblock'
   ./Modules/posixmodule.c:5558: error: 'struct rusage' has no member named 'ru_oublock'
   ./Modules/posixmodule.c:5559: error: 'struct rusage' has no member named 'ru_msgsnd'
   ./Modules/posixmodule.c:5560: error: 'struct rusage' has no member named 'ru_msgrcv'
   ./Modules/posixmodule.c:5561: error: 'struct rusage' has no member named 'ru_nsignals'
   ./Modules/posixmodule.c:5562: error: 'struct rusage' has no member named 'ru_nvcsw'
   ./Modules/posixmodule.c:5563: error: 'struct rusage' has no member named 'ru_nivcsw'
   ./Modules/posixmodule.c: In function 'posix_wait3':
   ./Modules/posixmodule.c:5592: warning: implicit declaration of function 'wait3'
   ./Modules/posixmodule.c: In function 'posix_wait4':
   ./Modules/posixmodule.c:5616: warning: implicit declaration of function 'wait4'
   ./Modules/posixmodule.c: In function 'posix_getloadavg':
   ./Modules/posixmodule.c:7963: warning: implicit declaration of function 'getloadavg'
   make: *** [Modules/posixmodule.o] Error 1
   

3. In the end, we told GCC that some system functions were not available (when they really were) by hacking the auto-generated "config.h" to change:

   #define HAVE_WAIT3
   #define HAVE_WAIT4
   

   to:

   #undef HAVE_WAIT3
   #undef HAVE_WAIT4
   

   This did result in a successful compile. Turns out that the branch of the code that that is compiled if the wait3 and wait4 functions are available references members of struct rusage that are not defined in Snow Leopard's version of the system library (but presumably are on other operating systems, including older versions of OS X). The struct itself is defined in Snow Leopard's system headers, and includes all POSIX standard members, but the Python source code references non-standard members of this struct rusage which Snow Leopard's system headers do not define. By pretending that the functions do not exist, this faulty branch of the code gets skipped by GCC.

So, sure, faking the non-availability of some fundamental system functions gets us a successful compile. But who knows what problems this may introduce into the installation? It was not a solution that we could live with.

At this stage, we gave up on trying to build Python 2.5 from scratch using automake, and turned to Plone's buildout recipe.

The Solution (That Worked)

The buildout provided by Plone is simple and slick, and "just works" (provided you use the system Python installation; see below):

$ cd /opt
$ svn export http://svn.plone.org/svn/collective/buildout/python
$ cd python
$ /System/Library/Frameworks/Python.framework/Versions/2.5/bin/python bootstrap.py
$ bin/buildout

And that's all there is to it! This will result in Python 2.4. 2.5, 2.6, 2.7, and 3.2 being installed to "/opt/python/python-2.4", "/opt/python/python-2.5", "/opt/python/python-2.6", "/opt/python/python-2.7", and "/opt/python/python-3.2" respectively.

No muss, no fuss: you now have five major versions of Python installed, independent of your system Python, to fulfill all your development needs and whims.

Icing on the Cake

Add the following to your "~/.bashrc" or equivalent:

export ALT_PYTHON_24=/opt/python/python-2.4
export ALT_PYTHON_25=/opt/python/python-2.5
export ALT_PYTHON_26=/opt/python/python-2.6
export ALT_PYTHON_27=/opt/python/python-2.7
export ALT_PYTHON_32=/opt/python/python-3.2

## Could be done using bash script/perl ...
## but why stoop so low?
strip_opt_python_from_path() {
    /usr/bin/python -c "
import os
import sys
path = os.environ['PATH']
paths = path.split(os.path.pathsep)
new_path = []
for p in paths:
    if not p.startswith('/opt/python') and p not in new_path:
        new_path.append(p)
sys.stdout.write(os.path.pathsep.join(new_path))
"
}

insert-opt-python-in-path() {
    if [[ -z $1 ]]
    then
        echo "Resetting to default native Python."
        export PATH="$(strip_opt_python_from_path)"
    else
        echo "Setting alternate Python: $1."
        export PATH="$1/bin:$(strip_opt_python_from_path)"
    fi
    echo "python is now: $(which python)."
    if ! type python3 >/dev/null 2>&1
    then
        echo "python3 is not on path."
    else
        echo "python3 is now: $(which python3)."
    fi
}

unimport-alt-python() {
    insert-opt-python-in-path
}

import-alt-python-24() {
    insert-opt-python-in-path $ALT_PYTHON_24
}

import-alt-python-25() {
    insert-opt-python-in-path $ALT_PYTHON_25
}

import-alt-python-26() {
    insert-opt-python-in-path $ALT_PYTHON_26
}

import-alt-python-27() {
    insert-opt-python-in-path $ALT_PYTHON_27
}

import-alt-python-32() {
    insert-opt-python-in-path $ALT_PYTHON_32
}

Now you can switch back and forth between any of these Python installations by invoking "import-alt-python-24"", "import-alt-python-25"", etc. from the shell. To go back to your default Python, simply type "unimport-alt-python".

Caveats and Issues with Running Plone's buildout Recipe

I could not get Plone's buildout recipe to work with my own installation of Python. I could only get it to work using the OS X system Python.

Like many Python developers, I prefer to not to use the operating system's Python installation (which lives in "/System/Library/Frameworks/Python.framework/") for day-to-day development. Instead I use my own "working" installation of Python that which is tweaked for my development needs (in conjunction with, of course, virtualenv). This allows me to freely update my working Python and its third-party libraries as needed, without worrying if I am going to break any version-specific or library-specific dependency in the operating system. For some reason, I just could not get Plone's buildout to work with my own Python installation.

The first thing that went wrong is that the bootstrapping script failed because my version of zc.buildout was newer than the default version required. Or, to be more precise, my version of zc.buildout was not exactly equal to the version required. By default the script requires 1.4.4. And only 1.4.4 will do. Not 1.4.3, or 1.4.5, or 1.4.6. And 1.5 is right out. If the zc.buildout version is different, then the script will fail with a "pkg_resources.VersionConflict" error, e.g.:

pkg_resources.VersionConflict: (zc.buildout 1.5.2 (/usr/platform/lib/python2.7/site-packages/zc.buildout-1.5.2-py2.7.egg), Requirement.parse('zc.buildout==1.4.4'))

This was an easy fix: I had to explicitly pass in the version of my zc.buildout installation using the "-v" or "--version" flag:

$ python bootstrap.py -v 1.5.2

Then I ran the build script:

$ bin/buildout

And all went well, with the compile (of 2.4) finishing successfully, but then failing upon installation:

python-2.4: Running '
/opt/python/bin/virtualenv-2.4 /opt/python/python-2.4
/opt/python/python-2.4/bin/easy_install -U collective.dist'
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/opt/python/parts/virtualenv/virtualenv.py", line 1965, in ?
    main()
  File "/opt/python/parts/virtualenv/virtualenv.py", line 795, in main
    never_download=options.never_download)
  File "/opt/python/parts/virtualenv/virtualenv.py", line 886, in create_environment
    site_packages=site_packages, clear=clear))
  File "/opt/python/parts/virtualenv/virtualenv.py", line 1024, in install_python
    import site
  File "/opt/python/parts/buildout/site.py", line 153
    with f:
         ^
SyntaxError: invalid syntax
/bin/sh: line 2: /opt/python/python-2.4/bin/easy_install: No such file or directory
python-2.4: Non zero exit code (127) while running command.
While:
  Installing python-2.4.
Error: Non zero exit code (127) while running command

It seems that "/opt/python/parts/buildout/site.py" uses the "with" construct, which is not available under Python 2.4, and hence the failure. Why was it trying to execute the script under Python 2.4? I have no idea. I refactored the code to achieve the same logic without using "with", but then the installation failed again. So I commented out the 2.4 build, to see how things would progress with 2.5. Same song, same dance. At this point, I gave up, and tried the whole palaver again with my operating system Python. That worked fine, as show above, and I did not bother with debugging the buildout failure with my "working" Python.

Closing Thoughts

I am aware that Snow Leopard has made some major changes to its libraries and headers etc., that have broken many tools and programs and builds. I am also aware that the versions of Python in question are obsolete, and it is difficult for Python core dev's to justify any time and effort in getting these versions building under Snow Leopard now. So this state of affairs is perfectly understandable.

At the same time, however, I am a little disturbed that it was not possible do a clean build of any Python prior to 2.6 on Snow Leopard using just the bog-standard autotools and GCC. Python 2.5 is still very much widespread in the wild (as measured by support queries and issues on software that I maintain), and will probably be so for a while to come. As a Python developer, I really would like to feel that I can count on building/installing this version of Python (as well as others) for testing and development purposes without hassle. It does not rest easy with me that building "historical" yet live-and-kicking versions of Python is not a straightforward issue on a (fairly) important development operating system.

It is only thanks to the efforts of the folks at Plone, that I managed to get a nice nest of Pythons all cozily installed on my Snow Leopard system. I am truly appreciative of that. THANK YOU, THANK YOU, THANK YOU, Plone people who gave us that buildout; I hope that this recipe continues to evolve and keep up with OS X releases, especially with Lion just around the corner.