Pyston is a new, under-development Python implementation built using LLVM and modern JIT techniques with the goal of achieving good performance.
We have a small website [pyston.org](http://pyston.org/), which for now just hosts the mailing lists and the [blog](http://blog.pyston.org/). We have two mailing lists: [pyston-dev@](http://lists.pyston.org/cgi-bin/mailman/listinfo/pyston-dev) for development-related discussions, and [pyston-announce@](http://lists.pyston.org/cgi-bin/mailman/listinfo/pyston-announce) which is for wider announcements (new releases, major project changes).
We have a small website [pyston.org](http://pyston.org/), which for now just hosts the mailing lists and the [blog](http://blog.pyston.org/). We have two mailing lists: [pyston-dev@](http://lists.pyston.org/cgi-bin/mailman/listinfo/pyston-dev) for development-related discussions, and [pyston-announce@](http://lists.pyston.org/cgi-bin/mailman/listinfo/pyston-announce) which is for wider announcements (new releases, major project changes). We also have a [gitter chat room](https://gitter.im/dropbox/pyston) where most discussion takes place.
### Current state
Pyston should be considered in early alpha: it "works" in that it can successfully run Python code, but it is still quite far from being useful for end-users.
Pyston should be considered in alpha: it "works" in that it can successfully run Python code, but it is still quite far from being useful for end-users.
Currently, Pyston targets Python 2.7, only runs on x86_64 platforms, and only has been tested on Ubuntu. Support for more platforms -- along with Python 3 compatibility -- is desired but currently not on the roadmap.
...
...
@@ -30,7 +30,7 @@ Pyston welcomes any kind of contribution; please see [CONTRIBUTING.md](https://g
- We have allowed performance to regress, sometimes considerably, but (hopefully) in places that allow for more efficient implementations as we have time.
...
...
@@ -45,16 +45,14 @@ Pyston welcomes any kind of contribution; please see [CONTRIBUTING.md](https://g
### Getting started
To get a full development environment for Pyston, you will need pretty recent versions of various tools. The docs/INSTALLING.md file contains information about what the tools are, how to get them, and how to install them; currently it can take up to an hour to get them all built on a quad-core machine.
We have some build instructions at [INSTALLING.md](https://github.com/dropbox/pyston/blob/master/docs/INSTALLING.md). If you have any issues, please feel free to file an issue in the issue tracker, or mention it via email or gitter.
To simply build and run Pyston, a smaller set of dependencies is required; see docs/INSTALLING.md, but skip the "OPTIONAL DEPENDENCIES" section. Once all the dependencies are installed, you should be able to do
Once you've followed those instructions, you should be able to do
```
$ make check -j4
$ make check
```
And see that hopefully all of the tests pass. (If they don't, please email pyston-dev.)
We also have a new CMake-based build system which is easier and faster to get running; see the end of INSTALLING.md for information.
And see that hopefully all of the tests pass. (If they don't, please let us know.)
All pull requests are built and tested by travis-ci.org running Ubuntu 12.04.
See [travis-ci.org/dropbox/pyston/builds](https://travis-ci.org/dropbox/pyston/builds).
Pyston currently only supports installing from source; the following instructions have fairly tested as working on Ubuntu, and are extensively verified as not working on Mac. (Please see issue #165 for discussion on enabling OSX support, which is pretty difficult.)
Pyston currently only supports installing from source; the following instructions have fairly tested as working on Ubuntu, and are extensively verified as not working on Mac. (Please see issue [#165](https://github.com/dropbox/pyston/issues/165) for discussion on enabling OSX support, which is pretty difficult.)
The build instructions assume that you will put the Pyston source code in `~/pyston` and put the dependencies in `~/pyston_deps`. Barring any bugs, you should be free to put them anywhere you'd like, though the instructions in this file would have to be altered before following. Also, if you want to change the dependency dir, you'll have to change the value of the the `DEPS_DIR` variable in `Makefile`.
# CMake build system
### Prerequisites
GNU make is required to build pyston.
Our CMake build system supersedes our old Makefile build system. The old Makefile rules still exist and can be activated by adding `USE_CMAKE=0` to your make invocation, but they will be removed at some point. We still use the Makefile as a sort of "lite development environment"; it helps manage multiple cmake configurations and has some testing helpers.
clang requires a fairly modern [host compiler](http://llvm.org/docs/GettingStarted.html#host-c-toolchain-both-compiler-and-standard-library), so typically you will have to install a new one. The easiest thing to do is to just create a fresh build of GCC:
ccache is a build tool that can help speed up redundant compilations. It's not strictly necessary but it's useful enough to be enabled by default; you can disable it by adding `USE_CCACHE := 0` to your Makefile.local. To get it, run:
LLVM and clang depend on a pretty modern compiler; the steps below assume you installed GCC 4.8.2 as described above. It should be possible to build using clang >= 3.1, such as what you might find on a Mac, but that will require changes to the way LLVM is configured (specified in Makefile) that I haven't tested.
There seem to be some lingering issues with the LLVM build that haven't been identified yet; if the last step fails with errors along the lines of "rm: could not find file foo.tmp", it is quite likely that simply running it again will cause it to continue successfully. You may have to do this multiple times, unfortunately.
Note: if you followed the previous version of the directions and installed libunwind globally, you'll need to uninstall it by doing the following:
```
cd ~/pyston_deps/libunwind-1.1
./configure
sudo make uninstall
```
and then repeat the correct process
### zsh
`zsh` is needed when running pyston tests.
```
sudo apt-get install zsh
```
### readline
`readline` is used for the repl.
```
sudo apt-get install libreadline-dev
```
### gmp
`gmp` is a multiprecision library used for implementing Python longs. It's also a dependency of gcc, so if you installed that you should already have it:
See the main README for more information about available make targets and options.
### LZ4
```
cd ~/pyston_deps
git clone git://github.com/Cyan4973/lz4.git
mkdir lz4-install
cd lz4/lib
DESTDIR="$HOME/pyston_deps/lz4-install" PREFIX="/" make install
```
---
#### CMake options
At this point you should be able to run `make check` (in the `~/pyston` directory) and pass the tests. See the main README for more information about available targets and options.
-`-DCMAKE_BUILD_TYPE=Debug` (defaults to Release)
-`-DENABLE_LLVM_DEBUG=1` for full LLVM debug
-`-DENABLE_CCACHE=0` to disable ccache
# Optional dependencies
There are a number of optional dependencies that the build system knows about, but aren't strictly necessary for building and running Pyston. Most of them are related to developing and debugging:
### CPython headers
We have some extension module tests, in which we compile the extension module code for both Pyston and CPython.
```
sudo apt-get install python-dev
```
### gdb
A new version of gdb is highly recommended since debugging a JIT tends to use new features of GDB:
A modern gdb is highly recommended; users on Ubuntu 12.04 should definitely update:
TODO: GDB should be able to determine its data directory. maybe it should be installed rather that run
from inside the source dir?
--->
#### getting gdb to pretty-print STL types
GDB ships with python scripts that tell it how to pretty-print STL containers, but they only work against the stdlib in the corresponding version of GCC. Unfortunately, we compile against an old version of gcc's C++ stdlib. Here's how to get GDB to use the *old* pretty-print scripts for GCC 4.8.2. (Note: Have only tested this with gdb 7.8.1-ubuntu4 and gcc-4.8.2.)
This will link pyston_dbg and pyston_debug against the debug version of libunwind (the release pyston build will still link against the release libunwind); to enable debug output, set the UNW_DEBUG_LEVEL environment variable, ex to 13.
### gperftools (-lprofiler)
```
download from http://code.google.com/p/gperftools/downloads/list
...
...
@@ -312,69 +171,173 @@ ninja docs
Generate doxygen documentation for Pyston. Requires using the cmake build system.
# (Experimental) CMake build system
To use the toolchain from this document, do:
# Deprecated Makefile build system
These are the old instructions for our original Makefile build system and toolchain. They will be disappearing shortly.
If your system provides a new enough GCC and cmake, you can just do:
### Compiler for clang
clang requires a fairly modern [host compiler](http://llvm.org/docs/GettingStarted.html#host-c-toolchain-both-compiler-and-standard-library), so typically you will have to install a new one. The easiest thing to do is to just create a fresh build of GCC:
```
mkdir ~/pyston-build && cd ~/pyston-build
cmake -GNinja ~/pyston
ninja check-pyston
sudo apt-get install libgmp-dev libmpfr-dev libmpc-dev make build-essential libtool zip gcc-multilib autogen
# Specifying LIBRARY_PATH is a workaround to get gcc to compile on newer Ubuntus with multiarch
LIBRARY_PATH=/usr/lib32 make -j4
make check
make install
```
**Ubuntu 12.04**
### ccache
ccache is a build tool that can help speed up redundant compilations. It's not strictly necessary but it's useful enough to be enabled by default; you can disable it by adding `USE_CCACHE := 0` to your Makefile.local. To get it, run:
LLVM and clang depend on a pretty modern compiler; the steps below assume you installed GCC 4.8.2 as described above. It should be possible to build using clang >= 3.1, such as what you might find on a Mac, but that will require changes to the way LLVM is configured (specified in Makefile) that I haven't tested.
There seem to be some lingering issues with the LLVM build that haven't been identified yet; if the last step fails with errors along the lines of "rm: could not find file foo.tmp", it is quite likely that simply running it again will cause it to continue successfully. You may have to do this multiple times, unfortunately.
`gmp` is a multiprecision library used for implementing Python longs. It's also a dependency of gcc, so if you installed that you should already have it:
This will link pyston_dbg and pyston_debug against the debug version of libunwind (the release pyston build will still link against the release libunwind); to enable debug output, set the UNW_DEBUG_LEVEL environment variable, ex to 13.
