Amazon AWS and iPython Notebook

Amazon Web Services (AWS) offer easy-to-use and cheap cloud computing on demand - the elastic cloud computing (EC2). I've been using it for several of my projects and I found that using spot instances is an extremely useful and effective way to test/experiment with your ideas.

In this post, I will show you how to set up iPython Notebook which you can then access from your web-browser. Please note, that I will be installing nVidia's drivers and CUDA so that you can use Theano to speed up your calculations using GPU.

Installing Ubuntu Packages

After you start your GPU large instance and connect to it using SSH just install these packages

sudo apt-get -y install gfortran  
sudo apt-get -y install libblas-dev libatlas-dev liblapack-dev  
sudo apt-get -y install freetype*  
sudo apt-get -y install pkg-config  
sudo apt-get -y install gcc  
sudo apt-get -y install g++  
sudo apt-get -y install build-essential  
sudo apt-get -y install git  
sudo apt-get -y install wget  
sudo apt-get -y install linux-image-generic  
sudo apt-get -y install cmake

# install python stuff
sudo apt-get -y install python-dev python-pip

# install virtual environments
sudo pip install virtualenvwrapper

# install CUDA
sudo wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_6.5-14_amd64.deb  
sudo dpkg -i cuda-repo-ubuntu1404_6.5-14_amd64.deb  
sudo apt-get update  
sudo apt-get -y install cuda  

Then, we can add CUDA's lib and bin to bashrc

echo -e "\nexport PATH=/usr/local/cuda-6.5/bin:$PATH\n\nexport LD_LIBRARY_PATH=/usr/local/cuda-6.5/lib64" >> .bashrc  

Reboot

sudo reboot  

And then test if CUDA works as it is supposed to be

#test CUDA
cuda-install-samples-6.5.sh ~/  
cd NVIDIA_CUDA-6.5_Samples/1_Utilities/deviceQuery  
make  
./deviceQuery

Setting-up Python

I prefer to use Python's virtual environments to keep each python environment separate.

mkdir ~/.virtualenvs  
cd .virtualenvs/

mkvirtualenv --no-site-packages quantTest  
workon quantTest

pip install nose  
pip install numpy  
pip install scipy  
pip install pygments  
pip install mysql-python  
pip install matplotlib  
pip install pandas

pip install feedparser  
pip install lxml  
pip install beautifulsoup4  
pip install requests  
pip install cython

pip install pyzmq

pip install tornado  
pip install jinja2  
pip install PySide

pip install ipython

pip install scikit-learn  
pip install scikit-image

pip install joblib  
pip install pyyaml  
pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git  
pip install --upgrade --no-deps git+git://github.com/benanne/Lasagne.git  
pip install --upgrade --no-deps git+git://github.com/dnouri/nolearn.git

git clone git://github.com/lisa-lab/pylearn2.git  
cd pylearn2  
python setup.py develop  

Setting-up iPython

First of all, create a password in python

from IPython.lib import passwd  
passwd()  
Enter password:  
Verify password:  
sha1:def70021a545:7b799f5ecfXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  

Then, create a certificate for notebook

mkdir .ssh  
cd .ssh  
openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem  

Followed by creating a iPython profile

ipython profile create quantServer  

Then, modify the profile .ipython/profile_quantServer

c = get_config()

# Kernel config
c.IPKernelApp.pylab = 'inline'  # if you want plotting support always

# Notebook config
c.NotebookApp.certfile = u'/home/ubuntu/.ssh/mycert.pem'  
c.NotebookApp.ip = '*'  
c.NotebookApp.open_browser = False  
c.NotebookApp.password = u'sha1:def70021a545:7b799f5ecfXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'  
# Put it on a fixed port
c.NotebookApp.port = 9999  

Run iPython Notebook

After all this, we can finally run iPython Notebook on our server by simply calling

ipython notebook --profile=quantServer  

And that's it! After this, you can connect to your server on port 9999 and enjoy iPython. Once finished, you can create an amazon image (AMI) and use it in future to set up your spot instance and avoid running all these scripts over and over again.