Leveraging Python Virtual Environments and pip
Introduction
Python virtual environments are a common means by which python applications can be isolated from one another and also segregated from the global directories and packages of an operating system. They can provde a pragmatic compromise between the flexibility needs of software development and the stability standards for server management. While new techniques like containers and Docker may point to the future of application deployment, virtual environments remain a solid choice for local Python development as well as application management across a set of Linux servers.
Installing virtualenv and pip
To take full advantage of python virtual environments, both virtualenv and pip should be installed. The virtualenv package provides environment management and isolation while pip provides python package installation and management within the virtual environment. These tools are not always available by default though. In order to bootstrap, it is often easiest to install virtualenv which contains pip.
Installation of virtualenv varies across different operating systems. Here, I will focus on Redhat Enterprise Linux / CentOS because of their common usage for our server installations.
For both RHEL 5 and RHEL 6, the EPEL repository is a great resource for obtaining virtualenv and related python packages. Installing this repository can be as simple as installing a single rpm manually, or having it added to the server configuration using your preferred management system such as puppet or chef. Details of this installation are beyond the scope of this article. Readers may also wish to instead investigate and use the newer RedHat Software Collections.
Once the EPEL repository is installed, installing virtualenv (and pip) is just a simple rpm installation. For RHEL 6, python 2.6 is the default python installation. For RHEL 5, you must install python2.6 from EPEL in addition to the virtualenv package. Note that this means paying close attention to 2.4 vs 2.6 usage on RHEL 5 systems.
Installing virtualenv (and pip) on RHEL 6:
# Requires EPEL repo
[root@host]$ yum install python-virtualenv
Installing virtualenv (and pip) on RHEL 5:
# Requires EPEL repo
# Requires python2.6 from EPEL
[root@host]$ yum install python26
[root@host]$ yum install python26-virtualenv
Creating and Activating a Virtual Environment
With the necessary pre-requisites, you can now create new virtual environments. The examples here illustrate usage on a Linux server. However, it is not necessary to have root privileges to uses virtual environments or pip. You can use them for local exploration and development, on automated testing hosts, and of course development and production servers.
Creating a new virtual environment:
# Choose a common location for your applications / virtual envs
[host]# cd /opt/company/
# Create a new virtual environment
[host]# virtualenv ./myapp/
Using the virtual environment requires that it be “activated”. This is a simple one step command.
Activate the virtual environment:
[root@host]# source ./myapp/bin/activate
(myapp)
[host]#
Using a Virtual Environment Together with an Application
How you choose to use a virtual environment for an application can vary dramatically per situation and personal preferences. Once a virtual environment is activated, you can navigate the server filesystem and use any python code you wish. Conceptually, the virtualenv can be wholly separate from an application. In practice, we often tie a single application to a single virtualenv. Below are just two examples to illustrate differences.
A virtualenv and a local working copy of code:
# Here's a virtual env and activation
source ~/myvirts/myapp/bin/activate
# And here is a working copy of trunk from myapp
cd ~/mycode/svn/myapp-trunk/
An application inside a virtual env directory:
# A virtual env root and an application root inside
/opt/company/myapp/ # the virtual env root
/opt/company/myapp/myapp/ # the application root
/opt/company/myapp/myapp/lib/ # the application library
/opt/company/myapp/myapp/bin/appd.py # the application
# So you can enter the virtual env and then run the application
cd /opt/company/myapp/
source ./bin/activate
cd ./myapp/
./bin/appd.py
While not a standard convention, placing the application inside the virtualenv directory like this allows for synchronizing a whole virtualenv and application from server to server, in a single isolated directory. You may prefer to decouple the directories for very valid reasons such as easier management in diverse server environment. As noted initially, conventions for joining a virtualenv and an app can vary widely.
Installing Packages with pip
The pip application allows you to install and manage applications. While it can be used globally, we focus here on usage within a virtual environment. Packages can be installed from several sources, with the most common being installations from the Python Package Index.
Example installation of a common package named “requests”:
# List the current packages
(myapp)
[host]# pip freeze
# Install a new package from pypi
(myapp)
[host]# pip install requests
Downloading/unpacking requests
Downloading requests-2.3.0.tar.gz (429kB): 429kB downloaded
Running setup.py egg_info for package requests
Installing collected packages: requests
Running setup.py install for requests
Successfully installed requests
Cleaning up...
(myapp)
[host]#
For a given application, you’ll want to keep a list of all the required external packages. The pip application provides a convention for this. You can place all the package names (and versions if desired) inside a file named “requirements.txt”. This file can then be stored in the root directory of your application, and managed like any other file in your version control system.
Example requirements.txt file
(myapp)
[host]$cat requirements.txt
requests
rawes
redis
Install requirements from file:
# Make sure the virtualenv is already activated!
(myapp)
[host]$pip install -r ./requirements.txt
Installing Packages from a Repository
The last item we’ll cover is the installation of packages from a repository. For a variety of reasons, the packages or versions you need to install with pip may not be available in pypi. A handy workaround can be to install a package straight from a repository. This may require some special setup effort on behalf of the package owner, but those details are beyond the scope of this article. This technique can be especially handy for installing internal packages straight from your company repository.
There are many possible variations to this installation method, which depend on the type of repository and other attributes. I’ll show two common examples.
Requirements entry which will install the langid package from the GitHub master branch:
(myapp)
[host]$cat requirements.txt
-e git+https://github.com/saffsd/langid.py.git#egg=langid
Requirements entry which will install a specific tagged version of an internal package (helper-lib) from a local svn repository:
(myapp)
[host]$cat requirements.txt
(dre-ops)
[rain dre-ops-trunk]$cat requirements.txt
-e svn+ssh://repo.mycompany.com/var/svn/repos/helper-lib/tags/REL-Stable#egg=helper-lib
Closing
The above examples illustrate some of the techniques used at Catalyst to isolate and manage python applications and their requirements. The flexibility of virtualenv and pip can allow you to use your own conventions for your own environment. Overall, we’ve found pip and virtualenv to be very useful for the management and deployment of python applications in our environment.
(Originally published on the Catalyst InsideOps blog)