Full Stack Python

Introduction

You're knee deep in learning the Python programming language. The syntax is beginning to make sense. The first few "AHA!" moments are hitting you. Now you are excited for direction to learn more.

Now what? Definitely read Kenneth Reitz's The Hitchhiker’s Guide to Python if you have not already. If you're not coming from a development background, you may find this post on Learning Python useful.

This guide has a different focus from the above resources. Here I focus on explaining several ways of setting up a Python web application stack from server infrastructure through JavaScript execution on a user's browser.

About the Author

I'm Matt Makai and I solve problems by developing software. I often use Python and occasionally work with languages in the Java Virtual Machine (JVM) stack.

In 2013 I will be traveling to 30 cities in the United States to visit with tech companies, attend and speak at tech meetups, and blog from an experienced software developer's perspective about the current state of technology in the United States. If you're reading this guide then you're likely someone I'm interested in meeting during my trip. Send me an email at matthew.makai@gmail.com or message me on Twitter. Read more about my upcoming trip that will take place between March 9 and August 15, 2013 at Coding Across America.

If there are typos, inaccurate statements, or general areas for improvement that you see in this guide, please issue a pull request on this website's Github repository. Thank you!

Servers

There are several options for setting up infrastructure to serve a web application:

  1. "Bare metal" servers
  2. Virtualized servers
  3. Infrastructure-as-a-service
  4. Platform-as-a-service

Bare metal

The term bare metal refers to purchasing the actual hardware and hooking it up to the Internet either through a business-class internet service provider (ISP) or co-locating the server with other servers.

The bare metal option offers the most control over the server configuration, usually has the highest performance for the price, but also is the most expensive upfront option and the highest ongoing maintenance. With bare metal servers the ongoing operating cost is the electricity the server(s) use as well as handling repairs when server components malfunction.

Buy actual hardware from a vendor either pre-built or as a collection of components that you assemble yourself.

Virtualized servers

Virtual private servers (VPSs) are slices of hardware on top of a larger bare metal server. Virtualization software such as Xen and VMWare allow providers such as Linode and prgmr (as well as a many others) to provide fractions of a full server that appear as their own instances. For example, a server with an 8-core Xeon processor and 16 gigabytes of memory can be sliced into 8 pieces with the equivalent of 1-core and 2 gigabytes of memory.

The primary disadvantage of virtualized servers is that there is resource overhead in the virtualization process. In addition, physical constraints such as heavy I/O operations by a single virtualized instance on persistent storage can cause performance bottlenecks for other virtualized instances on the shared server. Choosing virtualized server hosting should be based on your needs for urgency of service ticket requests and the frequency you require for ongoing maintenance such as persistent storage backups.

Infrastructure-as-a-service

Infrastructure-as-a-service (IaaS) overlaps with virtualized servers because the resources are often presented in the same way. The difference between virtualized servers and IaaS is the granularity of the billing cycle. IaaS generally encourages a finer granularity based on minutes or hours of server usage instead of on monthly billing cycles.

IaaS can be used in combination with virtualized servers to provide dynamic upscaling for heavy traffic. When traffic is low then virtualized servers can solely be used. This combination of resources reduces cost at the expense of greater complexity in the dynamically scaled infrastructure.

Platform-as-a-service

Platform-as-a-service (PaaS) refers to a class of services that will host an application with a layer between the server and the application. The layer defines how the application accesses resources such as computing time, files, and external services. The tradeoff to PaaS is usually ease of deployment to a specific architecture as long as you do things "their way."

If you go the PaaS route, you can skip over the operating system and web server sections because they are baked into PaaS offerings.

Server Monitoring

There are several important resources to monitor on the server level of a web stack. Technically some of these cross into the operating system level (next section) but they are listed here:

  1. Server uptime
  2. CPU utilization
  3. Memory utilization
  4. Persistence storage consumed versus free
  5. Network bandwidth and latency

Operating System

The operating system sits directly on top of the server or virtual server and provides access to computing resources. The only operating system I can recommend for production Python web stack deployments is a flavor of Linux. Ubuntu Long Term Support (LTS) releases, Red Hat Enterprise Linux, and CentOS are all viable options.

Side note: Mac OS X is fine for development activities but not for test/production deployments.

Ubuntu

Ubuntu is a Linux distribution packaged by the Canonical Ltd company. Ubuntu uses the Debian distribution as a base for packages, including the aptitude package manager. For desktop versions of Ubuntu, GNOME (until the 11.04 release) or Unity (11.10 through current) is added to the distribution to provide a user interface.

Red Hat and CentOS

Red Hat Enterprise Linux (RHEL) and Community enterprise Operating System (CentOS) are essentially the same distribution. The key difference is that CentOS is an open source free derivative of RHEL.

Web Server

A web server handles Hypertext Transfer Protocol (HTTP) requests from clients. The client is usually a browser such as Internet Explorer, Firefox, and Chrome, but it can also be a headless browser like phantomjs (commonly used for testing), a commandline utility like wget or curl, or a web crawler. The web server processes the request and sends a status code and depending on the status code a content response as well.

In a simple case, the client will request a static asset such as a picture or JavaScript file. The file sits on the file system in a location the web server is authorized to access and the web server sends the file to the clinet with a 200 status code. If the client already requested the file and the file has not changed, the web server will pass back a 304 "Not modified" response indicating the client already has the latest version of that file.

Sending static assets can eat up a large amount of bandwidth which is why using a Content Delivery Network (CDN) is important when possible (see the content delivery network section for a more detailed explanation).

Database

A database provides an abstraction on top of a file system for creating, reading, updating, and deleting persistent data. The abstraction most commonly represented is a set of relational tables, known as a relational database management system (RDBMS). Alternative abstractions include graph databases where data is stored in both nodes and edges of a graph.

Database resources

PostgreSQL and MySQL are two of the most common open source databases.

SQLite is a database that is stored in a single file on disk. SQLite is built into Python but is only built for access by a single connection at a time.

Web Framework

A web application framework is a collection of libraries that provide functionality to accomplish common operations for the web. These common operations include:

  1. URL routing
  2. HTML, XML, JSON, and other output templating
  3. Database manipulation
  4. Cross-site request forgery (CSRF) and Cross-site scripting (XSS) protection

Not all web frameworks include functionality for all of the above functionality. Frameworks must balance between "being all things to all people but very complicated" or doing only certain things well without prescription for how to do other functions.

For example, the Django web application framework includes an Object-Relational Mapping (ORM) layer that abstracts relational database read, write, query, and delete operations. However, the ORM layer in Django does not work (without modification) on non-relational databases such MongoDB and Riak. Other web frameworks such as Flask and Pyramid are generally easier to use with non-relational databases by incorporating external Python libraries.

Web Framework Resources

Django, Flask, Bottle, Pyramid, and web.py are the most common Python web frameworks.

WSGI Server

A Web Server Gateway Interface (WSGI) server implements the web server side of the WSGI interface for running Python web applications. The WSGI standard v1.0 is specified in PEP 0333. As of September 2010, WSGI v1.0 is superseded by PEP 3333, which defines the v1.0.1 WSGI standard.

WSGI Resources

PEP 0333 WSGI v1.0 and PEP 3333 WSGI v1.0.1 specifications.

Green Unicorn, mod_wsgi, uWSGI, and gvent are common WSGI server implementations.

Content Delivery Network

A content delivery network (CDN) serves static file assets. Amazon CloudFront, Akamai, and Rackspace Cloud Files are examples of CDNs. The purpose of a CDN is to remove the load of static file requests from web servers that are handling dynamic web content. For example, if you have an nginx server that handles both static files and acts as a front for a Green Unicorn WSGI server on a 512 megabyte virtual private server, the nginx server will run into resource constraints under heavy traffic. A CDN can remove the need to serve static assets from that nginx server so it can purely act as a pass through for requests to the Green Unicorn WSGI server.

CDNs distribute request load globally by using data centers in different locations.

Change Log

2012.12.30: Updated introduction with improved wording. Working on CDN section. Added very basic database content and removed stub from that section.

2012.12.27: Added beginning of section on Python web frameworks. Added stub for content delivery network.

2012.12.25: Initial incomplete release on fullstackpython.com.

Matt Makai 2013