~2012

Introduction to python web programming

It's time to say goodbye to PHP and replace it with Python? This article is a quick intro into two approaches of using python instead of PHP. Yes you can use python embedded in your HTML like with PHP however that approach is for kids and not covered here ;-).

I'm assuming you have a recent version (2.6>) of Python installed. I'm only covering Apache as a production webserver. Most tests can just be done inside python anyway. I'm only covering UNIX platforms. So if you use Windows just read the python docs about Windows, shouldn't be too different. Especially in the second method of python web development there's a very positive side effect: You don't need a dedicated webserver! 8-)

Approach 1: Good old school Python CGI

The most simple setup of using Python for webdevelopment is through CGI.

The most simple test.py application would look like this:

#!/usr/bin/env python
print("Content-Type: text/plain")   # Plaintext is following
print("")                           # blank line, end of headers
print("Hello world")

Which just prints out "Hello World".

First let's try to run this on a webserver.

I'm not going to cover a full apache setup. I've just got a simple apache setup using userdirs for users to use the webserver. I've edited the userdir module configuration as follows:

<IfModule mod_userdir.c>
        UserDir www
        UserDir disabled root

<Directory /home/*/www>
                AllowOverride FileInfo AuthConfig Limit Indexes Options
                Options MultiViews -Indexes SymLinksIfOwnerMatch IncludesNoExec ExecCGI
                <Limit GET POST OPTIONS>
                        Order allow,deny
                        Allow from all
                </Limit>
                <LimitExcept GET POST OPTIONS>
                        Order deny,allow
                        Deny from all
                </LimitExcept>
        </Directory>
    <Directory /home/*/www/cgi-bin>
        AddHandler cgi-script .py
        Options +ExecCGI
    </Directory>
</IfModule>

So this makes sure we can execure python scripts inside the cgi-bin directory. There's probably a better setup but this works for me. For a production server I don't recommended this!

So now you should be able browse to

http://your.webserver.com/~username/cgi-bin/test.py

and it should display something familiar. If it's not working see your webserver logs for hints.

CGI Magic with python

We just covered the most useless, but most created, application ever. So let's add two things to this. 1. Error handling: we would like to see what's going wrong if... cgitb module 2. Input handling: cgi module

If I would create an error in our script like this:

#!/usr/bin/env python
print("Content-Type: text/plain")
print("")
print("Hello world")
bla

We would only notice that in our server logs. This is usually not comfortable while developing. So let's add the CGI TraceBack module:

#!/usr/bin/env python

# enable debugging
import cgitb
cgitb.enable()

print("Content-Type: text/plain")
print("")
print("Hello World!")
bla

If you would point your browser to the test.py file now it will display lots of debug information. However it's formatted in HTML and the browser renders it as plaintext. Time to change the Content-Type.

#!/usr/bin/env python

# enable debugging
import cgitb
cgitb.enable()

print("Content-Type: text/html")
print("")
print("Hello World!")
bla

You can also change the format output of the cgitb module. It’s very helpful to use this feature during development. The reports produced by cgitb provide information that can save you a lot of time in tracking down bugs. You can always remove the cgitb line later.

User input

The cgi module provides methods to handle input. Let's keep it simple. The following code renders an input form only if there is no input named 'name'.

#!/usr/bin/env python

# enable debugging
import cgitb
import cgi
cgitb.enable()

print("Content-Type: text/html")
print("")

form = cgi.FieldStorage()
if "name" not in form:
    print("Please fill in your name:")
    print('<form action="test.py" method="get">')
    print('Name <input type="text" name="name" />')
    print('<input type="submit" value="submit" />')
    print('</form>')
else:
    print("<p>Hello", form["name"].value)

#Just a simple output of all arguments passed
print("<h1>arguments passed</h1>")
for item in form:
        print(item, form[item].value)

Well that covers the basic python cgi pipeline. The problem with CGI is that it needs to spawn a new python process for every request. In a simple setup this is no problem but when the server needs to process a lot of requests this can get problematic.

So let's see a more modern approach

Approach 2: Python Web Server Gateway Interface (WSGI)

A standard has been developed to interface between python and the webserver. This standard is called WSGI. I'd like to keep things simple so I'm not going in depth about this. I'm just showing how to get started doing things with it.

It all simply comes down to a function being called with 2 arguments: * a dictionary of environment variables * a function to be called to send header data back to the webserver The function must return the body data which will be send back to the server as a iterable [^1]

[^1]: So a list or dictionary, not a single string.

Python has a wsig reference module implemented which comes with a basic server. This is very usefull for testing your code on your own workstation.

So a simple hello world would look like this:

#!/usr/bin/env python

from wsgiref.simple_server import make_server

def hello_world_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/plain')] # HTTP Headers
    start_response(status, headers)

# The returned object is going to be printed
    return ["Hello World"]

httpd = make_server('', 8000, hello_world_app)
print "Serving on port 8000..."

# Serve until process is killed
httpd.serve_forever()

The great thing is you can just run this on your workstation and point your browser to http://localhost:8000

Now let's try to recreate our previous cgi app.

#!/usr/bin/python
from wsgiref.simple_server import make_server
from cgi import parse_qs, escape

def hello_user_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/html')] # HTTP Headers
    start_response(status, headers)

form =  parse_qs(environ['QUERY_STRING'])
    html=""
    if "name" not in form:
        html += "Please fill in your name:"
        html += '<form action="wsig.py" method="get">'
        html += 'Name <input type="text" name="name" />'
        html += '<input type="submit" value="submit" />'
        html += '</form>'
    else:
        html += "<p>Hello "
        # Always escape user input to avoid script injection
        html += escape(form['name'][0])

# The returned object is going to be printed
    return [html]

if **name** == '**main**':
    from wsgiref.simple_server import make_server
    httpd = make_server('', 8000, hello_user_app)
    print "Serving on port 8000..."

# Serve until process is killed
    #httpd.serve_forever()
    # Alternative: serve once and exit
    #httpd.handle_request()

Onto a production server

To get the WSGI approach working in Apache you need to install the Apache wsgi module (mod_wsgi). In Debian/Ubuntu you just issue:

sudo apt-get install libapache2-mod-wsgi
/etc/init.d/apache2 reload

// Ofcourse because of security you'll need to do some configuration telling the webserver where and what python scripts can be executed. The configuration I'm proposing is by no means secure and not recommended for production use. Read the mod-wsgi documentation about a secure setup. //

I'm going to configure a dedicated directory in the users home dir for the wsgi scripts. So I need to edit the Apache userdir module like we did before with CGI:

<IfModule mod_userdir.c>
        UserDir www
        UserDir disabled root

<Directory /home/*/www>
                AllowOverride FileInfo AuthConfig Limit Indexes Options
                Options MultiViews -Indexes SymLinksIfOwnerMatch IncludesNoExec ExecCGI
                <Limit GET POST OPTIONS>
                        Order allow,deny
                        Allow from all
                </Limit>
                <LimitExcept GET POST OPTIONS>
                        Order deny,allow
                        Deny from all
                </LimitExcept>
        </Directory>
    <Directory /home/*/www/cgi-bin>
        AddHandler cgi-script .py
        Options +ExecCGI
    </Directory>
    <Directory /home/*/www/wsgi-bin>
        Options ExecCGI
        AllowOverride FileInfo
        AddHandler wsgi-script .wsgi
    </Directory>
</IfModule>

As you can see I'm adding a handler with a specific file extension (.wsgi). Since WSGI is handled different from normal python scripts I don't want to use the .py extension. This is a bit more secure but also forces you to adapt to WSGI.

Now to run your application on the production server we need to configure one more thing. Apache's mod_wsgi directly calls the 'application' function. In our example we named our function 'hello_user_app'. So if we would copy our script to the production server it wouldn't do a thing. The webserver will error with:

...test.wsgi' does not contain WSGI application 'application'...

You could rename your function to 'application' however I would recommended add a function 'application' which calls your function. This function can be made more intelligent to call different functions depending on input. So the final script would look like this (Note the file extension .wsgi!):

#!/usr/bin/python
from cgi import parse_qs, escape

def hello_user_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/html')] # HTTP Headers
    start_response(status, headers)

form =  parse_qs(environ['QUERY_STRING'])
    html=""
    if "name" not in form:
        html += "Please fill in your name:"
        html += '<form action="wsig.py" method="get">'
        html += 'Name <input type="text" name="name" />'
        html += '<input type="submit" value="submit" />'
        html += '</form>'
    else:
        html += "<p>Hello "
        # Always escape user input to avoid script injection
        html += escape(form['name'][0])

#Just a simple output of all arguments passed
    html += "<h1>Arguments passed</h1>"
    for item in form:
        html += "%s : %s" %(item, form[item][0])

# The returned object is going to be printed
    return [html]

def application(environ, start_response):
    return hello_user_app(environ, start_response)

if **name** == '**main**':
    from wsgiref.simple_server import make_server
    httpd = make_server('', 8000, hello_user_app)
    print "Serving on port 8000..."

# Serve until process is killed
    httpd.serve_forever()
    # Serve once
    #httpd.handle_request()

Note: Because of the

if **name** == '**main**':

//line no http server will be started by python. That piece of code will only be run if it is run directly by the python interpreter. This is not the case when run through the WSGI module. //

This covers the basic of python web programming. I'll start a new post about webframeworks following this post to make life even easier.