streamhacker.com Weotta be Hacking

21Mar/100

Python and Django Testing and Continuous Integration Links

Django Continuous Integration:

Python Testing:

8Mar/105

jQuery Validation with Django Forms

Django has everything you need to do server-side validation, but it's also a good idea to do client-side validation. Here's how you can integrate the jQuery Validation plugin with your Django Forms.

jQuery Validation Rules

jQuery validation works by assigning validation rules to each element in your form. These rules can be assigned a couple different ways:

  1. Class Rules
  2. Metadata Rules
  3. Rules Object

Django Form Class Rules

The simplest validation rules, such as required, can be assigned as classes on your form elements. To do this in Django, you can specify custom widget attributes.

from django import forms
from django.forms import widgets

class MyForm(forms.Form):
    title = forms.CharField(required=True, widget=widgets.TextInput(attrs={
        'class': 'required'
    }))

In Django 1.2, there's support for a required css class, but you can still use the technique above to specify other validation rules.

Django Form Metadata Rules

For validation methods that require arguments, such minlength and maxlength, you can create metadata in the class attribute. You'll have to include the jQuery metadata plugin for this style of rules.

from django import forms
from django.forms import widgets

class MyForm(forms.Form):
    title = forms.CharField(required=True, minlength=2, maxlength=100, widget=widgets.TextInput(attrs={
        'class': '{required:true, minlength:2, maxlength:100}'
    }))

jQuery Validate Rules Object

If your validation requirements are more complex, or you don't want to use the metadata plugin or class based rules, you can create a rules object to pass as an option to the validate method. This object can be generated in your template like so:

<script type="text/javascript">
FORM_RULES = {
    '{{ form.title.name }}': 'required'
};

$(document).ready(function() {
    $('form').validate({
        rules: FORM_RULES
    });
});
</script>

The reason I suggest generating the rules object in your template is to avoid hardcoding the field name in your javascript. A rules object can also be used in conjunction with class and metadata rules, so you could have some rules assigned in individual element classes or metadata, and other rules in your rules object.

Error Messages

If you want to keep the client-side validation error messages consistent with Django's validation error messages, you'll need to copy Django's error messages and specify them in the metadata or in a messages object.

Metadata Messages

Messages must be specified per-field, and per-rule. Here's an example where I specify the minlength message for the title field.

from django import forms
from django.forms import widgets

class MyForm(forms.Form):
    title = forms.CharField(minlength=2, widget=widgets.TextInput(attrs={
        'class': '{minlength:2, messages:{minlength:"Ensure this value has at least 2 characters"}}'
    }))

Messages Object

Messages can also be specified in javascript object, like so:

<script type="text/javascript">
FORM_RULES = {
    '{{ form.title.name }}': 'required'
};

FORM_MESSAGES = {
    '{{ form.title.name }}': 'This field is required'
};

$(document).ready(function() {
    $('form').validate({
        rules: FORM_RULES,
        messages: FORM_MESSAGES
    });
});
</script>

Just like with validation rules, messages in element metadata can be used in conjunction with a global messages object. Note: if an element has a title attribute, then the title will be used as the default error message, unless you specify ignoreTitle: false in the jQuery validate options.

Error Labels vs Errorlist

Django's default error output is an error list, while the default for jQuery Validation errors is a label with class="error". So in order to unify your validation errors, there's 2 options:

  1. make jQuery Validation output an error list
  2. output error labels instead of an error list in the template

Personally, I prefer the simple error labels produced by jQuery validation. To make Django generate those instead of an error list, you can do the following in your templates:

{{ field }}
{% if field.errors %}
{# NOTE: must use id_NAME for jquery.validation to overwrite error label #}
<label class='error' for='id_{{ field.name }}' generated="true">{{ field.errors|join:". " }}</label>
{% endif %}

You could also create your own error_class for outputting the error labels, but then you'd lose the ability to specify the for attribute.

If you want to try to make jQuery validation produce an error list, that's a bit harder. You can specify a combination of jQuery validation options and get a list, but there's not an obvious way to get the errorlist class on the ul.

$('form').validate({
    errorElement: 'li',
    wrapper: 'ul'
});

Other options you can look into are errorLabelContainer, errorContainer, and a highlight function.

Final Recommendations

I find it's easiest to specify class and metadata rules in custom widget attributes 90% of the time, and use a rules object only when absolutely necessary. For example, if I want to require only the first elements in a formset, but not the rest, then I may use a rules object in addition to class and metadata rules. For error messages, I generally use a field template like the above example that I include for each field:

{% with form.title as field %}{% include "field.html" %}{% endwith %}

Or if the form is really simple, I do

{% for field in form %}{% include "field.html" %}{% endfor %}
1Mar/101

Django Model Formsets

Django model formsets provide a way to edit multiple model instances within a single form. This is especially useful for editing related models inline. Below is some knowledge I've collected on some of the lesser documented and undocumented features of Django's model formsets.

Model Formset Factory Methods

Django Model Formsets are generally created using a factory method. The default is modelformset_factory, which wraps formset_factory to create Model Forms. You can also create inline formsets to edit related objects, using inlineformset_factory. inlineformset_factory wraps modelformset_factory to restrict the queryset and set the initial data to the instance's related objects.

Adding Fields to a Model Formset

Just like with a normal Django formset, you can add additional fields to a model formset by creating a base formset class with an add_fields method, then passing it in to the factory method. The only difference is the class you inherit from. For inlineformset_factory, you should inherit from BaseInlineFormSet.

If you're using modelformset_factory, then you should import and inherit from BaseModelFormSet instead. Also remember that form.instance may be used to set initial data for the fields you're adding. Just check to make sure form.instance is not None before you try to access any properties.

from django.forms.models import BaseInlineFormSet, inlineformset_factory

class BaseFormSet(BaseInlineFormSet):
    def add_fields(self, form, index):
        super(BasePlanItemFormSet, self).add_fields(form, index)
        # add fields to the form

FormSet = inlineformset_factory(MyModel, MyRelatedModel, formset=BaseFormSet)

Changing the Default Form Field

If you'd like to customize one or more of the form fields within your model formset, you can create a formfield_callback function and pass it to the formset factory. For example, if you want to set required=False on all fields, you can do the following.

def custom_field_callback(field):
    return field.formfield(required=False)

FormSet = modelformset_factory(model, formfield_callback=custom_field_callback)

field.formfield() will create the default form field with whatever arguments you pass in. You can also create different fields, and use field.name to do field specific customization. Here's a more advanced example.

def custom_field_callback(field):
    if field.name == 'optional':
        return field.formfield(required=False)
    elif field.name == 'text':
        return field.formfield(widget=Textarea)
    elif field.name == 'integer':
        return IntegerField()
    else:
        return field.formfield()

Deleting Models in a Formset

Pass can_delete=True to your factory method, and you'll be able to delete the models in your formsets. Note that inlineformset_factory defaults to can_delete=True, while modelformset_factory defaults to can_delete=False.

Creating New Models with Extra Forms

As with normal formsets, you can pass an extra argument to your formset factory to create extra empty forms. These empty forms can then be used to create new models. Note that when you have extra empty forms in the formset, you'll get an equal number of None results when you call formset.save(), so you may need to filter those out if you're doing any post-processing on the saved objects.

If you want to set an upper limit on the number of extra forms, you can use the max_num argument to restrict the maximum number of forms. For example, if you want up to 6 forms in the formset, do the following:

MyFormSet = inlineformset_factory(MyModel, MyRelatedModel, extra=6, max_num=6)

Saving Django Model Formsets

Model formsets have a save method, just like with model forms, but in this case, you'll get a list of all modified instances instead of a single instance. Unmodified instances will not be returned. As mentioned above, if you have any extra empty forms, then those list elements will be None.

If you want to create custom save behavior, you can override 2 methods in your BaseFormSet class: save_new and save_existing. These methods look like this:

from django.forms.models import BaseInlineFormSet

class BaseFormSet(BaseInlineFormSet):
    def save_new(self, form, commit=True):
        # custom save behavior for new objects, form is a ModelForm
        return super(BaseFormSet, self).save_new(form, commit=commit)

    def save_existing(self, form, instance, commit=True):
        # custom save behavior for existing objects
        # instance is the existing object, and form has the updated data
        return super(BaseFormSet, self).save_existing(form, instance, commit=commit)

Inline Model Admin

Django's Admin Site includes the ability to specify InlineModelAdmin objects. Subclasses of InlineModelAdmin can use all the arguments of inlineformset_factory, plus some admin specific arguments. Everything mentioned above applies equally to InlineModelAdmin arguments: you can specify the number of extra forms, the maximum number of inline forms, and even your own formset with custom save behavior.

10Jan/102

Far Future Expires Header with django-storages S3Storage

One way to decrease your site's load time is to set a far future Expires header on all your static content. This doesn't help first-time visitors, but can greatly improve the experience of returning visitors. And you get to decrease your bandwidth needs at the same time, because all your static content will be cached by their browser.

S3

weotta puts all of its awesome plan images in Amazon's S3 using django-storages S3Storage backend, which by default does not set any Expires header. To remedy this, I set AWS_HEADERS in settings.py like so

from datetime import date, timedelta
tenyrs = date.today() + timedelta(days=365*10)
# Expires 10 years in the future at 8PM GMT
AWS_HEADERS = {
	'Expires': tenyrs.strftime('%a, %d %b %Y 20:00:00 GMT')
}

Now every uploaded file gets an Expires header set to 10 years in the future.

upload_to

One potential drawback to using a far future Expires header is that if you change the file content without also changing the file name, no one will notice because they'll keep using the old cached version of the file. Luckily, Django makes it easy to create (mostly) unique new file names by letting you include strftime formatting codes in a FileField or ImageField upload_to path, such as upload_to='images/%Y/%m/%d'. This way, every uploaded file automatically gets stored by date, which means it would take some deliberate effort to change the contents of a file without also changing the file name.

Tagged as: , , , , 2 Comments
9Nov/090

Django Tools and Links

Using Django
Social Apps
Forms
Notifications
Geolocation
Misc
29Sep/090

Django IA: Registration-Activation

django-registration is a pluggable Django app that implements a common registration-activation flow. This flow is quite similar to the password reset flow, but slightly simpler with only 3 views:

  1. register
  2. registration_complete
  3. activate

The basic idea is that an anonymous user can create a new account, but cannot login until they activate their account by clicking a link they'll receive in an activation email. It's a way to automatically verify that the new user has a valid email address, which is generally an acceptable proxy for proving that they're human. Here's an Information Architecture diagram, again using jjg's visual vocabulary.

Django Registration IA

Here's a more in-depth walk-thru with our fictional user named Bob:

  1. Bob encounters a section of the site that requires an account, and is redirected to the login page.
  2. But Bob does not have an account, so he goes to the registration page where he fills out a registration form.
  3. After submitting the registration form, Bob is taken to a page telling him that he needs to activate his account by clicking a link in an email that he should be receiving shortly.
  4. Bob checks his email, finds the activation email, and clicks the activation link.
  5. Bob is taken to a page that tells him his account is active, and he can now login.

As with password reset, I think the last step is unnecessary, and Bob should be automatically logged in when his account is activated. But to do that, you'll have to write your own custom activate view. Luckily, this isn't very hard. If you take a look at the code for registration.views.activate, the core code is actually quite simple:

from registration.models import RegistrationProfile

def activate(request, activation_key):
    user = RegistrationProfile.objects.activate_user(activation_key.lower())

    if not user:
        # handle invalid activation key
    else:
        # do stuff with the user, such as automatically login, then redirect

The rest of the custom activate view is up to you.

19Sep/097

Django IA: Auth Password Reset

Django comes with a lot of great built-in functionality. One of the most useful contrib apps is authentication, which (among other things) provides views for login, logout, and password reset. Login & logout are self-explanatory, but resetting a password is, by nature, somewhat complicated. Because it's a really bad idea to store passwords as plaintext, you can't just send a user their password when they forget it. Instead, you have to provide a secure mechanism for users to change their password themselves, even if they can't remember their original password. Lucky for us, Django auth provides this functionality out of the box. All you need to do is create the templates and hook-up the views. The code you need to write to make this happen is pretty simple, but it can be a bit tricky to understand how it all works together. There's actually 4 separate view functions that together provide a complete password reset mechanism. These view functions are

  1. password_reset
  2. password_reset_done
  3. password_reset_confirm
  4. password_reset_complete

Here's an Information Architecture diagram showing how these views fit together, using Jesse James Garrett's Visual Vocabulary. The 2 black dots are starting points, and the circled black dot is an end point.

Django Auth Password Reset IA

Here's a more in-depth walk-thru of what's going on, with a fictional user named Bob:

  1. Bob tries to login and fails, probably a couple times. Bob clicks a "Forgot your password?" link, which takes him to the password_reset view.
  2. Bob enters his email address, which is then used to find his User account.
  3. If Bob's User account is found, a password reset email is sent, and Bob is redirected to the password_reset_done view, which should tell him to check his email.
  4. Bob leaves the site to check his email. He finds the password reset email, and clicks the password reset link.
  5. Bob is taken to the password_reset_confirm view, which first validates that he can reset his password (this is handled with a hashed link token). If the token is valid, Bob is allowed to enter a new password. Once a new password is submitted, Bob is redirected to the password_reset_complete view.
  6. Bob can now login to your site with his new password.

This final step is the one minor issue I have with Django's auth password reset. The user just changed their password, why do they have to enter it again to login? Why can't we eliminate step 6 altogether, and automatically log the user in after they reset their password? In fact, you can eliminate step 6 with a bit of hacking on your own authentication backend, but that's a topic for another post.

26Aug/090

Django Forms, Utilities, OAuth, and OpenID Links

Form Customization
Utility Apps
OAuth and OpenID
27Jul/090

Various Django App Links

Django version control, design patterns, tree models, and pluggable applications:

26Apr/090

Deploying Django with Mercurial, Fab and Nginx

Writing web apps with Django can be a lot of fun, but deploying them can be a chore, even if you're using Apache. Here's a setup I've been using that makes deployment fast and easy. This all assumes you've got sudo access on a remote server running Ubuntu or something similar.

Mercurial

This setup assumes you've got 2 mercurial repositories: 1 on your local machine, and 1 on the remote server you're deploying to. In the remote repository, add the following to .hg/hgrc

[hooks]
changegroup = hg up

This makes mercurial run hg up whenever you push new code. Then in your local repo's .hg/hgrc, make sure the default path is to your remote repo. Here's an example

[paths]
default = ssh://user@domain.com/repo

Now when you run hg push, you don't need to include the path to the repo, and your code will be updated immediately.

Django FastCGI Deployment

Since I'm using nginx instead of Apache, we'll be deploying Django with FastCGI. Here's an example script you can use to start and restart your Django FastCGI server. Add this script to your mercurial repo as run_fcgi.sh.

#!/bin/bash
PIDFILE="/tmp/django.pid"
SOCKET="/tmp/django.sock"

# kill current fcgi process if it exists
if [ -f $PIDFILE ]; then
    kill `cat -- $PIDFILE`
    rm -f -- $PIDFILE
fi

python manage.py runfcgi socket=$SOCKET pidfile=$PIDFILE method=prefork

Important note: the FastCGI socket file will need to be readable & writable by nginx worker processes, which run as the www-data user in Ubuntu. This will be handled by the fab restart command below, or you could add chmod a+w $SOCKET to the end of the above script.

Nginx FastCGI Proxy

Nginx is a great high performance web server with simple configuration. Here's a simple example server config for proxying to your Django FastCGI process. Add this config to your mercurial repo as django.nginx.

server {
    listen 80;
    # change to your FQDN
    server_name YOUR.DOMAIN.COM;

    location / {
        # must be the same socket file as in the above fcgi script
        fastcgi_pass unix:/tmp/django.sock;
    }
}

On the remote server, make sure the following lines are in the http section of /etc/nginx/nginx.conf

include /etc/nginx/sites-enabled/*;
# fastcgi_params should contain a lot of fastcgi_param variables
include /etc/nginx/fastcgi_params;

You must also make sure there is a link in /etc/nginx/sites-enabled to your django.nginx config. Don't worry if django.nginx doesn't exist yet, it will once you run fab nginx the first time.

you@remote.ubuntu$ cd /etc/nginx/sites-enabled
you@remote.ubuntu$ sudo ln -s ../sites-available/django.nginx django.nginx

Python Fabric

Fab, or properly Fabric, is my favorite new tool. It's designed specifically for making remote deployment simple and easy. You create a fabfile where each function is a fab command that can run remote and sudo commands on one or more remote hosts. So let's deploy Django using fab. Here's an example fabfile with 2 commands: restart and nginx. These commands should only be run after you've done a hg push.

config.fab_hosts = ['YOUR.DOMAIN.COM']
config.projdir = '/PATH/TO/YOUR/REMOTE/HG/REPO'

def restart():
    sudo('cd %(projdir)s; run_fcgi.sh', user='www-data', fail='abort')

def nginx():
    sudo('cp %(projdir)s/django.nginx /etc/nginx/sites-available/', fail='abort')
    sudo('killall -HUP nginx', fail='abort')

fab restart

You only need to run fab restart if you've changed the actual Django python code. Changes to templates or static files don't require a restart and will be used automatically (because of the hg up changegroup hook). Executing run_fcgi.sh as the www-data user ensures that nginx can read & write the socket.

fab nginx

If you've changed your nginx server config, you can run fab nginx to install and reload the new server config without restarting the nginx server.

Wrap Up

Now that everything is setup, the next time you want to deploy some new code, it's as simple as hg push && fab restart. And if you've only changed templates, all you need to do is hg push. I hope this helps make your Django development life easier. It has certainly done so for me :)