jQuery Validation with Django Forms

Django has everything you need to do server-side validation, but it’s also a good idea to do client-side validation. Here’s how you can integrate the jQuery Validation plugin with your Django Forms.

Validation Rules

jQuery validation works by assigning validation rules to each element in your form. These rules can be assigned a couple different ways:

  1. Class Rules
  2. Metadata Rules
  3. Rules Object

Class Rules

The simplest validation rules, such as required, can be assigned as classes on your form elements. To do this in Django, you can specify custom widget attributes.

from django import forms
from django.forms import widgets

class MyForm(forms.Form):
    title = forms.CharField(required=True, widget=widgets.TextInput(attrs={
        'class': 'required'
    }))

In Django 1.2, there’s support for a required css class, but you can still use the technique above to specify other validation rules.

Metadata Rules

For validation methods that require arguments, such minlength and maxlength, you can create metadata in the class attribute. You’ll have to include the jQuery metadata plugin for this style of rules.

from django import forms
from django.forms import widgets

class MyForm(forms.Form):
    title = forms.CharField(required=True, minlength=2, maxlength=100, widget=widgets.TextInput(attrs={
        'class': '{required:true, minlength:2, maxlength:100}'
    }))

Rules Object

If your validation requirements are more complex, or you don’t want to use the metadata plugin or class based rules, you can create a rules object to pass to the validate method. This object can be generated in your template like so:

<script type="text/javascript">
FORM_RULES = {
    '{{ form.title.name }}': 'required'
};

$(document).ready(function() {
    $('form').validate({
        rules: FORM_RULES
    });
});
</script>

The reason I suggest generating the rules object in your template is to avoid hardcoding the field name in your javascript. A rules object can also be used in conjunction with class and metadata rules, so you could have some rules assigned in individual element classes or metadata, and other rules in your rules object.

Error Messages

If you want to keep the client-side validation error messages consistent with Django’s validation error messages, you’ll need to copy Django’s error messages and specify them in the metadata or in a messages object.

Metadata Messages

Messages must be specified per-field, and per-rule. Here’s an example where I specify the minlength message for the title field.

from django import forms
from django.forms import widgets

class MyForm(forms.Form):
    title = forms.CharField(minlength=2, widget=widgets.TextInput(attrs={
        'class': '{minlength:2, messages:{minlength:"Ensure this value has at least 2 characters"}}'
    }))

Messages Object

Messages can also be specified in javascript object, like so:

<script type="text/javascript">
FORM_RULES = {
    '{{ form.title.name }}': 'required'
};

FORM_MESSAGES = {
    '{{ form.title.name }}': 'This field is required'
};

$(document).ready(function() {
    $('form').validate({
        rules: FORM_RULES,
        messages: FORM_MESSAGES
    });
});
</script>

Just like with validation rules, messages in element metadata can be used in conjunction with a global messages object. Note: if an element has a title attribute, then the title will be used as the default error message, unless you specify ignoreTitle: false in the validation options.

Error Labels vs Errorlist

Django’s default error output is an error list, while the default for jQuery Validation errors is a label with class="error". So in order to unify your validation errors, there’s 2 options:

  1. make jQuery Validation output an error list
  2. output error labels instead of an error list in the template

Personally, I prefer the simple error labels produced by jQuery validation. To make Django generate those instead of an error list, you can do the following in your templates:

{{ field }}
{% if field.errors %}
{# NOTE: must use id_NAME for jquery.validation to overwrite error label #}
<label class='error' for='id_{{ field.name }}' generated="true">{{ field.errors|join:". " }}</label>
{% endif %}

You could also create your own error_class for outputting the error labels, but then you’d lose the ability to specify the for attribute.

If you want to try to make jQuery validation produce an error list, that’s a bit harder. You can specify a combination of options to validate and get a list, but there’s not an obvious way to get the errorlist class on the ul.

$('form').validate({
    errorElement: 'li',
    wrapper: 'ul'
});

Other options you can look into are errorLabelContainer, errorContainer, and a highlight function.

Final Recommendations

I find it’s easiest to specify class and metadata rules in custom widget attributes 90% of the time, and use a rules object only when absolutely necessary. For example, if I want to require only the first elements in a formset, but not the rest, then I may use a rules object in addition to class and metadata rules. For error messages, I generally use a field template like the above example that I include for each field:

{% with form.title as field %}{% include "field.html" %}{% endwith %}

Or if the form is really simple, I do

{% for field in form %}{% include "field.html" %}{% endfor %}

cheap web hosting

Django Model Formsets

Django model formsets provide a way to edit multiple model instances within a single form. This is especially useful for editing related models inline. Below is some knowledge I’ve collected on some of the lesser documented and undocumented features of Django’s model formsets.

Formset Factory Methods

Model Formsets are generally created using a factory method. The default is modelformset_factory, which wraps formset_factory to create Model Forms. You can also create inline formsets to edit related objects, using inlineformset_factory. inlineformset_factory wraps modelformset_factory to restrict the queryset and set the initial data to the instance’s related objects.

Adding Fields to a Formset

Just like with a normal formset, you can add additional fields to a model formset by creating a base formset class with an add_fields method, then passing it in to the factory method. The only difference is the class you inherit from. For inlineformset_factory, you should inherit from BaseInlineFormSet.

If you’re using modelformset_factory, then you should import and inherit from BaseModelFormSet instead. Also remember that form.instance may be used to set initial data for the fields you’re adding. Just check to make sure form.instance is not None before you try to access any properties.

from django.forms.models import BaseInlineFormSet, inlineformset_factory

class BaseFormSet(BaseInlineFormSet):
    def add_fields(self, form, index):
        super(BasePlanItemFormSet, self).add_fields(form, index)
        # add fields to the form

FormSet = inlineformset_factory(MyModel, MyRelatedModel, formset=BaseFormSet)

Changing the Default Field

If you’d like to customize one or more of the form fields within your model formset, you can create a formfield_callback function and pass it to the formset factory. For example, if you want to set required=False on all fields, you can do the following.

def custom_field_callback(field):
    return field.formfield(required=False)

FormSet = modelformset_factory(model, formfield_callback=custom_field_callback)

field.formfield() will create the default form field with whatever arguments you pass in. You can also create different fields, and use field.name to do field specific customization. Here’s a more advanced example.

def custom_field_callback(field):
    if field.name == 'optional':
        return field.formfield(required=False)
    elif field.name == 'text':
        return field.formfield(widget=Textarea)
    elif field.name == 'integer':
        return IntegerField()
    else:
        return field.formfield()

Deleting Models in a Formset

Pass can_delete=True to your factory method, and you’ll be able to delete the models in your formsets. Note that inlineformset_factory defaults to can_delete=True, while modelformset_factory defaults to can_delete=False.

Creating New Models with Extra Forms

As with normal formsets, you can pass an extra argument to your formset factory to create extra empty forms. These empty forms can then be used to create new models. Note that when you have extra empty forms in the formset, you’ll get an equal number of None results when you call formset.save(), so you may need to filter those out if you’re doing any post-processing on the saved objects.

If you want to set an upper limit on the number of extra forms, you can use the max_num argument to restrict the maximum number of forms. For example, if you want up to 6 forms in the formset, do the following:

MyFormSet = inlineformset_factory(MyModel, MyRelatedModel, extra=6, max_num=6)

Saving Model Formsets

Model formsets have a save method, just like with model forms, but in this case, you’ll get a list of all modified instances instead of a single instance. Unmodified instances will not be returned. As mentioned above, if you have any extra empty forms, then those list elements will be None.

If you want to create custom save behavior, you can override 2 methods in your BaseFormSet class: save_new and save_existing. These methods look like this:

from django.forms.models import BaseInlineFormSet

class BaseFormSet(BaseInlineFormSet):
    def save_new(self, form, commit=True):
        # custom save behavior for new objects, form is a ModelForm
        return super(BaseFormSet, self).save_new(form, commit=commit)

    def save_existing(self, form, instance, commit=True):
        # custom save behavior for existing objects
        # instance is the existing object, and form has the updated data
        return super(BaseFormSet, self).save_existing(form, instance, commit=commit)

Inline Model Admin

Django’s Admin Site includes the ability to specify InlineModelAdmin objects. Subclasses of InlineModelAdmin can use all the arguments of inlineformset_factory, plus some admin specific arguments. Everything mentioned above applies equally to InlineModelAdmin arguments: you can specify the number of extra forms, the maximum number of inline forms, and even your own formset with custom save behavior.

Mnesia Records to MongoDB Documents

I recently migrated about 50k records from mnesia to MongoDB using my fork of emongo, which adds supervisors with transparent connection restarting, for reasons I’ll explain below.

Why Mongo instead of Mnesia

mnesia is great for a number of reasons, but here’s why I decided to move weotta’s place data into MongoDB:

Converting Records to Docs and vice versa

First, I needed to convert records to documents. In erlang, mongo documents are basically proplists. Keys going into emongo can be atoms, strings, or binaries, but keys coming out will always by binaries. Here’s a simple example of record to document conversion:

record_to_doc(Record, Attrs) ->
    % tl will drop record name
    lists:zip(Attrs, tl(tuple_to_list(Record))).

This would be called like record_to_doc(MyRecord, record_info(fields, my_record)). If you have nested dicts then you’ll have to flatten them using dict:to_list. Also note that list values are coming out of emongo are treated like yaws JSON arrays, i.e. [{key, {array, [val]}}]. For more examples, check out the emongo docs.

Heavy Write Load

To do the migration, I used etable:foreach to insert each document. Bulk insertion would probably be more efficient, but etable makes single record iteration very easy.

I started using the original emongo with a pool size of 10, but it was crashy when I dumped records as fast as possible. So initially I slowed it down with timer:sleep(200), but after adding supervised connections, I was able to dump with no delay. I’m not exactly sure what I fixed in this case, but I think the lesson is that using supervised gen_servers will give you reliability with little effort.

Read Performance

Now that I had data in mongo to play with, I compared the read performance to mnesia. Using timer:tc, I found that mnesia:dirty_read takes about 21 microseconds, whereas emongo:find_one can take anywhere from 600 to 1200 microseconds, querying on an indexed field. Without an index, read performance ranged from 900 to 2000 microseconds. I also tested only requesting specific fields, as recommended on the MongoDB Optimiziation page, but with small documents (<10 fields) that did not seem to have any effect. So while mongodb queries are pretty fast at 1ms, mnesia is about 50 times faster. Further inspection with fprof showed that nearly half of the cpu time of emongo:find is taken by BSON decoding.

Heavy Read Load

Under heavy read load (thousands of find_one calls in less than second), emongo_conn would get into a locked state. Somehow the process had accumulated unparsable data and wouldn’t reply. This problem went away when I increased the size of the pool size to 100, but that’s a ridiculous number of connections to keep open permanently. So instead I added some code to kill the connection on timeout and retry the find call. This was the main reason I added supervision. Now, every pool is locally registered as a simple_one_for_one supervisor that supervises every emongo_server connection. This pool is in turn supervised by emongo_sup, with dynamically added child specs. All this supervision allowed me to lower the pool size back to 10, and made it easy to kill and restart emongo_server connections as needed.

Why you may want to stick with Mnesia

Now that I have experience with both MongoDB and mnesia, here’s some reasons you may want to stick with mnesia:

Despite all that, I’m very happy with MongoDB. Installation and setup were a breeze, and schema-less data storage is very nice when you have variable fields and a high probability of adding and/or removing fields in the future. It’s simple, scalable, and as mentioned above, it’s very easy to access from many different languages. emongo isn’t perfect, but it’s open source and will hopefully benefit from more exposure.

A/B Testing Links

A/B Testing Links:

A/B Testing with Mixpanel:

Proxy Links

Programming Philosophy Links

Far Future Expires Header with django-storages S3Storage

One way to decrease your site’s load time is to set a far future Expires header on all your static content. This doesn’t help first-time visitors, but can greatly improve the experience of returning visitors. And you get to decrease your bandwidth needs at the same time, because all your static content will be cached by their browser.

S3

weotta puts all of its awesome plan images in Amazon’s S3 using django-storages S3Storage backend, which by default does not set any Expires header. To remedy this, I set AWS_HEADERS in settings.py like so

from datetime import date, timedelta
tenyrs = date.today() + timedelta(days=365*10)
# Expires 10 years in the future at 8PM GMT
AWS_HEADERS = {
	'Expires': tenyrs.strftime('%a, %d %b %Y 20:00:00 GMT')
}

Now every uploaded file gets an Expires header set to 10 years in the future.

upload_to

One potential drawback to using a far future Expires header is that if you change the file content without also changing the file name, no one will notice because they’ll keep using the old cached version of the file. Luckily, Django makes it easy to create (mostly) unique new file names by letting you include strftime formatting codes in a FileField or ImageField upload_to path, such as upload_to='images/%Y/%m/%d'. This way, every uploaded file automatically gets stored by date, which means it would take some deliberate effort to change the contents of a file without also changing the file name.

Dealing with SSH Bruteforce Attacks

erldis – an Erlang Redis Client

Since it’s now featured on the redis homepage, I figure I should tell people about my fork of erldis, an erlang redis client focused on synchronous operations.

Synchronicity

The original client, which still exists as erldis_client.erl, implements asynchronous pipelining. This means you send a bunch of redis commands, then collect all the results at the end. This didn’t work for me, as I needed a client that could handle parallel synchronous requests from multiple concurrent processes. So I copied erldis_client.erl to erldis_sync_client.erl and modified it to send replies back as soon as they are received from redis (in FIFO order). Many thanks to dialtone_ for writing the original erldis app as I’m not sure I would’ve created the synchronous client without it. And thanks to cstar for patches, such as making erldis_sync_client the default client for all functions in erldis.erl.

Extras

In addition to the synchronous client, I’ve added some extra functions and modules to make interfacing with redis more erlangy. Here’s a brief overview…

erldis_sync_client:transact

erldis_sync_client:transact is analagous to mnesia:transaction in that it does a unit of work against a redis database, like so:

  1. starts erldis_sync_client
  2. calls your function with the client PID as the argument
  3. stops the client
  4. returns the result of your function

The goal being to reduce boilerplate start/stop code.

erldis_dict module

erldis_dict provides similar semantics as the dict module in stdlib, using redis key-value commands.

erldis_list module

erldis_list provides a number of functions operating on redis lists, inspired by the array, lists, and queue modules in stdlib. You must pass in both the client PID and a redis list key.

erldis_sets module

erldis_sets works like the sets module, but you have to provide both the client PID and a redis set key.

Usage

Despite the low version numbers, I’ve been successfully using erldis as a component in parallel/distributed information retrieval (in conjunction with plists), and for accessing data shared with python / django apps. It’s a fully compliant erlang application that you can include in your target system release structure.

If also you’re using erldis for your redis needs, I’d love to hear about it.

Execnet vs Disco for Distributed NLTK

There’s a number of options for distributed processing and mapreduce in python. Before execnet surfaced, I’d been using Disco to do distributed NLTK. Now that I’ve happily switched to distributed NLTK with execnet, I can explain some of the differences and why execnet is so much better for my purposes.

Disco Overhead

Disco is a mapreduce framework for python, with an erlang core. This is very cool, but unfortunately introduces overhead costs when your functions are not pure (meaning they require external code and/or data). And part of speech tagging with NLTK is definitely not pure; the map function requires a part of speech tagger in order to do anything. So to use a part of speech tagger within a Disco map function, it must be loaded inline, which means unpickling the object before doing any work. And since a pickled part of speech tagger can easily exceed 500K, unpickling it can take over 2 seconds. When every map call has a fixed overhead of 2 seconds, your mapreduce task can take orders of magnitude longer to complete.

As an example, let’s say you need to do 6000 map calls, at 1 second of pure computation each. That’s 100 minutes, not counting overhead. Now add in the 2s fixed overhead on each call, and you’re at 300 minutes. What should be just over 1.6 hours of computation has jumped to 5 hours.

Execnet FTW

execnet provides a very different computational model: start some gateways and communicate thru message channels. In my case, all the fixed overhead can be done up-front, loading the part of speech tagger once per gateway, resulting in greatly reduced compute times. I did have to change my old Disco based code to work with execnet, but I actually ended up with less code that’s easier to understand.

Conclusion

If you’re just doing pure mapreduce computations, then consider using Disco. After the one time setup (which can be non-trivial), writing the functions will be relatively easy, and you’ll get a nice web UI for configuration and monitoring. But if you’re doing any dirty operations that need expensive initialization procedures, or can’t quite fit what you need into a pure mapreduce framework, then execnet is for you.