Posts tagged python
jQuery Validation with Django Forms
Mar 8th
Django has everything you need to do server-side validation, but it’s also a good idea to do client-side validation. Here’s how you can integrate the jQuery Validation plugin with your Django Forms.
Validation Rules
jQuery validation works by assigning validation rules to each element in your form. These rules can be assigned a couple different ways:
- Class Rules
- Metadata Rules
- Rules Object
Class Rules
The simplest validation rules, such as required, can be assigned as classes on your form elements. To do this in Django, you can specify custom widget attributes.
from django import forms
from django.forms import widgets
class MyForm(forms.Form):
title = forms.CharField(required=True, widget=widgets.TextInput(attrs={
'class': 'required'
}))
In Django 1.2, there’s support for a required css class, but you can still use the technique above to specify other validation rules.
Metadata Rules
For validation methods that require arguments, such minlength and maxlength, you can create metadata in the class attribute. You’ll have to include the jQuery metadata plugin for this style of rules.
from django import forms
from django.forms import widgets
class MyForm(forms.Form):
title = forms.CharField(required=True, minlength=2, maxlength=100, widget=widgets.TextInput(attrs={
'class': '{required:true, minlength:2, maxlength:100}'
}))
Rules Object
If your validation requirements are more complex, or you don’t want to use the metadata plugin or class based rules, you can create a rules object to pass to the validate method. This object can be generated in your template like so:
<script type="text/javascript">
FORM_RULES = {
'{{ form.title.name }}': 'required'
};
$(document).ready(function() {
$('form').validate({
rules: FORM_RULES
});
});
</script>
The reason I suggest generating the rules object in your template is to avoid hardcoding the field name in your javascript. A rules object can also be used in conjunction with class and metadata rules, so you could have some rules assigned in individual element classes or metadata, and other rules in your rules object.
Error Messages
If you want to keep the client-side validation error messages consistent with Django’s validation error messages, you’ll need to copy Django’s error messages and specify them in the metadata or in a messages object.
Metadata Messages
Messages must be specified per-field, and per-rule. Here’s an example where I specify the minlength message for the title field.
from django import forms
from django.forms import widgets
class MyForm(forms.Form):
title = forms.CharField(minlength=2, widget=widgets.TextInput(attrs={
'class': '{minlength:2, messages:{minlength:"Ensure this value has at least 2 characters"}}'
}))
Messages Object
Messages can also be specified in javascript object, like so:
<script type="text/javascript">
FORM_RULES = {
'{{ form.title.name }}': 'required'
};
FORM_MESSAGES = {
'{{ form.title.name }}': 'This field is required'
};
$(document).ready(function() {
$('form').validate({
rules: FORM_RULES,
messages: FORM_MESSAGES
});
});
</script>
Just like with validation rules, messages in element metadata can be used in conjunction with a global messages object. Note: if an element has a title attribute, then the title will be used as the default error message, unless you specify ignoreTitle: false in the validation options.
Error Labels vs Errorlist
Django’s default error output is an error list, while the default for jQuery Validation errors is a label with class="error". So in order to unify your validation errors, there’s 2 options:
- make jQuery Validation output an error list
- output error labels instead of an error list in the template
Personally, I prefer the simple error labels produced by jQuery validation. To make Django generate those instead of an error list, you can do the following in your templates:
{{ field }}
{% if field.errors %}
{# NOTE: must use id_NAME for jquery.validation to overwrite error label #}
<label class='error' for='id_{{ field.name }}' generated="true">{{ field.errors|join:". " }}</label>
{% endif %}
You could also create your own error_class for outputting the error labels, but then you’d lose the ability to specify the for attribute.
If you want to try to make jQuery validation produce an error list, that’s a bit harder. You can specify a combination of options to validate and get a list, but there’s not an obvious way to get the errorlist class on the ul.
$('form').validate({
errorElement: 'li',
wrapper: 'ul'
});
Other options you can look into are errorLabelContainer, errorContainer, and a highlight function.
Final Recommendations
I find it’s easiest to specify class and metadata rules in custom widget attributes 90% of the time, and use a rules object only when absolutely necessary. For example, if I want to require only the first elements in a formset, but not the rest, then I may use a rules object in addition to class and metadata rules. For error messages, I generally use a field template like the above example that I include for each field:
{% with form.title as field %}{% include "field.html" %}{% endwith %}Or if the form is really simple, I do
{% for field in form %}{% include "field.html" %}{% endfor %}Django Model Formsets
Mar 1st
Django model formsets provide a way to edit multiple model instances within a single form. This is especially useful for editing related models inline. Below is some knowledge I’ve collected on some of the lesser documented and undocumented features of Django’s model formsets.
Formset Factory Methods
Model Formsets are generally created using a factory method. The default is modelformset_factory, which wraps formset_factory to create Model Forms. You can also create inline formsets to edit related objects, using inlineformset_factory. inlineformset_factory wraps modelformset_factory to restrict the queryset and set the initial data to the instance’s related objects.
Adding Fields to a Formset
Just like with a normal formset, you can add additional fields to a model formset by creating a base formset class with an add_fields method, then passing it in to the factory method. The only difference is the class you inherit from. For inlineformset_factory, you should inherit from BaseInlineFormSet.
If you’re using modelformset_factory, then you should import and inherit from BaseModelFormSet instead. Also remember that form.instance may be used to set initial data for the fields you’re adding. Just check to make sure form.instance is not None before you try to access any properties.
from django.forms.models import BaseInlineFormSet, inlineformset_factory
class BaseFormSet(BaseInlineFormSet):
def add_fields(self, form, index):
super(BasePlanItemFormSet, self).add_fields(form, index)
# add fields to the form
FormSet = inlineformset_factory(MyModel, MyRelatedModel, formset=BaseFormSet)
Changing the Default Field
If you’d like to customize one or more of the form fields within your model formset, you can create a formfield_callback function and pass it to the formset factory. For example, if you want to set required=False on all fields, you can do the following.
def custom_field_callback(field):
return field.formfield(required=False)
FormSet = modelformset_factory(model, formfield_callback=custom_field_callback)
field.formfield() will create the default form field with whatever arguments you pass in. You can also create different fields, and use field.name to do field specific customization. Here’s a more advanced example.
def custom_field_callback(field):
if field.name == 'optional':
return field.formfield(required=False)
elif field.name == 'text':
return field.formfield(widget=Textarea)
elif field.name == 'integer':
return IntegerField()
else:
return field.formfield()
Deleting Models in a Formset
Pass can_delete=True to your factory method, and you’ll be able to delete the models in your formsets. Note that inlineformset_factory defaults to can_delete=True, while modelformset_factory defaults to can_delete=False.
Creating New Models with Extra Forms
As with normal formsets, you can pass an extra argument to your formset factory to create extra empty forms. These empty forms can then be used to create new models. Note that when you have extra empty forms in the formset, you’ll get an equal number of None results when you call formset.save(), so you may need to filter those out if you’re doing any post-processing on the saved objects.
If you want to set an upper limit on the number of extra forms, you can use the max_num argument to restrict the maximum number of forms. For example, if you want up to 6 forms in the formset, do the following:
MyFormSet = inlineformset_factory(MyModel, MyRelatedModel, extra=6, max_num=6)
Saving Model Formsets
Model formsets have a save method, just like with model forms, but in this case, you’ll get a list of all modified instances instead of a single instance. Unmodified instances will not be returned. As mentioned above, if you have any extra empty forms, then those list elements will be None.
If you want to create custom save behavior, you can override 2 methods in your BaseFormSet class: save_new and save_existing. These methods look like this:
from django.forms.models import BaseInlineFormSet
class BaseFormSet(BaseInlineFormSet):
def save_new(self, form, commit=True):
# custom save behavior for new objects, form is a ModelForm
return super(BaseFormSet, self).save_new(form, commit=commit)
def save_existing(self, form, instance, commit=True):
# custom save behavior for existing objects
# instance is the existing object, and form has the updated data
return super(BaseFormSet, self).save_existing(form, instance, commit=commit)
Inline Model Admin
Django’s Admin Site includes the ability to specify InlineModelAdmin objects. Subclasses of InlineModelAdmin can use all the arguments of inlineformset_factory, plus some admin specific arguments. Everything mentioned above applies equally to InlineModelAdmin arguments: you can specify the number of extra forms, the maximum number of inline forms, and even your own formset with custom save behavior.
Mnesia Records to MongoDB Documents
Feb 1st
I recently migrated about 50k records from mnesia to MongoDB using my fork of emongo, which adds supervisors with transparent connection restarting, for reasons I’ll explain below.
Why Mongo instead of Mnesia
mnesia is great for a number of reasons, but here’s why I decided to move weotta’s place data into MongoDB:
- easy to access from python and other languages
- schema-less data, so you’re not constrained to records, and will never have to do mnesia:transform_table ever again
- don’t have to keep everything in memory (or only on disk as the case may be)
- simple & flexible indexing & querying
Converting Records to Docs and vice versa
First, I needed to convert records to documents. In erlang, mongo documents are basically proplists. Keys going into emongo can be atoms, strings, or binaries, but keys coming out will always by binaries. Here’s a simple example of record to document conversion:
record_to_doc(Record, Attrs) ->
% tl will drop record name
lists:zip(Attrs, tl(tuple_to_list(Record))).
This would be called like record_to_doc(MyRecord, record_info(fields, my_record)). If you have nested dicts then you’ll have to flatten them using dict:to_list. Also note that list values are coming out of emongo are treated like yaws JSON arrays, i.e. [{key, {array, [val]}}]. For more examples, check out the emongo docs.
Heavy Write Load
To do the migration, I used etable:foreach to insert each document. Bulk insertion would probably be more efficient, but etable makes single record iteration very easy.
I started using the original emongo with a pool size of 10, but it was crashy when I dumped records as fast as possible. So initially I slowed it down with timer:sleep(200), but after adding supervised connections, I was able to dump with no delay. I’m not exactly sure what I fixed in this case, but I think the lesson is that using supervised gen_servers will give you reliability with little effort.
Read Performance
Now that I had data in mongo to play with, I compared the read performance to mnesia. Using timer:tc, I found that mnesia:dirty_read takes about 21 microseconds, whereas emongo:find_one can take anywhere from 600 to 1200 microseconds, querying on an indexed field. Without an index, read performance ranged from 900 to 2000 microseconds. I also tested only requesting specific fields, as recommended on the MongoDB Optimiziation page, but with small documents (<10 fields) that did not seem to have any effect. So while mongodb queries are pretty fast at 1ms, mnesia is about 50 times faster. Further inspection with fprof showed that nearly half of the cpu time of emongo:find is taken by BSON decoding.
Heavy Read Load
Under heavy read load (thousands of find_one calls in less than second), emongo_conn would get into a locked state. Somehow the process had accumulated unparsable data and wouldn’t reply. This problem went away when I increased the size of the pool size to 100, but that’s a ridiculous number of connections to keep open permanently. So instead I added some code to kill the connection on timeout and retry the find call. This was the main reason I added supervision. Now, every pool is locally registered as a simple_one_for_one supervisor that supervises every emongo_server connection. This pool is in turn supervised by emongo_sup, with dynamically added child specs. All this supervision allowed me to lower the pool size back to 10, and made it easy to kill and restart emongo_server connections as needed.
Why you may want to stick with Mnesia
Now that I have experience with both MongoDB and mnesia, here’s some reasons you may want to stick with mnesia:
- very fast in-memory reads
- transactional
- simple master-master replication
- great for distributed read-heavy applications
Despite all that, I’m very happy with MongoDB. Installation and setup were a breeze, and schema-less data storage is very nice when you have variable fields and a high probability of adding and/or removing fields in the future. It’s simple, scalable, and as mentioned above, it’s very easy to access from many different languages. emongo isn’t perfect, but it’s open source and will hopefully benefit from more exposure.
Related Reading:
Scalable Database Links
Oct 26th
Redis:
- Redis vs MySQL vs Tokyo Tyrant (on EC2) « Colin Howe’s Blog
- Key-Value Stores for Ruby (Part 4): To Redis or Not To Redis? | Engine Yard Blog
Cassandra:
- Jonathan Ellis’s Programming Blog – Spyced: Why I like the Cassandra distributed database
- ieure’s python-cassandra at master – GitHub
- digg’s lazyboy at master – GitHub
Performance Tradeoffs:
- Debunking a Myth: Column-Stores vs. Indexes – The Database Column
- Debunking Another Myth: Column-Stores vs. Vertical Partitioning – The Database Column
- Code Monkeyism: Essential storage tradeoff: Simple Reads vs. Simple Writes
Other:
- Hypertable: An Open Source, High Performance, Scalable Database
- Welcome to django-denorm’s documentation! — django-denorm v0.1 documentation
Related Reading:
Django IA: Registration-Activation
Sep 29th
django-registration is a pluggable Django app that implements a common registration-activation flow. This flow is quite similar to the password reset flow, but slightly simpler with only 3 views:
registerregistration_completeactivate
The basic idea is that an anonymous user can create a new account, but cannot login until they activate their account by clicking a link they’ll receive in an activation email. It’s a way to automatically verify that the new user has a valid email address, which is generally an acceptable proxy for proving that they’re human. Here’s an Information Architecture diagram, again using jjg’s visual vocabulary.

Here’s a more in-depth walk-thru with our fictional user named Bob:
- Bob encounters a section of the site that requires an account, and is redirected to the login page.
- But Bob does not have an account, so he goes to the registration page where he fills out a registration form.
- After submitting the registration form, Bob is taken to a page telling him that he needs to activate his account by clicking a link in an email that he should be receiving shortly.
- Bob checks his email, finds the activation email, and clicks the activation link.
- Bob is taken to a page that tells him his account is active, and he can now login.
As with password reset, I think the last step is unnecessary, and Bob should be automatically logged in when his account is activated. But to do that, you’ll have to write your own custom activate view. Luckily, this isn’t very hard. If you take a look at the code for registration.views.activate, the core code is actually quite simple:
from registration.models import RegistrationProfile
def activate(request, activation_key):
user = RegistrationProfile.objects.activate_user(activation_key.lower())
if not user:
# handle invalid activation key
else:
# do stuff with the user, such as automatically login, then redirect
The rest of the custom activate view is up to you.
Django IA: Auth Password Reset
Sep 19th
Django comes with a lot of great built-in functionality. One of the most useful contrib apps is authentication, which (among other things) provides views for login, logout, and password reset. Login & logout are self-explanatory, but resetting a password is, by nature, somewhat complicated. Because it’s a really bad idea to store passwords as plaintext, you can’t just send a user their password when they forget it. Instead, you have to provide a secure mechanism for users to change their password themselves, even if they can’t remember their original password. Lucky for us, Django auth provides this functionality out of the box. All you need to do is create the templates and hook-up the views. The code you need to write to make this happen is pretty simple, but it can be a bit tricky to understand how it all works together. There’s actually 4 separate view functions that together provide a complete password reset mechanism. These view functions are
password_resetpassword_reset_donepassword_reset_confirmpassword_reset_complete
Here’s an Information Architecture diagram showing how these views fit together, using Jesse James Garrett’s Visual Vocabulary. The 2 black dots are starting points, and the circled black dot is an end point.

Here’s a more in-depth walk-thru of what’s going on, with a fictional user named Bob:
- Bob tries to login and fails, probably a couple times. Bob clicks a “Forgot your password?” link, which takes him to the
password_resetview. - Bob enters his email address, which is then used to find his User account.
- If Bob’s User account is found, a password reset email is sent, and Bob is redirected to the
password_reset_doneview, which should tell him to check his email. - Bob leaves the site to check his email. He finds the password reset email, and clicks the password reset link.
- Bob is taken to the
password_reset_confirmview, which first validates that he can reset his password (this is handled with a hashed link token). If the token is valid, Bob is allowed to enter a new password. Once a new password is submitted, Bob is redirected to thepassword_reset_completeview. - Bob can now login to your site with his new password.
This final step is the one minor issue I have with Django’s auth password reset. The user just changed their password, why do they have to enter it again to login? Why can’t we eliminate step 6 altogether, and automatically log the user in after they reset their password? In fact, you can eliminate step 6 with a bit of hacking on your own authentication backend, but that’s a topic for another post.
Cloud Computing Links
Sep 1st
Amazon Web Services:
- boto – Project Hosting on Google Code
- The Codebelay Blog » How to Load Balance and Auto Scale with Amazon’s EC2
- Getting Good IO from Amazon’s EBS
Python Libraries:
GlusterFS:
A Few Database Links
Aug 19th
- Lux IO – Yet Another Fast Database Manager
- WELCOME TO CLOUDSTORE
- MemcacheDB, Tokyo Tyrant, Redis performance test
- NoSQL: If Only It Was That Easy
- Riak – A Decentralized Database
- PyTables – Getting the most *out* of your data
- Durus: a Software Package from MNX
- Python Package Index : shove 0.2.1




































