Testing Command Line Scripts with Roundup
As nltk-trainer becomes more stable, I realized that I needed some way to test the command line scripts. My previous ad-hoc method of "test whatever script options I can remember" was becoming unwieldy and unreliable. But how do you make repeatable tests for a command line script? It doesn't really fit into the standard unit testing model.
Enter roundup by Blake Mizerany. (NOTE: do not try to do apt-get install roundup. You will get an issue tracking system, not a script testing tool).
Roundup provides a great way to prevent shell bugs by creating simple test functions within a shell script. Here's the first dozen lines of train_classifier.sh, which you can probably guess tests train_classifier.py:
#!/usr/bin/env roundup
describe "train_classifier.py"
it_displays_usage_when_no_arguments() {
./train_classifier.py 2>&1 | grep -q "usage: train_classifier.py"
}
it_cannot_find_foo() {
last_line=$(./train_classifier.py foo 2>&1 | tail -n 1)
test "$last_line" "=" "ValueError: cannot find corpus path for foo"
}
describe is like the name of a module or test case, and all test functions begin with test_. Within the test functions, you use standard shell commands that should produce no output on success (like grep -q or the test command). You can also match multiple lines of output, as in:
it_trains_movie_reviews_paras() {
test "$(./train_classifier.py movie_reviews --no-pickle --no-eval --fraction 0.5 --instances paras)" "=" "loading movie_reviews
2 labels: ['neg', 'pos']
1000 training feats, 1000 testing feats
training NaiveBayes classifier"
}
Once you've got all your test functions defined, make sure your test script is executable and roundup is installed, then run your test script. You'll get nice output that looks like:
nltk-trainer$ tests/train_classifier.sh train_classifier.py it_displays_usage_when_no_arguments: [PASS] it_cannot_find_foo: [PASS] it_cannot_import_reader: [PASS] it_trains_movie_reviews_paras: [PASS] it_trains_corpora_movie_reviews_paras: [PASS] it_cross_fold_validates: [PASS] it_trains_movie_reviews_sents: [PASS] it_trains_movie_reviews_maxent: [PASS] it_shows_most_informative: [PASS] ========================================================= Tests: 9 | Passed: 9 | Failed: 0
So far, roundup has been a perfect tool for testing all the nltk-trainer scripts, and the only downside is the one-time manual installation. I highly recommend it for anyone writing custom commands and scripts, no matter what language you use to write them.
Hierarchical Classification
Hierarchical classification is an obscure but simple concept. The idea is that you arrange two or more classifiers in a hierarchy such that the classifiers lower in the hierarchy are only used if a higher classifier returns an appropriate result.
For example, the text-processing.com sentiment analysis demo uses hierarchical classification by combining a subjectivity classifier and a polarity classifier. The subjectivity classifier is first, and determines whether the text is objective or subjective. If the text is objective, then a label of neutral is returned, and the polarity classifier is not used. However, if the text is subjective (or polar), then the polarity classifier is used to determine if the text is positive or negative.
Hierarchical classification is a useful way to combine multiple binary classifiers, if you have a hierarchy of labels that can modeled as a binary tree. In this model, each branch of the tree either continues on to a new pair of branches, or stops, and at each branching you use a classifier to determine which branch to take.
The Perils of Web Crawling
I've written a lot of web crawlers, and these are some of the problems I've encountered in the process. Unfortunately, most of these problems have no solution, other than quitting in disgust. Instead of looking for a way out when you're already deep in it, take this article as a warning of what to expect before you start down the perilous path of making a web spider. Now to be clear, I'm referring to deep web crawlers coded for a specific site, not a generic link harvester or image downloader. Generic crawlers probably have their own set of problems on top of everything mentioned below.
Most web pages suck
I have yet to encounter a website whose HTML didn't generate at least 3 WTFs.
Seriously, I now have a deep respect for web browsers that are able to render the mess of HTML that exists out there. If only browsers were better at memory management, I might actually like them again (see unified theory of browser suckage).
Inconsistent information & navigation
2 pages on the same site, with the same url pattern, and the same kind of information, may still not be the same. One will have additional information that you didn't expect, breaking all your assumptions about what HTML is where. Another may not have essential information you expected all similar pages to have, again throwing your HTML parsing assumptions out the window.
Making things worse, the same kind of page on different sections of a site may be completely different. This generally occurs on larger sites where different sections have different teams managing them. The worst is when the differences are non-obvious, so that you think they're the same, until you realize hours later that the underlying HTML structure is completely different, and CSS had been fooling you, making it look the same when it's not.
90% of writing a crawler is debugging
Your crawler will break. It's not a matter of if but when. Sometimes it breaks immediately after you write it, because it can't deal with the malformed HTML garbage that you're trying to parse. Sometimes it happens randomly, weeks or months later, when the site makes a minor change to their HTML that sends your crawler into a chaotic death spiral. Whatever the case, it'll happen, it may not be your fault, but you'll have to deal with it (I recommend cursing profusely). If you can't deal with random sessions of angry debugging because some stranger you don't know but want to punch in the face made a change that broke your crawler, quit now.
You can't make their connection any faster
If your crawler is slow, it's probably not your fault. Chances are, your incoming connection is much faster and less congested than their outgoing connection. The internet is a series of tubes, and while their tube may be a lot larger than yours, it's filled with other people's crap, slowing down your completely legitimate web crawl to, well, a crawl. The only solution is to request more pages in parallel, which leads directly to the next problem...
Sites will ban you
Many web sites don't want to be crawled, at least not by you. They can't really stop you, but they'll try. In the end, they only win if you let them, but it can be quite annoying to deal with a site that doesn't want you to crawl them. There's a number of techniques for dealing with this:
- route your crawls thru TOR, if you don't mind crawling slower than a pentium 1 on with a 2.4 bps modem (if you're too young to know what that means, think facebook or myspace, but 100 times slower, and you're crying because you're peeling a dozen onions while you wait for the page to load)
- use anonymous proxies, which of course are always completely reliable, and never randomly go offline for no particular reason
- slow down your crawl to the point a one armed blind man could do it faster
- ignore robots.txt - what right do these sites have to tell you what you can & can't crawl?!? This is America, isn't it!?!
Conclusion
Don't write a web crawler. Seriously, it's the most annoying programming work in the world. Of course, chances are that if you're actually considering writing a crawler, it's because there's no other option. In that case, good luck, and stock up on liquor.
Programming Philosophy Links
- Software Is Hard
- Maker's Schedule, Manager's Schedule
- programming | Quotes Archive
- How to be a Programmer: A Short, Comprehensive, and Personal Summary
- The "free electron" programmer
- Software Carpentry
- Edited Contributions - Programmer 97-things
- It's OK Not to Write Unit Tests
- Psychology and Security Resource Page
- How To Make Life Suck Less (While Making Scalable Systems)
- The myth of “undesigned”
- The Problem with Design and Implementation
Machine Learning Links
- Feature Column from the AMS on singular value decomposition
- The Geomblog: Correlation Clustering: I don't like you, but I like them...
- linkiblog | How to Build a Popularity Algorithm You can be Proud of
- Semisupervised Learning Approaches
- Gaussian Processes for Machine Learning: Book webpage
- pybrain's pybrain at master - GitHub
- Introduction to Statistical Thought
- Netflix prize tribute: Recommendation algorithm in Python | This Number Crunching Life
How to Deploy hgwebdir.fcgi behind Nginx with Fab
If you're managing multiple mercurial repositories, it's nice to see them all in one place, using a simple web-based repository browser. There's various ways to publish mercurial repositories, but hgwebdir is the only method that supports multiple repos. Since I prefer fastcgi and nginx, I decided to use hgwebdir.fcgi, which unfortunately isn't documented on the mercurial wiki.
hgweb.config
Let's start by creating hgweb.config, which tells hgwebdir where the repos are and what the web UI should look like.
[paths] /REPO1NAME = /PATH/TO/REPO1 /REPO2NAME = /PATH/TO/REPO2 [web] base = style = monoblue
There's a few different included themes you can choose from, I like the monoblue style. The empty base= line is apparently required to make everything work.
hgwebdir.fcgi
Next, make a copy of hgwebdir.fcgi, which in Ubuntu can be found in /usr/share/doc/mercurial/examples. Below is a simplified version with all comments removed. The one line you may want to change is the path to hgweb.config on the server, but I'll assume you'll want it in /etc/mercurial.
from mercurial import demandimport; demandimport.enable()
from mercurial.hgweb.hgwebdir_mod import hgwebdir
from mercurial.hgweb.request import wsgiapplication
from flup.server.fcgi import WSGIServer
def make_web_app():
return hgwebdir("/etc/mercurial/hgweb.config")
WSGIServer(wsgiapplication(make_web_app)).run()
hg_server.conf
This is a simple nginx fastcgi config you can modify for your own purposes. It forwards all requests for hg.DOMAIN.COM to the hgwebdir.fcgi socket we'll be starting below.
server {
listen 80;
server_name hg;
server_name hg.DOMAIN.COM;
access_log /var/log/hg_access.log;
error_log /var/log/hg_error.log;
location / {
fastcgi_pass unix:/var/run/hgwebdir.sock;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
}
}
The hg_server.conf file will need a link from /etc/nginx/sites-enabled to its location in /etc/nginx/sites-available, assuming that you're using the default nginx config which includes every server conf found in /etc/nginx/sites-available.
fab hgweb restart_nginx
To make deployment easy, I use fab, so that if I make any changes to hgweb.config or hg_server.conf, I can simply run fab hgweb restart_nginx. For starting hgwebdir.fcgi, we can use spawn-fcgi, which usually comes with lighttpd, so you'll need that installed too.
hgweb copies hgweb.config and hgwebdir.fcgi to appropriate locations on the server, then starts the fastcgi process with a socket at /var/run/hgwebdir.sock.
restart_nginx copies hg_server.conf to the server and tells nginx to reload its config.
def hgweb():
env.runpath = '/var/run'
put('hgweb.config', '/tmp')
put('hgwebdir.fcgi', '/tmp')
sudo('mv /tmp/hgwebdir.fcgi /usr/local/bin/')
sudo('chmod +x /usr/local/bin/hgwebdir.fcgi')
sudo('mv /tmp/hgweb.config /etc/mercurial/hgweb.config')
sudo('kill `cat %s/hgwebdir.pid`' % env.runpath)
sudo('spawn-fcgi -f /usr/local/bin/hgwebdir.fcgi -s %s/hgwebdir.sock -P %s/hgwebdir.pid' % (env.runpath, env.runpath), user='www-data')
def restart_nginx():
put('hg_server.conf', '/tmp/')
sudo('mv /tmp/hg_server.conf /etc/nginx/sites-available/')
sudo('killall -HUP nginx')
Once you've got these commands in your fabfile.py, you can run fab hgweb restart_nginx to deploy.
hgrc
Now that you've got hgwebdir.fcgi running (you can make sure it works by going to http://hg.DOMAIN.COM), you'll probably want to customize the info about each repo by editing .hg/hgrc.
[web] description = All about my repo contacts = Me
And that's it, you should now have a fast web-based browser for multiple repos 
mapfilter
Have you ever mapped a list, then filtered it? Or filtered first, then mapped? Why not do it all in one pass with mapfilter?
mapfilter?
mapfilter is a function that combines the traditional map & filter of functional programming by using the following logic:
- if your function returns false, then the element is discarded
- any other return value is mapped into the list
Why?
Doing a map and then a filter is O(2N), whereas mapfilter is O(N). That's twice as a fast! If you are dealing with large lists, this can be a huge time saver. And for the case where a large list contains small IDs for looking up a larger data structure, then using mapfilter can result in half the number of database lookups.
Obviously, mapfilter won't work if you want to produce a list of boolean values, as it would filter out all the false values. But why would you want to map to a list booleans?
Erlang Code
Here's some erlang code I've been using for a while:
mapfilter(F, List) -> lists:reverse(mapfilter(F, List, [])). mapfilter(_, [], Results) -> Results; mapfilter(F, [Item | Rest], Results) -> case F(Item) of false -> mapfilter(F, Rest, Results); Term -> mapfilter(F, Rest, [Term | Results]) end.
Has anyone else done this for themselves? Does mapfilter exist in any programming language? If so, please leave a comment. I think mapfilter is a very simple & useful concept that should be a included in the standard library of every (functional) programming language. Erlang already has mapfoldl (map-reduce in one pass), so why not also have mapfilter?
Test Driven Development in Python
One of my favorite aspects of Python is that it makes practicing TDD very easy. What makes it so frictionless is the doctest module. It allows you to write a test at the same time you define a function. No setup, no boilerplate, just write a function call and the expected output in the docstring. Here's a quick example of a fibonacci function.
def fib(n):
'''Return the nth fibonacci number.
>>> fib(0)
0
>>> fib(1)
1
>>> fib(2)
1
>>> fib(3)
2
>>> fib(4)
3
'''
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n - 1) + fib(n - 2)
If you want to run your doctests, just add the following three lines to the bottom of your module.
if __name__ == '__main__':
import doctest
doctest.testmod()
Now you can run your module to run the doctests, like python fib.py.
So how well does this fit in with the TDD philosophy? Here's the basic TDD practices.
- Think about what you want to test
- Write a small test
- Write just enough code to fail the test
- Run the test and watch it fail
- Write just enough code to pass the test
- Run the test and watch it pass (if it fails, go back to step 4)
- Go back to step 1 and repeat until done
And now a step-by-step breakdown of how to do this with doctests, in excruciating detail.
1. Define a new empty method
def fib(n): '''Return the nth fibonacci number.''' pass if __name__ == '__main__': import doctest doctest.testmod()
2. Write a doctest
def fib(n):
'''Return the nth fibonacci number.
>>> fib(0)
0
'''
pass
3. Run the module and watch the doctest fail
python fib.py
**********************************************************************
File "fib1.py", line 3, in __main__.fib
Failed example:
fib(0)
Expected:
0
Got nothing
**********************************************************************
1 items had failures:
1 of 1 in __main__.fib
***Test Failed*** 1 failures.
4. Write just enough code to pass the failing doctest
def fib(n):
'''Return the nth fibonacci number.
>>> fib(0)
0
'''
return 0
5. Run the module and watch the doctest pass
python fib.py
6. Go back to step 2 and repeat
Now you can start filling in the rest of function, one test at time. In practice, you may not write code exactly like this, but the point is that doctests provide a really easy way to test your code as you write it.
Unit Tests
Ok, so doctests are great for simple tests. But what if your tests need to be a bit more complex? Maybe you need some external data, or mock objects. In that case, you'll be better off with more traditional unit tests. But first, take a little time to see if you can decompose your code into a set of smaller functions that can be tested individually. I find that code that is easier to test is also easier to understand.
Running Tests
For running my tests, I use nose. I have a tests/ directory with a simple configuration file, nose.cfg
[nosetests] verbosity=3 with-doctest=1
Then in my Makefile, I add a test command so I can run make test.
test:
@nosetests --config=tests/nose.cfg tests PACKAGE1 PACKAGE2
PACKAGE1 and PACKAGE2 are optional paths to your code. They could point to unit test packages and/or production code containing doctests.
And finally, if you're looking for a continuous integration server, try Buildbot.
Programming As Information Architecture
Code = Information. [1] Therefore, Software Architecture can be approached as Information Architecture. Information Architecture can be defined as
- The structural design of shared information environments.
- The art and science of shaping information products.
The above definitions and much of the inspiration for this article comes from the book Information Architecture for the World Wide Web. My goal is to explain some of what an Information Architect does, and that software developers, especially the lead developer, should approach their code as an information system, applying the principles of Information Architecture. Why? Because it will lead to more organized, better structured, easier to understand code, which will reduce maintenance costs, decrease training time, and generally make it easier for you and your team to get things done.
So, what are the core focus areas for an Information Architect?
- Organization Systems
- Labeling Systems
- Navigation and Search Systems
- Controlled Vocabularies and Metadata
- Research
- Strategy
- Design and Documentation
Organization Systems
Organization Systems are exactly what you think they are: systems to organize information. Imagine if you had to come into your current code base completely fresh, knowing nothing about it. Does that thought horrify you? If your code isn't organized, then it can be very hard for new developers to come in and figure out what's going on. Think of your code repository as a shared information environment. If you are the only one that can navigate it, let alone modify it, then you'll always be stuck maintaining it. Hopefully your goal is not job security, but to provide an environment conducive to change.
So how should you organize your code? Unfortunately, that's not something that's really taught anywhere. My general practice is to follow the recommendations of the language/platform. If they say all code should go in a directory called src/, then that's where I put it. If every class is supposed to be in its own file, then that's what I do. And if the platform documentation doesn't specify how to do something, I'll find a major open source project and see how they do things. The key to an organization system is to maintain logical consistency. Then, as long as you know the logic, you can figure out where things are or where something should go.
Labeling Systems
Labeling Systems are basically standard naming practices. In IA, a labeling system specifies what label goes with each element in every context. For programming, you'll want a consistent naming scheme to make sure the all your code objects are consistently and clearly labeled. Good labels are simple, make sense in context, and hint at the details of the labeled object. The goal is to communicate information efficiently. You are not just writing code for yourself, you're writing code for the team. The best code is not only functional, it's readable, concise, and even beautiful. Clear labeling goes a long way towards achieving that ideal.
Navigation and Search Systems
Navigation and Search Systems are very important to information focused websites, but they don't apply much to code by itself. However, good Navigation and Search Systems are essential for API documentation. I believe that the quality of the API documentation has a huge effect on the adoption rate of libraries and platforms. [2] Good API docs can be great resource for quickly looking up a function and understanding how to use it. But if a developer can't navigate and search your API documentation, then how will they figure out how to that function works? Luckily for us programmers, navigation is usually provided for free with a documentation generator. And google can handle the search for you.
Controlled Vocabularies and Metadata
While all programming languages have a controlled vocabulary, in IA this refers to domain knowledge. The principle is to use words and jargon that are common to whatever domain you are developing for.
Metadata, in this case, is information about the code, such as comments and documentation. Just as an Information Architect is in charge of the language use within a system, the lead developer should be in charge of the domain language and how to use it.
Research
The goal of IA Research is to understand what needs to be designed and built before doing the work. In programming, you are often presented with problems you've never solved before. Hacking, or exploratory programming, is a way to figure out and evaluate possible solutions. Hacking is Research. The goal of research oriented hacking is to figure out possible solutions, evaluate platforms and technologies, and understand the constraints that come with each technology and solution. The knowledge you gather from research is used to drive your strategic choices.
Strategy
IA Strategy is about platform, process and design. What programming language(s) will you use? What are the core design patterns and architectural choices? What version control system will the team use? How will you track progress? Your strategic decisions will set the design constraints of the implementation and drive the development process.
Design and Documentation
In software development, the code is the design, but not everyone will want to read your code to understand how things work. You may need to communicate the design in other ways, such as with diagrams, comments, and documentation. And if the code is being written by someone else, then it's your job to communicate how their code will fit in to the rest of the system. Design documents aren't for you, they're for the other people on the team. You do want other team members to understand your code, right? And if your diagrams and documentation are good enough, you might even get business people to think they understand your software too
Conclusion
Information Architecture provides a top-down view of your software system. As a lead developer or software architect, IA principles and practices can help make sure that your system is well designed and that the design is communicated clearly to all team members. For further reading, I recommend Information Architecture for the World Wide Web and Documenting Software Architectures.
Notes
[1] Code = Data, Data = Information, Code = Information.
[2] I wish I had some data to back this up, but it's certainly how I behave. Lack of clear documentation = fail.
Programming as Design
Some say programming is engineering, others call it an art. A few might (mistakenly) think it's a science. But both the art and engineering can be encapsulated under the umbrella of design. The best design is functional art, and a huge part of the artistic beauty of a product is a result of carefully engineered functionality. Products that are not carefully designed and engineered generally suck to use, and that applies to everything from cameras to software APIs.
Design Principles
There are 4 major principles of graphic design.
Alignment
The principle of alignment is that everything on a page should be connected to something else on the page. The goal of alignment in graphic design is to create visual associations, often using a grid based layout. In software, we can apply this principle to the connectedness of data, such as the object inheritance hierarchy and relational data structures. Ideally, all your objects fit nicely into a well-defined hierarchy and your data structures relate to each other in an intuitive fashion. Of course, the real world of programming is never as clean as you'd like, but keep this principle in mind whenever you create a new object, add a new dependency, or modify relational structures.
- Does the object cleanly fit within the existing hierarchy? If not, do you need to change the new object, or re-align the hierarchy?
- How does this data structure relate to that other data structure? What will happen if the relations change? Will you need to re-align the relational structures?
Keeping your objects and data structures neatly aligned will result in easier to understand relations and hierarchies.
Proximity
The principle of proximity is that related items should be grouped together. Grouping things together is simple way to show relatedness. In software development, that generally means putting related functions into the same module, and related modules into the same package. Helper functions should be located near the functions that call them. Basically, group blocks of code in a logically consistent manner. And if possible, put documentation and tests close to the code too (python's doctest module provides a great way to do that). One of the major benefits of following this principle is that it reduces the amount of time you'll spend searching thru and understanding your own code. If all your code is organized in logical groups, and related functions are near each other in the same file, then it's much easier to find a particular block of code.
Contrast
The principle of contrast is that if two things are not the same, then make them very different. The goal with contrast is to make different things distinctive from each other. Naming is great place to apply this principle. Names are all you have to distinguish between objects, functions, variables, and modules, so make sure that your names are distinctive and descriptive. Good names can tell you exactly what something is, and even imply its properties and behavior. Use different naming styles for different types of things. Private variables could be prefixed with an underscore, like _private, versus public variables like public, and CamelCase class names, as in MyClassName. Having a distinctive naming style lets you know at a glance whether something is a class, variable, or function, making your code much more readable. Whatever naming style you choose, use it consistently.
Consistency
The principle of consistency, or repetition, is that you repeat design elements. Repetition helps patterns become internalized and instantly recognizable. For programming, that means keeping a consistent code style, with consistent naming practices. Also, try to use well known standard conventions and protocols, shared libraries, and design patterns. Your code should make sense, or at least be readable, to those familiar with the language and domain. You're not just writing code for yourself, you might be writing code for other programmers, maybe your manager, but most importantly, you're writing code for your future self. It always sucks coming back to code you haven't touched in months and not knowing what the hell is going on. Consistent style and software design can save you from that headache.
Programming as Design
If all this seems like obvious common sense to you, then great! But common sense isn't always so common. The point of this article is make you aware that everyday software programming is filled with design choices. Naming a variable is a design choice. Creating a new module is a design choice. The layout of your working directory is a design choice. Be conscious of these choices and use the above principles and to inform your decisions. The choices you make communicate how the software works and how the code fits together. Make every choice deliberate and justifiable. Use refactoring to improve the design without affecting the functionality.
Python NLTK Cookbook
Subscribe Here
Strata Conf
Popular Posts
- Text Classification for Sentiment Analysis - Naive Bayes Classifier
- Text Classification for Sentiment Analysis - Stopwords and Collocations
- Text Classification for Sentiment Analysis - Eliminate Low Information Features
- Django Model Formsets
- Chunk Extraction with NLTK
- jQuery Validation with Django Forms
- Part of Speech Tagging with NLTK Part 4 - Brill Tagger vs Classifier Taggers
- Text Classification for Sentiment Analysis - Precision and Recall
Recent Posts
- Upcoming Talks
- Fuzzy String Matching in Python
- NLTK Overview at SF Python
- Python 3 Web Development Review
- PyCon NLTK Tutorial Suggestions
Affiliate Links
Pages
Post Categories
- blog (2)
- books (7)
- design (6)
- erlang (11)
- javascript (6)
- links (20)
- programming (10)
- python (53)
- talks (1)
- Uncategorized (2)
- weotta (2)
