Part of Speech Tagging with NLTK Part 4 – Brill Tagger vs Classifier Taggers «streamhacker.com

12Apr/1012

Part of Speech Tagging with NLTK Part 4 – Brill Tagger vs Classifier Taggers

Hello there! If you are new here, you might want to subscribe to the RSS feed for updates on this topic.

In previous installments on part-of-speech tagging, we saw that a Brill Tagger provides significant accuracy improvements over the Ngram Taggers combined with Regex and Affix Tagging.

With the latest 2.0 beta releases (2.0b8 as of this writing), NLTK has included a ClassifierBasedTagger as well as a pre-trained tagger used by the nltk.tag.pos_tag method. Based on the name, then pre-trained tagger appears to be a ClassifierBasedTagger trained on the treebank corpus using a MaxentClassifier. So let's see how a classifier tagger compares to the brill tagger.

Training Sets

For the brown corpus, I trained on 2/3 of the reviews, lore, and romance categories, and tested against the remaining 1/3. For conll2000, I used the standard train.txt vs test.txt. And for treebank, I again used a 2/3 vs 1/3 split.

import itertools
from nltk.corpus import brown, conll2000, treebank

brown_reviews = brown.tagged_sents(categories=['reviews'])
brown_reviews_cutoff = len(brown_reviews) * 2 / 3
brown_lore = brown.tagged_sents(categories=['lore'])
brown_lore_cutoff = len(brown_lore) * 2 / 3
brown_romance = brown.tagged_sents(categories=['romance'])
brown_romance_cutoff = len(brown_romance) * 2 / 3

brown_train = list(itertools.chain(brown_reviews[:brown_reviews_cutoff],
	brown_lore[:brown_lore_cutoff], brown_romance[:brown_romance_cutoff]))
brown_test = list(itertools.chain(brown_reviews[brown_reviews_cutoff:],
	brown_lore[brown_lore_cutoff:], brown_romance[brown_romance_cutoff:]))

conll_train = conll2000.tagged_sents('train.txt')
conll_test = conll2000.tagged_sents('test.txt')

treebank_cutoff = len(treebank.tagged_sents()) * 2 / 3
treebank_train = treebank.tagged_sents()[:treebank_cutoff]
treebank_test = treebank.tagged_sents()[treebank_cutoff:]

Classifier Taggers

There are 3 new taggers referenced below:

cpos is an instance of ClassifierBasedPOSTagger using the default NaiveBayesClassifier. It was trained by doing ClassifierBasedPOSTagger(train=train_sents)
craubt is like cpos, but has the raubt tagger from part 2 as a backoff tagger by doing ClassifierBasedPOSTagger(train=train_sents, backoff=raubt)
bcpos is a BrillTagger using cpos as its initial tagger instead of raubt.

The raubt tagger is the same as from part 2, and braubt is from part 3.

postag is NLTK's pre-trained tagger used by the pos_tag function. It can be loaded using nltk.data.load(nltk.tag._POS_TAGGER).

Accuracy Evaluation

Tagger accuracy was determined by calling the evaluate method with the test set on each trained tagger. Here are the results:

brill vs classifier tagger accuracy chart

Conclusions

The above results are quite interesting, and lead to a few conclusions:

Training data is hugely significant when it comes to accuracy. This is why postag takes a huge nose dive on brown, while at the same time can get near 100% accuracy on treebank.
A ClassifierBasedPOSTagger does not need a backoff tagger, since cpos accuracy is exactly the same as for craubt across all corpora.
The ClassifierBasedPOSTagger is not necessarily more accurate than the bcraubt tagger from part 3 (at least with the default feature detector). It also takes much longer to train and tag (more details below) and so may not be worth the tradeoff in efficiency.
Using BrillTagger will nearly always increase the accuracy of your initial tagger, but not by much.

I was also surprised at how much more accurate postag was compared to cpos. Thinking that postag was probably trained on the full treebank corpus, I did the same, and re-evaluated:

cpos = ClassifierBasedPOSTagger(train=treebank.tagged_sents())
cpos.evaluate(treebank_test)

The result was 98.08% accuracy. So the remaining 2% difference must be due to the MaxentClassifier being more accurate than NaiveBayesClassifier, and/or the use of a different feature detector. I tried again with classifier_builder=MaxentClassifier.train and only got to 98.4% accuracy. So I can only conclude that a different feature detector is used. Hopefully the NLTK leaders will publish the training method so we can all know for sure.

Efficiency

On the nltk-users list, there was a question about which tagger is the most computationaly economic. I can't tell you the right answer, but I can definitely say that ClassifierBasedPOSTagger is the wrong answer. During accuracy evaluation, I noticed that the cpos tagger took a lot longer than raubt or braubt. So I ran timeit on the tag method of each tagger, and got the following results:

Tagger	secs/pass
raubt	0.00005
braubt	0.00009
cpos	0.02219
bcpos	0.02259
postag	0.01241

This was run with python 2.6.4 on an Athlon 64 Dual Core 4600+ with 3G RAM, but the important thing is the relative times. braubt is over 246 times faster than cpos! To put it another way, braubt can process over 66666 words/sec, where cpos can only do 270 words/sec and postag only 483 words/sec. So the lesson is: do not use a classifier based tagger if speed is an issue.

Here's the code for timing postag. You can do the same thing for any other pickled tagger by replacing nltk.tag._POS_TAGGER with a nltk.data accessible path with a .pickle suffix for the load method.

import nltk, timeit
text = nltk.word_tokenize('And now for something completely different')
setup = 'import nltk.data, nltk.tag; tagger = nltk.data.load(nltk.tag._POS_TAGGER)'
t = timeit.Timer('tagger.tag(%s)' % text, setup)
print 'timing postag 1000 times'
spent = t.timeit(number=1000)
print 'took %.5f secs/pass' % (spent / 1000)

File Size

There's also a significant difference in the file size of the pickled taggers (trained on treebank):

Tagger	Size
raubt	272K
braubt	273K
cpos	3.8M
bcpos	3.8M
postag	8.2M

Fin

I think there's a lot of room for experimentation with classifier based taggers and their feature detectors. But if speed is an issue for you, don't even bother. In that case, stick with a simpler tagger that's nearly as accurate and orders of magnitude faster.

Tagged as: brill, classification, nlp, nltk, python, tagging, treebank Leave a comment

Comments (12) Trackbacks (0) ( subscribe to comments on this post )

tdflatline
April 17th, 2010 - 19:27
Did you try a Brill tagger with the MaxEnt classifier as the initial tagger? It does look like Brill is buying you a little over a percent over your original classifier tagger. That might almost explain the remaining gap.
Also, I am willing to wager heavily that the primary reason nltk.pos_tag has such a high error rate on brown is because the tags are substantially different between brown and treebank, more so than any difference in the actual corpus material. Doing a simple set subtraction on the set of brown tags vs treebank tags makes me wonder how it had any accuracy on brown at all. Did you do any form of tag normalization?
Jacob Perkins
April 17th, 2010 - 20:05
No, I did not try Brill with a MaxentClassifier tagger. It probably would give another percent of accuracy, but I don't think that's what the pre-trained tagger for pos_tag does, as the repr of the tagger is from the ClassifierBasedTagger.
Yes, I'm sure the tag differences contributed significantly to the inaccuracy. I would have normalized them if I knew a simple method to do so, but I'm not sure that exists yet And then I saw on the nltk-users list that someone tried to train with brown & simplify_tags=True, and the results were even worse. The real point of testing against brown was to illustrate the importance of using the right training data, and I think that came across loud and clear
tdflatline
April 18th, 2010 - 01:07
Actually, I'm not sure if you have conclusively demonstrated the importance of training data. You need to compare apples to apples for that. Perhaps something like training on brown romance and testing against science_fiction, and/or different combinations of brown corpus categories.
At any rate, that doesn't detract from your excellent work here, which definitely elegantly showcases what can be done with different taggers under different constraints, but it's perhaps something to consider once nltk.pos_tag's source is published. Agility and generalization for different datasets with the same tags would be a useful final data point.
Oli
April 22nd, 2010 - 09:31
Hi Jacob,
as I mentioned on the google-group, your post is very interesting. Thanks for it. I am still seeking for the real implementation of pos_tag().
You mentioned a different feature_detector. What do you think about the following idea, for getting the same feature_detector as pos_tag()?
#load default tagger
t0 = nltk.data.load('taggers/maxent_treebank_pos_tagger/english.pickle');
train_set = []
for tagged_sent in train_sents:
untagged_sent = nltk.tag.untag(tagged_sent)
history = []
for i, (word, tag) in enumerate(tagged_sent):
featureset = t0.feature_detector(untagged_sent, i, history)
train_set.append( (featureset, tag) )
history.append(tag)
Now you can use train_set for training the classifier with the same feature_detector as the pos_tag(). Right?
I would like to test it, but I got a problem. When I use the simple example, u mentioned in your post:
cpos =ClassifierBasedPOSTagger(train=treebank.tagged_sents(),classifier_builder=MaxentClassifier.train)
It took about 1 hour for 1 iteration (out of 100). Even when I am only using 50 sents of the corpus for training its still taking about 10 minutes to finish all 100 iterations. So how did you train your ClassifierBasedPOSTagger on the whole corpus without waiting 1 week?
I hope, my English is not too bad and you understand me
all the best
Oli
Jacob Perkins
April 22nd, 2010 - 10:25
Hi Oli,
Yes, the default MaxentClassifier algorithm is unfortunately slow. If you create a custom training function that calls MaxentClassifier.train with different parameters, you can speed it up. Take a look at http://nltk.googlecode.com/svn/trunk/doc/api/nl... for more details. I generally set min_lldelta to 0.01 for the default algorithm and often stop the iterations before it gets to 10 iterations by using Ctrl-C.
Lately I've been using the 'megam' algorithm instead. It's much faster but requires installing the megam package: http://nltk.googlecode.com/svn/trunk/doc/api/to...
Great idea on using the feature extractor from the default tagger. I'll have to try that out.
And your english is great, I would have assumed you're a native if you hadn't mentioned anything
Jacob
Oli
April 22nd, 2010 - 13:36
Thanks for your fast answer, Jacob!
I tried different things now, but without any results. I am facing different issues:
- scipy-algorithms do not work for some reason
- there are no binary releases of megam for windows…and I wasn't able to compile it on my machine
So…I am not able to test the stuff, because the default-algo is toooo slow. But thank you for the hint with ctrl+c. You can also use max_iter=5 to limit the iterations.
I am also a bit confused how to init the ClassifierBasedPOSTagger. Do you build a MaxentClassifier and use it with classifier=YourMaxentClassifier, or do you use the classifier_builder-function? It would be great, if you could post your code for your testing.
thanks in advance!
Oli
Jacob Perkins
April 22nd, 2010 - 18:31
I've used classifier_builder=MaxentClassifier.train, or to pass in custom parameters something like
cb = lamba toks: MaxentClassifier.train(toks, max_iter=5)
and then classifier_builder=cb. I haven't tried any of scipy algorithms; all I can recommend is to be sure numpy and scipy are correctly installed. That's too bad about megam, maybe someone on the mailing list can help out.
I've been thinking about doing an evaluation of each of the training algorithms for speed and memory consumption. If I do that, I'll definitely post results & code.
Oli
April 22nd, 2010 - 20:36
Thanks for your fast answer, Jacob!
I tried different things now, but without any results. I am facing different issues:
- scipy-algorithms do not work for some reason
- there are no binary releases of megam for windows…and I wasn't able to compile it on my machine
So…I am not able to test the stuff, because the default-algo is toooo slow. But thank you for the hint with ctrl+c. You can also use max_iter=5 to limit the iterations.
I am also a bit confused how to init the ClassifierBasedPOSTagger. Do you build a MaxentClassifier and use it with classifier=YourMaxentClassifier, or do you use the classifier_builder-function? It would be great, if you could post your code for your testing.
thanks in advance!
Oli
Jacob Perkins
April 23rd, 2010 - 01:31
I've used classifier_builder=MaxentClassifier.train, or to pass in custom parameters something like
cb = lamba toks: MaxentClassifier.train(toks, max_iter=5)
and then classifier_builder=cb. I haven't tried any of scipy algorithms; all I can recommend is to be sure numpy and scipy are correctly installed. That's too bad about megam, maybe someone on the mailing list can help out.
I've been thinking about doing an evaluation of each of the training algorithms for speed and memory consumption. If I do that, I'll definitely post results & code.
Dannii
August 13th, 2010 - 14:11
You said: “A ClassifierBasedPOSTagger does not need a backoff tagger, since cpos accuracy is exactly the same as for craubt across all corpora.”
This is probably because you didn’t set the classifier’s cutoff_prob parameter. Without it the tagger will never consult its backoff. I don’t use a backoff tagger with a classifier tagger, as anything else is only going to be less accurate.
Dannii
August 13th, 2010 - 14:11
You said: “A ClassifierBasedPOSTagger does not need a backoff tagger, since cpos accuracy is exactly the same as for craubt across all corpora.”
This is probably because you didn’t set the classifier’s cutoff_prob parameter. Without it the tagger will never consult its backoff. I don’t use a backoff tagger with a classifier tagger, as anything else is only going to be less accurate.
Jacob Perkins
August 13th, 2010 - 19:31
Yes, I found that cutoff_prob parameter later and did some experiments, coming to the same conclusion as you: a backoff tagger with a classifier based tagger generally doesn’t help.

No trackbacks yet.

blog comments powered by Disqus

Linguistic and Natural Language Processing Links » « Python Logging Filters

Hosting by Just Host

Subscribe Here

Analytics by Clicky

My Code

Twitter Feed

NLTK Tokenization Demo for @mat_kelcey http://bit.ly/cG6GIN #python 7 hrs ago
Software Engineering - The Doomed Discipline - E. W. Dijkstra http://bit.ly/bKAwQ1 2 days ago
Ctrl-V, Ctrl-X, Ctrl-Z: I just wanted to post this photo because as geeky as it is, I love it. I found it on Like... http://bit.ly/clJgvY 4 days ago
More updates...

streamhacker.com Weotta be Hacking