I’ve recently pushed some updates to nltk-trainer, so that it now supports Python 3.7 and NLTK 3.4.5 or greater. NLTK also just released version 3.5.
One significant change is that the default part of speech tagger is now the PerceptronTagger, originally written by Matthew Honnibal (author of spacy) before it was ported to NLTK.
We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs
Below are more highlights from Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form–function relationship in language, our “composed” word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages
it is manifestly clear that similarity in form is neither a necessary nor sufficient condition for similarity in function: small orthographic differences may correspond to large semantic or syntactic differences (butter vs. batter), and large orthographic differences may obscure nearly perfect functional correspondence (rich vs. affluent). Thus, any orthographically aware model must be able to capture non-compositional effects in addition to more regular effects due to, e.g., morphological processes. To model the complex form–function relationship, we turn to long short-term memories (LSTMs), which are designed to be able to capture complex non-linear and non-local dynamics in sequences
our character-based model is able to generate similar representations for words that are semantically and syntactically similar, even for words are orthographically distant (e.g., October and January)
The goal of our work is not to overcome existing benchmarks, but show that much of the feature engineering done in the benchmarks can be learnt automatically from the task specific data. More importantly, we wish to show large dimensionality word look tables can be compacted into a lookup table using characters and a compositional model allowing the model scale better with the size of the training data. This is a desirable property of the model as data becomes more abundant in many NLP tasks.
The authors have also released Java code for training neural networks.
NLTK 2.0.1, a.k.a NLTK 2, was recently released, and what follows is my favorite changes, new features, and highlights from the ChangeLog.
The SVMClassifier adds support vector machine classification thru SVMLight with PySVMLight. This is a much needed addition to the set of supported classification algorithms. But even more interesting…
The SklearnClassifier provides a general interface to text classification with scikit-learn. While scikit-learn is still pre-1.0, it is rapidly becoming one of the most popular machine learning toolkits, and provides more advanced feature extraction methods for classification.
NLTK has moved development and hosting to github, replacing google code and SVN. The primary motivation is to make new development easier, and already a Python 3 branch is under active development. I think this is great, since github makes forking & pull requests quite easy, and it’s become the de-facto “social coding” site.
Coinciding with the github move, the documentation was updated to use Sphinx, the same documentation generator used by Python and many other projects. While I personally like Sphinx and restructured text (which I used to write this post), I’m not thrilled with the results. The new documentation structure and NLTK homepage seem much less approachable. While it works great if you know exactly what you’re looking for, I worry that new/interested users will have a harder time getting started.
Since the 0.9.9 release, a number of new corpora and corpus readers have been added:
And here’s a few final highlights:
I think NLTK’s ideal role is be a standard interface between corpora and NLP algorithms. There are many different corpus formats, and every algorithm has its own data structure requirements, so providing common abstract interfaces to connect these together is very powerful. It allows you to test the same algorithm on disparate corpora, or try multiple algorithms on a single corpus. This is what NLTK already does best, and I hope that becomes even more true in the future.
NLTK Trainer includes 2 scripts for analyzing both a tagged corpus and the coverage of a part-of-speech tagger.
You can get part-of-speech tag statistics on a tagged corpus using analyze_tagged_corpus.py. Here’s the tag counts for the treebank corpus:
$ python analyze_tagged_corpus.py treebank loading nltk.corpus.treebank 100676 total words 12408 unique words 46 tags Tag Count ======= ========= # 16 $ 724 '' 694 , 4886 -LRB- 120 -NONE- 6592 -RRB- 126 . 3874 : 563 CC 2265 CD 3546 DT 8165 EX 88 FW 4 IN 9857 JJ 5834 JJR 381 JJS 182 LS 13 MD 927 NN 13166 NNP 9410 NNPS 244 NNS 6047 PDT 27 POS 824 PRP 1716 PRP$ 766 RB 2822 RBR 136 RBS 35 RP 216 SYM 1 TO 2179 UH 3 VB 2554 VBD 3043 VBG 1460 VBN 2134 VBP 1321 VBZ 2125 WDT 445 WP 241 WP$ 14 WRB 178 `` 712 ======= =========
By default, analyze_tagged_corpus.py sorts by tags, but you can sort by the highest count using <span class="pre">--sort</span> count <span class="pre">--reverse</span>. You can also see counts for simplified tags using <span class="pre">--simplify_tags</span>:
$ python analyze_tagged_corpus.py treebank --simplify_tags
loading nltk.corpus.treebank
100676 total words
12408 unique words
31 tags
Tag Count
======= =========
7416
# 16
$ 724
'' 694
( 120
) 126
, 4886
. 3874
: 563
ADJ 6397
ADV 2993
CNJ 2265
DET 8192
EX 88
FW 4
L 13
MOD 927
N 19213
NP 9654
NUM 3546
P 9857
PRO 2698
S 1
TO 2179
UH 3
V 6000
VD 3043
VG 1460
VN 2134
WH 878
`` 712
======= =========
You can analyze the coverage of a part-of-speech tagger against any corpus using analyze_tagger_coverage.py. Here’s the results for the treebank corpus using NLTK’s default part-of-speech tagger:
$ python analyze_tagger_coverage.py treebank loading tagger taggers/maxent_treebank_pos_tagger/english.pickle analyzing tag coverage of treebank with ClassifierBasedPOSTagger Tag Found ======= ========= # 16 $ 724 '' 694 , 4887 -LRB- 120 -NONE- 6591 -RRB- 126 . 3874 : 563 CC 2271 CD 3547 DT 8170 EX 88 FW 4 IN 9880 JJ 5803 JJR 386 JJS 185 LS 12 MD 927 NN 13166 NNP 9427 NNPS 246 NNS 6055 PDT 21 POS 824 PRP 1716 PRP$ 766 RB 2800 RBR 130 RBS 33 RP 213 SYM 1 TO 2180 UH 3 VB 2562 VBD 3035 VBG 1458 VBN 2145 VBP 1318 VBZ 2124 WDT 440 WP 241 WP$ 14 WRB 178 `` 712 ======= =========
If you want to analyze the coverage of your own pickled tagger, use <span class="pre">--tagger</span> PATH/TO/TAGGER.pickle. You can also get detailed metrics on Found vs Actual counts, as well as Precision and Recall for each tag by using the <span class="pre">--metrics</span> argument with a corpus that provides a tagged_sents method, like treebank:
$ python analyze_tagger_coverage.py treebank --metrics loading tagger taggers/maxent_treebank_pos_tagger/english.pickle analyzing tag coverage of treebank with ClassifierBasedPOSTagger Accuracy: 0.995689 Unknown words: 440 Tag Found Actual Precision Recall ======= ========= ========== ============= ========== # 16 16 1.0 1.0 $ 724 724 1.0 1.0 '' 694 694 1.0 1.0 , 4887 4886 1.0 1.0 -LRB- 120 120 1.0 1.0 -NONE- 6591 6592 1.0 1.0 -RRB- 126 126 1.0 1.0 . 3874 3874 1.0 1.0 : 563 563 1.0 1.0 CC 2271 2265 1.0 1.0 CD 3547 3546 0.99895833333 0.99895833333 DT 8170 8165 1.0 1.0 EX 88 88 1.0 1.0 FW 4 4 1.0 1.0 IN 9880 9857 0.99130434782 0.95798319327 JJ 5803 5834 0.99134948096 0.97892938496 JJR 386 381 1.0 0.91489361702 JJS 185 182 0.96666666666 1.0 LS 12 13 1.0 0.85714285714 MD 927 927 1.0 1.0 NN 13166 13166 0.99166034874 0.98791540785 NNP 9427 9410 0.99477911646 0.99398073836 NNPS 246 244 0.99029126213 0.95327102803 NNS 6055 6047 0.99515235457 0.99722414989 PDT 21 27 1.0 0.66666666666 POS 824 824 1.0 1.0 PRP 1716 1716 1.0 1.0 PRP$ 766 766 1.0 1.0 RB 2800 2822 0.99305555555 0.975 RBR 130 136 1.0 0.875 RBS 33 35 1.0 0.5 RP 213 216 1.0 1.0 SYM 1 1 1.0 1.0 TO 2180 2179 1.0 1.0 UH 3 3 1.0 1.0 VB 2562 2554 0.99142857142 1.0 VBD 3035 3043 0.990234375 0.98065764023 VBG 1458 1460 0.99650349650 0.99824868651 VBN 2145 2134 0.98852223816 0.99566473988 VBP 1318 1321 0.99305555555 0.98281786941 VBZ 2124 2125 0.99373040752 0.990625 WDT 440 445 1.0 0.83333333333 WP 241 241 1.0 1.0 WP$ 14 14 1.0 1.0 WRB 178 178 1.0 1.0 `` 712 712 1.0 1.0 ======= ========= ========== ============= ==========
These additional metrics can be quite useful for identifying which tags a tagger has trouble with. Precision answers the question “for each word that was given this tag, was it correct?”, while Recall answers the question “for all words that should have gotten this tag, did they get it?”. If you look at PDT, you can see that Precision is 100%, but Recall is 66%, meaning that every word that was given the PDT tag was correct, but 6 out of the 27 words that should have gotten PDT were mistakenly given a different tag. Or if you look at JJS, you can see that Precision is 96.6% because it gave JJS to 3 words that should have gotten a different tag, while Recall is 100% because all words that should have gotten JJS got it.
NLTK trainer makes it easy to train part-of-speech taggers with various algorithms using train_tagger.py.
The fastest algorithms are the sequential backoff taggers. You can specify the backoff sequence using the <span class="pre">--sequential</span> argument, which accepts any combination of the following letters:
a: |
AffixTagger |
u: |
UnigramTagger |
b: |
BigramTagger |
t: |
TrigramTagger |
For example, to train the same kinds of taggers that were used in Part of Speech Tagging with NLTK Part 1 – Ngram Taggers, you could do the following:
python train_tagger.py treebank --sequential ubt
You can rearrange ubt any way you want to change the order of the taggers (though ubt is generally the most accurate order).
The <span class="pre">--sequential</span> argument also recognizes the letter a, which will insert an AffixTagger into the backoff chain. If you do not specify the <span class="pre">--affix</span> argument, then it will include one AffixTagger with a 3-character suffix. However, you can change this by specifying one or more <span class="pre">--affix</span> N options, where N should be a positive number for prefixes, and a negative number for suffixes. For example, to train an aubt tagger with 2 AffixTaggers, one that uses a 3 character suffix, and another that uses a 2 character prefix, specify the <span class="pre">--affix</span> argument twice:
python train_tagger.py treebank --sequential aubt --affix -3 --affix 2
The order of the <span class="pre">--affix</span> arguments is the order in which each AffixTagger will be trained and inserted into the backoff chain.
To train a BrillTagger in a similar fashion to the one trained in Part of Speech Tagging Part 3 – Brill Tagger (using FastBrillTaggerTrainer), use the <span class="pre">--brill</span> argument:
python train_tagger.py treebank --sequential aubt --brill
The default training options are a maximum of 200 rules with a minimum score of 2, but you can change that with the <span class="pre">--max_rules</span> and <span class="pre">--min_score</span> arguments. You can also change the rule template bounds, which defaults to 1, using the <span class="pre">--template_bounds</span> argument.
Many of the arguments used by train_classifier.py can also be used to train a ClassifierBasedPOSTagger. If you don’t want this tagger to backoff to a sequential backoff tagger, be sure to specify <span class="pre">--sequential</span> ''. Here’s an example for training a NaiveBayesClassifier based tagger, similar to what was shown in Part of Speech Tagging Part 4 – Classifier Taggers:
python train_tagger.py treebank --sequential '' --classifier NaiveBayes
If you do want to backoff to a sequential tagger, be sure to specify a cutoff probability, like so:
python train_tagger.py treebank --sequential ubt --classifier NaiveBayes --cutoff_prob 0.4
Any of the NLTK classification algorithms can be used for the <span class="pre">--classifier</span> argument, such as Maxent or MEGAM, and every algorithm other than NaiveBayes has specific training options that can be customized.
You can also include phonetic algorithm features using the following arguments:
<span class="pre">--metaphone</span>: |
Use metaphone feature |
<span class="pre">--double-metaphone</span>: |
Use double metaphone feature |
<span class="pre">--soundex</span>: |
Use soundex feature |
<span class="pre">--nysiis</span>: |
Use NYSIIS feature |
<span class="pre">--caverphone</span>: |
Use caverphone feature |
These options create phonetic codes that will be included as features along with the default features used by the ClassifierBasedPOSTagger. The <span class="pre">--double-metaphone</span> algorithm comes from metaphone.py, while all the other phonetic algorithm have been copied from the advas project (which appears to be abandoned).
I created these options after discussions with Michael D Healy about Twitter Linguistics, in which he explained the prevalence of regional spelling variations. These phonetic features may be able to reduce that variation where a tagger is concerned, as slightly different spellings might generate the same phonetic code.
A tagger trained with any of these phonetic features will be an instance of nltk_trainer.tagging.taggers.PhoneticClassifierBasedPOSTagger, which means nltk_trainer must be included in your PYTHONPATH in order to load & use the tagger. The simplest way to do this is to install nltk-trainer using python setup.py install.
Following up on the previous post showing the tag coverage of the NLTK 2.0b9 default tagger on the treebank corpus, below are the same metrics applied to the conll2000 corpus, using the analyze_tagger_coverage.py script from nltk-trainer.
The default tagger is 93.9% accurate on the conll2000 corpus, which is to be expected since both treebank and conll2000 are based on the Wall Street Journal. You can see all the metrics shown below for yourself by running python analyze_tagger_coverage.py conll2000 --metrics. In many cases, the Precision and Recall metrics are significantly lower than 1, even when the Found and Actual counts are similar. This happens when words are given the wrong tag (creating false positives and false negatives) while the overall tag frequency remains about the same. The CC tag is a great example of this: the Found count is only 3 higher than the Actual count, yet Precision is 68.75% and Recall is 73.33%. This tells us that the number of words that were mis-tagged as CC, and the number of CC words that were not given the CC tag, are approximately equal, creating similar counts despite the false positives and false negatives.
| Tag | Found | Actual | Precision | Recall |
| # | 46 | 47 | 1 | 1 |
| $ | 2122 | 2134 | 1 | 0.6 |
| ‘ | 1811 | 1809 | 1 | 1 |
| ( | 0 | 351 | None | 0 |
| ) | 0 | 358 | None | 0 |
| , | 13160 | 13160 | 1 | 1 |
| -LRB- | 351 | 0 | 0 | None |
| -NONE- | 59 | 0 | 0 | None |
| -RRB- | 358 | 0 | 0 | None |
| . | 10800 | 10802 | 1 | 1 |
| : | 1288 | 1285 | 0.7143 | 1 |
| CC | 6589 | 6586 | 0.6875 | 0.7333 |
| CD | 10325 | 10233 | 0.972 | 0.9919 |
| DT | 22301 | 22355 | 0.7826 | 1 |
| EX | 229 | 254 | 1 | 1 |
| FW | 1 | 42 | 1 | 0.0455 |
| IN | 27798 | 27835 | 0.7315 | 0.7899 |
| JJ | 15370 | 16049 | 0.7372 | 0.7303 |
| JJR | 1114 | 1055 | 0.5412 | 0.575 |
| JJS | 611 | 451 | 0.6912 | 0.7966 |
| LS | 13 | 0 | 0 | None |
| MD | 2616 | 2637 | 0.7143 | 0.75 |
| NN | 38023 | 36789 | 0.7345 | 0.8441 |
| NNP | 24967 | 24690 | 0.8752 | 0.9421 |
| NNPS | 589 | 550 | 0.4553 | 0.3684 |
| NNS | 17068 | 16653 | 0.8572 | 0.9527 |
| PDT | 24 | 65 | 0.6667 | 1 |
| POS | 2224 | 2203 | 0.6667 | 1 |
| PRP | 4620 | 4634 | 0.8438 | 0.7941 |
| PRP$ | 2292 | 2302 | 0.6364 | 1 |
| RB | 7681 | 7961 | 0.8076 | 0.8582 |
| RBR | 288 | 392 | 0.5 | 0.3684 |
| RBS | 90 | 240 | 0.5 | 0.1667 |
| RP | 634 | 95 | 0.1176 | 1 |
| SYM | 0 | 6 | None | 0 |
| TO | 6257 | 6259 | 1 | 0.75 |
| UH | 2 | 17 | 1 | 0.1111 |
| VB | 6681 | 7286 | 0.9042 | 0.8313 |
| VBD | 8501 | 8424 | 0.7521 | 0.8605 |
| VBG | 3730 | 4000 | 0.8493 | 0.8603 |
| VBN | 5763 | 5867 | 0.8164 | 0.8721 |
| VBP | 3232 | 3407 | 0.6754 | 0.6638 |
| VBZ | 5224 | 5561 | 0.7273 | 0.6906 |
| WDT | 1156 | 1157 | 0.6 | 0.5 |
| WP | 637 | 639 | 1 | 1 |
| WP$ | 38 | 39 | 1 | 1 |
| WRB | 566 | 571 | 0.9 | 0.75 |
| “ | 1855 | 1854 | 0.6667 | 1 |
The conll2000 corpus has 0 words tagged with -NONE-, yet the default tagger is unable to identify 50 unique words. Here’s a sample: boiler-room, so-so, Coca-Cola, top-10, AC&R, F-16, I-880, R2-D2, mid-1992. For the most part, the unknown words are symbolic names, acronyms, or two separate words combined with a “-“. You might think this can solved with better tokenization, but for words like F-16 and I-880, tokenizing on the “-” would be incorrect.
The default tagger apparently does not recognize parentheses or the SYM tag, and has trouble with many of the more rare tags, such as FW, LS, RBS, and UH. These failures highlight the need for training a part-of-speech tagger (or any NLP object) on a corpus that is as similar as possible to the corpus you are analyzing. At the very least, your training corpus and testing corpus should share the same set of part-of-speech tags, and in similar proportion. Otherwise, mistakes will be made, such as not recognizing common symbols, or finding -LRB- and -RRB- tags where they do not exist.
For some research I’m doing with Michael D. Healy, I need to measure part-of-speech tagger coverage and performance. To that end, I’ve added a new script to nltk-trainer: analyze_tagger_coverage.py. This script will tag every sentence of a corpus and count how many times it produces each tag. If you also use the --metrics option, and the corpus reader provides a tagged_sents() method, then you can get detailed performance metrics by comparing the tagger’s results against the actual tags.
Below is a table showing the performance details of the NLTK 2.0b9 default tagger on the treebank corpus, which you can see for yourself by running python analyze_tagger_coverage.py treebank --metrics. The default tagger is 99.57% accurate on treebank, and below you can see exactly on which tags it fails. The Found column shows the number of occurrences of each tag produced by the default tagger, while the Actual column shows the actual number of occurrences in the treebank corpus. Precision and Recall, which I’ve explained in the context of classification, show the performance for each tag. If the Precision is less than 1, that means the tagger gave the tag to a word that it shouldn’t have (a false positive). If the Recall is less than 1, it means the tagger did not give the tag to a word that it should have (a false negative).
| Tag | Found | Actual | Precision | Recall |
| # | 16 | 16 | 1 | 1 |
| $ | 724 | 724 | 1 | 1 |
| ‘ | 694 | 694 | 1 | 1 |
| , | 4887 | 4886 | 1 | 1 |
| -LRB- | 120 | 120 | 1 | 1 |
| -NONE- | 6591 | 6592 | 1 | 1 |
| -RRB- | 126 | 126 | 1 | 1 |
| . | 3874 | 3874 | 1 | 1 |
| : | 563 | 563 | 1 | 1 |
| CC | 2271 | 2265 | 1 | 1 |
| CD | 3547 | 3546 | 0.999 | 0.999 |
| DT | 8170 | 8165 | 1 | 1 |
| EX | 88 | 88 | 1 | 1 |
| FW | 4 | 4 | 1 | 1 |
| IN | 9880 | 9857 | 0.9913 | 0.958 |
| JJ | 5803 | 5834 | 0.9913 | 0.9789 |
| JJR | 386 | 381 | 1 | 0.9149 |
| JJS | 185 | 182 | 0.9667 | 1 |
| LS | 12 | 13 | 1 | 0.8571 |
| MD | 927 | 927 | 1 | 1 |
| NN | 13166 | 13166 | 0.9917 | 0.9879 |
| NNP | 9427 | 9410 | 0.9948 | 0.994 |
| NNPS | 246 | 244 | 0.9903 | 0.9533 |
| NNS | 6055 | 6047 | 0.9952 | 0.9972 |
| PDT | 21 | 27 | 1 | 0.6667 |
| POS | 824 | 824 | 1 | 1 |
| PRP | 1716 | 1716 | 1 | 1 |
| PRP$ | 766 | 766 | 1 | 1 |
| RB | 2800 | 2822 | 0.9931 | 0.975 |
| RBR | 130 | 136 | 1 | 0.875 |
| RBS | 33 | 35 | 1 | 0.5 |
| RP | 213 | 216 | 1 | 1 |
| SYM | 1 | 1 | 1 | 1 |
| TO | 2180 | 2179 | 1 | 1 |
| UH | 3 | 3 | 1 | 1 |
| VB | 2562 | 2554 | 0.9914 | 1 |
| VBD | 3035 | 3043 | 0.9902 | 0.9807 |
| VBG | 1458 | 1460 | 0.9965 | 0.9982 |
| VBN | 2145 | 2134 | 0.9885 | 0.9957 |
| VBP | 1318 | 1321 | 0.9931 | 0.9828 |
| VBZ | 2124 | 2125 | 0.9937 | 0.9906 |
| WDT | 440 | 445 | 1 | 0.8333 |
| WP | 241 | 241 | 1 | 1 |
| WP$ | 14 | 14 | 1 | 1 |
| WRB | 178 | 178 | 1 | 1 |
| “ | 712 | 712 | 1 | 1 |
Suprisingly, the treebank corpus contains 6592 words tags with -NONE-. But it’s not that bad, since it’s only 440 unique words, and they are not regular words at all: *EXP*-2, *T*-91, *-106, and many more similar looking tokens.
If you liked the NLTK demos, then you’ll love the text processing APIs. They provide all the functionality of the demos, plus a little bit more, and return results in JSON. Requests can contain up to 10,000 characters, instead of the 1,000 character limit on the demos, and you can do up to 100 calls per day. These limits may change in the future depending on usage & demand. If you’d like to do more, please fill out this survey to let me know what your needs are.
If you want to see what NLTK can do, but don’t want to go thru the effort of installation and learning how to use it, then check out my Python NLTK demos.
It currently demonstrates the following functionality:
If you like it, please share it. If you want to see more, leave a comment below. And if you are interested in a service that could apply these processes to your own data, please fill out this NLTK services survey.
Here’s a list of similar resources on the web:
A number of links related to natural language processing and linguistics: