Whilst I have your attention, do you know if its possible to print a list of NE tags used in Chunk or extend the tags? I’m a little new to this and have been reading through the NLTK book but couldn’t find anything this specific.

Thanks.

Yes, it would be nice to see a working example for the more challenged of us.


tagged_toks = self.tagger.tag(sentence)
(words, tags) = zip(*tagged_toks)
chunks = self.chunker.tag(tags)
return [(w, t, c) for (w, (t, c)) in zip(words, chunks)]


Hope that helps. Perhaps I should write an article about putting it all together.

import nltk
from nltk.tag import brill
import logging
logger = logging.getLogger(“ballyclare.tagger”)
# see: http://streamhacker.wordpress.com/2008/12/03/part-of-speech-tagging-with-nltk-part-3/

class Tagger:

def __init__(self, sentences=1000, corpus=nltk.corpus.brown):
logger.debug(‘training with ‘ + str(sentences) + ‘ sentences’)
train_sents = corpus.tagged_sents()[:sentences]

def backoff_tagger(tagged_sents, tagger_classes, backoff=None):
if not backoff:
backoff = tagger_classes[0](tagged_sents)
del tagger_classes[0]

for cls in tagger_classes:
tagger = cls(tagged_sents, backoff=backoff)
backoff = tagger

return backoff

word_patterns = [
(r'^-?[0-9]+(.[0-9]+)?$’, ‘CD’),
(r’.*ould$’, ‘MD’),
(r’.*ing$’, ‘VBG’),
(r’.*ed$’, ‘VBD’),
(r’.*ness$’, ‘NN’),
(r’.*ment$’, ‘NN’),
(r’.*ful$’, ‘JJ’),
(r’.*ious$’, ‘JJ’),
(r’.*ble$’, ‘JJ’),
(r’.*ic$’, ‘JJ’),
(r’.*ive$’, ‘JJ’),
(r’.*ic$’, ‘JJ’),
(r’.*est$’, ‘JJ’),
(r’^a$’, ‘PREP’),
]

raubt_tagger = backoff_tagger(train_sents, [nltk.tag.AffixTagger,
nltk.tag.UnigramTagger, nltk.tag.BigramTagger, nltk.tag.TrigramTagger],
backoff=nltk.tag.RegexpTagger(word_patterns))

templates = [
brill.SymmetricProximateTokensTemplate(brill.ProximateTagsRule, (1,1)),
brill.SymmetricProximateTokensTemplate(brill.ProximateTagsRule, (2,2)),
brill.SymmetricProximateTokensTemplate(brill.ProximateTagsRule, (1,2)),
brill.SymmetricProximateTokensTemplate(brill.ProximateTagsRule, (1,3)),
brill.SymmetricProximateTokensTemplate(brill.ProximateWordsRule, (1,1)),
brill.SymmetricProximateTokensTemplate(brill.ProximateWordsRule, (2,2)),
brill.SymmetricProximateTokensTemplate(brill.ProximateWordsRule, (1,2)),
brill.SymmetricProximateTokensTemplate(brill.ProximateWordsRule, (1,3)),
brill.ProximateTokensTemplate(brill.ProximateTagsRule, (-1, -1), (1,1)),
brill.ProximateTokensTemplate(brill.ProximateWordsRule, (-1, -1), (1,1))
]

trainer = brill.FastBrillTaggerTrainer(raubt_tagger, templates)
logger.debug(‘starting training’)
braubt_tagger = trainer.train(train_sents, max_rules=100, min_score=3)
logger.debug(‘finished training’)
self.tagger = braubt_tagger

def tag(self,sentence):
return self.tagger.tag(sentence)

and it gives me something like:

[('Further', 'AP'), ('snow', None), ('is', 'BEZ'), ('expected', 'VBN'), ('to', 'TO'), ('push', None), ('into', 'IN'), ('many', 'AP'), ('southern', 'JJ-TL'), ('and', 'CC'), ('eastern', 'JJ-TL'), ('parts', 'NNS'), ('of', 'IN'), ('England,', None), ('including', 'IN'), ('London,', None), ('overnight', 'NN'), ('and', 'CC'), ('during', 'IN'), ('the', 'AT'), ('day', 'NN'), ('on', 'IN'), ('Friday.', None)]

However when I feed this into the chunker I still get nothing:

[(('Further', 'AP'), None), (('snow', None), None), (('is', 'BEZ'), None), (('expected', 'VBN'), None), (('to', 'TO'), None), (('push', None), None), (('into', 'IN'), None), (('many', 'AP'), None), (('southern', 'JJ-TL'), None), (('and', 'CC'), None), (('eastern', 'JJ-TL'), None), (('parts', 'NNS'), None), (('of', 'IN'), None), (('England,', None), None), (('including', 'IN'), None), (('London,', None), None), (('overnight', 'NN'), None), (('and', 'CC'), None), (('during', 'IN'), None), (('the', 'AT'), None), (('day', 'NN'), None), (('on', 'IN'), None), (('Friday.', None), None)]

Is it I wonder because not all tokens get tags?

Thanks for your help so far.

It looks like you left out a step: part of speech tagging. The chunker requires tagged tokens, like [('foo', 'JJ'), ('bar', 'NN')] in order to extract chunks. So you’ll have to train a part of speech tagger as well as the chunker, then run the tokens thru the tagger, and use that output as input to the chunker. Check out my articles about part of speech tagging, starting with Part 1. You also may want to look at the NLTK Chunking Guide.

class Chunker:

def __init__(self):
def conll_tag_chunks(chunk_sents):
tag_sents = [nltk.chunk.tree2conlltags(tree) for tree in chunk_sents]
return [[(t, c) for (w, t, c) in chunk_tags] for chunk_tags in tag_sents]
train_sents = nltk.corpus.conll2000.chunked_sents()
train_chunks = conll_tag_chunks(train_sents)
logger.debug(‘training u_chunker’)
u_chunker = UnigramTagger(train=train_chunks)
logger.debug(‘training ub_chunker’)
ub_chunker = BigramTagger(train=train_chunks, backoff=u_chunker)
#ubt_chunker = nltk.tag.TrigramTagger(train_chunks, backoff=ub_chunker)
#ut_chunker = nltk.tag.TrigramTagger(train_chunks, backoff=u_chunker)
#utb_chunker = nltk.tag.BigramTagger(train_chunks, backoff=ut_chunker)
logger.debug(‘finished training’)
self.chunker = ub_chunker

def chunk(self, tokens):
return self.chunker.tag(tokens)

and tried to do this:

chunker = Chunker()
s = “Since then, we’ve changed how we use Python a ton internally.”
tokens = s.split()
chunked = chunker.chunk(tokens)
print chunked

which gives:

[(u'Since', None), (u'then,', None), (u"we've", None), (u'changed', None), (u'how', None), (u'we', None), (u'use', None), (u'Python', None), (u'a', None), (u'ton', None), (u'internally.', None)]

In other words, nothing at all gets chunked.

Have I missed something?

Col