Chunk extraction is a useful preliminary step to information extraction, that creates parse trees from unstructured text with a chunker. Once you have a parse tree of a sentence, you can do more specific information extraction, such as named entity recognition and relation extraction.
Chunking is basically a 3 step process:
- Tag a sentence
- Chunk the tagged sentence
- Analyze the parse tree to extract information
I’ve already written about how to train a NLTK part of speech tagger and a chunker, so I’ll assume you’ve already done the training, and now you want to use your pos tagger and iob chunker to do something useful.
IOB Tag Chunker
The previously trained chunker is actually a chunk tagger. It’s a Tagger that assigns IOB chunk tags to part-of-speech tags. In order to use it for proper chunking, we need some extra code to convert the IOB chunk tags into a parse tree. I’ve created a wrapper class that complies with the nltk ChunkParserI interface and uses the trained chunk tagger to get IOB tags and convert them to a proper parse tree.
import nltk.chunk import itertools class TagChunker(nltk.chunk.ChunkParserI): def __init__(self, chunk_tagger): self._chunk_tagger = chunk_tagger def parse(self, tokens): # split words and part of speech tags (words, tags) = zip(*tokens) # get IOB chunk tags chunks = self._chunk_tagger.tag(tags) # join words with chunk tags wtc = itertools.izip(words, chunks) # w = word, t = part-of-speech tag, c = chunk tag lines = [' '.join([w, t, c]) for (w, (t, c)) in wtc if c] # create tree from conll formatted chunk lines return nltk.chunk.conllstr2tree('\n'.join(lines))
Now that we have a proper NLTK chunker, we can use it to extract chunks. Here’s a simple example that tags a sentence, chunks the tagged sentence, then prints out each noun phrase.
# sentence should be a list of words tagged = tagger.tag(sentence) tree = chunker.parse(tagged) # for each noun phrase sub tree in the parse tree for subtree in tree.subtrees(filter=lambda t: t.node == 'NP'): # print the noun phrase as a list of part-of-speech tagged words print subtree.leaves()
Each sub tree has a phrase tag, and the leaves of a sub tree are the tagged words that make up that chunk. Since we’re training the chunker on IOB tags, NP stands for Noun Phrase. As noted before, the results of this natural language processing are heavily dependent on the training data. If your input text isn’t similar to the your training data, then you probably won’t be getting many chunks.