Joining phrases with “_” then splitting out later can definitely work, because it would let you use a manual dictionary UnigramTagger to define the tags, ensuring they don’t get missed. After chunking, it should be easy to split up any word with “_” in it.

An alternative solution is to transform your list of phrases into a corpus of tagged & chunked phrases, then train a tagger and chunker on it. Basically, each line could look like “[General/NN Electric/NN ...]“. The brackets are used by the BracketParseCorpusReader (used by treebank) to define noun phrases. This method can definitely work, but it may take a lot more effort on your part to tag & chunk every phrases.

Thanks for your fast reply! I was away (on vacation) but now I’m back to work. I’m following your recommendations, also learned NLTK Trainer and using for training.

As I know, key-phrases extraction highly depends on the POS-tagger efficiency. The thing is that POS-tagger cannot work perfectly due to unknown words and words or phrases that have special meaning (names of trademarks, companies, products and so on).
I want to use categorized phrases/words lists in order to parse information more efficiently. For example, I want to ensure that “General Electric” phrase won’t be broken or “iphone” won’t be assigned the “CD” POS-tag (as it is now in my application).
So I want to assign correct POS-tags to a set of non-standard words. It’s clear about tagging single words. But when it comes to multiple-word phrases, I’m not sure about the solution. So I want some multi-word phrases to be kept as they are and treated just like single nouns on the chunking stage (as I think, it may improve chunking efficiency).

I’ve been thinking about how to do this and that’s the most obvious solution I see:
Before the stage of tokenizing sentences into words the application finds all the occurrences from the lists (i.e. categorized lists of phrases/words) and replaces spaces with “_” (for example) so that “General Electric” will become “General_Electric” and won’t be split into separate words (at least by TreebankWordTokenizer.tokenize()). This way using my own POS tagger as the first in the chain will tag “General_Electric” as a single noun and chunking stage may be more successful. Also I’ll have to somehow remember that “General_Electric” refers to the original “General Electric” and belongs to “Companies” list.
OR
Apply some function that will join 2 words “General” “Electric” into a single “General Electric”.

That’s not the solution that I like. I’m sure Python or NLTK provide some suitable functionality. Could you suggest a better way to do such things? I would appreciate any thoughts.

Sounds like you’re clear the overall structure, though you may want to think about removing the stemming & stop word filtering steps, as transforming chunks & words can often change the meaning. But that depends on your needs.

To improve chunking, yes, you could define a few select chunk rules where you’re close to 100% sure that if they work, then it’s a good chunk, and if you get nothing, then use a trained chunker. In other words, you define a few high-precision regular expressions, and then rely on the trained chunker to find/recall chunks the manual chunker misses.

The other thing I recommend is if you’re using a treebank trained chunker, then you should use a treebank trained tagger. Or consider training on conll2000 as well. Checkout https://github.com/japerk/nltk-trainer for some scripts I made to make training easier.

However, the best option (but also the most time consuming) is to create your own tagged & chunked corpus, then train a tagger & chunker on that. I recommend a bootstrap approach, where you’d use an existing tagger & chunker to create an initial corpus, then go in and hand-correct before training a custom tagger & chunker. This is the only way I know of to end up with highly accurate results, and also allows you to define custom chunk structures that aren’t found in treebank, because you have control of the training data.

This is probably because you didn’t set the classifier’s cutoff_prob parameter. Without it the tagger will never consult its backoff. I don’t use a backoff tagger with a classifier tagger, as anything else is only going to be less accurate.