import nltkimport collectionsimport randomimport nltk.metricsfrom nltk.corpus.reader import PlaintextCorpusReaderfrom nltk.corpus.util import LazyCorpusLoaderdirty_teen_corpus = LazyCorpusLoader(‘cookbook’, PlaintextCorpusReader, ['dirty_teen.txt'])clean_teen_corpus = LazyCorpusLoader(‘cookbook’, PlaintextCorpusReader, ['clean_teen.txt'])raw_dataset = ([(sentence, "Dirty") for sentence in dirty_teen_corpus.sents()] +           [(sentence, "Clean") for sentence in clean_teen_corpus.sents()])def sexy_text(sentence):    return dict([(word.lower(), True) for word in sentence])featuresets = [(sexy_text(sentence), label) for (sentence, label) in raw_dataset]random.shuffle(featuresets)train_feats, test_feats = featuresets[:270], featuresets[270:]classifier = nltk.NaiveBayesClassifier.train(train_feats)refsets = collections.defaultdict(set)testsets = collections.defaultdict(set)for i, (feats, label) in enumerate(test_feats): refsets[label].add(i) observed = classifier.classify(feats) testsets[observed].add(i)print ‘Accuracy:’, nltk.classify.accuracy(classifier, test_feats)print “dirty precision:”, nltk.metrics.precision(refsets["Dirty"], testsets["Dirty"])print “dirty recall:”, nltk.metrics.recall(refsets["Dirty"], testsets["Dirty"])print “clean precision:”, nltk.metrics.precision(refsets["Clean"], testsets["Clean"])print “clean recall:”, nltk.metrics.recall(refsets["Clean"], testsets["Clean"])print classifier.show_most_informative_features(15)

I’m classifying text as either vulgar or clean. I ran the metrics on the data several times and got slightly different results each time. For example, notice that every score has changed slightly from the first to the second evaluation even though it’s the exact same data. Any idea why this might be? 

Accuracy: 0.875968992248
dirty precision: 0.965517241379
dirty recall: 0.8
clean precision: 0.802816901408
clean recall: 0.966101694915
Most Informative Features
                       ! = True            Dirty : Clean  =     19.8 : 1.0
                       . = None            Dirty : Clean  =      9.2 : 1.0
                     ass = True            Dirty : Clean  =      6.8 : 1.0
                    even = True            Clean : Dirty  =      6.1 : 1.0
                    feel = True            Clean : Dirty  =      5.7 : 1.0
               something = True            Clean : Dirty  =      5.3 : 1.0
                    your = True            Dirty : Clean  =      5.2 : 1.0
                     had = True            Dirty : Clean  =      5.2 : 1.0
                   would = True            Clean : Dirty  =      4.7 : 1.0
                   being = True            Clean : Dirty  =      4.5 : 1.0

Accuracy: 0.744186046512dirty precision: 1.0dirty recall: 0.521739130435clean precision: 0.645161290323clean recall: 1.0Most Informative Features                       ! = True            Dirty : Clean  =     19.5 : 1.0                      be = True            Clean : Dirty  =      8.7 : 1.0                     don = True            Clean : Dirty  =      7.9 : 1.0                     get = True            Clean : Dirty  =      6.2 : 1.0                     not = True            Clean : Dirty  =      6.2 : 1.0               something = True            Clean : Dirty  =      6.2 : 1.0                    this = True            Clean : Dirty  =      6.2 : 1.0                      by = True            Clean : Dirty  =      6.2 : 1.0                   would = True            Clean : Dirty  =      6.2 : 1.0                   never = True            Clean : Dirty  =      6.2 : 1.0                     off = True            Dirty : Clean  =      6.1 : 1.0                   being = True            Clean : Dirty  =      5.4 : 1.0                     did = True            Clean : Dirty  =      5.4 : 1.0                    feel = True            Clean : Dirty  =      5.2 : 1.0                       . = None            Dirty : Clean  =      5.2 : 1.0

1) Yes, it’s pretty simple. Just reduce a lot your training and test sets. In the classifier shown here (http://streamhacker.com/2010/05/10/text-classification-sentiment-analysis-naive-bayes-classifier/) just add these below line #9.

negids = negids[:10]
posids = posids[:10]

We’re reducing our sets to 10 files for each label. Obviously this is not what we want in a real environment and this doesn’t happen by training a relatively large corpus, but it just caught my attention to have ‘None’ values for words that appear both in the training and test sets. Shouldn’t all be True?

2) Great! I have to make a similar implementation using the Spanish language so it’s time for me to compile a neutral corpus

1) I haven’t seen None before, but it doesn’t sound it should be there. Could you post the output, and maybe the code you use to train?

2) Exactly, and I outlined the method in http://streamhacker.com/2011/01/05/hierarchical-classification/

1) When training the classifier with smaller corpus sometimes the most informative features function shows words with the value ‘None’. According to NLTK’s documentation: “The feature value ‘None’ is reserved for unseen feature values” but using your example I get None values for words which are in both training and test sets. Why? I don’t get it.

2) To implement neutrality you just used another classifier with the subjectivity dataset, right?

Thanks in advance!!