or

raw_dataset = [([word.lower() for word in sentence], “physics”) for sentence in physics.sents()]

physics_corpus = LazyCorpusLoader(‘cookbook’, PlaintextCorpusReader, ['physics.txt'])
def bag_of_words(sentence): return dict([(word, True) for word in sentence])

Then I label the training data

raw_dataset = [(sentence, "physics") for sentence in physics.sents()]

I would have preferred that raw_dataset be this instead:

raw_dataset2 = [(word.lower(), "physics") for word in physics.words()]

But the problem is that if I use raw_dataset2 to create my featuresets to train the classifier like this:

featuresets = [(bag_of_words(word), label) for (word, label) in raw_dataset2]

Then I get this:

[({'h': True, 'e': True, 'T': True}, 'physics'), ({'a': True, 'c': True, 'i': True, 'h': True, 'l': True, 'p': True, 's': True, 'y': True}, 'physics')

Not what I want. But with plain old raw_dataset:

raw_dataset = [(sentence, "physics") for sentence in physics.sents()]

featuresets = [(bag_of_words(sentence), label) for (sentence, label) in raw_dataset]

It returns whole words as I want:

({‘and’: True, ‘distances’: True, ‘scales’: True, ‘subatomic’: True, ‘over’: True, ‘challenges’: True, ‘meters’: True}, ‘physics’)

So my dilemma is that I’m stuck with physics.sents() so that bag_of_words returns whole words rather than letters. But I can’t lowercase sentences so a list comprehension like [word.lower() for word in physics.sents()] is not an option. And that’s why I put word.lower() in the bag_of_words() function.  I’m having trouble seeing where I can apply word.lower() . I tried converting raw_dataset to a string so I could lowercase the words and then convert back to a list, but I should have known it’s inane. Any insights?

thnx

I’m trying to classify song lyrics, where bigrams matter. My program is reading from custom corpuses full of lyrics from the Web. I need to lowercase all the unigrams and bigrams, and that’s where there’s an issue. First, the bag of words function

def bag_of_words(sentence): return dict([(word.lower(), True) for word in sentence])

Then the bigram extractor

def bag_of_bigrams_words(sentence, score_fn = BigramAssocMeasures.chi_sq, n = 200): bigram_finder = BigramCollocationFinder.from_words(sentence) bigrams = bigram_finder.nbest(score_fn, n) return bag_of_words(sentence + bigrams)

So, I use bag_of_bigrams_words() on a simple sentence like 

bag_of_bigrams_words(['Joey', 'Plays', 'the', 'Guitar'])

 and I get the following error

Traceback (most recent call last):  File “”, line 1, in     bag_of_bigrams_words(['Joey', 'Plays', 'the', 'Guitar'])  File “”, line 4, in bag_of_bigrams_words    return bag_of_words(sentence + bigrams)  File “”, line 2, in bag_of_words    return dict([(word.lower(), True) for word in sentence])AttributeError: ‘tuple’ object has no attribute ‘lower’

It seems that as long as word.lower() is in bag_of_words(), it’s incompatible with the bigram tuples. What’s the best way around this, considering that I need word.lower() in bag_of_words() in order to reduce dimensionality ?