or

raw_dataset = [([word.lower() for word in sentence], “physics”) for sentence in physics.sents()]

physics_corpus = LazyCorpusLoader(‘cookbook’, PlaintextCorpusReader, ['physics.txt'])
def bag_of_words(sentence): return dict([(word, True) for word in sentence])

Then I label the training data

raw_dataset = [(sentence, "physics") for sentence in physics.sents()]

I would have preferred that raw_dataset be this instead:

raw_dataset2 = [(word.lower(), "physics") for word in physics.words()]

But the problem is that if I use raw_dataset2 to create my featuresets to train the classifier like this:

featuresets = [(bag_of_words(word), label) for (word, label) in raw_dataset2]

Then I get this:

[({'h': True, 'e': True, 'T': True}, 'physics'), ({'a': True, 'c': True, 'i': True, 'h': True, 'l': True, 'p': True, 's': True, 'y': True}, 'physics')

Not what I want. But with plain old raw_dataset:

raw_dataset = [(sentence, "physics") for sentence in physics.sents()]

featuresets = [(bag_of_words(sentence), label) for (sentence, label) in raw_dataset]

It returns whole words as I want:

({‘and’: True, ‘distances’: True, ‘scales’: True, ‘subatomic’: True, ‘over’: True, ‘challenges’: True, ‘meters’: True}, ‘physics’)

So my dilemma is that I’m stuck with physics.sents() so that bag_of_words returns whole words rather than letters. But I can’t lowercase sentences so a list comprehension like [word.lower() for word in physics.sents()] is not an option. And that’s why I put word.lower() in the bag_of_words() function.  I’m having trouble seeing where I can apply word.lower() . I tried converting raw_dataset to a string so I could lowercase the words and then convert back to a list, but I should have known it’s inane. Any insights?

thnx