Comments on: Execnet vs Disco for Distributed NLTK

By: Jacob Perkins

Jacob Perkins — Thu, 09 Sep 2010 02:46:00 +0000

Thanks for the comment. I’m not really a fan of visual programming tools, but I have been meaning to checkout stackless. Could be a good combination with execnet, where execnet handles the distribution, and stackless handles processing on each node.

By: Eric Gaumer

Eric Gaumer — Tue, 07 Sep 2010 02:28:00 +0000

Great blog for NTLK users. Lot’s of helpful insight. We do a lot of NLP work as a precursor to indexing content for search. It’s pretty typical for us to process tens of millions of documents so we’ve designed a highly scalable pipeline framework that allows you to build document processing clusters.

In terms of the problem you’ve described here, we’re using stackless Python to model pipeline stages (referred to as Components). Since tasklets are true co-routines, they’re instantiated once and run forever making them ideal for loading data models and then using those models across huge volumes of data.

Check it out, it might fit into your needs at some point.

http://www.pypes.org/

A simple example of writing a component.

http://bitbucket.org/diji/pypes/wiki/Reverse_Field

We also provide a visual designer that allows you graphically design data flow graphs from the components you write.

http://bitbucket.org/diji/pypes/wiki/Screenshots

By: Jacob

Jacob — Sat, 19 Dec 2009 20:16:58 +0000

That’s what I thought at first too, but it turns out the Params object (and objects attached to it) are unpickled before every map call. And when unpickling takes 2 seconds, it’s prohibitively expensive.

By: Justin

Justin — Sat, 19 Dec 2009 19:45:03 +0000

Can’t you just use this to cache the object?

http://discoproject.org/doc/faq.html#how-to-maintain-state-across-many-map-reduce-calls