Comments on: Execnet vs Disco for Distributed NLTK http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/#utm_source=feed&utm_medium=feed&utm_campaign=feed Weotta be Hacking Thu, 19 Apr 2012 12:53:00 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Hadoop MapReduce http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-951 Hadoop MapReduce Thu, 19 Apr 2012 12:53:00 +0000 http://streamhacker.com/?p=765#comment-951 Nice Post dude.. Keep it up. Nice Post dude.. Keep it up.

]]> By: Jacob Perkins http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-641 Jacob Perkins Thu, 09 Sep 2010 02:46:00 +0000 http://streamhacker.com/?p=765#comment-641 Thanks for the comment. I'm not really a fan of visual programming tools, but I have been meaning to checkout stackless. Could be a good combination with execnet, where execnet handles the distribution, and stackless handles processing on each node. Thanks for the comment. I’m not really a fan of visual programming tools, but I have been meaning to checkout stackless. Could be a good combination with execnet, where execnet handles the distribution, and stackless handles processing on each node.

]]> By: Eric Gaumer http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-640 Eric Gaumer Tue, 07 Sep 2010 02:28:00 +0000 http://streamhacker.com/?p=765#comment-640 Great blog for NTLK users. Lot's of helpful insight. We do a lot of NLP work as a precursor to indexing content for search. It's pretty typical for us to process tens of millions of documents so we've designed a highly scalable pipeline framework that allows you to build document processing clusters. In terms of the problem you've described here, we're using stackless Python to model pipeline stages (referred to as Components). Since tasklets are true co-routines, they're instantiated once and run forever making them ideal for loading data models and then using those models across huge volumes of data. Check it out, it might fit into your needs at some point. http://www.pypes.org/ A simple example of writing a component. http://bitbucket.org/diji/pypes/wiki/Reverse_Field We also provide a visual designer that allows you graphically design data flow graphs from the components you write. http://bitbucket.org/diji/pypes/wiki/Screenshots Great blog for NTLK users. Lot’s of helpful insight. We do a lot of NLP work as a precursor to indexing content for search. It’s pretty typical for us to process tens of millions of documents so we’ve designed a highly scalable pipeline framework that allows you to build document processing clusters.

In terms of the problem you’ve described here, we’re using stackless Python to model pipeline stages (referred to as Components). Since tasklets are true co-routines, they’re instantiated once and run forever making them ideal for loading data models and then using those models across huge volumes of data.

Check it out, it might fit into your needs at some point.

http://www.pypes.org/

A simple example of writing a component.

http://bitbucket.org/diji/pypes/wiki/Reverse_Field

We also provide a visual designer that allows you graphically design data flow graphs from the components you write.

http://bitbucket.org/diji/pypes/wiki/Screenshots

]]> By: Jacob http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-299 Jacob Sat, 19 Dec 2009 20:16:58 +0000 http://streamhacker.com/?p=765#comment-299 That's what I thought at first too, but it turns out the Params object (and objects attached to it) are unpickled before every map call. And when unpickling takes 2 seconds, it's prohibitively expensive. That’s what I thought at first too, but it turns out the Params object (and objects attached to it) are unpickled before every map call. And when unpickling takes 2 seconds, it’s prohibitively expensive.

]]> By: Jacob Perkins http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-693 Jacob Perkins Sat, 19 Dec 2009 20:16:00 +0000 http://streamhacker.com/?p=765#comment-693 That's what I thought at first too, but it turns out the Params object (and objects attached to it) are unpickled before every map call. And when unpickling takes 2 seconds, it's prohibitively expensive. That’s what I thought at first too, but it turns out the Params object (and objects attached to it) are unpickled before every map call. And when unpickling takes 2 seconds, it’s prohibitively expensive.

]]> By: Justin http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-298 Justin Sat, 19 Dec 2009 19:45:03 +0000 http://streamhacker.com/?p=765#comment-298 Can't you just use this to cache the object? http://discoproject.org/doc/faq.html#how-to-maintain-state-across-many-map-reduce-calls Can’t you just use this to cache the object?

http://discoproject.org/doc/faq.html#how-to-maintain-state-across-many-map-reduce-calls

]]> By: Justin http://streamhacker.com/2009/12/14/execnet-disco-distributed-nltk/comment-page-1/#comment-692 Justin Sat, 19 Dec 2009 19:45:00 +0000 http://streamhacker.com/?p=765#comment-692 Can't you just use this to cache the object? http://discoproject.org/doc/faq.html#how-to-maintain-state-across-many-map-reduce-calls Can’t you just use this to cache the object?

http://discoproject.org/doc/faq.html#how-to-maintain-state-across-many-map-reduce-calls

]]>