Sunday April 23, 2017


Grub - Distributed Web Crawling Project
  Posted by: AceBHound on Mar 14th, 2005 5:29 PM
Here's a neat concept for a distributed computing project -- rather than search for aliens or compute the value of pi, you can help crawl the information super-highway with Grub (sorry, couldn't help putting a Cable Guy reference in there). The project's goal is to eventually get to the point where they can crawl the entire web each day and to use this information in a number of search engines.

So what's the incentive? Well, if you run a website or multiple websites, Grub will crawl your content before crawling other content. This also helps in the long-run with bandwidth consumption since webmasters are crawling their own content. This could mean early adopters will have a jump in marketing on other websites. At the current time, they have approximately 32,000 members -- with 525 active members this month. They estimate that there are over 10 billion web pages worth of content on the internet, with 2 million new pages or pages revised each day. Check out their FAQ for more info on the project. Looks like the WiseNut search engine is the first to benefit from Grub. I'm definitely gonna jump on the bandwagon.

[Update: Huh.. it appears that there are known problems with WindowsXP SP2 & the Grub Client. Nothing happens when you try to run the program. Looks like I'll have to wait for a fix or install on a linux box sometime.]

Read More

View All Articles