Your own search engine w/ spiders?

[qxh]

How would you go about starting you own "mini" search engine with spiders? Would you need SSH to actually send the spiders out or would that be done via a script? What database backend would you use? Do you even need a database backend?

So many questions.. Does anyone have any kinda tutorials or something for it, or anymore information?

Thanks,
Scott

[IDN]

I don't think you're a moron

[merk]

Why would you do such a thing?

Have you read about what google actually runs to achieve what they do?

http://www.google.com/technology/pigeonrank.html

[qxh]

I don't think you're a moron

LOL

It's just a mini search engine which I need to develop for certain small sites..though it needs a spider

[wandrer]

How would you go about starting you own "mini" search engine with spiders?

http://search.mnogo.ru/

mnoGoSearch (formerly known as UdmSearch) is a full-featured web search engine software for intranet and internet servers. mnoGoSearch for UNIX is a free software covered by the GNU General Public License and mnoGoSearch for Windows is a commercial search software version.

mnoGoSearch software has a number of unique features, which makes it appropriate for a wide range of applications from search within your site to specialized search systems such as cooking recipes or newspaper searches, ftp archive search, MP3 search, news articles search or even national-wide portal search engine.

[qxh]

Thanks wandrer, I'll be looking into it

[merk]

I still think you need a Pigeon Cluster!

[Beorn]

Read lynx(1) and look for -crawl and -traversal.

Use that to spider the whole site. After that, loop through each file you downloaded. Create a search index like vB's search.

[merk]

Totally offtopic Beorn, but shouldnt the fury line read

dumbasspeoplesmoney--; ??

smartasslawyermoney++;, too

[Beorn]

Totally offtopic Beorn, but shouldnt the fury line read

dumbasspeoplesmoney--; ??

smartasslawyermoney++;, too

Not really, because the dumbasspeoplesmoney keeps growing as long as the court system is screwed up....like the woman who sued 7-11 because her coffee was too hot....she put it between her legs, and while driving she hit a bump, it spilled all over her and burned her. Supposedly, this is 7-11's fault.

[Shining Arcanine]

Learn ASP.NET and then start coding. It is ideal for large applications like this. That is, if you want to code a search engine for the entire internet.

[qxh]

I don't want to use ASP, I would like to ideally use PHP and mySQL database..CGI/Perl maybe!

[Icheb]

I don't want to use ASP, I would like to ideally use PHP and mySQL database..CGI/Perl maybe!

*wonders if he knows what CGI/Perl really is*

Ok, never mind.

"Do you even need a database backend?"

Seriously, if you don't know that, you shouldn't start coding a search engine.

[merk]

Not really, because the dumbasspeoplesmoney keeps growing as long as the court system is screwed up....like the woman who sued 7-11 because her coffee was too hot....she put it between her legs, and while driving she hit a bump, it spilled all over her and burned her. Supposedly, this is 7-11's fault.

True i guess, though, you could also consider that the lawyers money grows as dumb people sue.

Lucky our court systems are not that bad around here

[0ptima]

http://phpdig.toiletoine.net/