New Dedicated Server question!

[JulianD]

Hi... currently I'm hosted on a P4 2.8 1GB RAM running Apache 1.3.x and PHP5.1.x / MySQL 4.1.

With a cookie timeout of 15mins I get an overall of 100-110 Users Online and 600 visitors... My server is currently with loads of 10 at peak hours (it's optimized.. just too many visitors) so I'm looking for a new dedicated server.

I was checking out at Servstra (http://www.servstra.com/servers/) and I need to make a choice:

Dual Opteron 242 2GB RAM or Dual Xeon 3GHz HT 1GB RAM

I can't go beyond this because my budget is limited... What do you think it's best for my needs?

thanks!

[Zachery]

Dual Opteron 242 2GB RAM

WAY better.

[ChrisLM2001]

For that many connections it might be better to curb the bots (just indexing a forum pages isn't enough to generate traffic anyway), and just get another P4 for mysql in the future. It's better to have 2 different servers instead of one box, as the I/O isn't saturated, and the connections aren't as limited (Apache connections that are limited to 1000 on one box, can be 2000 on two, for example). Duallies have to share the same outside connections, and that's where resources are eaten up.

You only have at most 200 members online at a time, and that doesn't justify getting a dually. So save your pennies for another P4 for now, and grow into your servers. That way you can justify the expense, and buy just what you need.

Web hosts will gladly sell you a quad Opteron, but be sensible and get what you need now with 10% to 20% overhead for peaks.

BTW, after reading some more on optimizations, you might like to get a kernal upgrade and add lingerd...

http://www.iagora.com/about/software/lingerd/

As that can divide the traffic load on your server. It's a lot more efficient than other methods (like proxy cacheing). Give that a try, if you haven't already, and see if that'll help before thinking on getting another server.

Chris

[JulianD]

Chris, thank you very much for your comments... I don't know what to do now :p I don't know if a P4 2.8 would hold with apache only and place the database on another box. My current provider doesn't allow private lans so I guess I'd have to switch providers anyway... Maybe I can try with the dual opteron and see how it performs, and if needed, I can get another box in the future for the webserver only or for the mysql... don't know... Adsense revenue is increasing so I think in a near future I can get two boxes for my site...

Guys, another question... I live in Colombia and colocation here is extremely expensive... that's why I always lease in the US... but what if my site keeps growing up... if it does, sometime in the future my site will need more and more server load balanced... there are providers in the US where I can lease such setups? Thanks!

[ChrisLM2001]

You need 2+ servers for performance -- no matter what type of processor is used.

The issue isn't what a processor can push, it's the I/O saturation and memory usage that is a bane to everything on computers (from memory; to video; to network speeds). Once saturated with connections (which Apache is notoriously inefficient with handling) even a dual/quad Opteron will crawl. So don't equate processing power with speed with servers. Servers operate differently than desktops. Performance with servers isn't about processing or memory speed, it's about eliminating I/O bottlenecks/memory limits/curbing mysql join table scans.

With Apache every single page will take 2+ megabytes of memory for just static page serving, which can add up with 600 bots on a site -- which is why folks need to tame them to handle their serve loads better. If you have max of 1000 connections, and have only 200 members online with 600 bots, you can see who's going to get shortchanged -- and it's not the bots.

When you grow, grow in piecemeal. Effective server performance is around load balancing and using more web front ends that feed off one or more fat mysql server(s). Web front ends can be but P3's (believe it or not) as, again, processing speed isn't an issue serving static pages/pics online. But dynamic page generation/serving with table scans from mysql (which again another notoriously inefficient db engine, and just beginning to catch up with more optimized ones <-- free ware comes with a price tag not in software, hardware!!) needs the fattest piped server you can find.

But for now you really don't need that type of hardware with but 200 max active users. Save your pennies for later (or use it for advertizing your forum -- or sell products to pay for the server rent and your time).

Chris

[bugzy]

what are tips to control bot/spiders?

[KingSpade]

what are tips to control bot/spiders?

Limit what they can access using robots.txt. A tutorial on how to use robots.txt is below.

http://www.searchengineworld.com/robots/robots_tutorial.htm


Very helpful indeed. Specifically, you'd like them to access index.php and those associated with your content, like forumdisplay.php, showthread.php and so forth, but they don't have a need to be trying to access profile.php, memberlist.php, register.php or anything else.

Since they can't access those files anyway, there's no need to allow them to be spidered.

[ChrisLM2001]

Yep, and faster the bots grab and disconnect the more server resources an owner will have.

Trick is getting them to index the juicy parts of a site, but not hanging around to suck up all of your connections. Get them off as quickly as possible after doing a crawl, and keep them off at intervals (let them index once a day, instead as needed -- faster indexing doesn't equate to more content being indexed, as delays occur before they're implimented anyway. Way better to wait a day and have more resources, than having 600 bots eating them up all day. That's wasted $$$).

Chris

[JulianD]

Hi Chris, thanks for your great contribution!

I liked the robots.txt part... Something that caught my attention was "let them index once a day, instead as needed". How can you do that?

[ChrisLM2001]

Use the "crawl-delay" syntax in your robots.txt file...

http://www.ysearchblog.com/archives/000078.html



Crawl-Delay: There's one last trick you can use to help reduce the bandwidth requirements of your site. You can use a special robots.txt (http://www.searchengineworld.com/robots/robots_tutorial.htm) directive, crawl-delay (http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html), to reduce the speed at which our crawlers make requests to your site. This allows webmasters to manage their bandwidth without restricting content on their site from crawlers and is being used effectively by sites like Slashdot (http://slashdot.org/). A safe value for this would be a delay that would allow us to fetch every page on your site in about five days. So a five second delay (crawl-delay: 5.0) would be fine for a site with 2,000 pages, but not for a site with 100,000 or more.



Better explained and simple syntax used (which can be used on other search spiders as well)...

http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html

Chris

[MRGTB]

Use the "crawl-delay" syntax in your robots.txt file...

http://www.ysearchblog.com/archives/000078.html



Better explained and simple syntax used (which can be used on other search spiders as well)...

http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html

Chris

Spiders can be a real pain at times, if you get them in big numbers hitting your site they can really slow things down.

[deatheyes]

crawl-delay can use for google bot ?