Site is Sluggish

[doctorcoug]

I HIGHLY doubt this is search engine robots.

You checked your error log...I want you to check your access logs. This is a ton of data, but you can use open source programs to filter it based upon parameters. You might find thousands of requests for a single page, which you can then block in the .htaccess, which will help with ram. I still think you should look at outbound logs lot make certain you don't have a Bank of America login page being hosted on an obscure folder on your server. You might be sending php phishing emails without your knowledge.

Also, I HIGHLY recommend you use a list of IP blocks, such as one found at countryipblocks.net. This will automatically block ALL users from a certain country. Be careful which countries you block, since the world is our campus.

With the captcha in vbulletin, you can ask a custom question that only live humans would know, such as "who was the greatest BYU football coach of all time". It is part of the standard VB package.

[Jeff Lebowski]

I HIGHLY doubt this is search engine robots.

I am starting to believe. I added several additional lines to the robots.txt file to include database intensive things like search.php, login.php, and newthread.php. No need to index any of those and they probably just keep pinging at them. Perhaps it is coincidence, but in the two hours since I made that change, the site has been as fast as I have seen it in a long time. I will keep my fingers crossed.

You checked your error log...I want you to check your access logs. This is a ton of data, but you can use open source programs to filter it based upon parameters. You might find thousands of requests for a single page, which you can then block in the .htaccess, which will help with ram.

I opened up the access log today, but it was 27 MB of text, for just one day of logging. Yikes.

If this keeps up, point me to one of these filtering programs.

I still think you should look at outbound logs lot make certain you don't have a Bank of America login page being hosted on an obscure folder on your server. You might be sending php phishing emails without your knowledge.

Sounds worth checking, but I don't recall seeing an outbound log anywhere in the hostgator control panel.

Also, I HIGHLY recommend you use a list of IP blocks, such as one found at countryipblocks.net. This will automatically block ALL users from a certain country. Be careful which countries you block, since the world is our campus.

I have quite a few subdomains blocked already. This is easy to do. But until we can trace the load down to some country, I am not sure I want to go that route.

With the captcha in vbulletin, you can ask a custom question that only live humans would know, such as "who was the greatest BYU football coach of all time". It is part of the standard VB package.

At one point we were swimming in new spam accounts. Then I installed a plug-in that requires verification of an e-mail address. That made a HUGE difference. I have not seen any evidence that our new accounts are anything but humans. It provides the same benefit as captcha, but I think it is even better. But once again, I don't see how either the captcha or the e-mail thing relates to our current problem, and I have now blocked the bots from going to the register page.

[Uncle Ted]

Wait. Looks like I can block access to that page using a robots.txt file (mine is currently empty).

http://www.robotstxt.org/robotstxt.html

Just tried that. Let's see if it works.

robots.txt is the honor code for bots. Evil bots don't honor the code.

[Jeff Lebowski]

robots.txt is the honor code for bots. Evil bots don't honor the code.

Of course, but it's a start.

[Uncle Ted]

Well, that's why I am skeptical about this diagnosis. The link in question:

http://cougaruteforum.com/register.php

is simply an "agree to these rules" page. You check the box and on you go to the next page where you enter an e-mail address (part of a verification process I added). How would a captcha change anything? Wouldn't the bots then simply start pinging the captcha page?

Also, this isn't really spam. Just your garden-variety bots pinging away at a site. How do you prevent that?

I wonder if this page gets dinged a lot because it is the first link in the header?

A CAPTCHA is a turing test. In short, it attempts to prove that it is a real person and not a computer.

There are CAPTCHA that are hosted off-site on a different server (I believe google has a CAPTCHA service). Therefore, the bot would be "pinging away" at someone else's server like google's.

Also, a well written anti-evil bot CAPTCHA would use a technique called tar pitting. As an evil-bot "pings away" the server intentionally responds slower and slower which basically traps the bot in doing less and less.

Attempt to use IP filters may work somewhat but the owner of the evil-bot is most likely using (a) "zombie server(s)" (ie. someone's Windose computer that has been taken over by malware) and these zoombies can move around in the IP address space fairly easily.

[Jeff Lebowski]

A CAPTCHA is a turing test. In short, it attempts to prove that it is a real person and not a computer.

Sure.

There are CAPTCHA that are hosted off-site on a different server (I believe google has a CAPTCHA service). Therefore, the bot would be "pinging away" at someone else's server like google's.

Also, a well written anti-evil bot CAPTCHA would use a technique called tar pitting. As an evil-bot "pings away" the server intentionally responds slower and slower which basically traps the bot in doing less and less.

Off-site would be the way to do it. But this would only make sense if there is one page primarily targeted by the bots. The only reason I think register.php was getting hammered is that it is the first one on the page and it involved a block. I will keep an eye on the marauder's map to see if a lot of bots are ignoring the robots.txt file.

Attempt to use IP filters may work somewhat but the owner of the evil-bot is most likely using (a) "zombie server(s)" (ie. someone's Windose computer that has been taken over by malware) and these zoombies can move around in the IP address space fairly easily.

Yeah, I don't think that will make much of a difference.

[Clark Addison]

Can you tell if a lot of these attacks are coming from Salt Lake City? I have a theory . . .

[Surfah]

CUF just timed out on me. Other websites in other tabs were opening just fine while CUF was timing out so it's not my ISP.

[Surfah]

CUF just timed out on me. Other websites in other tabs were opening just fine while CUF was timing out so it's not my ISP.

Just happened again. Twice in 5 minutes.

[TripletDaddy]

Maybe I'm doing something wrong but I haven't really noticed anything at all these days. CUF works fine for me. Is anyone else NOT having a problem with load times? Weird that this problem is so widespread.

[Donuthole]

Not having any problems whatsoever. Viewing via Firefox, if that matters at all.

[Jeff Lebowski]

Super fast for me today.

[old_gregg]

i cleared my cache and haven't had issues

[Jeff Lebowski]

The hostgator tech guy did an analysis of which files were being hammered most by the bots and added them to the robots.txt file. Looking through the who's online shows that almost all bots are behaving now. I also found an add-on that blocks bots that ignore the robots.txt file. I will check it out when I get a little time.

[Flystripper]

Just happened again. Twice in 5 minutes.

It continues to happen to me as well, but I think it is a resuilt of my company only allowing an antiquated version of IE.