I have been watching our sites as they have been running slow for a couple of days. On some days it would max out the server ram enough to cause the server to cut resources. In our shopping cart software it kept showing that we had 90+ “users” on our site at a time. This is very strange for us as even during the holidays we would only reach maybe 40-50 during the busy morning/early afternoon hours. I’ve never seen it say that we had over 50, for more than a few minutes, let alone pretty much all day.
At this point I logged in to my cpanel and went to the “Latest Visitors” and saw that I have had TONS of visits or connections from IP 213 dot 186 dot 122 dot 2 which from what I could see, is something in the Ukraine. So I then went IP Deny Manager and blocked the IP address. Within about 5 minutes the “users online” that showed within the shopping cart dropped down to 9. Likewise the server increased in speed.
Does anyone know anything about this IP? It appears to be some sort of bot, but how can I know this “bot” did not leave a script or something? Has anyone encountered this before? Is this the best way to block them?
From the Apache access logs, it looks like it was a bot that was blocked by our robots.txt as this shows up repeatedly:
[Fri Feb 04 22:33:03 2011] [error] [client 213dot186dot122dot2] client denied by server configuration:
So it was trying to hit our index.php and was being denied but it kept trying and trying page after page.
Our host describes it as a “rogue search engine bot that was spamming your site”.
I’m honestly not for sure if I should post the ip without the “dot” spelled out or not. I’m not really for sure if it is trying to hack or find holes or what. I do know it can cause havoc on your server speed.
Where are you drawing your conclusion about robots.txt? That is a voluntary file that kind bots are “supposed” to read and obey but there is no requirement for them to do so.
You are probably returning a 404 header to the bot based on your server level security for the IP address. And you might not be returning a valid header at all. Check with your host and find out what they return when they block the IP. If they just drop the connection (disconnect) then there’s nothing telling the bot there isn’t anything there for them. They know they’ve already been there so they keep trying.
But you are approaching this correctly by blocking the IP at the server level and not from the cart. Much more efficient since the cart would have to initialize before it could quit. But it would return a 404 or 500 error to the bot.
I have mine blocked in .htaccess, is this as efficient or should i go via cpanel ?
thanks
JOhn
[quote name=‘johnbol1’]I have mine blocked in .htaccess, is this as efficient or should i go via cpanel ?
JOhn[/QUOTE]
How did you blocked in .htaccess? I can’t imagine it would be bad to try multiple ways to block a rogue bot that doesn’t know how to listen.
@tbirnseth
Yeah, the robot.txt is only as good as the bot that is willing to obey it. I’ve read that some people feel when you tell a robot not to scan certain areas that rogue robots will be more likely to scan that area.
The site was just blocking the bot and then it would just disconnect, the problem is it was doing this hundreds of times and just KILLING the speed of the site. It would look like it was trying to visit about 50 (guess) pages a minute. It is a guess because I didn’t count all the hits the bot was trying to do.
Bottom line is we don’t really market outside the US and do not really sell much out of the US, so the best way for me was to just block the IP.
Oh yeah, I’m really not for on how much we can bank on the “Access restrictions” in the add-ons, but I went ahead and turned that on and tweaked the settings. Once again, anything I can do to make it more difficult just in case this bot was trying to hack.
The host did say that they checked the server and it did not show that any malicious scripts were planted. So I’m really not for sure what the heck a bot from the Ukraine would need from my site so bad that they were hitting it tons of times…unless it was some sort of “attack”.
This may help,
[url]http://www.parkansky.com/china.htm[/url]
[url]http://www.countryipblocks.net/[/url]
my .htaccess has
Country: CHINA
ISO Code: CN
Total Networks: 1,770
Total Subnets: 251,395,584
deny from 1.12.0.0/14
deny from 1.24.0.0/13
deny from 1.48.0.0/15
deny from 1.56.0.0/13
and then thousands more ip addresses after this.
I havent had any china visits since this.
JOhn
There are basically 3 levels you can block at (4 if you count a firewall router before it even gets to your server).
- Linux (or server) firewall level
- Apache
- The application
Why even let apache or the cart initialize or startup to simply reject the IP. The lower level you can block it, the better.
Note that the majority of the cart has to initialize (all the core stuff, registry read, addons loaded, etc.) before the IP address is checked. Why go through all of that load on your server if you don’t have to.
Do it at #1 if possible.
OK Thanks
Will try the server method
JOhn
Get ConfigServer for WHM - Fixes a lotta shit like that. I use it myself.