Robot And Bandwidth

General General Questions
1

There is a robot tha crawling every day my site : (compatible; MJ12bot/v1.4.4; MJ12Bot | Home | from Majestic).

It crawls a lot of pages and i think that it is the reason that my bandwidth reached its limits the past two months.

What can i do for this?

Thanks!

2

[quote name='akistdm' timestamp='1390055381' post='175438']

There is a robot tha crawling every day my site : (compatible; MJ12bot/v1.4.4; [url=“MJ12Bot | Home | from Majestic”]MJ12Bot | Home | from Majestic).

It crawls a lot of pages and i think that it is the reason that my bandwidth reached its limits the past two months.

What can i do for this?

Thanks!

[/quote]



Add this to your robots.txt file



User-agent: MJ12bot

Disallow: /

3

Ok i hope this will do the work because bandwidth is at 80% right now.

Thanks a lot!

4

As i can see there is another one (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

that is crawling thousands pages.

I think that this is not the real bing bot.

Does someone know something for this?

5

http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2012/08/31/how-to-verify-that-bingbot-is-bingbot.aspx



[url=“http://www.bing.com/webmaster/help/how-to-use-the-verify-bingbot-tool-2195837e”]http://www.bing.com/...t-tool-2195837e[/url]

6

Hi



bad Robots are a pain in the @$$ and there is not really a way to stop them. They wont respect robots.txt that's for sure.



You could try to use cloudflare for a while to recognize them and cut them through the block list that cloudflare has not to mention that you will save bandwidth in most cases and lower your server/hosting load.



It has a free subscription that really works www.cloudflare.com



Fotis

7

Thank you all!

I am starting to fix this problem.