301 redirect. How to make an exception for the robots.txt

I've got this pretty strange issue.

I have my shop installed in a directory /shop/ on the server (Apache).

The .htaccess has the following code:

Options +Indexes

Options +FollowSymlinks

RewriteEngine on

RewriteBase /shop

RewriteCond %{HTTP_HOST} ^mysite.com$ [OR]

RewriteCond %{HTTP_HOST} ^www.mysite.com$

RewriteRule ^(.*)$ “http://www.mysite.com/shop/$1” [R=301,L]

The script works fine: all the links are redirected to [http://www.mysite.com/shop/] but for the search robot don't see the robots.txt, which is as well redirected to (and located in) the /shop/ directory.

The question is: how to add an exception to the code in .htaccess for the file robots.txt?

P.S. File is opened in browser fine, but the robot still ignores it, because of 301 redirect (according to an answer from the search engine site support.

P.P.S. It's not google.


It sounds like you didn't set it up as a sub domain. If you set it up as a sub domain then other directories will not interfere with the /shop directory.

For those who might be interested.

I've resolved the issue by simply adding this record in .htaccess:

RewriteCond %{REQUEST_URI} !^/robots.txt$

before the last line.

Then moved the file robots.txt to the root dir and changed it accordingly, adding /shop/ directory to every exception (Disallow statement).

For example.

Disallow: /js/


Disallow: /shop/js/


Now everything works fine.

You still should have set it up as a sub domain (http://www.shop.mysite.com). You may run into other problems in the future.