Jump to content

301 redirect. How to make an exception for the robots.txt Rate Topic   - - - - -

 
  • d.saveliev
  • Junior Member
  • Members
  • Join Date: 20-Nov 10
  • 34 posts

Posted 07 September 2011 - 09:05 PM #1

I've got this pretty strange issue.
I have my shop installed in a directory /shop/ on the server (Apache).
The .htaccess has the following code:

Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteBase /shop
RewriteCond %{HTTP_HOST} ^mysite.com$ [OR]
RewriteCond %{HTTP_HOST} ^www.mysite.com$
RewriteRule ^(.*)$ "http\:\/\/www\.mysite\.com\/shop\/$1" [R=301,L]

The script works fine: all the links are redirected to [http://www.mysite.com/shop/] but for the search robot don't see the robots.txt, which is as well redirected to (and located in) the /shop/ directory.

The question is: how to add an exception to the code in .htaccess for the file robots.txt?

P.S. File is opened in browser fine, but the robot still ignores it, because of 301 redirect (according to an answer from the search engine site support.

P.P.S. It's not google.

Thanx!

 
  • The Tool
  • Senior Member
  • Members
  • Join Date: 30-Mar 07
  • 2,639 posts

Posted 07 September 2011 - 11:37 PM #2

It sounds like you didn't set it up as a sub domain. If you set it up as a sub domain then other directories will not interfere with the /shop directory.

 
  • d.saveliev
  • Junior Member
  • Members
  • Join Date: 20-Nov 10
  • 34 posts

Posted 12 September 2011 - 08:29 PM #3

For those who might be interested.

I've resolved the issue by simply adding this record in .htaccess:

RewriteCond %{REQUEST_URI} !^/robots\.txt$

before the last line.

Then moved the file robots.txt to the root dir and changed it accordingly, adding /shop/ directory to every exception (Disallow statement).

For example.

Disallow: /js/
for
Disallow: /shop/js/

etc.

Now everything works fine.

 
  • The Tool
  • Senior Member
  • Members
  • Join Date: 30-Mar 07
  • 2,639 posts

Posted 12 September 2011 - 09:32 PM #4

You still should have set it up as a sub domain (http://www.shop.mysite.com). You may run into other problems in the future.