Jump to content

  • You cannot start a new topic
  • You cannot reply to this topic

Disable Robots.txt Rate Topic   - - - - -

 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 461 posts

Posted 26 January 2019 - 11:37 AM #1

We have three different types of robots.txt and should put all of them in one and the same docroot. The other two domains are parked and display different language versions of the base domain. This is the solution I have found

 

Create three robots texts

site1.com.txt

site2.com.txt

site2.com.txt

 

Put them in directory /robots/

 

Add to htaccess the following

# URL Rewrite solution for robots.txt for multidomains on single docroot
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond robots/%{HTTP_HOST}.txt -f
RewriteRule ^robots\.txt$ robots/%{HTTP_HOST}.txt [L]

Now we have to somehow disable the cs-cart robots.txt



 
  • tbirnseth
  • CS Cart Expert
  • Authorized Reseller
  • Join Date: 08-Nov 08
  • 11306 posts

Posted 26 January 2019 - 05:32 PM #2

Rename it to Xrobots.txt or robots.old??


EZ Merchant Solutions: Custom (USA based) B2B Development, Consulting, Development and Special Projects (get a quote here).
Commercial addons, payment methods and modifications to meet your business and operations needs.


 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 461 posts

Posted 26 January 2019 - 06:08 PM #3

Rename it to Xrobots.txt or robots.old??

 

There is no robots.txt in the root directory. It is generated by CS-Cart, just like sitemap.xml



 
  • The Tool
  • Been Here Way Too Long Member
  • Members
  • Join Date: 30-Mar 07
  • 3800 posts

Posted 26 January 2019 - 07:25 PM #4

This is just a guess...

 

Create /robots.txt and put what you want in it.

Delete files generated by CSC.

Don't use CSC to edit robots.txt.



 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 461 posts

Posted 26 January 2019 - 07:30 PM #5

This is just a guess...

 

Create /robots.txt and put what you want in it.

Delete files generated by CSC.

Don't use CSC to edit robots.txt.

 

I need three different robots.txt files... as said in the starting post

 

Even if you leave the CSC robots.txt blank - Google will open a blank page for it, and there will be nothing in the root. This is a dynamic file not static.

 

BTW, CSC documentation says that if there is no robots.txt in the root, it will be generated from the DB.

 

Probably I should create a blank robots.txt, put it in the root and this will disable the CSC generated robots,txt

 

The question is will the above rewrite rule work in this case...

 

Update. Uploaded a blank robots.txt and there was nothing in the browser. Deleted all robots.txt from the root and the browser opened what is in the DB

User-agent: *
Disallow: /app/
Disallow: /design/
Disallow: /upgrades/
Disallow: /var/
Disallow: /store_closed.html
Disallow: /adm
Disallow: /con
Disallow: /login


 
  • The Tool
  • Been Here Way Too Long Member
  • Members
  • Join Date: 30-Mar 07
  • 3800 posts

Posted 26 January 2019 - 07:55 PM #6

It is not clear to me what you're trying to do.  I have no idea why you would need a rewrite?

 

Why can't you have one robots.txt with all of your content in it?



 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 461 posts

Posted 26 January 2019 - 08:03 PM #7

It is not clear to me what you're trying to do.  I have no idea why you would need a rewrite?

 

Why can't you have one robots.txt with all of your content in it?

 

Because I have all languages on separate domains, so need to disallow the unspecific languages, say on .de must be indexed only pages in German, on .ru - pages in Russian, and on .com - pages in English.

 

And since all three domains share one and the same docroot and one installation of CS-Cart, one robots.txt will not do the trick.



 
  • tbirnseth
  • CS Cart Expert
  • Authorized Reseller
  • Join Date: 08-Nov 08
  • 11306 posts

Posted 27 January 2019 - 12:30 AM #8

Ignore this.

EZ Merchant Solutions: Custom (USA based) B2B Development, Consulting, Development and Special Projects (get a quote here).
Commercial addons, payment methods and modifications to meet your business and operations needs.


 
  • tbirnseth
  • CS Cart Expert
  • Authorized Reseller
  • Join Date: 08-Nov 08
  • 11306 posts

Posted 27 January 2019 - 12:35 AM #9

The docs say you can have separate robtots.tx for each storefront...  Confused as to the problem.  If your storefronts are language specific and the robots.txt is storefront specific, seems it's designed to do exactly what you want it to do.

 

From the docs at: https://docs.cs-cart...ite/robots.html

Before 4.6.1
  • The instructions were stored in the robots.txt file.
  • You couldn’t specify different instructions for different storefronts.
  • FTP access was required for editing the robots.txt file.
In 4.6.1
  • The instructions are stored in the database and can be different for each storefront.
  • No FTP access is required to edit the instructions.
  • If a robots.txt file exists in the root directory of a CS-Cart/Multi-Vendor installation, then only the instructions from this file will be used, and the instructions from the database will be ignored.

EZ Merchant Solutions: Custom (USA based) B2B Development, Consulting, Development and Special Projects (get a quote here).
Commercial addons, payment methods and modifications to meet your business and operations needs.


 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 461 posts

Posted 27 January 2019 - 07:25 AM #10

 

 Confused as to the problem. 

 

We can have multiple robots.txt from DB for Multi-Srorefronts ONLY edition of CS-Cart. Not for Multi-Vendor Editions. Our store is MV Plus. Technically we can have only ONE storefront.

 

BUT, there are addons allowing the vendors to have their own frontend on different subdomains. Crippled addons, because they only add a subdomain in the cPanel and redirect it to the vendor store in the MVE - mod_rewritten or not. When you click on a product, you are transported back to the MVE.

 

We have developed a method to generate different frontends for the different languages, and as long as you don't change the language, you stay on one domain.

 

Here comes the problem - the language specific domains are essentially parked domains and share one root with the main domain.

In order to do what you suggest, CS-Cart have to develop a Multi-Vendor Multi-Store Multi-Frontends Edition...

 

Judging by the multiplication price logic of CS-Cart, this edition will cost probably $10,000.

 

My solution costed me $100.

 

What remains to be done is creating virtual robots.txt for each domain. When I manually create a robot.txt and put it in the root, it will be valid for all domains, because there is no way to give contradictory instructions or domain specific instructions, say, e.g.

 

User-agent: *

Disallow: /*sl=en$

Disallow: /sachgebiet/

Disallow: /предмет/

Host https://www.mysite.en/

 

User-agent: *

Disallow: /*sl=de$

Disallow: /subject/

Disallow: /предмет/

Host https://www.mysite.de/

 

User-agent: *

Disallow: /*sl=ru$

Disallow: /subject/

Disallow: /sachgebiet/

Host https://www.mysite.ru/

 

Or this is possible? We don't know the logic of the Google or Yandex bot. We only know that robots.txt is valid for that domain in whose root it is placed... Will Host and Sitemap directive really differentiate the domains?

 

Only Brin knows.

Or probably He doesn't.

 

What we know for sure is that the creation of robots.txt from DB must be disabled. What function or php file is responsible for this?



 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 461 posts

Posted 27 January 2019 - 09:09 AM #11

Here is the solution

 

https://forum.cs-car...v-installation/