Disable Robots.txt

We have three different types of robots.txt and should put all of them in one and the same docroot. The other two domains are parked and display different language versions of the base domain. This is the solution I have found

Create three robots texts

site1.com.txt

site2.com.txt

site2.com.txt

Put them in directory /robots/

Add to htaccess the following

# URL Rewrite solution for robots.txt for multidomains on single docroot
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond robots/%{HTTP_HOST}.txt -f
RewriteRule ^robots\.txt$ robots/%{HTTP_HOST}.txt [L]

Now we have to somehow disable the cs-cart robots.txt

Rename it to Xrobots.txt or robots.old??

Rename it to Xrobots.txt or robots.old??

There is no robots.txt in the root directory. It is generated by CS-Cart, just like sitemap.xml

This is just a guess...

Create /robots.txt and put what you want in it.

Delete files generated by CSC.

Don't use CSC to edit robots.txt.

This is just a guess...

Create /robots.txt and put what you want in it.

Delete files generated by CSC.

Don't use CSC to edit robots.txt.

I need three different robots.txt files... as said in the starting post

Even if you leave the CSC robots.txt blank - Google will open a blank page for it, and there will be nothing in the root. This is a dynamic file not static.

BTW, CSC documentation says that if there is no robots.txt in the root, it will be generated from the DB.

Probably I should create a blank robots.txt, put it in the root and this will disable the CSC generated robots,txt

The question is will the above rewrite rule work in this case...

Update. Uploaded a blank robots.txt and there was nothing in the browser. Deleted all robots.txt from the root and the browser opened what is in the DB

User-agent: *
Disallow: /app/
Disallow: /design/
Disallow: /upgrades/
Disallow: /var/
Disallow: /store_closed.html
Disallow: /adm
Disallow: /con
Disallow: /login

It is not clear to me what you're trying to do. I have no idea why you would need a rewrite?

Why can't you have one robots.txt with all of your content in it?

It is not clear to me what you're trying to do. I have no idea why you would need a rewrite?

Why can't you have one robots.txt with all of your content in it?

Because I have all languages on separate domains, so need to disallow the unspecific languages, say on .de must be indexed only pages in German, on .ru - pages in Russian, and on .com - pages in English.

And since all three domains share one and the same docroot and one installation of CS-Cart, one robots.txt will not do the trick.

Ignore this.

The docs say you can have separate robtots.tx for each storefront... Confused as to the problem. If your storefronts are language specific and the robots.txt is storefront specific, seems it's designed to do exactly what you want it to do.

From the docs at: https://docs.cs-cart.com/4.6.x/user_guide/website/robots.html

Before 4.6.1

In 4.6.1

Confused as to the problem.

We can have multiple robots.txt from DB for Multi-Srorefronts ONLY edition of CS-Cart. Not for Multi-Vendor Editions. Our store is MV Plus. Technically we can have only ONE storefront.

BUT, there are addons allowing the vendors to have their own frontend on different subdomains. Crippled addons, because they only add a subdomain in the cPanel and redirect it to the vendor store in the MVE - mod_rewritten or not. When you click on a product, you are transported back to the MVE.

We have developed a method to generate different frontends for the different languages, and as long as you don't change the language, you stay on one domain.

Here comes the problem - the language specific domains are essentially parked domains and share one root with the main domain.

In order to do what you suggest, CS-Cart have to develop a Multi-Vendor Multi-Store Multi-Frontends Edition...

Judging by the multiplication price logic of CS-Cart, this edition will cost probably $10,000.

My solution costed me $100.

What remains to be done is creating virtual robots.txt for each domain. When I manually create a robot.txt and put it in the root, it will be valid for all domains, because there is no way to give contradictory instructions or domain specific instructions, say, e.g.

User-agent: *

Disallow: /*sl=en$

Disallow: /sachgebiet/

Disallow: /предмет/

Host https://www.mysite.en/

User-agent: *

Disallow: /*sl=de$

Disallow: /subject/

Disallow: /предмет/

Host https://www.mysite.de/

User-agent: *

Disallow: /*sl=ru$

Disallow: /subject/

Disallow: /sachgebiet/

Host https://www.mysite.ru/

Or this is possible? We don't know the logic of the Google or Yandex bot. We only know that robots.txt is valid for that domain in whose root it is placed... Will Host and Sitemap directive really differentiate the domains?

Only Brin knows.

Or probably He doesn't.

What we know for sure is that the creation of robots.txt from DB must be disabled. What function or php file is responsible for this?

Here is the solution

https://forum.cs-cart.com/topic/54201-robotstxt-for-multiple-domains-in-one-mv-installation/