Here is the story
We can have multiple robots.txt from DB for Multi-Srorefronts ONLY edition of CS-Cart. Not for Multi-Vendor Editions. Our store is MV Plus. Technically we can have only ONE storefront.
BUT, there are addons allowing the vendors to have their own frontend on different subdomains. Crippled addons, because they only add a subdomain in the cPanel and redirect it to the vendor store in the MVE - mod_rewritten or not. When you click on a product, you are transported back to the MVE.
We have developed a method to generate different frontends for the different languages, and as long as you don't change the language, you stay on one domain.
Here comes the problem - the language specific domains are essentially parked domains and share one root with the main domain.
In order to do what you suggest, CS-Cart have to develop a Multi-Vendor Multi-Store Multi-Frontends Edition...
Judging by the multiplication price logic of CS-Cart, this edition will cost probably $10,000.
My solution costed me $100.
What remains to be done is creating virtual robots.txt for each domain. When I manually create a robot.txt and put it in the root, it will be valid for all domains, because there is no way to give contradictory instructions or domain specific instructions, say, e.g.
User-agent: *
Disallow: /*sl=en$
Disallow: /sachgebiet/
Disallow: /предмет/
Host https://www.mysite.en/
User-agent: *
Disallow: /*sl=de$
Disallow: /subject/
Disallow: /предмет/
Host https://www.mysite.de/
User-agent: *
Disallow: /*sl=ru$
Disallow: /subject/
Disallow: /sachgebiet/
Host https://www.mysite.ru/
Or this is possible?
Solution to the problem:
Add this line to htaccess in mod_rewrite section
RewriteRule ^robots.txt$ robots.php
Compose robots.php according to your needs, so that one language must exclude indexing of pages from the other languages
if($_SERVER[‘HTTP_HOST’]==‘www.mysite.de’){
echo 'User-agent: *
Disallow: /*sl=de$
Disallow: /книга
Disallow: /book
Disallow: /books/
Disallow: /предмет/
Disallow: /genres/
Disallow: /жанры/
Disallow: /format/
Disallow: /формат/
Disallow: /language/
Disallow: /Ñзык/
Disallow: /lifestyle/
Disallow: /жизнь/
Disallow: /app/
Disallow: /design/
Disallow: /upgrades/
Disallow: /var/
Disallow: /store_closed.html
Disallow: /adm
Disallow: /con
Disallow: /login
Host: https://www.mysite.de/
Sitemap: https://www.mysite.de/sitemap-de.xml
';
} else if($_SERVER[‘HTTP_HOST’]==‘www.mysite.ru’){
echo 'User-agent: *
Disallow: /*sl=ru$
Disallow: /buch
Disallow: /book
Disallow: /books/
Disallow: /gebiet/
Disallow: /genres/
Disallow: /genre/
Disallow: /format/
Disallow: /form/
Disallow: /language/
Disallow: /sprache/
Disallow: /lifestyle/
Disallow: /leben/
Disallow: /app/
Disallow: /design/
Disallow: /upgrades/
Disallow: /var/
Disallow: /store_closed.html
Disallow: /adm
Disallow: /con
Disallow: /login
Host: https://www.mysite.ru/
Sitemap: https://www.mysite.ru/sitemap-ru.xml
';
} else if($_SERVER[‘HTTP_HOST’]==‘www.mysite.com’){
echo 'User-agent: *
Disallow: /*sl=en$
Disallow: /книга
Disallow: /buch
Disallow: /adm
Disallow: /con
Disallow: /gebiet/
Disallow: /предмет/
Disallow: /genre/
Disallow: /жанры/
Disallow: /form/
Disallow: /формат/
Disallow: /sprache/
Disallow: /Ñзык/
Disallow: /leben/
Disallow: /жизнь/
Disallow: /app/
Disallow: /design/
Disallow: /upgrades/
Disallow: /var/
Disallow: /store_closed.html
Disallow: /login
Host: https://www.mysite.com/
Sitemap: https://www.mysite.com/sitemap.xml
';
} else {
echo 'User-agent: *
Disallow: /
';
}
?>
Create a language specific Google map by hiding the other languages, download it, change the .tld and upload accordingly as sitemap-de and sitepap-ru to the root of your main site.
That's it. You have now three separate robots.txt in one docroot:
mysite.com/robots.txt
mysite.de/robots.txt
mysite.ru/robots.txt
PS. The default CSC robots.txt will be ignored, no matter what you put in there.
Open to suggestions how to make the disallows more relevant.