Jump to content

  • You cannot start a new topic
  • You cannot reply to this topic

Robots.txt For Multiple Domains In One Mv Installation Rate Topic   - - - - -

 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 452 posts

Posted 27 January 2019 - 12:30 PM #1

Here is the story
 

We can have multiple robots.txt from DB for Multi-Srorefronts ONLY edition of CS-Cart. Not for Multi-Vendor Editions. Our store is MV Plus. Technically we can have only ONE storefront.
 
BUT, there are addons allowing the vendors to have their own frontend on different subdomains. Crippled addons, because they only add a subdomain in the cPanel and redirect it to the vendor store in the MVE - mod_rewritten or not. When you click on a product, you are transported back to the MVE.
 
We have developed a method to generate different frontends for the different languages, and as long as you don't change the language, you stay on one domain.
 
Here comes the problem - the language specific domains are essentially parked domains and share one root with the main domain.
In order to do what you suggest, CS-Cart have to develop a Multi-Vendor Multi-Store Multi-Frontends Edition...
 
Judging by the multiplication price logic of CS-Cart, this edition will cost probably $10,000.
 
My solution costed me $100.
 
What remains to be done is creating virtual robots.txt for each domain. When I manually create a robot.txt and put it in the root, it will be valid for all domains, because there is no way to give contradictory instructions or domain specific instructions, say, e.g.
 
User-agent: *
Disallow: /*sl=en$
Disallow: /sachgebiet/
Disallow: /предмет/
Host https://www.mysite.en/
 
User-agent: *
Disallow: /*sl=de$
Disallow: /subject/
Disallow: /предмет/
Host https://www.mysite.de/
 
User-agent: *
Disallow: /*sl=ru$
Disallow: /subject/
Disallow: /sachgebiet/
Host https://www.mysite.ru/
 
Or this is possible? 

 
Solution to the problem:
 
Add this line to htaccess in mod_rewrite section

RewriteRule ^robots.txt$ robots.php

Compose robots.php according to your needs, so that one language must exclude indexing of pages from the other languages

<?php
  header('Content-Type: text/plain; charset=utf-8');

  if($_SERVER['HTTP_HOST']=='www.mysite.de'){
    echo 'User-agent: *
    Disallow: /*sl=de$
    Disallow: /книга
    Disallow: /book
    Disallow: /books/
    Disallow: /предмет/
    Disallow: /genres/
    Disallow: /жанры/
    Disallow: /format/
    Disallow: /формат/
    Disallow: /language/
    Disallow: /язык/
    Disallow: /lifestyle/
    Disallow: /жизнь/
    Disallow: /app/
    Disallow: /design/
    Disallow: /upgrades/
    Disallow: /var/
    Disallow: /store_closed.html
    Disallow: /adm
    Disallow: /con
    Disallow: /login
    Host: https://www.mysite.de/
    Sitemap: https://www.mysite.de/sitemap-de.xml
    ';
  } else if($_SERVER['HTTP_HOST']=='www.mysite.ru'){
    echo 'User-agent: *
    Disallow: /*sl=ru$
    Disallow: /buch
    Disallow: /book
    Disallow: /books/
    Disallow: /gebiet/
    Disallow: /genres/
    Disallow: /genre/
    Disallow: /format/
    Disallow: /form/
    Disallow: /language/
    Disallow: /sprache/
    Disallow: /lifestyle/
    Disallow: /leben/	
    Disallow: /app/
    Disallow: /design/
    Disallow: /upgrades/
    Disallow: /var/
    Disallow: /store_closed.html
    Disallow: /adm
    Disallow: /con
    Disallow: /login
    Host: https://www.mysite.ru/
    Sitemap: https://www.mysite.ru/sitemap-ru.xml
    ';
  } else if($_SERVER['HTTP_HOST']=='www.mysite.com'){
    echo 'User-agent: *
    Disallow: /*sl=en$
    Disallow: /книга
    Disallow: /buch
    Disallow: /adm
    Disallow: /con
    Disallow: /gebiet/
    Disallow: /предмет/
    Disallow: /genre/
    Disallow: /жанры/
    Disallow: /form/
    Disallow: /формат/
    Disallow: /sprache/
    Disallow: /язык/
    Disallow: /leben/
    Disallow: /жизнь/
    Disallow: /app/
    Disallow: /design/
    Disallow: /upgrades/
    Disallow: /var/
    Disallow: /store_closed.html
    Disallow: /login
    Host: https://www.mysite.com/
    Sitemap: https://www.mysite.com/sitemap.xml
    ';	
  } else {
    echo 'User-agent: *
    Disallow: /
    ';
  }
?>

Create a language specific Google map by hiding the other languages, download it, change the .tld and upload accordingly as sitemap-de and sitepap-ru to the root of your main site.
 
That's it. You have now three separate robots.txt in one docroot:
   mysite.com/robots.txt
   mysite.de/robots.txt
   mysite.ru/robots.txt
 
PS. The default CSC robots.txt will be ignored, no matter what you put in there.
 
Open to suggestions how to make the disallows more relevant.



 
  • imago
  • Senior Member
  • Members
  • Join Date: 18-Oct 18
  • 452 posts

Posted 28 January 2019 - 10:08 AM #2

Submitted both cover domains to Google and Yandex. They were both approved and verified with 0 errors. Google only protested against disallowing all contact pages.

 

So had to change Disallow: /con to Disallow: /conf



 
  • 12ka4
  • Senior Member
  • Trial users
  • Join Date: 21-Feb 18
  • 164 posts

Posted 28 January 2019 - 05:01 PM #3

good stuff