Jump to content

  • You cannot start a new topic
  • This topic is locked This topic is locked

robots.txt Rate Topic   - - - - -

 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 28 March 2010 - 07:00 PM #21

Is this supposed to represent some widespread disregard for the meta tag?

No, it's not.
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • gabrieluk
  • Senior Member
  • Members
  • Join Date: 21-Jul 09
  • 133 posts

Posted 28 March 2010 - 10:28 PM #22

Hi Indy!
I have this urls being indexed many times...
http://www.fmydomain...ducts=Y&page=90

the only difference between them is the end of the url "Y&page= "

should i
Disallow: /index.php?target=gift_certificates&mode=free_products&search_products=Y&page=

??:rolleyes:
Number 1

 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 29 March 2010 - 09:37 AM #23

should i
Disallow: /index.php?target=gift_certificates&mode=free_products&search_products=Y&page=

??:rolleyes:

Yes, right.
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • hyteckit
  • Senior Member
  • Members
  • Join Date: 24-Mar 08
  • 122 posts

Posted 31 March 2010 - 03:19 PM #24

Avoid duplicate content?

Disallow: /*.html?subcats=Y
Disallow: /*.php?subcats=Y
inkWOW.com - Printer ink and toner cartridges.
Cart Sidebox - Free CS-Cart 2.0 Addon for adding shopping cart to sidebar

 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 31 March 2010 - 03:42 PM #25

Avoid duplicate content?

Disallow: /*.html?subcats=Y
Disallow: /*.php?subcats=Y

You can still add it to the file, there is nothing wrong. The only thing are the "wildcards" [ * ]. Not all bots support those.
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • ALEXsei_
  • Senior Member
  • Members
  • Join Date: 27-Jun 08
  • 1423 posts

Posted 04 May 2010 - 09:22 PM #26

+
IMHO

Disallow: /?sort_by=*
Disallow: /?sl=*

 
  • dlm3089
  • Junior Member
  • Members
  • Join Date: 23-Jan 10
  • 21 posts

Posted 14 May 2010 - 01:04 PM #27

Is it possible to prevent the generation of dynamic links for cs-cart? That way, the sitemap won't index any?

Thanks!

 
  • johnbol1
  • Never Re
  • Members
  • Join Date: 23-Feb 10
  • 4641 posts

Posted 14 May 2010 - 01:09 PM #28

http://www.robotstxt.org/

Custom printed hi visibility clothing sale the UK's online hivis safety shop
v4.5.2


 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 14 May 2010 - 01:38 PM #29

Is it possible to prevent the generation of dynamic links for cs-cart? That way, the sitemap won't index any?

Thanks!

You can use something like that:

Disallow: /index.php?dispatch=
It depends, which links do you want to disallow. The example above will disallow indexing of all URLs which contain '/index.php?dispatch=' in the URL.
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • albertpro
  • Member
  • Members
  • Join Date: 25-Nov 07
  • 118 posts

Posted 17 June 2010 - 04:52 PM #30

You can use something like that:

Disallow: /index.php?dispatch=
It depends, which links do you want to disallow. The example above will disallow indexing of all URLs which contain '/index.php?dispatch=' in the URL.


Is it ok to use Disallow actual URL instead to prevent the duplication.

Disallow: /display-cabinets.html?subcats=Y
Disallow: /display-cabinets/corner-display-cabinets.html?sort_by=
Disallow: /display-cabinets/corner-display-cabinets.html?subcats=Y

Thank you.

CS-CART: version 4.6.1


 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 17 June 2010 - 05:11 PM #31

Is it ok to use Disallow actual URL instead to prevent the duplication.

Disallow: /display-cabinets.html?subcats=Y
Disallow: /display-cabinets/corner-display-cabinets.html?sort_by=
Disallow: /display-cabinets/corner-display-cabinets.html?subcats=Y

Thank you.

What do you mean with 'duplication'?
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • albertpro
  • Member
  • Members
  • Join Date: 25-Nov 07
  • 118 posts

Posted 17 June 2010 - 05:19 PM #32

What do you mean with 'duplication'?


I should have said, prevent crawling for Filters and Sorting features.

Because I use extensive filtering, and google is crawling all of my pages and I am scare to be spammed.
I thought to use actual path to prevent them to be crawled.

I know u are expert, and I used suggested robot.txt sample.
Is that correct or I should take it out.

Thank you.

CS-CART: version 4.6.1


 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 17 June 2010 - 06:55 PM #33

I should have said, prevent crawling for Filters and Sorting features.

Because I use extensive filtering, and google is crawling all of my pages and I am scare to be spammed.
I thought to use actual path to prevent them to be crawled.

I know u are expert, and I used suggested robot.txt sample.
Is that correct or I should take it out.

Thank you.

And what the filter begin the string with?
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • albertpro
  • Member
  • Members
  • Join Date: 25-Nov 07
  • 118 posts

Posted 18 June 2010 - 12:10 AM #34

And what the filter begin the string with?


Something like:
http://domain.com/fl...ures_hash=V1275

CS-CART: version 4.6.1


 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 18 June 2010 - 10:44 AM #35

Something like:
http://domain.com/fl...ures_hash=V1275

Hi you can try this one:

Disallow: features_hash
Disallow: /index.php?type=extended&search_performed
Disallow: /index.php?subcats=Y
This should disallow all URLs which contain the word 'features_hash'
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • albertpro
  • Member
  • Members
  • Join Date: 25-Nov 07
  • 118 posts

Posted 18 June 2010 - 03:04 PM #36

Hi you can try this one:

Disallow: features_hash
Disallow: /index.php?type=extended&search_performed
Disallow: /index.php?subcats=Y
This should disallow all URLs which contain the word 'features_hash'


Thank you.

CS-CART: version 4.6.1


 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 18 June 2010 - 06:25 PM #37

Thank you.

You're welcome... of course I added the two other lines to the code just to disallow some searching strings...
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • nedd
  • Senior Member
  • Members
  • Join Date: 13-Jan 08
  • 125 posts

Posted 21 June 2010 - 04:19 PM #38

... and how to disallow the following strings:

.html?sort_by=position&sort_order=asc
.html?sort_by=popularity&sort_order=desc&layout=products_multicolumns
.html?sort_by=product&sort_order=asc

by adding:

Disallow: ?sort_by=

or

Disallow: /?sort_by=

or

Disallow: position&sort_order
Disallow: popularity&sort_order
Disallow: product&sort_order

or...?

 
  • indy0077
  • Senior Member
  • Banned
  • Join Date: 03-Nov 09
  • 1431 posts

Posted 21 June 2010 - 04:30 PM #39

... and how to disallow the following strings:

.html?sort_by=position&sort_order=asc
.html?sort_by=popularity&sort_order=desc&layout=products_multicolumns
.html?sort_by=product&sort_order=asc

by adding:

Disallow: ?sort_by=

or

Disallow: /?sort_by=

or

Disallow: position&sort_order
Disallow: popularity&sort_order
Disallow: product&sort_order

or...?

this one

Disallow: /?sort_by=

is wrong, because there isn't a '/' before '?sort_by='

It depends on what do you want to disallow. If you choose 'html?sort_by=position&sort_order=asc' then the disallowed URL must contain the whole string. If you say e.g. just '?sort_by=' then all URLs which contain the string, will be disallowed.
.
CS-Cart Professional €160.00 | CS-Cart Multi-Vendor €625.00 | CS-Cart Hosting | SSL Certificates
.
CS-Cart Optimized Servers *** USA & UK VPS Servers

 
  • ather
  • Junior Member
  • Members
  • Join Date: 29-Apr 10
  • 16 posts

Posted 10 May 2011 - 11:00 AM #40

hey Indy,

I have 84,263 duplicate title ,meta descriptions tags due to filters will you please guide what should i disallow in my robot file Posted Image
Ather Sheikh
www.360bin.com