Adding 1000s of products to my website not to be crawled

General General Questions
1

I want to add 1000s of product pages to my website that doesnt have to be seo in anyway, I am relying on selling the products to exisiting customers.

My exisisting site does very well for seo , and the other products are different, I dont want these to water down any seo ranking.
It would be 1 main category, with around 150 sub categories, each containing dozens of clothing products of mamy sizes and colours.

What woud be the best way to add these and stop crawling, is it just robots text,
Disallow: /category/widgets2/*

or is there any better way to ensure 100% they arent crawled directly.

thanks
John

2

Will category Availability User groups setup work for you? Or else a custom mod or addon to auto apply noindex meta tag on those categories and products can meet your needs.

3

Hmm
I want all customers to see the products, but dont want them crawled by search index.
Sounds like custom noindex is the answer ?

4

Yes, please feel free to PM us.

1 Like
5

You can also add a rule to your web server, that will return 403 for any requests to this URL, which contain bot|crawler in their user-agents.

1 Like